Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove use of Docker Hub in tests #2743

Closed
phillebaba opened this issue Jul 21, 2024 · 0 comments
Closed

Remove use of Docker Hub in tests #2743

phillebaba opened this issue Jul 21, 2024 · 0 comments
Assignees
Milestone

Comments

@phillebaba
Copy link
Member

Describe what should be investigated or refactored

We have been seeing an increase in flaky tests caused by rate limits enforce by Docker Hub. The e2e tests depend on a couple of different images that need to be pulled from Docker Hub. When we have a high rate of contributions the tests will start to fail. Rerunning the tests will resolve the problem after some time, but this is not an optimal solution.

We have two options to resolve this problem long term. Either we remove any dependency on Docker Hub in any test. Alternatively we add credentials to the tests so that we can increase the rate limit, The latter option could be considered the easiest but has its limitations. It would not solve the problem for people running the tests locally nor for people contributing to the project using a fork.

Depending on other projects images for test stability could be questioned as a whole. While it does test real world situations it also means that we are depending on these artifacts being available for as long as we need them. Additionally we assume that these images will not change. We could probably achieve similar types of tests by having our own images in the orgs image registry instead. Promising stability and removing rate limit problems.

Links to any relevant code

stdOut, stdErr, err = e2e.Zarf("package", "create", "examples/helm-charts", "-o", "build", "--registry-override", "ghcr.io/stefanprodan=docker.io/stefanprodan", "--tmpdir", tmpdir, "--confirm")

Additional context

Relates to #2477

@AustinAbro321 AustinAbro321 self-assigned this Jul 23, 2024
@schristoff schristoff closed this as completed by moving to Done in Zarf Aug 1, 2024
@schristoff schristoff added this to the v0.38.0 milestone Aug 1, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Status: Done
Development

No branches or pull requests

3 participants