You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Describe what should be investigated or refactored
We have been seeing an increase in flaky tests caused by rate limits enforce by Docker Hub. The e2e tests depend on a couple of different images that need to be pulled from Docker Hub. When we have a high rate of contributions the tests will start to fail. Rerunning the tests will resolve the problem after some time, but this is not an optimal solution.
We have two options to resolve this problem long term. Either we remove any dependency on Docker Hub in any test. Alternatively we add credentials to the tests so that we can increase the rate limit, The latter option could be considered the easiest but has its limitations. It would not solve the problem for people running the tests locally nor for people contributing to the project using a fork.
Depending on other projects images for test stability could be questioned as a whole. While it does test real world situations it also means that we are depending on these artifacts being available for as long as we need them. Additionally we assume that these images will not change. We could probably achieve similar types of tests by having our own images in the orgs image registry instead. Promising stability and removing rate limit problems.
Describe what should be investigated or refactored
We have been seeing an increase in flaky tests caused by rate limits enforce by Docker Hub. The e2e tests depend on a couple of different images that need to be pulled from Docker Hub. When we have a high rate of contributions the tests will start to fail. Rerunning the tests will resolve the problem after some time, but this is not an optimal solution.
We have two options to resolve this problem long term. Either we remove any dependency on Docker Hub in any test. Alternatively we add credentials to the tests so that we can increase the rate limit, The latter option could be considered the easiest but has its limitations. It would not solve the problem for people running the tests locally nor for people contributing to the project using a fork.
Depending on other projects images for test stability could be questioned as a whole. While it does test real world situations it also means that we are depending on these artifacts being available for as long as we need them. Additionally we assume that these images will not change. We could probably achieve similar types of tests by having our own images in the orgs image registry instead. Promising stability and removing rate limit problems.
Links to any relevant code
zarf/src/test/e2e/25_helm_test.go
Line 58 in 1e4399e
Additional context
Relates to #2477
The text was updated successfully, but these errors were encountered: