Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
jenkinsfiles: Fix order of ginkgo tests
The ginkgo tests are currently executed in a different order for each run in our CI. Unfortunately, the order in which we run the tests often matters and can impact the test results. That's because there are a lot of side effects (leftover Linux state, leftover Cilium conntrack entries, leftover Kubernetes state, etc.) that are hard to control for. This undeterminism facilities the introduction of flakes: a first pull request introduces a flake, but it doesn't show up until a later pull request runs the tests in a particular order. The random order also makes flake a bit order to reproduce locally, though that can easily be worked around by passing the proper seed to ginkgo when reproducing. One argument in favor of this randomness is that it surfaces those side effects and allows us to identify and clean them. In practice, that doesn't work great: - Being side effects, they are often hard to identify and are a huge time cost for contributors to debug. Our ever increasing list of flake issues is a good testimony of that. - There are not always easy to clean up [1]. - There are often not representative of bugs likely to affect users. Our users rarely restart Cilium 50 times in the span of an hour with different configurations. In the end, our tests were not meant to uncover those side effects and they are therefore inadequate at exposing them properly. If we want to identify those side effects, then we should have tests specifically for that. Other, non-ginkgo CI jobs run their tests in a fixed order [2] or on completely independent clusters [3]. This commit therefore fixes the order of the ginkgo tests. To that end, a ginkgo seed was selected using $SRANDOM and will be the same for all subsequent runs. 1 - #17459 2 - cilium/cilium-cli#558 3 - https://github.com/cilium/cilium/blob/main/.github/workflows/conformance-datapath.yaml Signed-off-by: Paul Chaignon <paul@cilium.io>
- Loading branch information