Skip to content

Commit

Permalink
jenkinsfiles: Fix order of ginkgo tests
Browse files Browse the repository at this point in the history
The ginkgo tests are currently executed in a different order for each
run in our CI. Unfortunately, the order in which we run the tests often
matters and can impact the test results. That's because there are a lot
of side effects (leftover Linux state, leftover Cilium conntrack
entries, leftover Kubernetes state, etc.) that are hard to control for.

This undeterminism facilities the introduction of flakes: a first pull
request introduces a flake, but it doesn't show up until a later pull
request runs the tests in a particular order.

The random order also makes flake a bit order to reproduce locally,
though that can easily be worked around by passing the proper seed to
ginkgo when reproducing.

One argument in favor of this randomness is that it surfaces those side
effects and allows us to identify and clean them. In practice, that
doesn't work great:
- Being side effects, they are often hard to identify and are a huge
  time cost for contributors to debug. Our ever increasing list of flake
  issues is a good testimony of that.
- There are not always easy to clean up [1].
- There are often not representative of bugs likely to affect users. Our
  users rarely restart Cilium 50 times in the span of an hour with
  different configurations.
In the end, our tests were not meant to uncover those side effects and
they are therefore inadequate at exposing them properly. If we want to
identify those side effects, then we should have tests specifically for
that.

Other, non-ginkgo CI jobs run their tests in a fixed order [2] or on
completely independent clusters [3].

This commit therefore fixes the order of the ginkgo tests. To that end,
a ginkgo seed was selected using $SRANDOM and will be the same for all
subsequent runs.

1 - #17459
2 - cilium/cilium-cli#558
3 - https://github.com/cilium/cilium/blob/main/.github/workflows/conformance-datapath.yaml
Signed-off-by: Paul Chaignon <paul@cilium.io>
  • Loading branch information
pchaigno committed Apr 21, 2023
1 parent a2a51b5 commit e84e99f
Show file tree
Hide file tree
Showing 2 changed files with 2 additions and 2 deletions.
2 changes: 1 addition & 1 deletion jenkinsfiles/ginkgo-kernel.Jenkinsfile
Original file line number Diff line number Diff line change
Expand Up @@ -238,7 +238,7 @@ pipeline {
}
steps {
sh 'env'
sh 'cd ${TESTDIR}; HOME=${GOPATH} INTEGRATION_TESTS=true ginkgo --focus="${FOCUS}" -v --failFast=${FAILFAST} -- -cilium.provision=false -cilium.timeout=${GINKGO_TIMEOUT} -cilium.kubeconfig=${TESTDIR}/vagrant-kubeconfig -cilium.passCLIEnvironment=true -cilium.runQuarantined=${RUN_QUARANTINED} -cilium.image=${CILIUM_IMAGE} -cilium.tag=${CILIUM_TAG} -cilium.operator-image=${CILIUM_OPERATOR_IMAGE} -cilium.operator-tag=${CILIUM_OPERATOR_TAG} -cilium.hubble-relay-image=${HUBBLE_RELAY_IMAGE} -cilium.hubble-relay-tag=${HUBBLE_RELAY_TAG} -cilium.operator-suffix="-ci"'
sh 'cd ${TESTDIR}; HOME=${GOPATH} INTEGRATION_TESTS=true ginkgo -seed=3898027111 -focus="${FOCUS}" -v --failFast=${FAILFAST} -- -cilium.provision=false -cilium.timeout=${GINKGO_TIMEOUT} -cilium.kubeconfig=${TESTDIR}/vagrant-kubeconfig -cilium.passCLIEnvironment=true -cilium.runQuarantined=${RUN_QUARANTINED} -cilium.image=${CILIUM_IMAGE} -cilium.tag=${CILIUM_TAG} -cilium.operator-image=${CILIUM_OPERATOR_IMAGE} -cilium.operator-tag=${CILIUM_OPERATOR_TAG} -cilium.hubble-relay-image=${HUBBLE_RELAY_IMAGE} -cilium.hubble-relay-tag=${HUBBLE_RELAY_TAG} -cilium.operator-suffix="-ci"'
}
post {
always {
Expand Down
2 changes: 1 addition & 1 deletion jenkinsfiles/ginkgo-runtime-kernel.Jenkinsfile
Original file line number Diff line number Diff line change
Expand Up @@ -136,7 +136,7 @@ pipeline {
TESTDIR="${GOPATH}/${PROJ_PATH}/test"
}
steps {
sh 'cd ${TESTDIR}; INTEGRATION_TESTS=true ginkgo --focus="$(python3 get-gh-comment-info.py "${ghprbCommentBody}" | sed "s/^$/Runtime/" | sed "s/K8s.*/NoTests/")" -v --failFast=${FAILFAST} -- -cilium.provision=false -cilium.timeout=${GINKGO_TIMEOUT} -cilium.runQuarantined=${RUN_QUARANTINED}'
sh 'cd ${TESTDIR}; INTEGRATION_TESTS=true ginkgo -seed=3898027111 -focus="$(python3 get-gh-comment-info.py "${ghprbCommentBody}" | sed "s/^$/Runtime/" | sed "s/K8s.*/NoTests/")" -v --failFast=${FAILFAST} -- -cilium.provision=false -cilium.timeout=${GINKGO_TIMEOUT} -cilium.runQuarantined=${RUN_QUARANTINED}'
}
post {
always {
Expand Down

0 comments on commit e84e99f

Please sign in to comment.