Add global LIMA_BATS_ALL_TESTS_RETRIES variable

[jandubois]

It will rerun all tests up to this number on failure.

Also provides a flaky function to set the retries to LIMA_BATS_FLAKY_TESTS_RETRIES to allow a greater number of retries to known flaky tests.

So in CI you could set both, to e.g. retry all tests at least once, but retry flaky tests up to 5 times.

❯ cat hack/bats/tests/chaos-monkey.bats
load "../helpers/load"

@test "flaky test that fails randomly" {
    flaky
    echo "Attempt ${BATS_TEST_TRY_NUMBER}/$((BATS_TEST_RETRIES + 1))" >&3
    ((RANDOM % 10 < 3))
}

❯ LIMA_BATS_ALL_TESTS_RETRIES=2 ./hack/bats/lib/bats-core/bin/bats -T ./hack/bats/tests/chaos-monkey.bats
chaos-monkey.bats
   flaky test that fails randomly                                                                                                                           1/1 in 0 sec
Attempt 1/3
   flaky test that fails randomly                                                                                                                           1/1 in 0 sec
Attempt 2/3
 ✗ flaky test that fails randomly [4]
Attempt 3/3
   (in test file hack/bats/tests/chaos-monkey.bats, line 6)
     `(( RANDOM % 10 < 3))' failed

1 test, 1 failure in 0 seconds

❯ LIMA_BATS_FLAKY_TESTS_RETRIES=7 ./hack/bats/lib/bats-core/bin/bats -T ./hack/bats/tests/chaos-monkey.bats
chaos-monkey.bats
   flaky test that fails randomly                                                                                                                           1/1 in 0 sec
Attempt 1/8
   flaky test that fails randomly                                                                                                                           1/1 in 1 sec
Attempt 2/8
 ✓ flaky test that fails randomly [5]
Attempt 3/8

1 test, 0 failures in 1 seconds

[jandubois]

Should we have a default LIMA_BATS_FLAKY_TESTS_RETRIES=2 so we always allow up to 3 attempts for tests calling flaky by default?

[AkihiroSuda]

Could you also update

lima/.github/workflows/test.yml

Lines 361 to 369 in 463f7e5

	- name: "Run BATS k8s tests"
	# Wish we could use BATS_TEST_RETRIES=3 as an environment variable here,
	# but bats does not seem to support it.
	uses: nick-fields/retry@ce71cc2ab81d554ebbe88c79ab5975992d79ba08 # v3.0.2
	with:
	timeout_minutes: 30
	retry_on: error
	max_attempts: 3
	command: ./hack/bats/lib/bats-core/bin/bats --timing ./hack/bats/extras/k8s.bats

[AkihiroSuda]

ping @jandubois 🙏

[AkihiroSuda]

Should we have a default LIMA_BATS_FLAKY_TESTS_RETRIES=2 so we always allow up to 3 attempts for tests calling flaky by default?

No, by default the test shouldn't be flaky.

[AkihiroSuda]

Needs rebase

[AkihiroSuda]

Thanks