Problem
In my testing, the bats suite takes ~95s locally and a similar time per CI job.
On the transcrypt-v3 branch it takes double that. It cannot be sped up with
bats --jobs N as-is because every test shares state:
- Each test builds its git repository directly in the
tests/
directory (init_git_repo targets $BATS_TEST_DIRNAME), so
concurrent tests would corrupt each other's repos. This also leaks
state between runs when a test fails before teardown.
- The two
transcrypt.crypt-dir tests share a hardcoded /tmp/crypt
path and rm -fR it at start, which is a race even between the two
of them.
- Tests inherit the developer's global and system git config. Settings
like commit.gpgsign or merge.conflictstyle diff3 change test
behavior and cause spurious failures (the conflictStyle case was
already worked around per-setting in the test helper).
Potential solution
- Build each test's repo in
$BATS_TEST_TMPDIR, which bats creates
per-test and cleans up automatically. This isolates tests from each
other and makes bats --jobs N safe; the nuke_git_repo /
cleanup_all safety machinery becomes unnecessary.
- Reference the script under test by absolute path (
$TRANSCRYPT)
instead of cwd-relative ../transcrypt.
- Export
GIT_CONFIG_GLOBAL=/dev/null and GIT_CONFIG_SYSTEM=/dev/null
in the test helper (git 2.32+) so tests are hermetic against
developer/system git config.
- Give the crypt-dir tests unique paths under
$BATS_TEST_TMPDIR.
- CI: install GNU parallel and run with
--jobs.
Measured locally om my M4 MacBook Pro: ~95s serial to ~26s with --jobs 16, 79/79 passing.
I have a working fix and can submit a PR, but this issue is open for discussion if there's questions or concerns.
Problem
In my testing, the bats suite takes ~95s locally and a similar time per CI job.
On the transcrypt-v3 branch it takes double that. It cannot be sped up with
bats --jobs Nas-is because every test shares state:tests/directory (
init_git_repotargets$BATS_TEST_DIRNAME), soconcurrent tests would corrupt each other's repos. This also leaks
state between runs when a test fails before teardown.
transcrypt.crypt-dirtests share a hardcoded/tmp/cryptpath and
rm -fRit at start, which is a race even between the twoof them.
like
commit.gpgsignormerge.conflictstyle diff3change testbehavior and cause spurious failures (the conflictStyle case was
already worked around per-setting in the test helper).
Potential solution
$BATS_TEST_TMPDIR, which bats createsper-test and cleans up automatically. This isolates tests from each
other and makes
bats --jobs Nsafe; thenuke_git_repo/cleanup_allsafety machinery becomes unnecessary.$TRANSCRYPT)instead of cwd-relative
../transcrypt.GIT_CONFIG_GLOBAL=/dev/nullandGIT_CONFIG_SYSTEM=/dev/nullin the test helper (git 2.32+) so tests are hermetic against
developer/system git config.
$BATS_TEST_TMPDIR.--jobs.Measured locally om my M4 MacBook Pro: ~95s serial to ~26s with
--jobs 16, 79/79 passing.I have a working fix and can submit a PR, but this issue is open for discussion if there's questions or concerns.