Skip to content

Test suite is too slow and was not designed to run properly in parallel #217

Description

@mattmc3

Problem

In my testing, the bats suite takes ~95s locally and a similar time per CI job.
On the transcrypt-v3 branch it takes double that. It cannot be sped up with
bats --jobs N as-is because every test shares state:

  1. Each test builds its git repository directly in the tests/
    directory (init_git_repo targets $BATS_TEST_DIRNAME), so
    concurrent tests would corrupt each other's repos. This also leaks
    state between runs when a test fails before teardown.
  2. The two transcrypt.crypt-dir tests share a hardcoded /tmp/crypt
    path and rm -fR it at start, which is a race even between the two
    of them.
  3. Tests inherit the developer's global and system git config. Settings
    like commit.gpgsign or merge.conflictstyle diff3 change test
    behavior and cause spurious failures (the conflictStyle case was
    already worked around per-setting in the test helper).

Potential solution

  • Build each test's repo in $BATS_TEST_TMPDIR, which bats creates
    per-test and cleans up automatically. This isolates tests from each
    other and makes bats --jobs N safe; the nuke_git_repo /
    cleanup_all safety machinery becomes unnecessary.
  • Reference the script under test by absolute path ($TRANSCRYPT)
    instead of cwd-relative ../transcrypt.
  • Export GIT_CONFIG_GLOBAL=/dev/null and GIT_CONFIG_SYSTEM=/dev/null
    in the test helper (git 2.32+) so tests are hermetic against
    developer/system git config.
  • Give the crypt-dir tests unique paths under $BATS_TEST_TMPDIR.
  • CI: install GNU parallel and run with --jobs.

Measured locally om my M4 MacBook Pro: ~95s serial to ~26s with --jobs 16, 79/79 passing.

I have a working fix and can submit a PR, but this issue is open for discussion if there's questions or concerns.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions