Add bats Tests for validate-qa-coverage.sh

[couimet]

Context

packages/rangelink-vscode-extension/scripts/validate-qa-coverage.sh is a shell script that cross-references TC IDs in the QA YAML file against TC IDs extracted from integration test .test.ts files. It validates three categories of the automated YAML field: true (must have a non-[assisted] integration test), assisted (must have an [assisted]-tagged integration test), and false (must NOT have any integration test -- parsed and counted but not actually validated). The script is invoked via pnpm validate:qa-coverage, which is called automatically by test-release-run.sh (line 168) for non-grep test runs. It is also smoke-tested by verify-qa-scripts.sh which currently just checks that the script exits 0 or creates a report artifact, with no assertion on correctness of the cross-reference logic. The script uses set -euo pipefail throughout and produces a timestamped report under qa/output/.

What to Test

• find_latest_qa_yaml() finds the latest YAML file correctly, handling the ASCII sort quirk: unsuffixed names (e.g., qa-test-cases-v1.1.0.yaml) sort AFTER suffixed names (qa-test-cases-v1.1.0-001.yaml) because . > - in ASCII. Verify the normalizing -000 suffix append produces correct ordering across scenarios: only unsuffixed, only suffixed, mixed, and reverse chronological order.
• find_latest_qa_yaml() exits 1 with error message when qa/ contains no matching YAML files at all.
• parse_ids_by_automated(yaml, target) correctly extracts TC IDs matching the given automated value. Test with single-quoted IDs ('link-navigation-001'), double-quoted IDs ("link-navigation-002"), and unquoted IDs. Test that non-matching automated values are excluded.
• parse_ids_by_automated(yaml, target) produces empty output when the YAML has no entries for the target value.
• find_automated_test_ids() correctly extracts TC ID patterns from integration test .test.ts files, filtering out [assisted] test names. Test with a temp directory containing test files with various TC ID placements (in it(), describe(), suite() calls) and non-matching lines.
• find_automated_test_ids() exits 1 when INTEGRATION_TEST_DIR does not exist.
• find_assisted_test_ids() correctly extracts TC IDs only from [assisted]-tagged tests, excluding non-tagged tests.
• find_assisted_test_ids() exits 1 when INTEGRATION_TEST_DIR does not exist (same error handling as find_automated_test_ids).
• Main validation logic correctly reports MISMATCH when YAML marks automated: true but no matching test ID exists (asymmetry A).
• Main validation logic correctly reports MISMATCH when a non-[assisted] test exists but YAML does not mark it automated: true (asymmetry B).
• Main validation logic correctly reports MISMATCH when YAML marks automated: assisted but no matching [assisted] test exists (asymmetry C).
• Main validation logic correctly reports MISMATCH when an [assisted] test exists but YAML does not mark it automated: assisted (asymmetry D).
• Main validation logic reports PASSED with exit 0 when all automated and assisted markers match.
• Main validation logic reports FAILED with exit 1 when any mismatch is found (the final grep -q "FAILED" on the report file).
• The automated: false IDs are parsed and counted correctly, but NOT validated against integration tests (the script has a gap here -- no comm check for false entries). This should be documented and a decision made on whether to add the missing validation.
• The relative_path() helper using python3 os.path.relpath produces the correct relative path between two absolute paths.
• Report file is created at the expected path under qa/output/ with the correct version-timestamp naming pattern.
• grep -c . || true pattern correctly handles empty ID lists (avoids set -e exit from grep -c on zero matches). Test with zero, one, and many IDs in each group.
• comm calls with grep . || true pipeline correctly handle empty input from either side (avoids set -e on comm with empty file descriptors).

PROS

• bats is a shell-agnostic testing framework well suited for validating the exact exit codes, stdout/stderr content, and file artifacts that shell scripts produce.
• The script is already isolated enough to test function-by-function: find_latest_qa_yaml, parse_ids_by_automated, find_automated_test_ids, and find_assisted_test_ids are defined as separate named functions that take parameters, making them naturally testable.
• The existing verify-qa-scripts.sh smoke test is a coarse pass/fail (exit 0 check) that would not catch logic bugs like a mangled sorting algorithm, a broken awk parser, or a regression in the [assisted] grep filter. bats tests would cover these cases.
• The QA YAML file structure and integration test naming conventions are enforced by project rules (QA001-QA008), so a change that breaks the cross-reference should be caught by the bats tests before it causes CI failures or silent validator passes.
• The script already uses temp-friendly patterns -- it reads from well-known directories (qa/, src/__integration-tests__/suite/) and writes to qa/output/, so bats tests can seed a temp qa/ and a temp suite/ directory and validate against them without touching real files.

CONS

• bats tests depend on the QA YAML file structure (multi-line list format, id: field, automated: field) and integration test file naming conventions (*.test.ts, [assisted] tag in test names). If either structure changes, the bats test fixtures must be updated in sync -- but this is the same coupling that the production script itself has.
• The script's file discovery (find_latest_qa_yaml) walks the real qa/ directory. bats tests that call this function as-is must manage the filesystem state carefully to avoid picking up real YAML files. The alternative is to source the script and pass a fixture directory, but sourcing exposes all internal functions.
• The relative_path() helper shells out to python3, creating a Python dependency for bats tests even though the test fixtures are simple paths. An alternative is to test through a different path or mock the Python call.
• Unlike the .github/scripts/ci/ scripts (whose bats tests live in root tests/ci/), this script lives in packages/rangelink-vscode-extension/scripts/. Test placement (root tests/ci/ vs package scripts/__tests__/) needs a decision to keep the CI workflow's bats tests/ci/ glob working.

Recommendation

Add bats tests for validate-qa-coverage.sh. The script has enough edge case logic -- the ASCII sort normalization in YAML discovery, the awk-based YAML parser, the [assisted] grep filter, the grep . || true pipe guards -- that the coarseness of the current smoke test in verify-qa-scripts.sh is a blind spot. Focus tests on the four named functions (find_latest_qa_yaml, parse_ids_by_automated, find_automated_test_ids, find_assisted_test_ids) sourced individually via . ./scripts/validate-qa-coverage.sh in a bats setup() that seeds temp directories. Keep the main exit-code integration test (full run on a real YAML vs real test files) as a single bats test to catch cross-function issues. The project already has bats infrastructure from #572 (tests/ci/test_helper.bash, pnpm test:bats, CI workflow). Place tests where the existing bats tests/ci/ glob picks them up, or extend the glob if tests live in the package directory.