ci: stop the required checks from failing on unrelated changes - #492
Merged
Conversation
Two independent defects were making `bats (macos-latest)` and `bats (windows-latest, install helpers)` — both REQUIRED checks — fail on pull requests that changed nothing relevant to them. 1. The macOS bats timeout was stale. `timeout-minutes: 15` was set when the suite ran in about 5 minutes. The suite is now 728 tests and macOS measures 11m57s-14m10s, so only 1-3 minutes of headroom remained. Nine of nineteen sampled runs were killed at ~15m18s, each at whatever test it happened to be executing (the stop point varied between tests 528 and 640, which is what distinguishes a timeout from a hang). Raised to 25, matching the app-test-windows job. 2. The Windows sqlite3 install was unverified. `choco install sqlite` occasionally fails when the chocolatey community feed returns 504, logging "Chocolatey installed 0/0 packages". The verification step ended in `|| echo "sqlite3 not found on PATH"`, so it exited zero and the job continued, failing about three minutes later inside the install tests with "sqlite3 is required but not found" — a message that reads like a defect in the pull request under test. The install now retries, and the verification step fails immediately when sqlite3 is genuinely absent. No test or product behaviour changes.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two independent CI defects were failing REQUIRED checks on pull requests that changed nothing relevant to them. Both are measured, not inferred.
1. The macOS bats timeout is stale
timeout-minutes: 15was chosen when the suite ran in about 5 minutes — its comment still says so. The suite is now 728 tests, and successful macOS runs measure 11m57s, 12m31s, 12m32s, 13m58s and 14m10s. That leaves 1-3 minutes of headroom.The result: 9 of 19 sampled runs were cancelled, every one of them at ~15m18s. The stop point varied (last passing test was 528, 572, 623, 627 or 640 on different runs), which is what separates a timeout from a hang — a hang stops at the same place every time.
Raised to 25, matching the
app-test-windowsjob in the same file.2. The Windows sqlite3 install is unverified
choco install sqliteintermittently fails when the chocolatey community feed returns 504:The verification step ended in
|| echo "sqlite3 not found on PATH", so it exited zero and the job carried on. About three minutes later the install tests failed withError: sqlite3 is required but not found.— which reads as a defect in whatever pull request happened to be running. The same pull request passed on re-run once chocolatey returnedChocolatey installed 1/1 packages, with an unchanged diff.The install step now retries up to three times, and the verification step fails immediately when sqlite3 is genuinely missing.
Effect
No test or product behaviour changes. This only stops unrelated pull requests from being blocked by infrastructure.