docs: re-derive the testsuite figures from the manifests that produce them - #7661
Merged
Conversation
… them `README.md` ships the exact `awk` command that reads a leg's expected-outcome manifest and prints its pass/fail/skip counts. Running it disagrees with every row of the table directly beneath it. | row | documented | re-derived | |---|---|---| | non-root, pipe | 257 / 3 / 85 | **259 / 1 / 85** | | root, pipe | 286 / 3 / 56 | **288 / 1 / 56** | | non-root, tcp | 101 / 21 / 33 | **108 / 14 / 33** | | root, tcp | 113 / 27 / 15 | **120 / 20 / 15** | `SECURITY.md` carries the same four rows and the same two summary figures, so both files drifted together. Corrected throughout, along with: - **"3 of 345 tests currently diverge"** across the full-corpus legs. It is **one** - `filter-merge-content-echo` - and the README names it now rather than leaving the reader to count. - **"29 across all four"** distinct failures. It is **23**, and across **five** manifests, not four: the README's own glob `tools/ci/upstream-3.5.0-expect.*.txt` has always matched the macOS file its prose excluded. - **The macOS leg was missing from both tables.** It runs on every PR as `upstream-testsuite-macos` (`ci.yml`), on the full 345-cell corpus, with its own committed manifest. It is the only leg that can observe a platform-conditional divergence - one of its three remaining failures *skips* on Linux, so it had never executed in this repository's CI before the leg existed. Added as a fifth row, with its non-required status stated. - **"the required checks being the two stdio-pipe legs."** All four Linux legs have been required contexts since #7408 wired the TCP pair into PR CI; the ruleset returns ten contexts, not eight. - **`proxy protocol hosts` described as "not yet implemented"** and "still under audit" in two places. It shipped in #7648: parsed into a `ProxyProtocolPolicy` that mirrors upstream's `allow_proxy_protocol_peer()`, rejecting every peer when the trusted list is empty or unset, and warning at startup on the combination upstream warns about. - **The `MAX_PROXY_LINE_BYTES` citation** pointed at `connect/proxy.rs:344`. The constant moved and, more usefully, stopped being a typed literal: it is now `PROXY_BUF_SIZE - 1`, so the doc records the derivation rather than a line number that will drift again. The two macOS-leg rationale comments in the workflows carried the same pre-fix counts and a failure list six entries out of date. Both are recounted from the manifests. `CHANGELOG.md` stopped at #7632, leaving the 26 PRs merged since then unrecorded. Added under Security / Fixed / Testing and CI / Documentation. ## Why the numbers were wrong in a way reading could not catch Every figure here is the outcome column of a committed manifest. The previous values were transcribed once and then maintained by hand, so they decayed as fixes landed - and a reader checking the table against the prose beside it would find them perfectly consistent with each other. The check that finds this class is re-running the derivation, not re-reading the text, so that is what was done: a script recomputes all five legs and both summary figures from `tools/ci/upstream-3.5.0-expect*.txt` and asserts the two documents contain the results, with the superseded strings blacklisted so a partial edit cannot pass. It also caught a live error in this changeset - #7659 landed mid-review and flipped `operator-path-partial-dir-daemon`, taking macOS from 236/4 to 237/3 and the distinct-failure count from 24 to 23. The figures here are derived from master with that merge in place.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
docs: re-derive the testsuite figures from the manifests that produce them
README.mdships the exactawkcommand that reads a leg's expected-outcomemanifest and prints its pass/fail/skip counts. Running it disagrees with every
row of the table directly beneath it.
SECURITY.mdcarries the same four rows and the same two summary figures, soboth files drifted together. Corrected throughout, along with:
one -
filter-merge-content-echo- and the README names it now ratherthan leaving the reader to count.
five manifests, not four: the README's own glob
tools/ci/upstream-3.5.0-expect.*.txthas always matched the macOS file itsprose excluded.
upstream-testsuite-macos(ci.yml), on the full 345-cell corpus, with itsown committed manifest. It is the only leg that can observe a
platform-conditional divergence - one of its three remaining failures skips
on Linux, so it had never executed in this repository's CI before the leg
existed. Added as a fifth row, with its non-required status stated.
have been required contexts since ci: run the 3.5.0 testsuite over loopback TCP on every pull request #7408 wired the TCP pair into PR CI; the
ruleset returns ten contexts, not eight.
proxy protocol hostsdescribed as "not yet implemented" and "stillunder audit" in two places. It shipped in fix(daemon): refuse a PROXY header from an untrusted direct peer #7648: parsed into a
ProxyProtocolPolicythat mirrors upstream'sallow_proxy_protocol_peer(),rejecting every peer when the trusted list is empty or unset, and warning at
startup on the combination upstream warns about.
MAX_PROXY_LINE_BYTEScitation pointed atconnect/proxy.rs:344.The constant moved and, more usefully, stopped being a typed literal: it is
now
PROXY_BUF_SIZE - 1, so the doc records the derivation rather than aline number that will drift again.
The two macOS-leg rationale comments in the workflows carried the same pre-fix
counts and a failure list six entries out of date. Both are recounted from the
manifests.
CHANGELOG.mdstopped at #7632, leaving the 26 PRs merged since thenunrecorded. Added under Security / Fixed / Testing and CI / Documentation.
Why the numbers were wrong in a way reading could not catch
Every figure here is the outcome column of a committed manifest. The previous
values were transcribed once and then maintained by hand, so they decayed as
fixes landed - and a reader checking the table against the prose beside it
would find them perfectly consistent with each other.
The check that finds this class is re-running the derivation, not re-reading
the text, so that is what was done: a script recomputes all five legs and both
summary figures from
tools/ci/upstream-3.5.0-expect*.txtand asserts the twodocuments contain the results, with the superseded strings blacklisted so a
partial edit cannot pass. It also caught a live error in this changeset -
#7659 landed mid-review and flipped
operator-path-partial-dir-daemon, takingmacOS from 236/4 to 237/3 and the distinct-failure count from 24 to 23. The
figures here are derived from master with that merge in place.