Skip to content

docs: re-derive the testsuite figures from the manifests that produce them - #7661

Merged
oferchen merged 1 commit into
masterfrom
docs/testsuite-leg-counts
Sep 4, 2026
Merged

docs: re-derive the testsuite figures from the manifests that produce them#7661
oferchen merged 1 commit into
masterfrom
docs/testsuite-leg-counts

Conversation

@oferchen

@oferchen oferchen commented Sep 4, 2026

Copy link
Copy Markdown
Owner

docs: re-derive the testsuite figures from the manifests that produce them

README.md ships the exact awk command that reads a leg's expected-outcome
manifest and prints its pass/fail/skip counts. Running it disagrees with every
row of the table directly beneath it.

row documented re-derived
non-root, pipe 257 / 3 / 85 259 / 1 / 85
root, pipe 286 / 3 / 56 288 / 1 / 56
non-root, tcp 101 / 21 / 33 108 / 14 / 33
root, tcp 113 / 27 / 15 120 / 20 / 15

SECURITY.md carries the same four rows and the same two summary figures, so
both files drifted together. Corrected throughout, along with:

  • "3 of 345 tests currently diverge" across the full-corpus legs. It is
    one - filter-merge-content-echo - and the README names it now rather
    than leaving the reader to count.
  • "29 across all four" distinct failures. It is 23, and across
    five manifests, not four: the README's own glob
    tools/ci/upstream-3.5.0-expect.*.txt has always matched the macOS file its
    prose excluded.
  • The macOS leg was missing from both tables. It runs on every PR as
    upstream-testsuite-macos (ci.yml), on the full 345-cell corpus, with its
    own committed manifest. It is the only leg that can observe a
    platform-conditional divergence - one of its three remaining failures skips
    on Linux, so it had never executed in this repository's CI before the leg
    existed. Added as a fifth row, with its non-required status stated.
  • "the required checks being the two stdio-pipe legs." All four Linux legs
    have been required contexts since ci: run the 3.5.0 testsuite over loopback TCP on every pull request #7408 wired the TCP pair into PR CI; the
    ruleset returns ten contexts, not eight.
  • proxy protocol hosts described as "not yet implemented" and "still
    under audit" in two places. It shipped in fix(daemon): refuse a PROXY header from an untrusted direct peer #7648: parsed into a
    ProxyProtocolPolicy that mirrors upstream's allow_proxy_protocol_peer(),
    rejecting every peer when the trusted list is empty or unset, and warning at
    startup on the combination upstream warns about.
  • The MAX_PROXY_LINE_BYTES citation pointed at connect/proxy.rs:344.
    The constant moved and, more usefully, stopped being a typed literal: it is
    now PROXY_BUF_SIZE - 1, so the doc records the derivation rather than a
    line number that will drift again.

The two macOS-leg rationale comments in the workflows carried the same pre-fix
counts and a failure list six entries out of date. Both are recounted from the
manifests.

CHANGELOG.md stopped at #7632, leaving the 26 PRs merged since then
unrecorded. Added under Security / Fixed / Testing and CI / Documentation.

Why the numbers were wrong in a way reading could not catch

Every figure here is the outcome column of a committed manifest. The previous
values were transcribed once and then maintained by hand, so they decayed as
fixes landed - and a reader checking the table against the prose beside it
would find them perfectly consistent with each other.

The check that finds this class is re-running the derivation, not re-reading
the text, so that is what was done: a script recomputes all five legs and both
summary figures from tools/ci/upstream-3.5.0-expect*.txt and asserts the two
documents contain the results, with the superseded strings blacklisted so a
partial edit cannot pass. It also caught a live error in this changeset -
#7659 landed mid-review and flipped operator-path-partial-dir-daemon, taking
macOS from 236/4 to 237/3 and the distinct-failure count from 24 to 23. The
figures here are derived from master with that merge in place.

… them

`README.md` ships the exact `awk` command that reads a leg's expected-outcome
manifest and prints its pass/fail/skip counts. Running it disagrees with every
row of the table directly beneath it.

| row | documented | re-derived |
|---|---|---|
| non-root, pipe | 257 / 3 / 85 | **259 / 1 / 85** |
| root, pipe | 286 / 3 / 56 | **288 / 1 / 56** |
| non-root, tcp | 101 / 21 / 33 | **108 / 14 / 33** |
| root, tcp | 113 / 27 / 15 | **120 / 20 / 15** |

`SECURITY.md` carries the same four rows and the same two summary figures, so
both files drifted together. Corrected throughout, along with:

- **"3 of 345 tests currently diverge"** across the full-corpus legs. It is
  **one** - `filter-merge-content-echo` - and the README names it now rather
  than leaving the reader to count.
- **"29 across all four"** distinct failures. It is **23**, and across
  **five** manifests, not four: the README's own glob
  `tools/ci/upstream-3.5.0-expect.*.txt` has always matched the macOS file its
  prose excluded.
- **The macOS leg was missing from both tables.** It runs on every PR as
  `upstream-testsuite-macos` (`ci.yml`), on the full 345-cell corpus, with its
  own committed manifest. It is the only leg that can observe a
  platform-conditional divergence - one of its three remaining failures *skips*
  on Linux, so it had never executed in this repository's CI before the leg
  existed. Added as a fifth row, with its non-required status stated.
- **"the required checks being the two stdio-pipe legs."** All four Linux legs
  have been required contexts since #7408 wired the TCP pair into PR CI; the
  ruleset returns ten contexts, not eight.
- **`proxy protocol hosts` described as "not yet implemented"** and "still
  under audit" in two places. It shipped in #7648: parsed into a
  `ProxyProtocolPolicy` that mirrors upstream's `allow_proxy_protocol_peer()`,
  rejecting every peer when the trusted list is empty or unset, and warning at
  startup on the combination upstream warns about.
- **The `MAX_PROXY_LINE_BYTES` citation** pointed at `connect/proxy.rs:344`.
  The constant moved and, more usefully, stopped being a typed literal: it is
  now `PROXY_BUF_SIZE - 1`, so the doc records the derivation rather than a
  line number that will drift again.

The two macOS-leg rationale comments in the workflows carried the same pre-fix
counts and a failure list six entries out of date. Both are recounted from the
manifests.

`CHANGELOG.md` stopped at #7632, leaving the 26 PRs merged since then
unrecorded. Added under Security / Fixed / Testing and CI / Documentation.

## Why the numbers were wrong in a way reading could not catch

Every figure here is the outcome column of a committed manifest. The previous
values were transcribed once and then maintained by hand, so they decayed as
fixes landed - and a reader checking the table against the prose beside it
would find them perfectly consistent with each other.

The check that finds this class is re-running the derivation, not re-reading
the text, so that is what was done: a script recomputes all five legs and both
summary figures from `tools/ci/upstream-3.5.0-expect*.txt` and asserts the two
documents contain the results, with the superseded strings blacklisted so a
partial edit cannot pass. It also caught a live error in this changeset -
#7659 landed mid-review and flipped `operator-path-partial-dir-daemon`, taking
macOS from 236/4 to 237/3 and the distinct-failure count from 24 to 23. The
figures here are derived from master with that merge in place.
@oferchen
oferchen merged commit 3f0bc50 into master Sep 4, 2026
10 checks passed
@oferchen
oferchen deleted the docs/testsuite-leg-counts branch September 4, 2026 06:34
@github-actions github-actions Bot added the documentation Improvements or additions to documentation label Sep 4, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant