Skip to content

Make report generators emit records in canonical order, not filesystem order - #86

Merged
juemerson-at-purestorage merged 2 commits into
dmann000:mainfrom
juemerson-at-purestorage:fix/deterministic-report-emit-order
Aug 3, 2026
Merged

Make report generators emit records in canonical order, not filesystem order#86
juemerson-at-purestorage merged 2 commits into
dmann000:mainfrom
juemerson-at-purestorage:fix/deterministic-report-emit-order

Conversation

@juemerson-at-purestorage

Copy link
Copy Markdown
Collaborator

Summary

The report generators emitted records in filesystem enumeration order, which differs
between platforms. Regenerating Reports/PfbFieldCmdletMap.json on a Linux runner instead of
a Windows workstation produced a 10,218-line diff containing zero semantic change — which
is what makes the update-api-capability-map auto-PR unreviewable, and why #85 asks for this
to land before that workflow is unblocked.

Closes #85.

The fix

tools/lib/PfbCmdletParamTools.ps1

  • Get-PfbCmdletParameterInventory sorts its records at emit (Cmdlet, Parameter, File, Line).

tools/lib/PfbApiDriftTools.ps1

  • Get-PfbModuleCalledEndpoints sorts its return (Cmdlet, Key, File).
  • Get-PfbParameterCoverageGaps sorts the Group-Object groups, whose order was
    first-appearance in the input, i.e. file-walk-derived — this drove parameterGaps row order.
  • Same function: Select-Object -UniqueSort-Object -Unique for the per-endpoint cmdlet
    list. -Unique preserves input order and does not sort; this was the observed intra-row flip
    (Get-PfbArray, Test-PfbConnectionTest-PfbConnection, Get-PfbArray). Matches the
    already-correct pattern in Get-PfbConventionStrength.
  • Get-PfbCentralInjectionSites sorted likewise (third instance of the same defect class).

Sorted at emit rather than only on the file list: FullName carries platform-specific
separators, so sorting the walk alone is the fragile fix. The three Get-ChildItem walks are
sorted too, but only as belt-and-braces so intermediate debugging output is stable. Every sort
pins -Culture '' (invariant) so the runner's locale cannot reintroduce the divergence.

Reviewing the artifact commit

Do not read 9bb74a3 line by line. Two of its four files are a pure re-sort. The review
that matters is the canonical comparison, which is reproduced in the commit message and
summarised here.

Generating Reports/PfbFieldCmdletMap.json from identical inputs at HEAD~1 and at this
branch gives files identical in byte length (586,461 both) that are order-insensitive-equal:

Collection Count Result
entries 2015 → 2015 multiset-equal, 2015/2015 repositioned
attributesOnly 68 → 68 multiset-equal, 65/68 repositioned
typedUnresolved 39 → 39 multiset-equal, 35/39 repositioned
PfbFieldCmdletMapping.md 161 lines both identical line multiset

Reports/PfbApiDriftReport.json came out raw-identical between those two runs: on NTFS the
unsorted walk already happened to yield the canonical order. That is precisely why this defect
was invisible from a Windows workstation and only appeared once CI regenerated on Linux.

Per #85, the two artifacts that must not move were regenerated and hashed to confirm they
did not — Data/PfbCapabilityMap.json (632 endpoints, 29 versions, -MaxVersion 2.28) and
Reports/PfbValueEnumMap.json are both byte-identical to their committed copies.

One real content change, separate from the re-sort

Reports/PfbApiDriftReport.json also picked up a legitimate refresh — the committed copy
predated #78/#81:

Collection Change
uncoveredEndpoints 98 → 96 — PUT /presets/workload, PUT /workloads/tags/batch
parameterGaps 436 → 438 — the same two, now as gap rows
systemicGaps 252 → 252 — context_names 269 → 270; 13 body-property names +1 each

Those PRs routed Set-PfbPresetWorkload/Set-PfbWorkloadTag through Invoke-PfbApiRequest,
which is the only thing the AST resolver can see. So the endpoints leave the uncovered list and
Set-PfbPresetWorkload immediately reappears carrying 14 missing body properties — "uncovered
→ covered" here means visible to the scanner, not finished.

This is also why Tests/Build-PfbApiDriftReport.Tests.ps1's "no serialization-only divergence"
invariant was already failing on main with 19 differences before this branch. Baselined by
stashing the fix: identical 19 with and without it, so the sort contributed none of them. It
passes again now the report is current.

Regression tests

Deliberately not "regenerate twice on one machine" — that is the assertion
Tests/Build-PfbApiDriftReport.Tests.ps1:380-390 already makes, and enumeration order is
stable within a single filesystem, so it can never fail. Instead, two fixture trees hold the
same cmdlets with the cmdlet-to-filename mapping swapped. Both trees carry identical file
names, so the walk visits them in the same sequence on any filesystem — only the cmdlet each
position yields differs. That reproduces the divergence on one platform.

They also need no tools/specs/, so they will not silently skip in a fresh clone or on a CI
runner (see #63).

5 new tests, watched fail first: 4 failed on pwsh 7 and 5 on WinPS 5.1 — the gap-row-order test
passed on Windows before the fix, which is the whole point.

Verification

  • End-to-end on Linux, which is the platform that exposed the bug. Dispatched
    update-api-capability-map.yml against this branch on a fork
    (run):
    all five generators ran on ubuntu-latest against freshly fetched specs, then
    Check for changes reported changed=falseSummarize new versions and
    Open pull request both skipped. A Linux regeneration now reproduces the committed
    artifacts byte-for-byte, which is exactly the property Report generators emit in filesystem order: a 10k-line phantom diff blocks the capability-map auto-PR #85 asks for and the thing
    neither a Windows run nor the normal Tests CI job can demonstrate (per CI silently skips ~23% of the test suite, including the absolute-path regression guards #63 the
    specs-gated tooling tests skip on a runner with no spec cache).
    Full suite there: 1695 passed / 0 failed / 2 skipped in 347s.
  • Scoped Pester over the 12 affected test files, both editions:
    pwsh 7 498 passed / 0 failed, WinPS 5.1 333 passed / 0 failed / 165 skipped,
    containers healthy on both.
  • Re-running both generators leaves git status clean (idempotent).
  • No live FlashBlade testing: this touches only tools/ and Reports/. No cmdlet, parameter,
    or wire behaviour changes.

Not in scope

No version bump or CHANGELOG entry, per the maintainer-owned release process.

#63's "cheap half" (re-pointing the three absolute-path guards at the committed report instead
of a regenerated one) was considered for this branch and deliberately left out: it would mean
re-pointing those guards at a file this same PR rewrites, making any failure ambiguous between
a bad artifact and a bad guard. It is cleaner once this has landed.

🤖 Generated with Claude Code

Get-PfbCmdletParameterInventory and Get-PfbModuleCalledEndpoints both walked
Public//Private/ with an unsorted recursive Get-ChildItem and let that walk
become the emit order of their records. Regenerating Reports/ on a Linux CI
runner instead of a Windows workstation therefore produced a 10,218-line diff
across PfbFieldCmdletMap.json/.md and PfbApiDriftReport.json/.md with zero
semantic change -- all 2015 entries moved, not one changed content, and the two
files were even identical in byte length. That is what makes the
update-api-capability-map auto-PR unreviewable.

Sorted at emit rather than only on the file list: FullName carries
platform-specific separators, so sorting the walk alone is the fragile fix. The
walk is sorted too, but only as belt-and-braces for intermediate debugging
output. Every sort pins -Culture '' (invariant) so the runner locale cannot
reintroduce the divergence.

Also replaces Select-Object -Unique with Sort-Object -Unique at the intra-row
cmdlet list in Get-PfbParameterCoverageGaps (-Unique preserves input order and
does not sort -- the observed `Get-PfbArray, Test-PfbConnection` flip), and
sorts the Group-Object groups themselves, whose order is first-appearance in
the input and so was also file-walk-derived.

The regression tests deliberately do not regenerate twice on one machine --
that is the assertion Tests/Build-PfbApiDriftReport.Tests.ps1 already makes, and
enumeration order is stable within a filesystem, so it can never fail. Instead
two fixture trees hold the same cmdlets with the cmdlet-to-filename mapping
swapped, which reproduces the divergence on any single platform and needs no
tools/specs (so it will not silently skip in a fresh clone -- see dmann000#63).

Refs dmann000#85

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Do NOT review this line by line. Two of the four files are a pure re-sort; the
review that matters is the canonical comparison, reproduced below.

Reports/PfbFieldCmdletMap.json and .md -- pure re-sort, zero content change:

  order-insensitive-equal vs committed: true
  entries         2015 -> 2015   multiset-equal   2015/2015 repositioned
  attributesOnly    68 -> 68     multiset-equal     65/68  repositioned
  typedUnresolved   39 -> 39     multiset-equal     35/39  repositioned
  markdown: identical line multiset, 161 lines both sides

Generating the same artifact with HEAD~1's tools/lib and with this branch, from
identical inputs, gives byte-length-identical files (586,461 both) that are
order-insensitive-equal -- so the sort provably reorders and never rewrites.
Reports/PfbApiDriftReport.json came out raw-identical between those two runs:
on NTFS the unsorted walk already happened to yield the canonical order, which
is precisely why this defect was invisible from a Windows workstation.

Reports/PfbApiDriftReport.json and .md -- a real, expected content refresh,
NOT part of the re-sort. The committed copies predate PRs dmann000#78/dmann000#81:

  uncoveredEndpoints  98 -> 96   PUT /presets/workload, PUT /workloads/tags/batch
  parameterGaps      436 -> 438  the same two, now as gap rows
  systemicGaps       252 -> 252  context_names 269 -> 270; 13 body-property
                                 names +1 each, all from Set-PfbPresetWorkload

Those PRs routed Set-PfbPresetWorkload/Set-PfbWorkloadTag through
Invoke-PfbApiRequest, which is the only thing the AST resolver can see -- so the
endpoints leave the uncovered list and Set-PfbPresetWorkload immediately
reappears carrying 14 missing body properties. "Uncovered -> covered" here means
visible to the scanner, not finished; see dmann000#85 and the dmann000#45/dmann000#44 caveat.

Per dmann000#85, the two artifacts that must NOT move were regenerated and hashed to
confirm they did not: Data/PfbCapabilityMap.json (632 endpoints, 29 versions,
-MaxVersion 2.28) and Reports/PfbValueEnumMap.json are both byte-identical to
their committed copies.

Tests/Build-PfbApiDriftReport.Tests.ps1's "no serialization-only divergence"
invariant was already failing on main with 19 differences, unchanged by the
sort, and passes again now the report is current.

Refs dmann000#85

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@juemerson-at-purestorage
juemerson-at-purestorage merged commit 5e5a269 into dmann000:main Aug 3, 2026
4 checks passed
@juemerson-at-purestorage
juemerson-at-purestorage deleted the fix/deterministic-report-emit-order branch August 9, 2026 18:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Report generators emit in filesystem order: a 10k-line phantom diff blocks the capability-map auto-PR

1 participant