Make report generators emit records in canonical order, not filesystem order - #86
Merged
juemerson-at-purestorage merged 2 commits intoAug 3, 2026
Conversation
Get-PfbCmdletParameterInventory and Get-PfbModuleCalledEndpoints both walked Public//Private/ with an unsorted recursive Get-ChildItem and let that walk become the emit order of their records. Regenerating Reports/ on a Linux CI runner instead of a Windows workstation therefore produced a 10,218-line diff across PfbFieldCmdletMap.json/.md and PfbApiDriftReport.json/.md with zero semantic change -- all 2015 entries moved, not one changed content, and the two files were even identical in byte length. That is what makes the update-api-capability-map auto-PR unreviewable. Sorted at emit rather than only on the file list: FullName carries platform-specific separators, so sorting the walk alone is the fragile fix. The walk is sorted too, but only as belt-and-braces for intermediate debugging output. Every sort pins -Culture '' (invariant) so the runner locale cannot reintroduce the divergence. Also replaces Select-Object -Unique with Sort-Object -Unique at the intra-row cmdlet list in Get-PfbParameterCoverageGaps (-Unique preserves input order and does not sort -- the observed `Get-PfbArray, Test-PfbConnection` flip), and sorts the Group-Object groups themselves, whose order is first-appearance in the input and so was also file-walk-derived. The regression tests deliberately do not regenerate twice on one machine -- that is the assertion Tests/Build-PfbApiDriftReport.Tests.ps1 already makes, and enumeration order is stable within a filesystem, so it can never fail. Instead two fixture trees hold the same cmdlets with the cmdlet-to-filename mapping swapped, which reproduces the divergence on any single platform and needs no tools/specs (so it will not silently skip in a fresh clone -- see dmann000#63). Refs dmann000#85 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Do NOT review this line by line. Two of the four files are a pure re-sort; the review that matters is the canonical comparison, reproduced below. Reports/PfbFieldCmdletMap.json and .md -- pure re-sort, zero content change: order-insensitive-equal vs committed: true entries 2015 -> 2015 multiset-equal 2015/2015 repositioned attributesOnly 68 -> 68 multiset-equal 65/68 repositioned typedUnresolved 39 -> 39 multiset-equal 35/39 repositioned markdown: identical line multiset, 161 lines both sides Generating the same artifact with HEAD~1's tools/lib and with this branch, from identical inputs, gives byte-length-identical files (586,461 both) that are order-insensitive-equal -- so the sort provably reorders and never rewrites. Reports/PfbApiDriftReport.json came out raw-identical between those two runs: on NTFS the unsorted walk already happened to yield the canonical order, which is precisely why this defect was invisible from a Windows workstation. Reports/PfbApiDriftReport.json and .md -- a real, expected content refresh, NOT part of the re-sort. The committed copies predate PRs dmann000#78/dmann000#81: uncoveredEndpoints 98 -> 96 PUT /presets/workload, PUT /workloads/tags/batch parameterGaps 436 -> 438 the same two, now as gap rows systemicGaps 252 -> 252 context_names 269 -> 270; 13 body-property names +1 each, all from Set-PfbPresetWorkload Those PRs routed Set-PfbPresetWorkload/Set-PfbWorkloadTag through Invoke-PfbApiRequest, which is the only thing the AST resolver can see -- so the endpoints leave the uncovered list and Set-PfbPresetWorkload immediately reappears carrying 14 missing body properties. "Uncovered -> covered" here means visible to the scanner, not finished; see dmann000#85 and the dmann000#45/dmann000#44 caveat. Per dmann000#85, the two artifacts that must NOT move were regenerated and hashed to confirm they did not: Data/PfbCapabilityMap.json (632 endpoints, 29 versions, -MaxVersion 2.28) and Reports/PfbValueEnumMap.json are both byte-identical to their committed copies. Tests/Build-PfbApiDriftReport.Tests.ps1's "no serialization-only divergence" invariant was already failing on main with 19 differences, unchanged by the sort, and passes again now the report is current. Refs dmann000#85 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
juemerson-at-purestorage
deleted the
fix/deterministic-report-emit-order
branch
August 9, 2026 18:52
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The report generators emitted records in filesystem enumeration order, which differs
between platforms. Regenerating
Reports/PfbFieldCmdletMap.jsonon a Linux runner instead ofa Windows workstation produced a 10,218-line diff containing zero semantic change — which
is what makes the
update-api-capability-mapauto-PR unreviewable, and why #85 asks for thisto land before that workflow is unblocked.
Closes #85.
The fix
tools/lib/PfbCmdletParamTools.ps1Get-PfbCmdletParameterInventorysorts its records at emit (Cmdlet, Parameter, File, Line).tools/lib/PfbApiDriftTools.ps1Get-PfbModuleCalledEndpointssorts its return (Cmdlet, Key, File).Get-PfbParameterCoverageGapssorts theGroup-Objectgroups, whose order wasfirst-appearance in the input, i.e. file-walk-derived — this drove
parameterGapsrow order.Select-Object -Unique→Sort-Object -Uniquefor the per-endpoint cmdletlist.
-Uniquepreserves input order and does not sort; this was the observed intra-row flip(
Get-PfbArray, Test-PfbConnection→Test-PfbConnection, Get-PfbArray). Matches thealready-correct pattern in
Get-PfbConventionStrength.Get-PfbCentralInjectionSitessorted likewise (third instance of the same defect class).Sorted at emit rather than only on the file list:
FullNamecarries platform-specificseparators, so sorting the walk alone is the fragile fix. The three
Get-ChildItemwalks aresorted too, but only as belt-and-braces so intermediate debugging output is stable. Every sort
pins
-Culture ''(invariant) so the runner's locale cannot reintroduce the divergence.Reviewing the artifact commit
Do not read
9bb74a3line by line. Two of its four files are a pure re-sort. The reviewthat matters is the canonical comparison, which is reproduced in the commit message and
summarised here.
Generating
Reports/PfbFieldCmdletMap.jsonfrom identical inputs atHEAD~1and at thisbranch gives files identical in byte length (586,461 both) that are order-insensitive-equal:
entriesattributesOnlytypedUnresolvedPfbFieldCmdletMapping.mdReports/PfbApiDriftReport.jsoncame out raw-identical between those two runs: on NTFS theunsorted walk already happened to yield the canonical order. That is precisely why this defect
was invisible from a Windows workstation and only appeared once CI regenerated on Linux.
Per #85, the two artifacts that must not move were regenerated and hashed to confirm they
did not —
Data/PfbCapabilityMap.json(632 endpoints, 29 versions,-MaxVersion 2.28) andReports/PfbValueEnumMap.jsonare both byte-identical to their committed copies.One real content change, separate from the re-sort
Reports/PfbApiDriftReport.jsonalso picked up a legitimate refresh — the committed copypredated #78/#81:
uncoveredEndpointsPUT /presets/workload,PUT /workloads/tags/batchparameterGapssystemicGapscontext_names269 → 270; 13 body-property names +1 eachThose PRs routed
Set-PfbPresetWorkload/Set-PfbWorkloadTagthroughInvoke-PfbApiRequest,which is the only thing the AST resolver can see. So the endpoints leave the uncovered list and
Set-PfbPresetWorkloadimmediately reappears carrying 14 missing body properties — "uncovered→ covered" here means visible to the scanner, not finished.
This is also why
Tests/Build-PfbApiDriftReport.Tests.ps1's "no serialization-only divergence"invariant was already failing on
mainwith 19 differences before this branch. Baselined bystashing the fix: identical 19 with and without it, so the sort contributed none of them. It
passes again now the report is current.
Regression tests
Deliberately not "regenerate twice on one machine" — that is the assertion
Tests/Build-PfbApiDriftReport.Tests.ps1:380-390already makes, and enumeration order isstable within a single filesystem, so it can never fail. Instead, two fixture trees hold the
same cmdlets with the cmdlet-to-filename mapping swapped. Both trees carry identical file
names, so the walk visits them in the same sequence on any filesystem — only the cmdlet each
position yields differs. That reproduces the divergence on one platform.
They also need no
tools/specs/, so they will not silently skip in a fresh clone or on a CIrunner (see #63).
5 new tests, watched fail first: 4 failed on pwsh 7 and 5 on WinPS 5.1 — the gap-row-order test
passed on Windows before the fix, which is the whole point.
Verification
update-api-capability-map.ymlagainst this branch on a fork(run):
all five generators ran on
ubuntu-latestagainst freshly fetched specs, thenCheck for changesreportedchanged=false—Summarize new versionsandOpen pull requestboth skipped. A Linux regeneration now reproduces the committedartifacts byte-for-byte, which is exactly the property Report generators emit in filesystem order: a 10k-line phantom diff blocks the capability-map auto-PR #85 asks for and the thing
neither a Windows run nor the normal
TestsCI job can demonstrate (per CI silently skips ~23% of the test suite, including the absolute-path regression guards #63 thespecs-gated tooling tests skip on a runner with no spec cache).
Full suite there: 1695 passed / 0 failed / 2 skipped in 347s.
pwsh 7 498 passed / 0 failed, WinPS 5.1 333 passed / 0 failed / 165 skipped,
containers healthy on both.
git statusclean (idempotent).tools/andReports/. No cmdlet, parameter,or wire behaviour changes.
Not in scope
No version bump or CHANGELOG entry, per the maintainer-owned release process.
#63's "cheap half" (re-pointing the three absolute-path guards at the committed report instead
of a regenerated one) was considered for this branch and deliberately left out: it would mean
re-pointing those guards at a file this same PR rewrites, making any failure ambiguous between
a bad artifact and a bad guard. It is cleaner once this has landed.
🤖 Generated with Claude Code