feat(build): #90 — build the GAUL lookup arrow-native, and refuse an empty one - #220
Merged
Merged
Conversation
…empty one
The lookup builder was the last runtime pandas import in the repository. The sources
are parquet and the output is parquet; pandas was only ever the thing in the middle.
Proven against the real producer, not asserted
Rebuilt land_gaul from the views-datafactory checkout and compared to the committed
artifact:
rows 64,742 -> 64,742
column order identical
schema types identical (including dictionary<string> on the four names)
ALL 10 COLUMNS VALUE-IDENTICAL
adr / region / n_cells / n_dropped_incomplete / lookup_version / source_provenance
byte-identical
That is #90's re-baselined acceptance criterion met exactly: contents identical, and
the flat declared `lookup_version` key reproduced byte-for-byte (C-60).
One deliberate difference: the written file no longer carries a `pandas` schema-metadata
blob. It described an index arrow does not have, the only consumer reads the file as
arrow (`contract.gaul_lookup.load`), and reproducing it would have meant keeping
knowledge of pandas' metadata format in the one script this change exists to remove
pandas from. Stated in the docstring rather than left to be discovered.
The committed artifact is NOT regenerated here. The builder is what #90 asks for;
rewriting a delivered artifact is a separate act with no delivery benefit.
C-76 closed, and the guard failed its own first test
`build()` now refuses a zero-row result instead of writing one and printing `cells=0`
as though that were an outcome. The first draft did not survive its mutation test: on
an empty table `pa.array([True] * 0)` infers NULL type, so `pc.and_` raised
ArrowNotImplementedError in the completeness filter BEFORE the zero-row check could
speak -- the confusing-late-failure C-76 exists to prevent, relocated by one function.
The mask is now explicitly typed.
C-75 decided: RETIRE, and not in this PR
The entry's Owner field required whoever took #90 to make the keep-or-retire call on
GaulLookupEnricher "not to defer a third time". Retire: zero production callers, its
"build/verification path" justification is spent now that this path is arrow-native, it
holds the package's last pandas reference, and its _gather duplicates the shipping
gather in contract/historical.py -- which four test files cover independently. C-45 is
the precedent and was resolved by deleting.
Not executed here: the retirement touches ten files including a CIC, ADR-012 and four
test files. Mixing that with a builder rewrite is what epic #148's S5 explicitly refused
to do. It is the next change.
Epic #85 and tracking #93 stay OPEN until it lands -- their claim only becomes true when
that module is gone.
373 passed / 40 xfailed / 0 failed. ruff clean. Register 83/15/68.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This was referenced Aug 4, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
B3 of the sequenced plan. Closes #90. The builder was the last runtime pandas import in the repository — the sources are parquet, the output is parquet, and pandas was only ever the thing in the middle.
Proven against the real producer, not asserted
Rebuilt
land_gaulfrom the views-datafactory checkout and compared to the committed artifact:dictionary<string>on the four name columnsadr/region/n_cells/n_dropped_incomplete/lookup_version/source_provenanceThat is #90's re-baselined acceptance criterion met exactly: contents identical, and the flat declared
lookup_versionkey reproduced byte-for-byte (C-60).One deliberate difference: the written file no longer carries a
pandasschema-metadata blob. It described an index arrow does not have, the only consumer reads the file as arrow (contract.gaul_lookup.load), and reproducing it would have meant keeping knowledge of pandas' metadata format in the one script this change exists to remove pandas from. Said in the docstring rather than left to be discovered.The committed artifact is not regenerated here. The builder is what #90 asks for; rewriting a delivered artifact is a separate act with no delivery benefit.
C-76 closed — and the guard failed its own first test
build()now refuses a zero-row result instead of writing one and printingcells=0as though that were an outcome.The first draft did not survive its mutation test. On an empty table
pa.array([True] * 0)infers NULL type, sopc.and_raisedArrowNotImplementedErrorin the completeness filter before the zero-row check could speak — the confusing-late-failure C-76 exists to prevent, relocated by one function. The mask is now explicitlypa.bool_(), and the test asserts the refusal a human can act on.C-75 decided: retire — and deliberately not here
The entry's Owner field required whoever took #90 to make the keep-or-retire call on
GaulLookupEnricher, "not to defer a third time."Retire. Zero production callers (only three test files import it); its "build/verification path" justification is spent now that this path is arrow-native; it holds the package's last pandas reference; and its
_gatherduplicates the shipping gather incontract/historical.py, which four test files cover independently. C-45 is the precedent and was resolved by deleting.Not executed in this PR. The retirement touches ten files — the module, its 39 tests, references in two other test files, its CIC, ADR-012,
gaul_lookup.py's docstring, the machinery list intest_clone_readiness.py, and the pandas-importer assertion intest_doc_accuracy.py. Mixing that with a builder rewrite is exactly what epic #148's S5 refused to do. It is the next change.Epic #85 and tracking #93 stay open until it lands — their claim, pandas pushed to the seams, only becomes true when that module is gone.
Verification
The fidelity suite's synthetic source is now an arrow table (
_load_sourcereturns a table with agidcolumn; arrow has no index) — same column vocabulary, so the builder's contract with that seam is unchanged.Register 83 / 15 open / 68 resolved.