Compress ordinals/forms data, drop build-only data retention - #277
Merged
Conversation
Add ordinals.json and address forms.json to the build.rs zstd table; replace their include_str! embeddings with OUT_DIR decode-on-use. Turn the symbols and org-types LazyLock statics into caller-owned decode functions: every reader is a cached tagger/replacer build, so the parsed source data now drops once those structures exist instead of staying resident for process life. Refresh the stale zstd comment in Cargo.toml and related doc drift. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
pudo
force-pushed
the
feat/data-embedding
branch
from
September 3, 2026 12:48
aaec2eb to
f6c50f1
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two related cleanups to the embedded-data handling in
rigour-core:Compression coverage.
data/text/ordinals.json(~82 KiB) anddata/addresses/forms.json(~11 KiB) were still embedded as plaininclude_str!blobs. Both are now entries in thebuild.rszstd table and are decoded fromOUT_DIRon use, like the other four data files. The committed plain-text sources are unchanged, so the CImake builddiff check is unaffected.Retention of build-only data. The parsed forms of
symbols.json(five HashMaps, ~85 KiB source) andorg_types.json(Vec<OrgTypeSpec>, ~126 KiB source) lived inLazyLockstatics for process life, even though every reader is a tagger/replacer build that is itself cached (TAGGER_CACHE/REPLACER_CACHE). Those statics are now caller-owned decode functions, following the modelperson_names.rsandterritories.rsalready use — the data drops as soon as the derived automata exist. Same for the ordinals spec list, whose only runtime consumer (ordinals_dict()→ Python) sits behind a Python-side@cache.Untouched: the stopwords/compare tables (their statics are the runtime structures, and the sources are 1–3 KiB).
Also refreshes the stale
zstdcomment inrust/Cargo.tomland related doc drift (build.rs header sizes,generate_text.pydocstring, CLAUDE.md data-resources section, territories module comment).Verified:
cargo clippy --all-targets -- -D warnings(with and without--features python),cargo test(259 passed),pytest --cov rigour(532 passed),mypy --strict rigourclean.🤖 Generated with Claude Code