Embed core data, lazy verified ephemeris, offline mode, platform data dirs (Phase 2) - #139
Merged
Conversation
ssmichael1
added a commit
that referenced
this pull request
Aug 30, 2026
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017fhrVLbjyHhmuGzU4ohEXE
Compile the core data tier into the library so frames and gravity work with no data directory and no network: IERS tables 5.2a/b/d and the EGM96/JGM2/JGM3/ITU_GRACE16 coefficient files (truncated to degree 70) are gzip'd by tools/embed_data.py into data/embedded/ (308,675 bytes, provenance in SOURCES.json) and included with include_bytes!. Loaders prefer a copy found in any data search directory, fall back to the embedded copy (one-time note, silenced by SATKIT_QUIET=1), and only then download. Rework datadir: resolution is a pure, testable function over an Env snapshot that yields an ordered list of *search* directories and a single platform *write* directory (SATKIT_DATA first for both; macOS ~/Library/Application Support/satkit-data, Linux $XDG_DATA_HOME or ~/.local/share/satkit-data, Windows %LOCALAPPDATA%\satkit-data). Directories next to the dylib and inside site-packages are searched but never created or written to. The provisioning marker is now the presence of a JPL ephemeris file rather than a sentinel. New API: search_dirs / find_data_file / add_search_dir; set_datadir keeps its meaning. The ephemeris is resolved across all search directories (highest DE version wins) and any manifest-pinned ephemeris is auto-downloaded on first use with a SHA-256 check. Add offline mode: SATKIT_OFFLINE=1 or utils::set_offline(true) (is_offline() reads back; the setter overrides the environment) forbids every download and returns download::Error::Offline listing the sources that would have been tried; a build without the `download` feature returns the same error. Search and embedded data are unaffected. Tests: embedded blobs match SOURCES.json, parse, and give bit-identical accelerations to the full files at the degree-40 cap; resolver order on macOS/Linux/Windows; offline fetch makes zero HTTP requests against an in-process server; setter-vs-env precedence; tests/offline_smoke.rs covers gravity, precession-nutation, time scales, SGP4, Kepler/Lambert with SATKIT_DATA empty and SATKIT_OFFLINE=1. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017fhrVLbjyHhmuGzU4ohEXE
…ne mode satkit-data is no longer a hard dependency (pip install "satkit[data]" keeps the optional full bundle); when the package is present its data directory is added to the search list at import. The embedded blobs are added to MANIFEST.in so an sdist build is self-contained. New in satkit.utils: data_search_dirs(), add_search_dir(path), set_offline(enabled), is_offline(); datadir() now documents the per-platform write location and set_datadir/update_datafiles/ datafiles_exist docstrings reflect the new semantics. Type stubs updated (stubtest clean). python/test/test_offline.py exercises the offline contract (gravity, precession-nutation, time scales, SGP4, Kepler/Lambert, search-dir API, set_offline round trip, typed error for a missing ephemeris). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017fhrVLbjyHhmuGzU4ohEXE
CI: run the offline_ unit tests, tests/offline_smoke.rs, a --no-default-features build, and python/test/test_offline.py with SATKIT_DATA pointing at an empty directory and SATKIT_OFFLINE=1. Docs: datafiles.md gains the three data tiers, "Where satkit looks for data, and where it writes" (per-platform search-order table), the environment-variable / API table (SATKIT_DATA, SATKIT_DATA_URL, SATKIT_OFFLINE, SATKIT_JPLEPHEM_FILE, SATKIT_QUIET, set_offline), verification, provisioning and ephemeris selection; installation.md, docs/index.md, docs/api/utils.md, README.md and data/README.md (rationale for the search/write split, no writes into site-packages, sentinel file dropped) updated to match. CHANGELOG entry marked Breaking for Python packaging. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017fhrVLbjyHhmuGzU4ohEXE
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017fhrVLbjyHhmuGzU4ohEXE
Stacked PRs whose base is a feature branch got no checks at all because neither the push nor the pull_request branch filter matched. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017fhrVLbjyHhmuGzU4ohEXE
A non-ephemeris file now fails with NotJplEphemeris instead of a header parse artefact, and load failures name the resolved path and the env var that selected it. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017fhrVLbjyHhmuGzU4ohEXE
…refix Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017fhrVLbjyHhmuGzU4ohEXE
- Concurrent first use: every download streams into its own <name>.part.<pid>.<seq>; a fetch that finishes second verifies the winner's file and discards its own. No lock files. Tested with 8 threads on one entry: one verified file, no leftovers. - No writable location: datadir::Error::NoWriteableDirectory now carries the reason (SATKIT_DATA unset + no HOME/XDG/LOCALAPPDATA, or the chosen directory not creatable / read-only, probed by writing a temp file) and the search directories consulted. path_for() is fallible; no caller falls back to the current directory any more. - Corrupt on-disk pinned file: ManifestEntry::ensure_verified hashes a manifest-known file once (DE440: ~0.2 s) and records a <name>.sha256-verified sidecar (hash, size, mtime) so later loads only stat; the lazy ephemeris load re-fetches a corrupt copy, or under offline mode returns download::Error::CorruptFile with the expected hash. fetch_static_file uses the same check, so repeat runs skip the hash. - Rename over an open file (Windows): retry_io retries 6x at 50 ms, then download::Error::ReplaceFailed names the destination. - Proxies: ureq 3.4's default agent already reads HTTPS_PROXY/HTTP_PROXY/ ALL_PROXY and NO_PROXY; tested and documented. - Docs: offline statement now says frames need an EOP file to be exact, failure-behaviour tables in datafiles.md and data/README.md, release-tag immutability policy; CHANGELOG line extended. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017fhrVLbjyHhmuGzU4ohEXE
ssmichael1
force-pushed
the
feat/data-embed
branch
from
August 30, 2026 01:06
9317b84 to
0d5b1f1
Compare
ssmichael1
marked this pull request as ready for review
August 30, 2026 01:07
This was referenced Aug 30, 2026
Merged
ssmichael1
added a commit
that referenced
this pull request
Aug 30, 2026
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017fhrVLbjyHhmuGzU4ohEXE
ssmichael1
added a commit
that referenced
this pull request
Aug 31, 2026
…y/truncated tables (#166) * ierstable: fall back to the compiled-in table when the on-disk copy is corrupt; reject empty and truncated tables A tab5.2*.txt in a search directory that fails to read or parse now warns and loads the compiled-in copy (the same IERS 2010 series, so the fallback is exact) instead of panicking in `table()`. That panic was the last reachable one after #139 embedded the tables; with this it is unreachable short of a build defect, which makes the planned breaking Result-threading (#125) unnecessary. The parser also now rejects text with no `j =` header (a proxy notice page or empty file previously loaded as six empty series and silently zeroed the nutation terms) and a table whose header promises more rows than the text delivers (a truncated transfer), so that such files actually trigger the fallback rather than loading as garbage. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PG969yapJ84DJceKt21Wen * changelog: #166 ierstable fallback entry Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PG969yapJ84DJceKt21Wen --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Phase 2 of the data-hosting plan, stacked on #137 (this PR's base is
feat/data-manifest, so the diff is Phase 2 only). Draft for design review.What changes for users
pip install satkitno longer pullssatkit-data(~105 MB). Core data — IERS nutation tables 5.2a/b/d and the four gravity models to degree 70 — is compiled into the library (309 KB gzip, inflated on first use). Frames, gravity, SGP4, time, Kepler and Lambert work with no data directory and no network.SATKIT_JPLEPHEM_FILE=lnxp1900p2053.421for DE421, 14 MB), from the GitHubdata-v1release first.satkit-dataremains an optional offline bundle (pip install satkit-data): an installed copy or a system directory is found read-only.SATKIT_OFFLINE=1orsatkit.utils.set_offline(True)/satkit::utils::set_offline(true)(setter wins once called). Blocks all downloads with a typed error naming the file and URLs; the no-downloadbuild returns the same error. Zero HTTP attempts under offline mode (tested).src/utils/datadir.rs): search locations are separate from the single write location. Search:SATKIT_DATA→ extra dirs (add_search_dir) → dylib-adjacent → site-packagessatkit_data→ platform user dir →~/.satkit-data(legacy) →/usr/share/satkit-data→/Library/Application Support/satkit-data. Write:SATKIT_DATA, else macOS~/Library/Application Support/satkit-data, Linux$XDG_DATA_HOME/satkit-data, Windows%LOCALAPPDATA%\satkit-data. Downloads never land next to the dylib or inside site-packages (they used to, in a fresh venv). Thetab5.2a.txtsentinel is gone; lookup is per file. Pureresolve()function with per-platform precedence tests.SATKIT_DATA,SATKIT_DATA_URL,SATKIT_OFFLINE,SATKIT_JPLEPHEM_FILE,SATKIT_QUIET(silences the one-time "using embedded copy" note).Sizes
Embedded 308,675 B · crate 986 KiB compressed (limit 10 MB) · sdist 3.3 MB (ships
data/embedded/*anddata/manifest.json) · wheel ≈ +309 KB.Verification
--no-default-featuresbuildscargo test --features chrono: 258 lib + integration + 18 GMAT (residuals unchanged) + 6offline_smokeSATKIT_DATA+SATKIT_OFFLINE=1: 10offline_lib tests, 6 smoke, Pythontest_offline.py— in-tree and from a clean-venv sdist install with no datapytest161 passed; stubtest clean; mkdocs builds with no new warnings (datafiles page: per-platform search/write tables, env vars, offline API)github.com/ssmichael1/satkit-data/releases/download/data-v1/…, EOP/SW from CelesTrak, 6.2 sDocs
getting-started/datafiles.md(tiers, search order, write location, env vars, offline capability),installation.md,index.md,api/utils.md,data/README.md(rationale), README,datadir()doc comment,utils.pyi.Follow-ups: short DE440 asset; conda recipe from the manifest; retire GCS.
🤖 Generated with Claude Code
https://claude.ai/code/session_017fhrVLbjyHhmuGzU4ohEXE