Releases: jakeryderv/usdata
Release list
v0.10.0
Added
- CO-OPS observed water levels for one station, explicit datum and UTC interval, with raw CSV quality fields, response validation, and a reproducible manifest example.
noaa:gsoyannual station CSVs with complete UTC year selection, shared NCEI geographic discovery, reproducible restoration, and a small manifest example.
Documentation
- Keep generated catalog files separate from handwritten documentation and combine dataset reference facts with usage guides in the local docs site.
- Make the dataset catalog easier to browse with Released, Source only, and Planned labels; explicit file formats and selection behavior; and automatic dataset navigation.
- Make the first-use walkthrough the site home, separate NOAA dataset guides from service research, and clarify source installation and reproducibility.
Development
- Add optional fast commit hooks, workflow linting, and PR-title validation; pin GitHub Actions to reviewed commits while retaining Dependabot updates.
- Allow focused manual live-test and notebook CI runs while retaining complete weekly coverage; summarize test failures, skips, and timings in Actions.
- Assemble release notes with Towncrier fragments, validate notes in PRs, and preview upcoming changes in the documentation.
- Generate the radar-site CSV with LF line endings so fresh worktrees stay clean, and apply lint fixes before formatting.
- Prepare releases on a branch before editing files, validate draft release PRs before review/merge, check navigation and notebook release notices, and provide guarded cleanup for merged local branches and worktrees.
v0.9.0
Documentation
-
Default the documentation site to dark mode, with a light-mode toggle.
-
Add a local documentation site with navigation, Mermaid diagrams, generated API/CLI references, saved notebook previews, and strict CI link checks. Separate generated dataset catalogs from provider access notes.
Added
-
Per-dataset live checks, independent example jobs, retained test/coverage and
failed-notebook diagnostics, and scheduled minimum-core-dependency validation. -
Shared adapter conformance scenarios for every available dataset, covering
deterministic listing, fetch destinations, byte preservation, and client ownership. -
Documented L0–L4 test levels, responsibility-based suites, and
just test-live
selection; existing integration command and marker aliases remain supported. -
Pure
select_by_timeasset selection with explicit tolerance and nearest/prior
direction, signed offsets, candidate counts, and an explicit no-match result. -
Explicit zero-based radar sweep selection with
FetchedAsset.open(sweep=...)
and an executed Storm Events / NEXRAD / GOES event-context notebook.
Fixed
-
Require Typer 0.18 or newer; the old declared minimum failed to construct the
CLI with modern union annotations and lacked current Click compatibility. -
Reject truncated S3 listings with missing or cycling continuation tokens instead
of returning incomplete assets or requesting pages indefinitely. -
Reject NEXRAD decoder tables that misalign moment and coordinate records,
including equal-length sweeps following a missing interior end marker; valid
explicitly selected sweeps remain readable without silently dropping others. -
Reject out-of-range or non-finite point coordinates and negative/non-finite
radii before bounding-box clipping; complete the manifest provider options for
GSOM, GOES ABI, and Storm Events.
v0.8.0
Added
- Optional local NetCDF4 opening with eager xarray loading, CF decoding,
closed file resources, source metadata, and an executed GOES infrared notebook. noaa:goes-abiCONUS Cloud and Moisture Imagery with explicit satellite/channel
selection, precise scan times and sizes, and checksum-verified NetCDF downloads.- Storm Events annual event-details gzip CSVs, selecting current creation-date
revisions with exact source-byte provenance and locked restoration; local
compressed CSV opening preserves event and geographic identifier strings. - Executed Storm Events notebook with local Oklahoma/date filters, event-record
and damage-rating plots, and source/reporting caveats. - Optional NEXRAD Level II opening with xradar sweep DataTrees, native units and
flag masking, source provenance, and an executed reflectivity notebook.
Changed
- Replace example analysis scripts with executed Jupyter notebooks containing
compact data previews, plots, and source provenance. Add an optional examples
environment, offline notebook checks, and isolated live execution/refresh commands. - Pin the development interpreter to Python 3.14.7 to avoid an upstream crash
affecting NetCDF decoding in older Linux uv Python builds.
v0.7.0
Added
- Opt-in GHCN station-search diagnostics with bounded response details, and
independent live checks for geographic discovery and explicit-station downloads. noaa:gsommonthly station CSVs through NCEI, with geographic discovery,
complete-month selection, reproducible restoration, and a pandas analysis example.- Terminal progress for CLI fetch and pull, showing known asset sizes, HTTP download
bytes and retry attempts, and validated cache hits. Disable with--no-progress;
redirected output retains its existing format.
v0.6.0
Added
- Optional
FetchedAsset.open()CSV reading throughusdata[pandas], including
ERDDAP units metadata, string identifiers/codes, explicit dtype/date/column/row
options, and a fetch-to-analysis example. Opening leaves raw bytes and provenance intact.
Changed
- Test core-only and pandas installations in CI and installed-wheel checks.
- Check versioned source-only documentation notices during validation so release
PRs cannot carry known stale availability wording. - Focus v0.6 on optional CSV readers and practical usage; NASA, model output,
climate grids, and remote caching remain follow-up work.
v0.5.0
Breaking
- Manifest pulls now fail when a required source resolves to no assets. Set
allow_empty: trueper source to permit an intentionally empty result. verifyrejects a manifest edited since locking (exit 2), matchingpull.- NOAA adapters reject unknown provider parameters, empty explicit identifiers,
invalid units, and conflicting or invalid radar selectors.
Fixed
- Generated provider and roadmap docs distinguish unreleased implementations
from support included in the declared package version. - GET metadata and download requests retry transient failures up to three attempts,
respecting boundedRetry-Afterdelays and preserving existing files on failure. - Release publishing promotes the distributions from successful CI for the exact
release commit, with version checks, instead of rebuilding.
Changed
- Narrow v0.5 to USGS, ERDDAP/CoastWatch, complete place lookup, and reliability;
additional NOAA adapters and optional readers remain follow-up work. - Enforce offline unit tests and smoke-test the installed wheel on Linux, macOS,
and Windows. Add release-tool regression tests. - Add a manifest reference and a small NOAA/USGS example with a live round-trip test.
Added
-
noaa:coastwatch-sst: bounded ERDDAP gridded CSV subsets by location, UTC
time, and variable, with optional spatial stride and reproducible restoration. -
Census 2025 lookup for 56 states/DC/territories and 3,235 counties/equivalents,
including county/state names and quoted FIPS. Add--locationas an alias for
--state. Generated bounds replace the six approximate seed boxes, so
geographic queries may select different stations; existing lockfiles stay pinned. -
usgs:water-daily: the first non-NOAA adapter, using the modern USGS Water Data
API for paginated CSV downloads by site or bbox, dates, parameter codes, and
statistic. Preserves units and quality metadata; supports cached fetches,
manifest pull, locked restoration, and verification without new dependencies.
v0.4.0
Breaking
- Manifest and source fields reject unknown keys instead of silently ignoring them;
provider-specific inputs remain supported underparams.
Fixed
- Verify cached bytes and source provenance before reuse or fresh lockfile creation.
- Reject unsafe cache paths, including symlinks escaping the cache root.
- Preserve existing files after failed downloads or checksum mismatches; replace downloads,
provenance, and lockfiles atomically using unique temporary files. - Reuse provider clients across downloads and close internally owned clients on success or failure.
- Report invalid dates, radar IDs, manifest YAML, and lockfiles with clear CLI input errors.
- Infer multiple domains correctly when constructing a custom registry without a domain catalog.
- Gate package publishing on successful CI for the exact main-branch release commit.
v0.3.0
Added
-
usdata pullfetches every source in a manifest and writes a lockfile pinning each asset's
checksum and provenance; a laterpullrestores from the lockfile without re-resolving, and
refuses (until--force) if the manifest changed.usdata verifyreports missing or altered
cached files. Python API:usdata.pull,usdata.verify. -
Datasets carry a
status(available, stub, planned).searchandinfoshow it; fetching a
planned dataset exits 3 with a clear message. -
Planned registry entries for USGS, NASA, EPA, FEMA, Census, USDA, and GOES so the roadmap is
visible fromusdata searchand the docs. -
Provider metadata (name, homepage) in the registry; README gains a generated per-provider table.
-
Datasets carry a
domain(shared taxonomy across providers),since(version shipped) or
target(planned phase orlater).searchandinfoshow them; the roadmap lists
datasets by target version, generated from the registry. -
26 NOAA registry entries spanning 18 domains, each verified for anonymous access
(except GHCN-Hourly, whose entry says its bulk path is unconfirmed), with target phases.
The NOAA provider page gains a data-landscape table mapping domains to entries.
Changed
usdata searchhides planned datasets unless--plannedis given.
Changed
- Dataset reference moved to
docs/providers/: one page per provider with hand-written access notes and a generated dataset table, plus a generated index.
v0.2.1
Added
- Type information is shipped (
py.typed). - Contributor docs:
CONTRIBUTING.md,SECURITY.md, a guide to adding datasets, issue and PR templates.
v0.2.0
First usable release.
- Curated dataset registry with keyword search filtered by provider, bbox, and time
noaa:ghcn-daily: station observations via the NCEI search and data servicesnoaa:nexrad-level2: Level II volume scans from the unidata-nexrad-level2 bucket, anonymous S3 over HTTPS- Local cache with sha256 verification and a provenance sidecar for every fetched file
- CLI:
usdata search,info,fetch(with--dry-run), and manifest validation inpull
Not yet implemented: ERDDAP adapter, pull downloads and lockfiles. See docs/roadmap.md.