Skip to content

v1.8.0 - Chronological observations, correct last_n, and clearer ABS 404s

Choose a tag to compare

@AnthonyPuggs AnthonyPuggs released this 09 Jun 22:03
· 26 commits to main since this release

v1.8.0 - Chronological observations, correct last_n, and clearer ABS 404s

This release fixes a family of correctness bugs around "latest value" retrieval that
affected every ABS dataset, and makes ABS error handling honest about what a 404 means.

🐛 Fixed: last_n returned the oldest ABS observations

The ABS SDMX API returns CSV rows in arbitrary order, but latest-N selection assumed
chronological input — so last_n=2 on headline_cpi returned 1986-Q2/Q3 instead of the
two most recent quarters. get_latest_observations and get_economic_series inherited
the same bug.

Observations are now sorted by parsed period before selection:

  • last_n selects the true most recent N observations per series, regardless of
    upstream row order.
  • All retrieval responses are now chronologically ascending — this is a contract
    guarantee, documented in the response schema docs.
  • Date-bound filtering compares parsed periods (interval overlap) instead of raw
    strings, so bounds like start=2025-11-15 interact correctly with quarterly data.

A new shared periods.py module handles all five period formats (YYYY, YYYY-QN,
YYYY-SN, YYYY-MM, YYYY-MM-DD), replacing duplicated parsing in derived.py and
bounds.py — and adding the previously missing semester support.

🐛 Fixed: ABS 404s are no longer one opaque error

The ABS API answers HTTP 404 for two very different situations. The server now tells
them apart by response body:

  • No data in the requested window (NoRecordsFound) → returns a valid empty payload
    with a plain-English metadata.warnings entry instead of raising.
  • Unknown / renamed / retired dataflow → raises AuseconNotFoundError with guidance
    to check search_datasets or list_catalogue.
  • 404s never fall back to stale cached data anymore; transient 5xx/timeout failures
    still do.

📦 Retail Trade (RT) marked as discontinued

The ABS ceased the Retail Trade publication after the June 2025 reference month. The
catalogue entry is now marked ceased (hidden from default discovery, available via
include_ceased=true) with a successor pointer to the Monthly Household Spending
Indicator
(household_spending concept, dataflow HSI_M), surfaced through
describe_dataset. The retail_turnover concept still retrieves the full historical
series (1982–2025).

✨ Improvements

  • New metadata.observations_dropped field records how many observations last_n
    removed; truncated is true exactly when this is non-zero.
  • describe_dataset now reports ceased and successor for every dataset.
  • ABS cache keys use structured encoding, eliminating a theoretical collision between
    absent parameters and literal "None" values.
  • Clearer last_n / count parameter descriptions on the MCP tool surface.

🧪 Testing

  • New regression suite drives a 64-row shuffled ABS fixture end-to-end — the exact
    failure mode that production CSVs exhibit and small ordered fixtures could never catch.
  • Coverage for both 404 paths, the stale-fallback exclusion, semester periods, and
    cache-key collisions.
  • Fixed two pre-existing Windows-only test failures (UTF-8 fixtures read with the
    platform default cp1252 encoding).

⚠️ Behavioural changes

  • ABS payload observation order changes from upstream-arbitrary to chronologically
    ascending (RBA/APRA were already chronological, now guaranteed).
  • ABS "no results in window" requests return an empty payload instead of raising.
  • RT no longer appears in default search_datasets / list_catalogue results.

No breaking schema changes — all new metadata fields are additive and optional.