test: make the suite offline by default, trim fixtures and snapshots - #359
Merged
Merged
Conversation
The suite depended on USGS uptime. 46 tests queried the real services on every push, so an unrelated commit's CI could go red because a gage did not report or a service returned a transient 503. Those tests also asserted on live data volumes (``len(df) > 0``, ``df.shape[1] == 97``), which is a claim about today's water, not about this package. Mock the request/response wiring instead, using real captured response shapes trimmed to two features -- property names, nesting, and value types verbatim, only the row count reduced. The assertions move to what we own: the outgoing request and the shape of the returned frame. Doing this surfaced seven wrong assumptions the live tests could not catch, including a water-year filter test that an empty frame satisfied and so could never fail. Keep the 12 tests whose subject genuinely is the upstream API behind a ``live`` marker, deselected by default and run nightly. A mock cannot detect upstream drift, because the mock is the thing that would need updating. Also shrink the test fixtures (53 MB -> 1.1 MB) and replace two brittle snapshots. The 22 MB fixtures existed to satisfy ``df.size == 5087443`` assertions; the signature and export snapshots failed on renames and reflows, which break no caller, while the property-based checks that replace them still catch a new required argument or a family growing a second public home. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The test suite depended on USGS uptime. 46 tests queried the real services on every push, so an unrelated commit's CI could go red because a gage didn't report or a service returned a transient 503. Those tests also asserted on live data volumes —
len(df) > 0,df.shape[1] == 97— which is a claim about today's water, not about this package.livemarker, deselected by default viaaddopts = "-m 'not live'"inpyproject.tomland run nightly by a newlive-api.ymlworkflow. A mock can't detect upstream drift, because the mock is the thing that would need updating.tests/data/goes 53 MB → 1.1 MB. The two 22 MB files existed only to satisfyassert df.size == 5087443-style magic numbers.str(inspect.signature(f))snapshot incontracts/public_api_test.pyand the frozen name sets inarchitecture_test.py. Both failed on renames, reorders, and annotation reflows — changes that break no caller — while the property-based checks that replace them still catch a new required argument, a stale re-export, or a family growing a second public home.Why this is worth the diff
Writing the mocks surfaced seven wrong assumptions the live tests could not catch, because a test that only asserts "some rows came back" passes regardless of whether the request was built correctly. The clearest case:
test_get_peaks_water_year_filterasserted only that returned rows fell inside the requested years — which an empty frame satisfies, so it could never fail. Others included the wrongskipGeometrycasing, a CQL2 body envelope that doesn't exist, and a multi-site fan-out that actually comma-joins into one request.Results
tests/data/livetests, nightlyNetwork isolation was verified empirically with a socket-blocking pytest plugin: the only test that opens a socket is
test_fan_out_outlives_pool_timeout_on_real_transport, which binds a localhost server on purpose (aMockTransportbypasses the connection pool it's testing).Test plan
pytest tests/— 761 passed, 12 deselected, 5.6 spytest tests/ -m live— 12 passed against the real APIsruff check/ruff format --check/mypyclean; all pre-commit hooks passget_queryablesand making a family re-export another family's getter both fail as intendedlive-api.ymlschedule fires once merged to the default branch🤖 Generated with Claude Code