Download each Google Sheet once per run, not once per xdist worker - #130
Merged
Conversation
GoogleSheetTestCases was the only place that knew how to download a Google Sheet without hitting the network once per xdist worker: it hashed the sheet ID into a cache file under cache_dir() and took a FileLock around the fetch. That logic is not specific to the test-case sheet, and the next caller needs it, so move it to sources.google_sheets.fetch_sheet_csv(). No behaviour change beyond the cache key, which now covers the tab name as well as the sheet ID: two tabs of one sheet are two different CSVs and must not share a cache file. Existing gsheet_*.csv files are orphaned by the new key, which costs one download; tests/conftest.py sweeps that glob at the start of every run anyway. The sheet ID is also no longer kept on the GoogleSheetTestCases instance. Nothing read it, and this object's str() reaches assertion messages that the dashboard publishes. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
requests puts the request URL into its exception message, and for an unauthenticated CSV export that URL contains the sheet ID — the capability that grants access to the sheet. So any transient Google error (a 429, a connect timeout) printed the ID into pytest's output, and dashboard.yaml runs pytest on a public repository, where that output is a public Actions log. Catch RequestException around the fetch and re-raise with only the status line, or the exception type when there was no response. `from None` rather than `from e` is the point of the change: pytest prints an exception chain in full, so a suppressed context would have published the original message anyway. Verified against a well-formed but nonexistent sheet ID: the traceback now carries neither the ID nor docs.google.com. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
test_blocklist.py builds its parametrization in pytest_generate_tests, which
runs in every xdist worker. The blocklist was the one collection-time sheet
download that did not go through a cache, so `-n auto` fired one request per
worker (14 on this laptop, 8 in dashboard.yaml) at Google's unauthenticated
gviz endpoint, simultaneously, with a 10s timeout and no retry.
Any one of those failing or returning something different from the others
makes that worker collect a different set of tests, and xdist then aborts the
entire run before executing anything:
ERROR tests/nameres/test_blocklist.py - requests.exceptions.ConnectTimeout
ERROR gw2 - Different tests were collected between gw0 and gw2
Reproduced by failing the fetch in a subset of workers. Routing it through
fetch_sheet_csv() takes the run from 14 downloads to 1, verified by logging
each fetch during a `--target ci-es -n auto tests/nameres/` run.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two invariants, each verified to fail without its fix (with __pycache__ cleared between runs, since a same-length edit can leave a stale .pyc): - Repeated downloads hit the network once, for the test-case sheet and for the blocklist. The blocklist case is the regression that started this: in an xdist run, "the second caller" is another worker. - A failed download names neither the sheet ID nor docs.google.com. The assertion walks the exception chain the way a traceback does — __cause__ always, __context__ only when `raise ... from None` has not suppressed it — because checking str(exc) alone would pass even if the original message, URL included, were still attached and printed. The blocklist CSV is built with the csv module rather than written as a literal: several of the real column names contain a comma or a quote, and a hand-written header silently shifts every field by one. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
cache_dir() read BABEL_VALIDATION_CACHE_DIR straight out of os.environ, but nothing loads .env except resolve_sheet_id(). So the answer changed part-way through a process: tests/conftest.py asks at import time, before any sheet has been resolved, and got ~/.cache/babel-validation, while the downloads ask afterwards and got the override. The visible effect is that pytest_configure's start-of-run gsheet_*.csv sweep has been globbing an empty directory for anyone with the override set. The caches it is meant to clear were never cleared; they only aged out on the one-hour TTL. That is its own route to "Different tests were collected between gw0 and gwN", and one that needs no network failure at all: a cache that expires between one worker's collection and the next has half the run reading the old sheet and half re-downloading it. It also left unlink_if_exists()'s containment check comparing paths against a cache_dir() that could move under it, which would abort the run outright. So cache_dir() now loads .env itself. Once per process, not per call: load_dotenv() leaves an existing key alone but re-sets a deleted one, and reloading each time would make the override impossible to unset in a test. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A collection-time fetch runs once per xdist worker, so it has to be cached, or one bad response aborts the run with "Different tests were collected". And a requests exception message carries the URL, which carries the sheet ID — the next outbound call built from a secret will need the same handling. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The fake load_dotenv wrote os.environ directly. Where there is no .env at all — CI — the monkeypatch.delenv above it has no previous value to restore, so that write outlived the test and test_cache_files_live_in_the_cache_dir then compared the issue cache's directory against a tmp_path. It passed locally only because .env supplies the key here, giving monkeypatch something to put back. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
pytest --target ci-es -n auto tests/nameres/intermittently died before running anything:test_blocklist.pybuilds its parametrization inpytest_generate_tests, which runs inevery xdist worker, and it was the one collection-time sheet download with no cache:
-n autofired 14 simultaneous requests at Google's unauthenticated gviz endpoint with a 10stimeout and no retry. Any one of them failing makes that worker collect a different set of
tests, and xdist then aborts the whole run.
dashboard.yamlruns-n 8against everytarget daily, so this was live in CI, where it surfaces as the "Target run broke"
annotation with no results written for that target.
GoogleSheetTestCasesalready had the fix — aFileLocked download cached undercache_dir()— so this PR lifts that intosources.google_sheets.fetch_sheet_csv()androutes both sheets through it. Verified by logging each fetch during a
--target ci-es -n auto tests/nameres/run: 14 blocklist downloads before, 1 after, with the same165/162/117/1950 outcome split.
Two other things this turned up
A failed download printed the sheet ID.
requestsputs the request URL in its exceptionmessage, and for an unauthenticated CSV export that URL is the capability that grants
access to the sheet. A transient Google error therefore wrote it into pytest's output — and
dashboard.yamlruns pytest on a public repository, so that is a public Actions log. Thehelper now re-raises with the status line only.
from Nonerather thanfrom eis thesubstance of it: pytest prints an exception chain in full, so suppressing the context is
what actually stops the original message being published. No rotation was needed — this was
found by injecting a failure locally, not in a published log.
The start-of-run cache sweep was globbing the wrong directory.
cache_dir()readBABEL_VALIDATION_CACHE_DIRstraight fromos.environ, but nothing loads.envexceptresolve_sheet_id(). So its answer changed part-way through a process:tests/conftest.pyasks at import time and got
~/.cache/babel-validation, while the downloads ask afterwardsand got the override. For anyone with the override set,
pytest_configure'sgsheet_*.csvsweep has been deleting nothing, and the caches it exists to clear only aged out on the
one-hour TTL.
That is a second, independent route to the same error, and one that needs no network
failure: a cache that expires between one worker's collection and the next leaves half the
run reading the old sheet and half re-downloading it. It also left
unlink_if_exists()'s containment check comparing against acache_dir()that could moveunder it, which raises and aborts the run outright.
cache_dir()now loads.envitself,once per process — not per call, because
load_dotenv()re-sets a key that has beendeleted, which would make the override impossible to unset in a test.
Gotchas worth keeping
two different CSVs and must not share a file. Existing
gsheet_*.csvare orphaned by thenew key, which costs one download.
GoogleSheetTestCasesinstance. Nothing read it,and that object's
str()reaches assertion messages the dashboard publishes.from Nonetofrom e, andputting the uncached
requests.getback — with__pycache__cleared between runs.csvmodule: several real column namescontain a comma or a quote, and a hand-written header silently shifts every field by one.
Testing
pytest -m unit— 201 passed (was 192).pytest --target ci-es -n auto tests/nameres/— same failure/pass split as before thechange, no xdist errors.
pytest tests --target dev -n 8 -k <no match>— the whole tree collects identicallyacross 8 workers.
The remaining
tests/nameresfailures againstci-esare findings about NameRes ES, notdefects in this repo; they are what the
validate-nameres-esbranch is for.