FRED from GitHub runners: no custom User-Agent; canary one leg per builder

[mmcky]

The first canary run over #114's builders failed on us_business_cycle_monthly.csv, twice, with a 60-second read timeout on the very first FRED request — and the workflow did what it should: classified it as a fetch failure, wrote nothing, and opened #115. The World Bank legs passed.

Root cause, measured rather than guessed. A throwaway probe run on a GitHub-hosted runner (#116, closed and branch deleted) hit fredgraph.csv?id=UNRATE with several user agents:

User-Agent	Result from the runner
Python-urllib/3.12 (urllib's default)	200, 14,181 B, ~50 ms
curl/8.5.0	200, 14,181 B, ~50 ms
qeld-builder (what _fred.py sent)	stalls until the read times out
Mozilla/5.0 …Chrome/128…	stalls until the read times out

The same requests all succeed from a workstation, which is how the custom agent survived local testing — and why the lecture repos' own builds (pandas, no custom agent) have always reached FRED from the same runners.

Fix. The Fred class sends no User-Agent by default (urllib's own), with the measurement recorded in its docstring so nobody re-adds one without measuring from where the builder runs; fred_data.py drops the same header (no byte change — it only ever ran locally). Verified: a dry run of business_cycle_fred.py with the default agent reproduces the committed vintage byte for byte.

Two workflow fixes found on the same runs. The canary ran a set-writing builder once per dataset — three identical World Bank fetches — so snapshots.py list --by-builder now gives it one leg per builder (named for its first dataset, datasets listed). And the dataset dispatch input filtered only the refresh matrix; it now filters the canary as well.

After merge I'll dispatch the workflow once more; a green FRED leg closes #115.

🤖 Generated with Claude Code

[Copilot]

Pull request overview

This PR addresses FRED fetch timeouts observed on GitHub-hosted runners by removing the custom User-Agent header from FRED requests, and refines the dynamic-snapshot canary planning so set-writing builders run once per builder (not once per dataset).

Changes:

• Stop sending a custom User-Agent in FRED fetches (builders/_fred.py, builders/fred_data.py) to avoid runner-specific stalls.
• Extend scripts/snapshots.py list with --dataset filtering and --by-builder grouping for canary planning.
• Update the refresh-snapshots workflow to build the canary matrix from snapshots.py list --by-builder and apply the dataset filter consistently.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 3 comments.

File	Description
scripts/snapshots.py	Adds dataset filtering and a by-builder listing mode for the canary matrix.
builders/fred_data.py	Removes the custom User-Agent from legacy FRED fetches and documents the runner behavior.
builders/_fred.py	Makes Fred default to “no custom UA” and documents the measured runner behavior.
.github/workflows/refresh-snapshots.yml	Switches canary planning to one leg per builder and applies the dataset filter to canary planning.

Suppressed comments (1)

scripts/snapshots.py:354

• The new argparse setup packs multiple statements onto single lines with semicolons. This is inconsistent with the surrounding Python style in this repo and makes future edits (e.g., adding help text / defaults) harder to review and maintain.

    p = sub.add_parser("list"); p.add_argument("--dataset"); p.add_argument("--by-builder", action="store_true")
    p.set_defaults(fn=cmd_list)
    p = sub.add_parser("due"); p.add_argument("--all", action="store_true"); p.add_argument("--dataset")
    p.set_defaults(fn=cmd_due)
    p = sub.add_parser("stamp"); p.add_argument("dataset"); p.add_argument("--summary", required=True)
    p.set_defaults(fn=cmd_stamp)
    p = sub.add_parser("pr-body"); p.add_argument("dataset"); p.add_argument("--summary", required=True)
    p.set_defaults(fn=cmd_pr_body)

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[mmcky]

Fixed in e2e7730 — the plan step builds due_flags and list_flags as bash arrays and expands them as "${arr[@]}", so the dataset input reaches the script as exactly one argument however it is spelled. Simulated the step's shell locally with and without a dataset: filtered to one leg, and both legs, respectively.