P4 groundwork: manifest, four-stage builder and provenance/ for business_cycle

[mmcky]

The on-ramp to the P4 dynamic-snapshot pilot: the last unmanifested dataset gets its manifest, its builder gets the four-stage contract, and the two World Bank metadata dumps leave the published tree. Closes the Phase 6 backfill and settles #13 (option 3, as recommended there on 2026-08-06). Two commits: the work, then a short policy note.

The finding that shaped it. Re-running the builder against live WDI (lastupdated 2026-07-13) does not reproduce the committed CSV: the World Bank revised 236 of the 320 overlap cells (63 of 64 year columns; median change 0.0 pp, 99th percentile 1.0, max 1.5) and appended YR2024/YR2025. This is a genuine tracking snapshot, so equality is the wrong overlap test and diverged is its normal state between refreshes.

What changes

• lectures/business_cycle_data.csv.yml — class: dynamic-snapshot, cadence: annual, integrity.upstream: diverged / upstream-moved with the measured delta, the first pattern column entry (YR\d{4}, as Builder architecture: four-stage template, traceback-clean failures, notebook policy (Phase 5) #14 recommended for a column set that grows), consumers: []. Intro's business_cycle still makes the live wbgapi call the builder reproduces; whether it adopts the snapshot is the P4 decision, not this PR's.
• builders/business_cycle.py — fetch → pre-process → validate → write. validate() asserts the grid, the five economies, percent units, the one structural null (YR1960) and recency, and bounds the overlap window against the committed snapshot at 5 pp (3× the largest routine revision observed) with a printed revision summary — the review surface for a future refresh PR. --out-dir gives a dry run. The bytes in lectures/ are unchanged; a refresh is a separate, reviewed PR.
• provenance/ — both dumps move here (git mv), with a README saying what they are evidence for. Not served (the Pages job assembles site/, lectures/ and audit.json only), no manifests, regenerated every run. Zero readers org-wide by code search and audit.
• Bookkeeping: migration.yml record (landed, P4), manifest-schema.yml documents the pattern form, CATALOG.md regenerated (41 datasets), counts corrected in AGENTS.md, builders/README.md and requirements.txt, and four PLAN.md boxes ticked (Phase 5 retrofit; Phase 6 manifest and classify; Phase 7 dynamic re-run — "and they do not agree, by design").
• Second commit: an AGENTS.md section, "Refresh, break, or schema change — who gets told", and a per-consumer on_refresh: rebuild | review field sketched in the schema. An upstream interface change fails validate(), writes nothing and opens an issue here (consumers unaffected by construction; the change is absorbed in pre_process); a successful refresh fans out to consumers per on_refresh; a deliberate schema change is a new filename plus hand-opened consumer issues. Recorded ahead of the Phase 5 workflows so they are written to a stated policy.

Judgement calls, flagged for review: retrieved: null (the bytes came from the builder at QuantEcon/data b857c5c, 2025-02-16, but AGENTS.md says not to reconstruct a date from git history — the vintage is pinned by content instead); the 5 pp bound; date_range.end: 2023 as the concrete newest column rather than the schema's null; the dumps moved rather than deleted, since they are what the licence fields cite. The catalog's summary line now says "1 awaiting repoint" for a landed record with no consumer — the renderer's vocabulary, left for a later tidy.

Gates. check_consumed_files: 41 manifests, 0 errors. Strict audit scan against the same local clones: problem set identical before and after. Builder dry run against live WDI: passes, prints the overlap summary above. The register entry this manifest's integrity.upstream points at is posted on #39 alongside this PR.

See #13, #14, #39.

🤖 Generated with Claude Code

[Copilot]

Pull request overview

Groundwork for the P4 dynamic-snapshot pilot by fully onboarding the business_cycle_data.csv dataset into the repo’s manifest/builder/provenance conventions, while keeping the published dataset bytes unchanged.

Changes:

• Added a full sidecar manifest for lectures/business_cycle_data.csv (classed as dynamic-snapshot, with cadence, schema patterning, and upstream-delta documentation).
• Refactored builders/business_cycle.py into the four-stage fetch → pre-process → validate → write structure and moved metadata/info dumps out of the published lectures/ tree into provenance/.
• Updated repo bookkeeping/docs (CATALOG.md, migration.yml, PLAN.md, AGENTS.md, builders/README.md, manifest-schema.yml, requirements.txt) to reflect completion of the Phase 6 backfill and the new pattern schema concept.

Reviewed changes

Copilot reviewed 9 out of 12 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
requirements.txt	Updates pandas usage commentary to include business_cycle.py.
provenance/README.md	Introduces and documents the new unserved provenance/ directory purpose.
provenance/business_cycle_metadata.md	Moves World Bank series metadata dump into provenance.
provenance/business_cycle_info.md	Moves wb.series.info(...) dump into provenance.
PLAN.md	Marks Phase 5/6/7 checkboxes as completed for business_cycle groundwork.
migration.yml	Adds a P4 business_cycle_data.csv record and status bookkeeping.
manifest-schema.yml	Documents columns[].pattern as an alternative to enumerated column names.
lectures/business_cycle_data.csv.yml	Adds the missing dataset manifest for the dynamic snapshot.
CATALOG.md	Regenerates catalog to include the new manifested dataset (41 total).
builders/README.md	Updates builder inventory to reflect business_cycle as a dynamic snapshot and provenance output relocation.
builders/business_cycle.py	Refactors builder into four stages with validation and provenance output.
AGENTS.md	Updates repo map/counts and adds a policy section about refresh/break/schema-change notifications.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.