Add generated dataset catalog (CATALOG.md + build_catalog.py) - #19
Merged
Conversation
Realises PLAN Phase 2's "index/catalog page from the manifests — doubles as the dataset registry". Migrated-only scope: a dataset appears once it has a manifest. - scripts/build_catalog.py reads lectures/*.yml and emits CATALOG.md — one row per dataset with class, source, licence + redistribution, integrity status, builder status, size and consumers (which lectures use it, linked). Generated, not hand-maintained, so it cannot drift from the manifests; a Phase 5 CI check will assert currency via `git diff --exit-code CATALOG.md`. - CATALOG.md — first generation: the 4 migrated datasets (P1 lingcod + P2 pandas_panel trio). Surfaces the honest gaps AGENTS.md promises the catalog shows: unverifiable upstream integrity, the countries.csv restricted licence (tracked in QuantEcon/workspace-lectures#20), and unrecovered builders. - requirements.txt: add PyYAML for the parser. - PLAN.md: tick the Phase 2 catalog item. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
Implements PLAN Phase 2’s “generated catalog / dataset registry” by adding a generator script that derives CATALOG.md from the authoritative per-dataset manifests in lectures/*.yml, keeping the registry in sync with migrated datasets.
Changes:
- Add
scripts/build_catalog.pyto generate a migrated-only dataset catalog from sidecar manifests. - Add
CATALOG.mdas the generated output (and updatePLAN.mdto mark Phase 2 catalog complete). - Add
PyYAMLtorequirements.txtto support manifest parsing.
Reviewed changes
Copilot reviewed 3 out of 4 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
| scripts/build_catalog.py | New generator that reads lectures/*.yml manifests and emits a Markdown registry (CATALOG.md). |
| requirements.txt | Adds PyYAML dependency required by the catalog builder script. |
| PLAN.md | Marks Phase 2 catalog work as complete and documents the generator + intended CI check. |
| CATALOG.md | Generated catalog output committed for the current set of migrated datasets. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Copilot review (data-lectures#19): Path.open()/write_text() used the platform default encoding. Manifests carry non-ASCII (en-dashes, curly quotes) and the catalog emits Unicode marks (checkmark/warning), so on a non-UTF-8 locale (e.g. Windows cp1252) reads/writes could raise UnicodeDecode/EncodeError. This script is headed for CI where the runner locale is not guaranteed. No output change on UTF-8 systems. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Generated dataset catalog (PLAN Phase 2)
Realises the PLAN Phase 2 item "index/catalog page from the manifests — doubles as the dataset registry", as a durable in-repo registry of what has been migrated.
Approach: generated, not hand-maintained
The per-dataset manifests (
lectures/*.yml) stay the single source of truth.scripts/build_catalog.pyreads them and emitsCATALOG.md— so the catalog cannot drift from the manifests (the same derive-don't-duplicate principle behind the #15 cutover tracker). Regenerate on any manifest change; a Phase 5 CI check will assert currency withgit diff --exit-code CATALOG.md, failing the build if someone edits a manifest without regenerating.What it shows
One row per migrated dataset: class, source (linked), licence +
redistribution, integrity (upstream.status), builder status, size, and consumers — the lectures that use it, linked to each file. It deliberately surfaces the honest gaps AGENTS.md says the catalog should show:unverifiableupstream integrity, thecountries.csvrestrictedlicence (tracked in QuantEcon/workspace-lectures#20), andunrecoveredbuilders.Scope: migrated-only
A dataset appears once it has a manifest — currently the 4 migrated in the pilot (P1
lingcod+ the P2pandas_paneltrio). Files inlectures/without a manifest are not yet migrated and stay tracked in PLAN Phase 9; they'll appear here automatically as their manifests land. Later feeds the Pages index (Phase 4).Part of QuantEcon/meta#338; registry surface for QuantEcon/QuantEcon.manual#108.