Skip to content

Retire the dashboard's high_dim_data references now the fold is complete - #70

Merged
mmcky merged 2 commits into
mainfrom
cleanup/retire-high-dim-data-strings
Aug 11, 2026
Merged

Retire the dashboard's high_dim_data references now the fold is complete#70
mmcky merged 2 commits into
mainfrom
cleanup/retire-high-dim-data-strings

Conversation

@mmcky

@mmcky mmcky commented Aug 11, 2026

Copy link
Copy Markdown
Contributor

Every consumer was repointed and migration.yml flipped in #69, so the dashboard was the last place still describing a repo the lecture family no longer reads. It named it once on the overview page.

Three changes, and none of them is a plain deletion — each of the two obvious deletions would have broken something.

audit_annotations.yml loses the six entries for the folded datasets. Annotations are curated judgment for refs that are not yet migrated; all six now carry full manifests, which are the authoritative record. The strict audit fails a ref that has neither, so this is only safe because the manifests landed first — the same ordering that governed the rest of the fold.

render_audit.py's external pattern label is generalised, not removed. It read QuantEcon/high_dim_data via raw and media (LFS) hosts, which describes one retired repo — but the entry is still live: migration.yml records prior_pattern: external for all six, and render_audit.py:610 looks the label up to render those rows. Deleting it would have degraded them to a raw enum string. It now reads "reads a data file from another QuantEcon repo by URL", which is true of the prior state and of any future external ref.

The branch-pin finding is restated as history. It described SCF_plus_mini_no_weights.csv as "now reads main" — on a repo the family has since stopped reading altogether. It now says the file was folded in here and every consumer repointed, so the pin is doubly retired.

Verification

check result
build_audit.py scan --strict exit 0, 0 warnings
render_audit.py exit 0
grep high_dim_data across the three generated pages 0, down from 1

Close-out of QuantEcon/workspace-lectures#23 step 3. Remaining there: tag an intro publish, then archive — never delete — high_dim_data.

🤖 Generated with Claude Code

@mmcky @claude
Retire the dashboard's high_dim_data references now the fold is complete
2dcfa6f 
Every consumer was repointed and migration.yml flipped in #69, so the
dashboard was the last place still describing a repo the lecture family
no longer reads. It named it once on the overview page.

Three changes, none of them a plain deletion:

- audit_annotations.yml loses the six entries for the folded datasets.
  Annotations are curated judgment for refs that are NOT yet migrated;
  all six now carry full manifests, which is the authoritative record.
  The strict audit fails a ref with neither, so this is only safe
  *because* the manifests landed first.

- render_audit.py's `external` pattern label is generalised rather than
  removed. It reads "QuantEcon/high_dim_data via raw and media (LFS)
  hosts", which describes one retired repo, but the entry is still live:
  migration.yml records `prior_pattern: external` for all six and
  render_audit.py:610 looks the label up to render it. Deleting it would
  have degraded those rows to a raw enum string.

- The branch-pin finding is restated as history. It described
  SCF_plus_mini_no_weights.csv as "now reads main" on a repo we have
  since stopped reading entirely; it now says the file was folded in and
  every consumer repointed, so the pin is doubly retired.

Verified locally: scan --strict exit 0 with zero warnings, render exit 0,
and grep for high_dim_data across the three generated pages returns 0,
down from 1.

Close-out of QuantEcon/workspace-lectures#23 step 3.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Copilot AI lite review requested due to automatic review settings August 11, 2026 05:29
Copilot AI reviewed Aug 11, 2026
View reviewed changes

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Retires the dashboard’s last remaining high_dim_data-specific references now that the fold is complete and the six datasets are fully represented by manifests + migration.yml.

Changes:

  • Removes the six folded datasets from scripts/audit_annotations.yml so migrated datasets are described only by their manifests.
  • Generalizes the external pattern label in scripts/render_audit.py so it’s not tied to the retired high_dim_data repo.
  • Rewrites the “branch-pin” finding prose to describe the prior state as historical context post-fold.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

File Description
scripts/render_audit.py Generalizes the “external” pattern description and updates the “what changed” narrative to remove high_dim_data-specific wording.
scripts/audit_annotations.yml Removes annotations for datasets that are now covered by manifests (authoritative record) after the fold.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread scripts/render_audit.py Outdated
@mmcky @claude
Describe external by what it excludes, not by whose repo it was
8cedc1e 
Copilot caught that the generalisation in 2dcfa6f was still wrong, just
one level up. classify_url reaches `external` by TWO routes: a
non-QuantEcon org (build_audit.py:180-181) and a QuantEcon repo that is
none of this repo, the consuming repo, a legacy name or a scanned
sibling (:190-191). "another QuantEcon repo" describes only the second,
which is the branch high_dim_data happened to take.

The pattern is defined by what it excludes, so the label now says that:
a GitHub repo outside the audited lecture set, any org. Added a comment
naming both branches so the next edit does not have to re-derive the
definition from classify_url -- this string has now been too narrow
twice, first naming one repo and then one org.

Note the description is currently dormant: PATTERN_META's label (index 1)
renders 30 times in migration.html for `prior_pattern`, but the
description (index 2) only surfaces when a live ref classifies external,
and none does today. A wrong one would have sat unnoticed until the next
external read appeared, which is the argument for fixing it rather than
deferring.

scan --strict exit 0 / 0 warnings; render exit 0; high_dim_data still 0
across the generated pages.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@mmcky
mmcky merged commit 960e1f0 into main Aug 11, 2026
4 checks passed
@mmcky
mmcky deleted the cleanup/retire-high-dim-data-strings branch August 11, 2026 05:44
This was referenced Aug 11, 2026
Merged
Merged
mmcky added a commit that referenced this pull request Aug 11, 2026
@mmcky
PLAN: bring the record up to date now the P3 fold is complete (#73)
ca10bc6 
Seven statements in PLAN.md and one comment in migration.yml still described
the six high_dim_data files as unrepointed. They were true when written on
2026-08-10 and false by the following afternoon, when PR set C landed and #69
flipped the records. PLAN.md's own line 9 is the diagnosis: "every figure below
that restates one is a copy that can drift."

Figures re-read from audit.json (2026-08-11), not carried forward:

- the headline: 18 of 41 migrated / 23 to go / 5 URL forms -> **24 of 41 / 17
  to go / 4 URL forms**
- rule 2: "8 of the 23 remaining datasets have two consuming repos" -> **2 of
  17**, both intro + wasm, both in step 4
- Track A: "17, of which 9 remain" -> **3 remain**; usa-gini's SCF dependency
  is discharged
- Phase 6: 21 non-.yml files / 18 manifests -> **27 / 24**
- Phase 9: 23 datasets to repoint -> **17**
- the CATALOG-vs-`migrated` paragraph explained a six-file gap that has since
  closed; it now explains when the gap appears rather than asserting one

Two boxes were unchecked for work that had landed — the 28 consuming repoints
(Phase 3) and P3 itself (Phase 8). Both are ticked, with the four repoint PRs
named, and P3 gains the record of its consumer half: the C0 -> C1 -> C2
ordering, the flip as its acceptance test (dry-run `landed` -> exit 1 with 6
warnings, `repointed` -> exit 0), and the independent validation against
QuantEcon/workspace-lectures#36.

Two things worth more than the arithmetic:

**Rule 6's tables are now labelled as the pre-fold record.** The "Current host"
column said `media` for four `data.ipynb:37` rows that have read
raw.githubusercontent since 2026-08-11 — a table titled "current" that
describes a state no repo is in is worse than a stale number, because it reads
as a description of today. It is the worked example now, not open work.

**Rule 2 gains the distinction the audit cannot make.** "Two consuming repos"
is `SCAN_REPOS`' count, not the consumer set: both remaining pairs have five
reference-holders each, and three of the five are invisible to every audit run.
That gap is exactly what rule 1 exists for, and rule 2 previously read as
though the dashboard's number were the whole answer.

P3's "three things it proved" list is now five, adding the two findings that
outlived the step: the `# i18n` markers were never exercised, so the hand-diff
is still what protects a localisation; and the translation sync is `.md`-only,
so no hand-localised `_static` asset can be carried by it
(QuantEcon/action-translation#271).

Also drops migration.yml's verbatim quote of a render_audit.py label that #70
deleted — a quoted string that outlives its source is an unfindable false
record. Strict audit green.
mmcky added a commit that referenced this pull request Aug 11, 2026
@mmcky
Land Track A's last two CSVs: life-expectancy and usa-gini (wave A4)
67af2ea 
The two intro statics that the Feb 2025 migration missed, because they sat
under lectures/_static/lecture_specific/ rather than in the datasets/ tree.
Both land at `status: landed` with `consumers: []` — no lecture is repointed by
this PR. Track A's third remaining item, graph.txt, is deliberately NOT here:
it is synthetic teaching data with null provenance in every field, and its one
URL reader gets an inline literal instead.

BYTE GATE, the thing that makes a repoint safe to merge later: both files are
byte-identical to the copies all four consuming repos read today, verified
against lecture-python-intro @ 0066783 —
  life-expectancy-vs-gdp-per-capita.csv  2,059,709 B  sha256 ec5d3235…
  usa-gini-nwealth-tincome-lincome.csv       1,268 B  sha256 bed9074a…
The wasm, zh-cn and canary copies of both paths carry the identical git blob.

DRY-RUN BOTH DIRECTIONS, per the discipline the P3 flip established:
  status: landed     -> strict audit exit 0, zero problems
  status: repointed  -> strict audit exit 1, "marked repointed but consumers
                        still read via ['own-repo', 'sibling']" x2
So `landed` is the only status that passes today, and the flip to `repointed`
has a measured failure to convert into a pass.

THE OWID DELTA IS LARGER THAN THE RECORD SAID, and one widely repeated
description of it is wrong. Fetched today: FOUR of eight columns are renamed,
not one, and `417485-annotations` is RENAMED and MOVED from position 6 to 8 —
not dropped. Rows 62,156 -> 29,912; Year -10000..2021 -> 1..2023; entities
317 -> 276. All four lectures hardcode the old life-expectancy label in
`usecols`, so a refresh raises `ValueError: Usecols do not match columns`;
and quietly, `year == 2018` returns 166 rows in BOTH vintages, so the final
scatter would look unchanged while every value shifted. Recorded as
`diverged` / `upstream-moved` with the full measurement, registered on #39.

THE usa-gini BUILDER LANDS FROZEN, for three independent reasons — any one
sufficient. It has no validate stage. It does not run: line 88 `np.asarray(y)`
returns a read-only array under pandas 3 copy-on-write, so `rd.shuffle` at :90
raises; the identical idiom in the LECTURE was fixed by
QuantEcon/lecture-python-intro#776 but the notebook never received it, because
lectures/_config.yml excludes _static/* from execution so nothing alerts. And
it is non-deterministic — an unseeded shuffle changes the float64 summation
order inside qe.gini_coefficient's prange loop, and the CSV is written at full
repr precision, so it cannot reproduce its own bytes. `unrecovered` would be a
lie: the builder is here, in four repos. Committed verbatim, and the pandas
defect is recorded rather than patched — editing a frozen builder destroys the
property that makes it worth keeping.

This is also the first dataset to answer #14's open sub-question ("a distinct
status value such as `notebook`, vs reusing `not-applicable` with a note —
decide when the first such dataset lands"): `committed-frozen`, no schema
change, matching webscrape_forbes.ipynb and generating_mini.md.

Renamed on landing, unlike the two frozen builders before it. Their upstream
names were meaningful; this one's was `data.ipynb`, which says nothing in a
flat builders/ directory. builders/README.md records both choices and why they
differ.

Annotations removed rather than left for the close-out. #70 retired P3's at
close-out, but usa-gini's note reads "built by inequality/data.ipynb (committed
beside it)" — which this PR makes false, and the audit renders an annotation's
`note` even when a manifest exists. The manifests carry richer, correct
versions; strict audit confirms zero missing_annotations after removal.

One value flagged for the eventual review rather than left to be re-discovered:
l_income at 1965 is 0.7488 against a 0.5159-0.6707 range in every other year.
It is in the published bytes and migrates unchanged (rule 4), and the frozen
non-deterministic builder means it cannot be settled by re-running.

Verified: consumed-file-check 26 manifests / 27 files hash-checked / 0 errors;
strict audit exit 0 with zero problems in all four categories.
mmcky added a commit that referenced this pull request Aug 12, 2026
@mmcky
Land Track A's last two CSVs: life-expectancy and usa-gini (wave A4) (#…
2caa2ee 
…74)

The two intro statics that the Feb 2025 migration missed, because they sat
under lectures/_static/lecture_specific/ rather than in the datasets/ tree.
Both land at `status: landed` with `consumers: []` — no lecture is repointed by
this PR. Track A's third remaining item, graph.txt, is deliberately NOT here:
it is synthetic teaching data with null provenance in every field, and its one
URL reader gets an inline literal instead.

BYTE GATE, the thing that makes a repoint safe to merge later: both files are
byte-identical to the copies all four consuming repos read today, verified
against lecture-python-intro @ 0066783 —
  life-expectancy-vs-gdp-per-capita.csv  2,059,709 B  sha256 ec5d3235…
  usa-gini-nwealth-tincome-lincome.csv       1,268 B  sha256 bed9074a…
The wasm, zh-cn and canary copies of both paths carry the identical git blob.

DRY-RUN BOTH DIRECTIONS, per the discipline the P3 flip established:
  status: landed     -> strict audit exit 0, zero problems
  status: repointed  -> strict audit exit 1, "marked repointed but consumers
                        still read via ['own-repo', 'sibling']" x2
So `landed` is the only status that passes today, and the flip to `repointed`
has a measured failure to convert into a pass.

THE OWID DELTA IS LARGER THAN THE RECORD SAID, and one widely repeated
description of it is wrong. Fetched today: FOUR of eight columns are renamed,
not one, and `417485-annotations` is RENAMED and MOVED from position 6 to 8 —
not dropped. Rows 62,156 -> 29,912; Year -10000..2021 -> 1..2023; entities
317 -> 276. All four lectures hardcode the old life-expectancy label in
`usecols`, so a refresh raises `ValueError: Usecols do not match columns`;
and quietly, `year == 2018` returns 166 rows in BOTH vintages, so the final
scatter would look unchanged while every value shifted. Recorded as
`diverged` / `upstream-moved` with the full measurement, registered on #39.

THE usa-gini BUILDER LANDS FROZEN, for three independent reasons — any one
sufficient. It has no validate stage. It does not run: line 88 `np.asarray(y)`
returns a read-only array under pandas 3 copy-on-write, so `rd.shuffle` at :90
raises; the identical idiom in the LECTURE was fixed by
QuantEcon/lecture-python-intro#776 but the notebook never received it, because
lectures/_config.yml excludes _static/* from execution so nothing alerts. And
it is non-deterministic — an unseeded shuffle changes the float64 summation
order inside qe.gini_coefficient's prange loop, and the CSV is written at full
repr precision, so it cannot reproduce its own bytes. `unrecovered` would be a
lie: the builder is here, in four repos. Committed verbatim, and the pandas
defect is recorded rather than patched — editing a frozen builder destroys the
property that makes it worth keeping.

This is also the first dataset to answer #14's open sub-question ("a distinct
status value such as `notebook`, vs reusing `not-applicable` with a note —
decide when the first such dataset lands"): `committed-frozen`, no schema
change, matching webscrape_forbes.ipynb and generating_mini.md.

Renamed on landing, unlike the two frozen builders before it. Their upstream
names were meaningful; this one's was `data.ipynb`, which says nothing in a
flat builders/ directory. builders/README.md records both choices and why they
differ.

Annotations removed rather than left for the close-out. #70 retired P3's at
close-out, but usa-gini's note reads "built by inequality/data.ipynb (committed
beside it)" — which this PR makes false, and the audit renders an annotation's
`note` even when a manifest exists. The manifests carry richer, correct
versions; strict audit confirms zero missing_annotations after removal.

One value flagged for the eventual review rather than left to be re-discovered:
l_income at 1965 is 0.7488 against a 0.5159-0.6707 range in every other year.
It is in the published bytes and migrates unchanged (rule 4), and the frozen
non-deterministic builder means it cannot be settled by re-running.

Verified: consumed-file-check 26 manifests / 27 files hash-checked / 0 errors;
strict audit exit 0 with zero problems in all four categories.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

Copilot code review Copilot
Copilot review effort
Applies to this pull request for everyone.Learn more about Copilot code review.
Copilot left review comments

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@mmcky