Audit: assert on the media host, and on data-lectures URL resolvability - #55
Merged
Conversation
`--strict` exited 0 on a fold that 404s every read. `pattern` is derived
from org and repo alone, so a `data-lectures` URL counted as fully migrated
no matter what host served it, what ref it pinned, or whether the file had
landed here yet. `lfs_media` was computed per reference at build_audit.py
and asserted on nowhere.
Two independent assertions, not one:
* repoint rule 6 — a `data-lectures` reference on
media.githubusercontent.com. That host is the LFS media endpoint and
routes per path, so it 404s plain-git files, and everything published
here is plain git.
* resolvability — ref is not `main`, path is not `lectures/<file>`, or
the file is not committed here yet.
They cannot be folded together: a media URL parses to exactly the same
(ref, path) as the raw URL beside it, so a resolvability check alone exits
0 on the media-host trap. A canonical url_form check is not available as a
shortcut either — 25 of the 33 current data-lectures refs use the
github.com/{org}/{repo}/raw/{ref}/ form, which is correct outside wasm.
The rule-5 comment said the media host was a valid target for LFS files,
which this makes false; PLAN's "not covered by CI" paragraph goes stale on
the same change. Both updated here.
Verified both ways. Today's tree: exit 0, zero warnings, on 33
data-lectures refs. A doctored lecture-wasm with four reads on the media
host, one pinning `master` and one keeping the source subdirectory: exit 1
with each class named, and rule 5 fires zero times on it — the host trap
is invisible to every other check in the file.
Gate 1 of QuantEcon/workspace-lectures#23 step 3. Closes #54.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
Tightens the audit’s strict mode so it fails fast on data-lectures URLs that are guaranteed to break at runtime (media host / wrong ref / wrong path / missing published bytes), closing the gap where --strict could exit 0 even when every consumer read 404s.
Changes:
- Enforces a new strict failure when a
data-lecturesreference usesmedia.githubusercontent.com(LFS media host) (repoint rule 6). - Adds strict “resolvability” assertions for
data-lecturesrefs:ref == main,path == lectures/<basename>, and the target file exists inlectures/. - Updates
PLAN.mdto reflect that these checks are now covered by CI for scanned repos, and clarifies what remains outside CI visibility.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.
| File | Description |
|---|---|
| scripts/build_audit.py | Adds strict assertions for media-host usage and for ref/path/file-existence resolvability of data-lectures URLs. |
| PLAN.md | Updates operational guidance to match the new strict audit behavior and current CI coverage limits. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
This was referenced Aug 10, 2026
mmcky
added a commit
that referenced
this pull request
Aug 10, 2026
) * Docs: the storage position, the URL rule, and the refs that drifted #57 deleted the "storage choice invisible to consumers" line from PLAN, but the same position survived in two more places, and one of them was telling authors to do the opposite of what CI enforces. README.md said the interim URL form "works for both plain-git and LFS-tracked files" and told readers to AVOID raw.githubusercontent.com — which is the only form lecture-wasm can fetch, and the form the strict audit demands there. Replaced with the runtime-keyed rule: CPython gets the github.com/*/raw/ form, browser consumers get raw.githubusercontent, and the media host is never valid for this repo. AGENTS.md's URL section had the same "works regardless of storage" claim and never mentioned repoint rule 5 or CORS at all — so an agent working from the conventions doc alone would write the CORS-broken form into a wasm repoint, which is exactly the incident data-lectures#46 records. Also adds the ladder for a published file approaching 100 MiB, which the repo had no answer for: subset, plain git, gzip in place, split, publish from outside git. Reasoning and measurements in #58. PLAN corrections, all verified against main today: * rule coverage — CI now checks rules 5, 6 and resolvability, not "only a corner" * the browser row of the URL table still offered the media host for LFS files; #55 made that a hard failure * the .gitattributes and lfs:false boxes tick, with the line refs moved (audit-dashboard.yml:51, consumed-file-check.yml:26) and the LFS quota restated as measured-and-not-binding, keeping the mechanism * the consumed-file check is rekeyed off consumers (#56) * generating_mini.md: do NOT uncomment its to_csv writes until #14 settles runnable-builder vs provenance — PLAN was instructing one branch of an open decision. The input-URL repoint stands either way * data.quantecon.org is NXDOMAIN at its own authoritative nameserver and the Pages cname is null, so Track Y has no external blocker left — but classify_url knows six GitHub hosts and neither canonical one, so `final` is a status the audit must report as broken until the classifier learns it * build_audit.py:45-46 -> :46-47 and :45-57 -> :46-57; wasm mirror-orphans "a dozen" -> five, matching Track X graph.txt is a 100-node digraph, not the 15-node toy audit_annotations described — that string renders into the public dashboard. Strict audit exit 0, consumed-file check 18/18. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * AGENTS: spell out access-control-allow-origin in the URL table The table used the full header name in one row and abbreviated it to ACAO in the next, and the abbreviation appears nowhere else in the repo's docs. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Gate 1 of QuantEcon/workspace-lectures#23 step 3. Closes #54.
--strictexits 0 today on a fold that 404s every read.patternis derived from org and repo alone, so adata-lecturesURL counts as fully migrated no matter which host serves it, which ref it pins, or whether the file has landed here yet — andlfs_mediahas been computed per reference since the dashboard was built, asserted on nowhere.Two assertions, deliberately independent
data-lecturesreference is onmedia.githubusercontent.com— the LFS media endpoint, which routes per path and 404s plain-git files, and everything published here is plain gitrefis notmain, orpathis notlectures/<file>, or the file is not committed here yet#54 proposed that a resolvability check would subsume the host one. It does not, and this is the load-bearing correction: a media URL parses to exactly the same
(ref, path)as the raw URL beside it — only the host differs — so a resolvability check alone exits 0 on the media-host trap it is meant to catch. A canonicalurl_formcheck is not available as a shortcut either: 25 of the 33 current data-lectures refs usegithub.com/{org}/{repo}/raw/{ref}/…, which is correct everywhere exceptlecture-wasm.Verified both ways
Today's tree — exit 0, zero warnings, across 33 data-lectures refs in 8 repos. Every one is
main+lectures/<basename>, every basename is present inlectures/, none is on the media host, so the patch is a no-op onmain.A doctored
lecture-wasm— exit 1, with each class named: fourheavy_tailsreads moved tomedia.githubusercontent.com/media/QuantEcon/data-lectures/main/lectures/…, onemleread pinningmaster, oneinequalityread keeping the source subdirectory (lectures/SCF_plus/SCF_plus_mini.csv). Rule 5 fires zero times on that tree — the host trap is invisible to every other check in the file, which is the whole point.Also in this PR
The rule-5 comment read "…or
media.githubusercontent.comfor LFS files", i.e. the file blessed exactly what the code below it now forbids.PLAN.md's "This is not covered by CI" paragraph goes stale on the same commit and is rewritten — including what is still not covered:SCAN_REPOSexcludeslecture-intro.zh-cnand the scan excludeslectures/_static/**, so 9 of the fold's 21 reads remain outside CI. The gate-2 grep in intro and wasm covers both of those repos'_staticnotebooks, which leaveslecture-intro.zh-cnas the only consumer with no automated check of any kind.What these assertions do not reach
Both checks are gated on
pattern == "data-lectures", andclassify_urlderives that from six regexes overgithub.com/raw.githubusercontent.com/media.githubusercontent.comonly. So neither reaches the GitHub Pages host,https://quantecon.github.io/data-lectures/lectures/<file>— which is live, CORS-clean and serving correctly today.The pre-existing consequence is sharper than "the new checks are silent there". Such a read classifies as
pattern: external-web, somigratedevaluates false, and amigration.ymlstatus ofrepointedthen fails with "marked repointed but consumers still read via ['external-web']". Sincefinalis defined as "every consumer readshttps://data.quantecon.org/lectures/…", the migration's terminal status is a state the audit is structurally guaranteed to report as broken.None of that is introduced here, and this PR adds no false positives on it — the resolvability loop
continues on any non-data-lecturespattern. Teachingclassify_urlthe canonical host is a follow-up, tracked in #58.One thing this run surfaced, worth recording
The doctored run also produced six pre-existing
in pending wave P3 but already read from data-lectureswarnings. That check has always been there, andmigratedis read-derived — so merging the step-3 lecture repoints turns this audit red on its own, with or without this patch. Green-on-broken only becomes reachable oncemigration.ymlmoves the six intodatasets:atrepointed. The real deadline for this check is therefore before PR B, not before PR set C, and the window between the repoints landing and themigration.ymlflip is red either way.🤖 Generated with Claude Code