Content extractor and the ContentCard union - #94
Merged
Conversation
The prototype's data files are not modules and several carry React markup beside their data, so a bank cannot be read by evaluating its file. Each is sliced out and evaluated alone. Finding where a declaration ends is the part worth explaining. Counting brackets is the obvious approach and it is wrong: the source contains `// Only 'match' cards.`, and an apostrophe inside a comment desynchronises any scanner tracking quotes without also tracking comments. Rather than re-implement JavaScript's lexer this asks the real one — a slice ending anywhere but the true end is unbalanced and fails to compile, so the first candidate V8 accepts is the end. The shape guard is belt-and-braces. A truncated slice cannot compile, but that argument rests on the prototype's formatting, and the cost of being wrong is a bank that is quietly short rather than a run that fails.
Every check answers one question: would this render as something broken, blank or wrong in the app? A pointer that no longer resolves, a graded card with no correct answer, a dictionary entry whose deeper material has lost the explanation it hangs off. None of these throw at runtime — they degrade quietly, which is why they are caught here. Violations are collected rather than thrown, so one run reports every problem and names the card and the reference in each. The graded kinds are the key set of the answer-check map rather than a second list beside it. A kind that gains a check is graded by construction, so the two cannot drift — and drifting is how mastery starts dividing by the wrong number. One rule bends: "exactly one correct answer" cannot hold for `multi`, whose answer is the whole set. It is checked for being neither empty nor everything instead.
Validate-and-refuse-to-write is the point of the tool. It reads every bank, checks the whole graph, and only then writes — so a run can never leave a stale mixture of old and new files behind. Brew challenges are extracted for the validator's sake, not the app's. No screen reads them, but their 31 lesson, module and collectible pointers are edges the graph would otherwise not check — including into m1l1 and c1, the content the first slice ships. Mini-games are deferred: `MINI_GAME_CONTENT` is not a self-contained literal, since one entry is a getter reading a global another file defines. Slice-and-evaluate hands back an empty array for that game, so it needs a different mechanism and no surface yet justifies building it. The prototype's field vocabulary is emitted verbatim; Dart takes idiomatic names through annotations. The known cost is that a prototype-side rename yields a runtime null rather than a compile error. The header carries the full register of what is joined, what is dropped, and where a rule bends — including that module and lesson `status` and `locked` are the prototype's demo progress and nothing should read them.
Fourteen variants, one per kind the prototype actually authors. `intro` and `takeaway` are absent on purpose: renderers for them survive but no card uses either, having been superseded by `predict` and `recall`. The ten graded kinds carry an empty `Gradable` marker. It is empty because they share no field, and it exists so a scoring seam can take `List<Gradable>` rather than `List<ContentCard>` — which makes the wrong-denominator bug unrepresentable rather than merely avoidable. Counting an ungraded card deflates every score; leaving a graded one out lets mastery exceed 100%, which the prototype shipped when `flavor` scored while missing from the graded list. The comment says so, because without it the marker reads as ceremony and gets deleted. Dispatch is a plain exhaustive switch — no registry, builder map or factory indirection. The sealed union makes an unhandled kind a compile error, and that guarantee is worth more than the indirection saves. The metrics exemption is the same case as mood_colors.dart: the method count is the kind count, so number-of-methods measures the prototype's content design rather than anything about this file.
Constructing every card the extractor emits is what turns a prototype field rename into a red build rather than a runtime null. Its complement matters as much: no card may carry a field the union silently drops, which is what a renamed optional would look like. The drift check is the one that makes the others honest. Reading the committed output means the committed output is what they keep agreeing with, so a rename in brew-path/ would stay green until someone happened to rerun the extractor. Asserting that the committed banks equal a fresh extraction closes that: the script is deterministic, so byte equality is the whole check. The graded-kind assertion spans the language boundary. The extractor decides which kinds get an answer check and the union decides which ones score; nothing else holds those two lists together, and mastery divides by them. The refusal path is driven as a subprocess because the behaviour under test is the process contract — what it writes, and what it declines to. Its source is a scratch copy, so brew-path/ stays read-only; one test asserts that directly.
The extractor test shells out to `node`. The runner ships one already, so this changes nothing today — but relying on an accident of the image is how it breaks later, silently, in a job that has nothing to do with it. Nothing to install: the script takes no npm packages, which is why it parses with V8 rather than pulling in acorn.
It landed between flutter_test and integration_test, splitting the two SDK deps that belong together. pubspec deps here are grouped by concern rather than sorted, so placement is the only thing carrying that.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #77.
Reads the prototype's authored content into the app's bundled JSON, and adds the type that receives it. These are one change because neither is testable alone: the union defines the wire format the extractor emits, and the extractor produces the cards the union's tests construct.
The extractor
node tool/extract_content.jsreads five banks — modules, lessons, collectibles, dictionary terms and brew challenges — validates the whole cross-reference graph, and only then writesassets/content/generated/. On any violation it names the offending card and the broken reference, writes nothing, and exits non-zero.Each bank is read by slicing its declaration and evaluating only that. Finding where a declaration ends goes through a real parser: counting brackets breaks on
// Only 'match' cards., where an apostrophe in a comment desynchronises any scanner that tracks quotes but not comments. Instead V8 decides — a slice ending anywhere but the true end is unbalanced and won't compile, so the first candidate that parses is the end. Verified against apostrophes and brackets inside comments, and interpolated template literals.brew-path/is opened for reading only; a test asserts it is never written to.Brew challenges are included for the validator's sake, not the app's. No screen reads them, but their 31 pointers (12 module + 7 lesson + 12 collectible) are edges the graph would otherwise not check — including into
m1l1andc1, the content slice 1 ships.Mini-games are deferred.
MINI_GAME_CONTENTis not a self-contained literal — one entry is a getter readingwindow.BAGPICK_ROUNDS, whichbean-anatomy.jsxdefines, so slice-and-evaluate returns an empty array for that game.MINI_GAMESandCARD_KIND_HELPare held back with it rather than split across two changes.ContentCardOne flat sealed union discriminating on
kind, with a variant per authored kind. The graded kinds carry an emptyGradablemarker whose comment names the wrong-denominator bug it prevents. Dispatch is a single exhaustive switch — no registry or factory indirection.The prototype authors 14 kinds, and
lesson.jsx:165'squizTotalgrades 10. Corroboration: the graded card total comes out at 144, exactly the figuredocs/design/06-content.mdstates, and that doc independently says "14 authored".introandtakeawaystill have renderers but zero authored cards — adding variants for them would have created cases no test could exercise.Decisions worth a reviewer's eye
MODULE_REWARDSis read as a sixth declaration and joined onto each module asreward. Without it the five module Field Guide collectibles have no words, since a collectible stores identity only.DICT_CATEGORIESis read but not emitted — it validates every term'scat; no dictionary surface exists yet.status/lockedare emitted verbatim and are the prototype's demo progress. Real progress lives in the database; nothing app-side should read those fields. They survive only because the extractor renames and drops nothing.multi, whose answer is the whole set; it is checked for being neither empty nor everything.tool/extract_content.js.Tests
13 new tests. Beyond the two the issue asks for:
brew-path/stays green until someone happens to rerun the extractor — the other tests read the committed output, so that is what they keep agreeing with. Confirmed to fail correctly: a simulatedexplain→explanationrename turns 6 tests red.flutter testremains the only test command. CI now pins Node, which the extractor test shells out to.Full suite green (405 passing),
flutter analyzeclean,dart formatclean,dart_code_lintermetrics gate clean.content_card.dartcarries a documentedmetrics-excludeentry — 14 factories plusfromJsonexceedsnumber-of-methods: 12, the same case asmood_colors.dart, where the method count is the token count.