Skip to content

SAT Question Pipeline

Jump to bottom
benjamindobbs edited this page Sep 28, 2026 · 3 revisions

SAT Question Pipeline

How questions get into the platform, and how the adaptive engine decides what to serve.

Source

Questions come from the College Board SAT Suite Question Bank — the official practice-question export. Each exported PDF block carries: Question ID, Assessment, Test, Domain, Skill, Difficulty (Easy/Medium/Hard), the question text, four choices, the correct letter, and a rationale.

Three assessments, six banks (all static JSON under SAT-Questions/, fetched directly by the browser):

Assessment English bank Math bank
SAT SAT-English-Questions.json SAT-Math-Questions.json
PSAT/NMSQT & PSAT 10 PSAT-English-Questions.json PSAT-Math-Questions.json
PSAT 8/9 PSAT89-English-Questions.json PSAT89-Math-Questions.json

Each record: { ID, Question, A, B, C, D, Answer, Rationale, Difficulty, Test, Assessment, Domain, Skill, image?, choiceSprite? }.

Both subjects are shown to students as images cropped from the source PDF, not as the extracted text: image is the question area, and choiceSprite packs the four answer choices into one PNG:

"choiceSprite": { "src": "…/choices/<ID>.png", "size": [W, H],
                  "A": [y, w, h], "B": [y, w, h], "C": [y, w, h], "D": [y, w, h] }
Math English
Question image math-images/<ID>.png english-images/<ID>.png (+ imageSize: [W, H])
Choice sprite math-images/choices/<ID>.png english-images/choices/<ID>.png
Text fields OCR — often wrong; not shown exact text layer — used as alt text and as the fallback if a question has no crop

Images keep what text extraction loses: tables, charts, underlined sentences the question refers to, superscripts (13C), and italics.

Extraction scripts (SAT-Questions/*.py)

extract_english.py — English / Reading & Writing

The R&W PDFs have a real text layer, so this uses exact text positions from PyMuPDF — no OCR. It rebuilds each English bank from scratch (text, metadata, and images in one pass):

Bank PDF (in SAT-Questions/ or SAT-Questions/.SAT-Question-Bank/, gitignored)
SAT-English-Questions.json SAT-English-Questions.pdf
PSAT-English-Questions.json PSAT-10-English-Questions.pdf
PSAT89-English-Questions.json PSAT-8-9-English-Questions.pdf
  • A question starts on each page headed Question ID: <id> and runs to the next one.
  • The metadata table is read by column position, so wrapped cells (Standard English / Conventions, Text Structure and / Purpose) come out whole. Cross-text is normalised to Cross-Text.
  • Headings (Question, Answer, Correct Answer: X, Rationale) are recognised by their font, which differs from body text; the A.–D. markers are body lines at the left margin between Answer and Correct Answer. (Continuation lines are indented, so an italic name like A. mexicanus wrapping onto a new line isn't mistaken for a marker.)
  • Crops are rendered at 150 DPI, trimmed to their ink, and stitched across page breaks. Plain text is saved as 4-level greyscale; crops with shading or charts keep 16 greys (32 colours if coloured).
  • Text is extracted with TEXT_INHIBIT_SPACES (otherwise the PDF's kerning splits words: repor ted). Rationale paragraphs are separated by a blank line (\n\n).
  • Questions it can't parse are left out and listed at the end of the run. As of the September 2026 export that's 4 questions: three copies of one question whose PDF is missing the D. marker (e3bbf2bf, 1a97fb49, 7ffce84a) and one page printed at a different scale (fb17d540).
pip install pymupdf pillow numpy
python extract_english.py                       # all three banks (~5 min, ~150 MB of images)
python extract_english.py --bank sat --ids f1bfbed3,de55ec71 --debug out/   # re-do a few, merged in
python extract_english.py --text-only           # text/metadata only, keep existing image fields

It replaces the old extract_questions.py (pdfplumber, line-based), which is kept for reference but should not be used — see English bank rebuild.

extract_math_choices.py — Math answer choices

The math choices are cropped into choiceSprites the same way; OCR is used only to locate the Answer / Correct Answer landmarks. See the script's docstring.

extract_math_questions.py — Math

The Math PDFs have no text layer — every page is vector paths. So:

  • Each page is rasterized at 200 DPI with PyMuPDF (fitz) and run through Tesseract OCR (pytesseract, --psm 3).
  • Same block/metadata/choice parsing as the English script, but with OCR-tolerant regexes (colons read as pipes, stray spaces in keywords, lowercase embedded choice letters like c. / d.).
  • Domain and Skill strings are fuzzy-matched (difflib.get_close_matches) to a canonical list per domain, so OCR noise still lands on the right label.
  • Student-Produced-Response (free-response, numeric answer, no ABCD) questions are detected and skipped — listed in math_spr.txt.
  • A merged-choices post-process splits "60 c. 700" into C: 700 when OCR runs choices together.
  • Flags: image-only-question, visual-keyword, merged-choices, incomplete-choices, missing-metadata, embedded-image. Image-needing flags go to math_flagged.txt with page numbers.

extract_math_images.py

Runs after the two above, using SAT-Math-Questions.json + math_flagged.txt. Per flagged question:

  1. If the PDF page has an embedded raster image, save the largest one.
  2. Otherwise (pure vector chart), use OCR word bounding boxes to find the Answer heading, then crop from near the top of the page down to just above it — capturing the question text and any figure. Falls back to the top 55% of the page.

Saves math-images/<ID>.png and adds "image": … to the JSON in place.

classify_domains.py

Backfills Domain/Skill on questions the parsers couldn't tag, by keyword-matching the combined question + rationale text against per-domain pattern sets. Overwrites the file in place.

patch_*.py

Targeted one-offs applied to specific banks: patch_ocr_choices.py, patch_merged_choices.py, patch_math_flags.py, patch_assessment.py. Not part of the normal flow — re-run only when re-extracting.

Re-extraction, end to end

pip install pymupdf pytesseract pillow numpy        # + a Tesseract binary for Math
python extract_english.py
python extract_math_questions.py <math.pdf> SAT-Math-Questions.json --assessment SAT
python extract_math_images.py --pdf <math.pdf> --json SAT-Math-Questions.json --flagged math_flagged.txt
python classify_domains.py SAT-Math-Questions.json
python extract_math_choices.py

Client load

js/SATmain.js / js/SATmathmain.js:

  1. Resolve assessmentType from the selected class (daily-progress?class_id=), default sat.
  2. fetch the matching bank.
  3. Filter out question_reports-suspended IDs (GET /api/questions/suspended).
  4. Bucket by domain into questionTypes[0..3].
  5. On a class switch, drop the cache (questionTypes = null) and reload — a different class may use a different bank.

Given a { domainIdx, skill, difficulty } from /next, pickQuestion tries exact skill+difficulty, then any difficulty for that skill, then difficulty only, then anything in the domain pool.

Rendering (English). buildQuestion shows image in the question card and one sprite slice per choice (js/choiceSprite.js, shared with math); a question without them falls back to its text. The English quiz column is widened to 1040px (.quiz-area--wide). Crops are displayed at 0.8 CSS px per crop px (9pt source text ≈ 15px); when the question's widest crop doesn't fit, every crop of that question shrinks by the same factor (--crop-scale, recomputed on resize) so the passage and the choices stay the same size. On a phone-width screen the text becomes very small.

Adaptive serving

GET /api/sat/next (and /api/sat-math/next). Constants: ACCURACY_THRESHOLD = 0.70, MIN_ATTEMPTS = 5, difficulties ['Easy','Medium','Hard'].

  1. Allowed domains — from the one class in ?class_id= (sat_english_domains / sat_math_domains on classes, CSV of 0–3). No class_id, or NULL column ⇒ all four. If the resolved set is empty, return a random allowed domain at Easy.

  2. Recency window — a SQL ROW_NUMBER() OVER (PARTITION BY domain_idx, skill, difficulty ORDER BY submitted_at DESC) keeps the last 25 attempts per (domain, skill, difficulty) for this user_key. Global — not filtered by class_id; a student's skill is the same person in every class.

  3. Accuracy map — aggregate those rows into { "d|skill|diff": { attempts, correct, accuracy } }.

  4. Candidates:

    • Every allowed domain never seen → { domain, skill:'', difficulty:'Easy', weight: 1.0 }.
    • Every seen (domain, skill) in an allowed domain:
      • Target difficulty by walking up the tiers: start Easy; advance to Medium only if the Easy tier has attempts ≥ 5 and accuracy > 0.70; same bar Medium→Hard; stop at the first tier that fails the bar.
      • Weight = max(0.05, 1 − accuracy_at_target). 0% → 1.0, 75% → 0.25, 100% → 0.05 (a floor, so mastered areas still resurface).

  5. Weighted random pick — a random number in [0, Σweights) walked through the candidate list. Returns { domainIdx, skill, difficulty }.

Conventions answers from before the skill split. Rows with domain_idx = 3 and skill = '' (recorded before the September 2026 rebuild) are fed into the window as history for both Conventions skills (Boundaries, Form, Structure, and Sense), and the '' combination itself is dropped. A student who had reached Hard in Conventions therefore starts both skills at Hard; as they answer real Boundaries / Form questions, those push the old rows out of each 25-attempt window. The mapping lives in LEGACY_CONVENTIONS (server/englishBankMigration.js).

The engine never sees question content — it works purely from the attempt log. Difficulty is therefore requested; the client falls back through pickQuestion if the bank has no exact match.

Domain indices

English (0–3): Information and Ideas · Craft and Structure · Expression of Ideas · Standard English Conventions.

Math (0–3): Algebra · Advanced Math · Problem-Solving and Data Analysis · Geometry and Trigonometry.

English bank rebuild (September 2026)

The English banks were rebuilt with extract_english.py, fixing several long-standing problems in the old extract_questions.py output:

  • No Standard English Conventions skills. Its line-based metadata parser broke on the wrapped Standard English / Conventions cell, leaving Domain and Skill empty on every Conventions question (the client special-cased Domain === ''). Now: Boundaries and Form, Structure, and Sense. The same wrap produced Text Structure and (for …and Purpose) and one Words in ContextMedium.
  • SAT and PSAT 10 banks were swapped. College Board gives the same question a different ID in each assessment's export; SAT-English-Questions.json held the PSAT 10 export's IDs and vice versa. The content overlaps ~95%, which is why it went unnoticed. IDs travel with their content.
  • Truncated choices where an italic species name (A. mexicanus) was read as a choice marker, and ~160 / 156 / 105 questions missing from the SAT / PSAT 10 / PSAT 8/9 banks.

All 4,814 questions present in both the old and new banks kept the same correct answer.

Data migration (server/englishBankMigration.js, run at startup from server/db.js):

  • Renames Text Structure and → Text Structure and Purpose (and the two other truncated names) in sat_scores. Runs on every startup — a no-op once clean — so a score posted from a stale tab after a deploy is fixed on the next restart.
  • Once per subject (recorded in schema_migrations as english-bank-rebuild-2026-09 and math-image-choices-2026-09): deletes that subject's question_reports and suppressed_questions. Nearly all reports were about garbled OCR/extracted text, which students no longer see now that both subjects are served as images.
  • Leaves Conventions rows with skill = '' alone (see Adaptive serving).

node tools/preview-english-migration.js <scores.db> reports what it would change without writing. Against the production data at the time: 110 rows renamed, 1 English and 16 math reports cleared, 973 legacy Conventions rows (105 students).

Home

For Teachers

Technical

Clone this wiki locally