Fixing a parse

Fixing a parse

A wrong parts list is fixed in one of two ways: change a parser, when the same mistake affects many boards, or add a hand correction, when the misread is a one-off. Either way the change is checked from cache, so a rerun costs no network.

Find the board

The circuit id is <vendor>:<slug>. The page URL uses the same id with every run of punctuation turned into a hyphen: https://akashic.cryptideffects.com/circuit/pedalpcb-pcb038 is pedalpcb:pcb038.

cd scraper
sqlite3 ../data/library.sqlite "select id, url, doc_url from circuits where id like '%pcb038%'"
sqlite3 ../data/library.sqlite "select ref, value, norm_value, category, notes from bom where circuit_id='pedalpcb:pcb038' order by position"
.venv/bin/pcblib scrape pedalpcb --only <part of its URL> --dump    # re-parse from cache and print it

--only matches any part of the adapter's target, usually the product URL. The cached document is under data/raw/<vendor>/ (the path is in the row's doc_local), and pdftotext -layout <file> - shows the text the table parsers see.

Changing a parser

1. Record the counts before.

   .venv/bin/pcblib stats > /tmp/stats-before.txt

   stats prints, per vendor: circuits, circuits with a BOM, with a schematic, and the average BOM rows.

2. Make the change in scraper/pcblib/pdf.py (or the adapter, or the shared module it uses). Prefer extending a parser over special-casing a vendor in a shared one.

3. Re-run the affected vendors from cache.

   .venv/bin/pcblib scrape <vendor>

   Every board is re-parsed from the cached documents and its parts rows replaced. Several vendors can run at once in separate shells. A plain rescrape is all a parser check needs. --reset is for when the adapter changes which products it keeps or how it names them: after a full run with no errors it drops the listed boards that run did not store. It never drops a delisted board, and it stops if more than a tenth of the listing would go.

4. Run at least two other vendors that go through the same parser. process_document and the OCR passes are shared by most PDF vendors, so a fix for one layout can break another. The table parsers all run and the longest result wins (see How the data is made), which protects old layouts, but not completely.

5. Compare the counts after.

   .venv/bin/pcblib stats > /tmp/stats-after.txt
   diff /tmp/stats-before.txt /tmp/stats-after.txt

   Fewer circuits with a BOM, or a drop in average rows, is a regression to explain.

6. Run the audit and read the flags it added.

   .venv/bin/pcblib audit              # --show N prints up to N added flags (default 20)

   The audit flags rows that do not look like valid parts: values that do not parse, values outside a sane range or the standard series, parts filed under the wrong kind, part numbers used once that are one character from a common one, transistors the transistor database does not know, and mangled or duplicated designators. It compares with the previous run and prints how many flags were cleared and added, the reasons whose counts changed, and the added flags, most severe first. Flags on boards that are new since the last run are counted separately.

   It writes to data/cache/audit/:

   File	Content
   parts-audit.html	the filterable report
   audit.csv, audit.json	every flag; audit.json is the baseline for the next run
   boards.json	the boards that existed at this run
   meta.json	run time and counts

   A flag is a lead to check against the document, not a verdict. Read every added flag before deploying. --fail-on-added exits with code 3 when flags were added to existing boards; the weekly refresh uses it to hold a deploy.

7. Look at the result in the app: pcblib export, then build and preview (see Getting started).

8. Write it up. The pull request carries the stats lines before and after. Add a line under Unreleased in CHANGELOG.md if users would notice. Remove the matching entry from TODO.md (the running list of known parse gaps) when the parser now handles it.

Hand corrections

scraper/pcblib/corrections.py holds fixes no parser rule can make, each checked against the source document by a person. They are applied in db.upsert_circuit every time a board is stored, after every parser has run, so a rescrape or the weekly refresh keeps them. After editing, re-store the board from cache:

.venv/bin/pcblib scrape <vendor> --only <part of its URL>

Corrected rows get the note corrected by hand and show the fixed mark.

CORRECTIONS: single rows

CORRECTIONS: dict[str, dict[Key, str | None]] = {
    # Dead End FX Zuul: OCR reads IC3 as 'LhI3Z24' (reported 2026-09-29).
    "deadendfx:zuul": {"IC3": "LM324", "D5": "1N4148", "D6": "1N4148"},
    # Lectric-FX Betty Boost: the scanned grid misreads four values and invents R0 and C0 (checked 2026-10-02).
    "lectricfx:betty-boost": {"R1": "10M", "R2": "10K", "R7": "18k", "R9": "18k", "R0": None, "C0": None},
    # one build variant only
    "otrfx:warm-fuzzies": {("Ritual Fuzz", "C"): "2N2222A"},
    # a shopping-list row ('×1') named by its value
    "moody:tremolito-kit": {"=2N5809": "2N5089"},
    ...
}

• Key: the circuit id, then the designator. Designators match regardless of case (CLEAN corrects a knob stored as Clean).
• Value: the corrected value, or None to drop the row.
• A designator the parsers missed is added, after the nearest lower designator of its kind.
• (variant, designator) limits a fix to one build variant.
• '=' + value targets a row whose designator other rows share (quantity-named rows).
• One comment per entry: what was wrong, where it came from, the date it was checked.

TRANSCRIBED: a whole parts list

For a document no parser can read (values printed on the parts of a wiring drawing), TRANSCRIBED[circuit_id] is the full list, which replaces whatever the parsers found:

"effectslayouts:lawn-darts": [
    ("R1", "1M", "R", ""), ("R2", "100R", "R", ""), ..., ("Level", "B10k", "POT", ""),
],

Each row is (designator, value, category, note), with an optional fifth field for the build variant. Rows without designators are named by quantity (×2).

transcribed/: readings by eye

Schematic images and layouts read whole by eye live one per file in scraper/pcblib/transcribed/<circuit id with ':' as '__'>.json, for example deadastronaut__tremshifter.json:

{
 "id": "deadastronaut:tremshifter",
 "read": "2026-10-11, by eye from the parts list on page 2 of the build document",
 "comment": "The build document is scanned pictures; OCR had missed the LDRs, ...",
 "uncertain": ["LDR1, LDR2: the source gives no part number, only '2k-24k'; ..."],
 "rows": [["R1", "220k", "R", ""], ["R2", "220k", "R", ""], ...]
}

Key	Meaning
id	the circuit id
read	the date, and what was read
comment	notes on the drawing and the reading
uncertain	values the reader was not sure of, in their words; shown as Check before you build
rows	as in TRANSCRIBED; an empty list drops wrong OCR rows when the drawing gives no values
source	"layout" for a stripboard, perfboard, tag strip or turret drawing (rows noted "read from the layout by eye"); absent for a schematic ("read from the schematic by eye")
fixes	corrections the source itself gives for its drawing, already applied to the rows and shown above the parts list

BASED_ON and CONTROLS

BASED_ON[circuit_id] replaces an original the source names wrongly. CONTROLS[circuit_id] sets the control names for a board whose document names its knobs but gives no values to list them as parts.

Keeping corrections honest

Remove an entry when a parser learns to read the board right. A correction whose value the parser now reads exactly is a no-op (the row is not marked fixed), but left in place it would hide a later regression on that row.