How the data is made

How the data is made

Everything is built on one machine, cached, and shipped as static JSON. There is no server and no database behind the live site.

vendor site ──> adapter (scraper/pcblib/vendors/<vendor>.py)
                  │  fetch.Fetcher: polite, cached in data/raw/<vendor>/
                  ▼
             document parsers (pdf.py, ibom.py, gsheet.py, docx.py, eagle.py, layouts.py, vision.py)
                  ▼
             enrich.py: schematic pairing for thin lists and unnamed knobs
                  ▼
             db.upsert_circuit: cleanup, normalize.py, corrections.py ──> data/library.sqlite
                  ▼
             pcblib export ──> app/static/data/*.json
                  ▼
             SvelteKit static PWA (app/) ──> Cloudflare

1. Adapters and the cache

Each source is one adapter class in scraper/pcblib/vendors/. It lists targets (product URLs, feed entries, handles) and turns each into a Circuit (models.py): name, URL, price and currency, stock, based on, category, enclosure, controls, document links and a list of BomRows. See Adding a vendor for the interface.

All HTTP goes through fetch.Fetcher:

• every response is cached under data/raw/<vendor>/, keyed by a hash of the URL, so a re-run costs no network;
• a 404 or 410 is remembered and not retried unless the cache is bypassed;
• requests to the same host are spaced 1.5 s apart by default (min_interval; one_clock puts every host of a vendor on one clock);
• --refresh-pages re-fetches pages (anything read with get_text: listings, product pages, JSON) and keeps cached documents and images (get_file); --refresh re-fetches everything;
• a network error or server error during a refresh falls back to the cached copy.

Renders, OCR text and other derived files go in data/cache/. Both directories are git-ignored.

2. Parsers

The shared parsers live in scraper/pcblib/. An adapter calls the ones that fit its documents; most text PDFs go through pdf.process_document.

Source format	Where	Notes
Text PDF tables	pdf.text_bom (via process_document)	pdftotext -layout, then every table parser (below)
KiCad interactive BOM	ibom.py	the embedded pcbdata JSON: exact, every designator
Vector schematic labels	pdf.schematic_bom	designators and values paired by position on the drawing
Scanned tables	pdf.ocr_bom, pdf.ocr_image_bom	tesseract, plus Apple Vision when available; a thorough pass (upscaled, thresholded) when the quick one is thin
Ruled grids	pdf._ocr_grid_rows	a scanned grid read cell by cell
Schematic images	pdf.ocr_schematic_bom, pdf.schematic_value_bom	OCR words paired by position
Google Sheets	gsheet.py	the sheet fetched as xlsx, every tab read
Word documents	docx.py	paragraphs and tables straight from the OOXML
Eagle files	eagle.py	XML .sch and .brd: every part's name and value, exact
Layout drawings	layouts.py	the legend and pot names on a stripboard, tag strip, turret or perfboard drawing
Readings by eye	transcribed/, corrections.py	see Fixing a parse

vision.py adds Apple Vision as a second OCR engine on macOS; it reads thin, small and colored type that tesseract garbles. Elsewhere it returns nothing and tesseract works alone.

The text-table rule

Vendors lay out their parts tables in several ways, and no one parser reads them all. pdf.text_bom runs every table parser on the document's text (parse_bom, parse_bom_qty_value_parts, parse_bom_columns, parse_bom_qty_value_ref, parse_bom_name_designator, parse_bom_cells, parse_bom_designator_qty_name) and keeps the one that returned the most rows. Then:

• a per-variant table (one value column per version, or repeated titled blocks) takes over the parts it covers when it covers at least 80% of the designators;
• a shopping list (value, type, quantity, no designators) is the fallback when nothing else read; its rows are named by quantity (×2);
• variant notes and mod charts add build variants.

Because the best result wins, a parser added for a new layout should not regress an old one. Check that anyway: see Fixing a parse.

Schematic enrichment

After the adapter returns, enrich.enrich_from_schematic runs on boards whose parts list is thin (under 8 rows), quantity-only, or names no knobs. It reads the schematic page and adds pot and switch names and, for a thin list, the designators the list is missing. It only adds rows for designators the list does not already have, so it never overwrites a row from a text table. Archive and fab-project sources skip it.

3. Provenance

Rows that were not read from a text table say so in their notes, and the circuit page shows a mark beside the value:

Note in the row	Mark on the site
contains OCR	ocr
contains from schematic	sch
contains corrected by hand	fixed (outranks the others)
read from the schematic by eye / read from the layout by eye	the note is shown as written

Text tables, interactive BOMs and CAD files carry no mark. OCR is best effort: it fills gaps and must never replace a row that came from a text table.

4. Storing a board

db.upsert_circuit cleans every board before writing it:

• expands range rows (Q1-2 2N5088 becomes Q1 and Q2);
• drops prose read as a part and impossible pot values;
• fixes stray digits and micro read as pico;
• applies the hand corrections (corrections.apply, last, so they key on the stored designators), BASED_ON and CONTROLS overrides;
• names the controls from the pot rows when the adapter could only count knobs;
• replaces the board's BOM rows.

normalize.normalize_row sets each row's category (R, C, D, Q, IC, POT, TRIM, SW, LED, OPTO, CONN, HW and others) and norm_value, so 1K5, 1.5k and 1k5 all become 1.5k, 4u7 becomes 4.7u, and A100K, 100KA and 100k log all become A100k. taxonomy.py maps each vendor's category words onto one shared list of effect categories and finds enclosure sizes; originals.py merges spellings of the same original (EHX Big Muff and Big Muff Pi are Electro-Harmonix Big Muff Pi).

The database is data/library.sqlite: vendors, circuits (one row per vendor board) and bom (one row per parts-list line).

5. Vendor kinds

What a source offers decides the buy button and the wording on the circuit page. The kind is set in db.VENDOR_KIND (for the export) and VENDOR_KIND in app/src/lib/types.ts (for the app); a vendor in neither is a shop.

Kind	Meaning	Examples
shop	buy the PCB from the vendor	PedalPCB, Aion FX, Fuzz Dog
projects	order the board from a fab	PCBWay: Glory to Ukraine, Jon Patton (OSH Park)
repo	download the gerbers from a code repository	RWL Pedals, Cryptid Effects
blog	read the post	Bent Fishbowl, Revolution Deux
archive	open the schematic	Experimentalists Anonymous
layout	build from a vero, perfboard, tagboard or turret drawing	Tagboard Effects, Dirtbox Layouts
kit	a complete kit: parts, enclosure and instructions	Mod Electronics

A layout the same vendor publishes beside its PCB is attached to that circuit as layout.also ({"also": [{"build": "vero", "url": ...}]}), not listed as a second circuit.

6. Delisting

Boards are never deleted when a vendor stops listing them. After a full scrape (no --only, no --limit, no errors), db.mark_listing:

• marks every board the scrape found as listed today (and clears an old delisted date);
• for each board the scrape did not find, checks its own page; only if the page is gone (404, 410, or a redirect to a page without the product's path) is it marked delisted with the date. A network error counts as alive;
• marks nothing if the scrape found fewer than half the boards the library lists for that vendor (when it lists at least 10): that is a broken scrape, not a closing-down sale.

The site keeps delisted boards and says they are no longer listed.

7. Export and the app

pcblib export writes the bundle to app/static/data/: the search index, parts cross-reference, per-circuit JSON, vendors, transistor substitutes, the What can I build data, exchange rates and the change history. The SvelteKit app (app/, adapter-static) prerenders every page from it; search runs in the browser with MiniSearch, and the service worker caches pages for offline use.

What the site shows, and what it doesn't

Akashic is an index. Every entry points builders back to the place the circuit was published, and the source's own page stays the place to build from.

What the scraper does. For each source, an adapter reads the source's public listing (a shop's product pages, a blog's feed) and the build documents those pages link to. It identifies itself as Akashic/0.1 (personal DIY pedal reference; contact via github.com/drlholloway) and waits 1.5 seconds between requests to the same host. Every page and document it reads is cached on the maintainer's computer (data/raw/, data/cache/), so later runs don't fetch them again. A full refresh runs once a week.

What a circuit page shows:

• the circuit's name, what it is based on, its effect category, controls and enclosure;
• the price and stock, for a source that sells boards;
• the parts list, with values normalized so circuits can be compared (where one could be read from a text table, a CAD file or, marked as such, by OCR);
• for a layout, the build type (vero, tagboard, turret, perfboard) and the board size;
• for a blog or layout site, the opening text of the post, under "From the post";
• links to the source's product page or post and to its build document.

What the site never shows: the build documents themselves, schematic images, layout drawings and photos. They are read on the maintainer's computer to get the parts list, the board size and the control names, and are never published. A visitor who wants the document or the drawing follows the link to the source.

Exceptions:

• data/archive/ is committed and holds the documentation of stores that have closed (Zero G IOD), so it outlives the store.
• The site owner's own designs (db.OWNER_VENDORS) have their schematic and photo shown.
• A hand-drawn KiCad fragment attached with pcblib attach-kicad is redistributable.
• A few discontinued datasheets are hosted in app/static/datasheets/; everything else links to the maker's sheet.

pcblib export --images bundles cached schematic renders for local use. Never deploy it; npm run deploy always exports without it.

Leaving Akashic

A designer, vendor or author who doesn't want their work in Akashic can ask, through a GitHub issue or by contacting the maintainer. Their source is then removed completely:

• every entry from it comes off the site;
• the cached copies of their pages, documents and drawings are deleted;
• the adapter is deleted, so their site is never fetched again.

If they're happy with it, the Workbench links page keeps a plain link to their site, so builders can still find it. Sources that have asked to leave are never added back, by adapter or by hand. Before adding a source, check with the maintainer.