Getting started

Getting started

This page takes you from a clone to a local copy of the site built from your own scrape. The scraper runs on macOS and Linux; the weekly refresh and Apple Vision OCR are macOS only.

Prerequisites

Tool	Needed for	Required
uv and Python 3.12+	the scraper (scraper/, package pcblib)	yes
poppler (pdftotext)	reading every text PDF	yes
Node 22+ and npm	the app (app/)	yes, to build the site
tesseract	OCR of scanned tables and schematic images	no, but image-only vendors (GuitarPCB, Dead End FX and others) come out thin without it
Xcode command-line tools (swiftc)	Apple Vision OCR, a second OCR engine beside tesseract	no, macOS only
bsdtar	zip and rar document sets (OP Electronics)	no
kicad-cli	pcblib attach-kicad renders	no
graphify (graphifyy on PyPI)	the Claude graph, see The graph	no

On macOS: brew install uv poppler tesseract node. The Vision helper (scraper/pcblib/vision_ocr/main.swift) is compiled with swiftc into data/cache/_bin the first time an OCR pass needs it; without it tesseract works alone. Without tesseract the OCR passes return nothing and the board keeps whatever the text parsers found.

The transistor substitutes need a private MySQL dump of a transistor parameter database (pcblib import-transistors <dump dir> builds data/transistors.sqlite). It is not in the repository and you can work without it; subs.json then comes out empty, and pcblib audit will flag transistors it cannot look up.

Set up the scraper

git clone https://github.com/drlholloway/akashic.git
cd akashic/scraper
uv sync                       # creates scraper/.venv from uv.lock (CI runs `uv sync --frozen`)
.venv/bin/pcblib vendors      # lists every registered adapter; a quick import check

uv venv .venv && uv pip install -e . works too. Every command below runs from scraper/. .venv/bin/pcblib --help and .venv/bin/pcblib <command> --help list the options.

A first scrape

Scrape one vendor. A small one finishes in seconds; a large one takes a while on the first run, since requests to each host are spaced 1.5 s apart.

.venv/bin/pcblib scrape zeppelin                 # one kit
.venv/bin/pcblib scrape pedalpcb --limit 5       # stop after five stored circuits
.venv/bin/pcblib scrape pedalpcb --only muffin   # only targets whose identifier contains 'muffin'

The vendor argument is the adapter's id as pcblib vendors prints it (pedalpcb, aionfx, pcbway-gtu, effectslayouts-blog...). Each stored circuit prints one line with its id, original, BOM row count, schematic page and price.

scrape options:

Option	Effect
--limit N	stop after N circuits stored
--only TEXT	only targets containing TEXT; a target is whatever the adapter lists, usually a product URL, sometimes a handle or slug
--reset	after a full run with no errors, also drop boards the vendor still lists that the run did not store (ones the adapter now skips or renames); delisted boards are never dropped, and it refuses --only and --limit
--refresh-pages	re-fetch listings and product pages; keep cached documents and images
--refresh	bypass the cache entirely
--dump	print each circuit as JSON

Everything fetched is cached under data/raw/<vendor>/, so a second run costs no network. Several vendors can scrape in parallel: the database is in WAL mode and each circuit commits on its own.

Check what you got

.venv/bin/pcblib stats    # per vendor: circuits, with BOM, with schematic, average BOM rows
.venv/bin/pcblib audit    # flags rows that do not look like valid parts; report in data/cache/audit/

The first audit run is the baseline; later runs compare with it. See Fixing a parse.

Export the data bundle

.venv/bin/pcblib export

This writes the JSON bundle to app/static/data/ (git-ignored): index.json, parts.json, vendors.json, per-circuit files and the rest. It also fetches exchange rates (frankfurter.dev, keeping the last file if offline) and reads git log for the site's changes page.

export --images also copies cached schematic renders into the bundle. That is for local use only: never deploy a bundle built with it (see How the data is made).

Run the app

cd ../app
npm install
npm run check     # svelte-check
npm run dev       # development server

To see what will be deployed:

npm run build     # static site in app/build, every circuit and part page prerendered
npm run preview   # serve app/build

The preview server caches its file list when it starts, so stop and restart it after every build. The app reads only the exported bundle; re-run pcblib export after a scrape, then build again.

Optional: attach a KiCad schematic

A circuit can carry a hand-drawn KiCad fragment, which becomes its schematic on the site:

.venv/bin/pcblib attach-kicad pedalpcb:pcb038 ~/kicad/muroidea.kicad_sch
.venv/bin/pcblib export

It copies the file to app/static/kicad/ and renders an SVG with kicad-cli.

Next: How the data is made for what happens inside a scrape.