Adding a vendor

Adding a vendor

Every source is one adapter module in scraper/pcblib/vendors/. Start from the closest existing one rather than from scratch.

Platform	Adapters to copy
WooCommerce (Store API)	fivecats.py, pcbguitarmania.py, lectricfx.py
WooCommerce without the API (sitemap and JSON-LD)	pedalpcb.py, jmk.py
Shopify (products.json)	sheepylove.py, ggg.py, zeppelin.py
Big Cartel (products.json)	deadendfx.py, moonn.py, zerogiod.py
Wix	deadastronaut.py, bentfishbowl.py
Static pages or file archives	madbean.py, parasit.py, expanon.py
Blogger layout blogs	dirtbox.py, tagboardeffects.py (on blogger.py)

Checklist

1. Adapter: scraper/pcblib/vendors/<vendor>.py, a class decorated with @register.
2. Import it in load_all() in scraper/pcblib/vendors/__init__.py (the long import line). The decorator only runs when the module is imported.
3. Registry in scraper/pcblib/db.py:
   • VENDORS["<vendor>"] = (name, url, license_note). The note says what the vendor permits and that documents are indexed and linked, not redistributed. Check with the maintainer before adding a new source: some sources have asked to be left out (see How the data is made#leaving-akashic).
   • VENDOR_KIND["<vendor>"] if it is not a shop (projects, repo, blog, archive, layout, kit; see How the data is made).
   • VENDOR_WARNING only if every page needs a caution beside the buy link.
4. App in app/src/lib/types.ts:
   • add the id to the Vendor union and to VENDOR_NAMES;
   • add it to VENDOR_KIND if it is not a shop (this map duplicates the one in db.py; keep them in step);
   • VENDOR_FAB, VENDOR_FILES, VENDOR_ACTION, VENDOR_STORE only if the buy button needs different wording.
5. README sources table: circuits (about), how the parts list is read, notes (platform, currency, quirks).
6. CHANGELOG line under Unreleased (### Added, "New source: ...").

scripts/refresh.sh scrapes every registered adapter, so a new vendor joins the weekly refresh without further changes.

The adapter interface

vendors/base.py:

class Adapter(ABC):
    vendor: str = ""                       # the id used on the command line and in circuit ids

    def __init__(self, fetcher: Fetcher):  # self.f is the cached, throttled fetcher
        self.f = fetcher

    @abstractmethod
    def list_targets(self) -> Iterable[str]:
        """Yield every candidate identifier (usually a URL) to parse."""

    @abstractmethod
    def parse(self, target: str) -> Circuit | None:
        """Parse one target into a Circuit, or None if it is not a PCB project."""

pcblib scrape <vendor> calls list_targets(), filters by --only, calls parse() on each target, runs schematic enrichment, and stores the result. An exception in parse() is printed and counted; the scrape goes on, but a run with errors does not update the listed/delisted state.

list_targets()

Yield strings. They are what --only matches against, so a product URL or handle is best. Listing calls usually cache what they learn for parse() to use (a Shopify adapter keeps each product's JSON in self.products keyed by handle).

parse()

Return a Circuit (scraper/pcblib/models.py) or None. Required fields are vendor (self.vendor), slug (stable and vendor-local: the circuit id is <vendor>:<slug>), name and url (the product page, the buy link). Fill what the source gives:

Field	Notes
based_on	the commercial original as the vendor names it
category, effect_type	taxonomy.classify(*hints) maps vendor words to the shared categories
enclosure	taxonomy.find_enclosure(*texts)
controls	knob names; a bare count ('3 knobs') is replaced by pot names later when possible
price, currency, in_stock, sku	list price in the vendor's currency
doc_url, extra_docs	the build document and any others ({label: url})
doc_local, schematic_local, schematic_page, doc_version, bom	from pdf.process_document
layout	for layout sources: {"build": "vero", "board": "18x18", "images": [...]}; on a PCB, {"also": [...]}

A typical PCB adapter ends like this:

pdf = self.f.get_file(c.doc_url, ".pdf")
if pdf:
    c.__dict__.update(process_document(pdf, self.vendor, slug))
return c

process_document runs the text-table parsers, finds and renders the schematic page, and returns bom, schematic_local, schematic_page, doc_version and doc_local. For anything else, reuse a shared parser before writing a new one: ocr_bom for scanned tables, schematic_bom for vector schematics, ocr_schematic_bom for schematic images, ibom.py, gsheet.py, docx.py, eagle.py. Rows you build yourself go through normalize.normalize_row(BomRow(ref=..., value=..., notes=...)), and is_plausible drops junk. Mark OCR rows with OCR in notes and schematic pairings with from schematic.

Helpers in base.py: clean_text, html_to_text, and on_host(url, *domains) for checking a link's host (it parses the URL, so a domain in a query string does not count).

Return None for what is not a circuit

Kits of parts, enclosures, knobs, faceplates for a board indexed on its own, gift cards, bundles and combo packs are not circuits. Skip them in parse() (or in list_targets() when the listing already says so). pedalpcb.py skips by store category, by "faceplate" in the name, and for products without a build document.

Dedupe regional and older listings

When a store lists the same board more than once (regional copies, a gerber-files-only listing, an older version), collapse them onto one target. pcbguitarmania.py ranks the listings of each base name and keeps the plain physical board; its document links are pooled by base name so the kept listing finds the PDF.

Layouts beside a PCB

A layout the same vendor publishes for its own PCB is not a second circuit. Attach it:

c.layout.setdefault("also", []).append({"build": "vero", "url": layout_url})

The circuit then counts under both PCB and Vero in the Build filter (parasit.py, revolutiondeux.py).

Blogger layout sites

A layout blog on Blogger needs only a subclass of BloggerLayouts:

@register
class Dirtbox(BloggerLayouts):
    vendor = "dirtbox"
    blog = "dirtboxlayouts.blogspot.com"
    build = "vero"                                     # unless a post's label says otherwise
    not_layouts = frozenset({"back-from-wilderness"})  # post slugs that are news, not layouts

The base reads the post feed 150 posts at a time, takes the maker, effect, build and Verified labels, picks the drawing, and reads the board size, cut and link counts and legend from the image (layouts.py). Other hooks: skip_labels, not_makers, and split for a post that holds several layouts (tagboardeffects.py). Passive values are printed turned on the parts, which OCR cannot read, so full parts lists for layouts come from readings by eye in transcribed/.

Politeness

• Fetch only through self.f (get_text for pages, get_file or get_bytes for documents and images). Everything is cached under data/raw/<vendor>/ and requests to a host are spaced 1.5 s apart.
• For a vendor that asks to be fetched slowly, raise the limits in the adapter's __init__: self.f.min_interval = max(self.f.min_interval, 10.0) and, for one clock across all its hosts, self.f.one_clock = True (pcbway.py; amplifiedparts.py uses 3 s).
• Prefer a platform's JSON (Store API, products.json, a feed, a sitemap) over crawling HTML.
• Some hosts need a header (a Referer, an Accept): pass headers= to the fetch call.
• The user agent names the project and links the GitHub account.

Test it

cd scraper
.venv/bin/pcblib vendors                                   # the new id is listed
.venv/bin/pcblib scrape <vendor> --limit 1 --dump          # one circuit, printed as JSON
.venv/bin/pcblib scrape <vendor> --only <part of a URL>    # a document you know well
.venv/bin/pcblib scrape <vendor>                           # the whole vendor
.venv/bin/pcblib stats
.venv/bin/pcblib audit
.venv/bin/pcblib export && cd ../app && npm run check && npm run build

Compare a few boards against their documents by hand. Put the vendor's stats line in the pull request. If you changed a shared parser on the way, follow Fixing a parse.