Card Scanning and Computer Vision

Card Scanning & Computer Vision

Bindarr features an instant, image-based identification system for Pokémon and Magic: The Gathering cards. It operates purely on visual artwork similarity—requiring no OCR, no typing, and no manual alignment.

---

How the Pipeline Works

The identification process splits computational work between the client browser and the server:

[Phone Camera Frame]
        │
        ▼ (In-Browser Worker)
┌─────────────────────────────────────────────────────────────┐
│ 1. Cornelius (ONNX): Finds 4 card corners in ~80ms         │
│ 2. Perspective Dewarp: Transforms card into 448x448 square  │
└─────────────────────────────────────────────────────────────┘
        │
        ▼ (Dewarped Image Upload to POST /api/scan-match)
┌─────────────────────────────────────────────────────────────┐
│ 3. Milo (ONNX): Encodes card artwork into 128-d unit vector │
│ 4. Cosine Sweep: Fast dot product against prebuilt catalog  │
│ 5. Gap Floor Gate: Checks if winner exceeds runner-ups      │
└─────────────────────────────────────────────────────────────┘
        │
        ▼
[Card Identified & Auto-Added or Presented for Review]

1. In-Browser Corner Detection (Cornelius)

• The client-side worker runs Cornelius (cornelius.onnx, 384×384 input, ~4.2 MB) using WebAssembly (onnxruntime-web).
• It tracks the card's four boundary corners at ~12 FPS in real time, rendering a green bounding polygon over the live video stream.
• When the camera is held steady, the browser perspective-dewarps the rectangular card frame into a normalized 448×448 square and uploads only that single processed frame.

2. Neural Artwork Embedding (Milo)

• On the backend, Milo (milo.onnx, 448×448 input, ~5.4 MB) converts the cropped artwork into a 128-dimensional L2-normalized feature vector.
• A brute-force dot product is executed against the prebuilt binary catalog of card vectors. Sweeping 22,000+ card vectors takes less than 6 milliseconds.

---

Fetching the Neural Network Models

Because Cornelius and Milo are licensed under AGPL-3.0 (developed by HanClinto), while Bindarr is licensed under MIT, they are not bundled directly inside the Docker image.

To download the models into your persistent storage volume:

# For Docker deployments:
docker exec bindarr node scripts/fetch-models.mjs

# For Bare Metal / Source installations:
cd backend && node scripts/fetch-models.mjs

Restart the container after downloading. If the models are missing, the server logs a notice at startup and returns a clear 503 notBuilt error when scanning is attempted.

---

Managing & Building Catalogs

A Catalog is a precomputed index of 128-d visual embeddings for every card in a specific game and language.

Downloading Prebuilt Catalogs (Fastest)

To download the official prebuilt English catalogs for Pokémon and MTG:

docker exec bindarr node scripts/fetch-models.mjs --catalogs

Building Custom Catalogs (Admin Panel)

You can build or rebuild catalogs for any game and language (e.g. Japanese Pokémon, German MTG) directly within the web UI:

1. Log in with an Admin account and navigate to Admin → Catalogs.
2. Select your Game and Language.
3. Click Build Catalog.
4. The build process runs in two distinct phases:
   • Phase 1: Cache Card Data: Walks the provider API (Scryfall, Pokémon TCG API, or TCGdex) to index set lists and card metadata into the local database.
   • Phase 2: Generate Embeddings: Downloads high-resolution card artwork and generates 128-d Milo embeddings into milo-<game>-<lang>-local.bin.

Build Resilience: Catalog builds are fully resumable. If you cancel or interrupt a build, previously embedded cards are preserved.

---

Scoped Scanning (Pinning Sets)

While Bindarr can identify a card across the entire 25,000+ card catalog, you can also pin a specific set in the scanner UI.

• Speed & Precision: Filtering by set skips non-matching rows before scoring.
• Reprint Disambiguation: When cards share identical artwork across reprints, pinning the set ensures the exact set and collector number you are opening is chosen automatically.

---

Stranger Rejection & the Gap Floor (CV_SCAN_GAP)

A standard cosine similarity sweep will always return the closest mathematical match—even if you hold up a credit card or a card not present in the catalog.

To prevent false positives, Bindarr uses a Gap Floor (GAP_FLOOR, configurable via CV_SCAN_GAP=0.10):

• Rather than checking absolute cosine similarity (which fluctuates heavily based on lighting, camera glare, and tilt), Bindarr measures how far the top match stands above ranks 2 through 11 in its catalog.
• If the winning score does not clear the 0.10 gap margin, the API returns notInCatalog: true.
• The scanner will refuse to auto-add the card, but will present the top candidates for manual review.

---

Multi-Language Scanning Fallbacks

Card artwork is identical across language printings, but API data coverage for non-English sets varies across upstream providers (for example, TCGdex covers only a portion of historical Japanese sets).

To handle this gracefully:

• When scanning in Japanese, French, German, etc., Bindarr sweeps both the localized catalog and the English catalog simultaneously.
• If a card is identified via the English catalog, Bindarr automatically maps it back to the target language printing (getPrintingInLang) before displaying it to the user.