Architecture

Architecture

A high-level overview that points you at the in-tree deep dives. The single source of truth is docs/architecture.md — this page is for orienting yourself before opening that file.

The shape

Two HTML pages talking to a small Python server:

paper_database.html ────┐
paper_scraper.html  ────┤   ←  HTTP  →  scq.server (Python)
dev.html            ────┘                   │
                                            ├─→  data/scientific_litter_scoop.db
                                            ├─→  arXiv proxy (avoid CORS)
                                            ├─→  /api/config/<domain> (read+write)
                                            └─→  /api/secret (write to OS keyring)

Both halves of the codebase share the same JSON Schemas + test vectors so JS-side and Python-side validators agree.

Frontend layers (top-down)

HTML pages            Markup + a thin inline boot block (kept thin)
  │
  ▼
src/ui/<page>/        DOM-coupled per-page modules (database, scraper, settings)
  │
  ▼
src/services/         Pure logic. NO DOM. Survives a future Vue port.
  │
  ▼
src/core/             Framework primitives — db, store, events, config,
                       search-config-bridge. NO DOM.

The dependency arrow points into the framework-agnostic layer. Three load-bearing rules in docs/architecture.md keep the seam coherent:

1. core/ and services/ must never touch the DOM.
2. ui/ modules import services, never the reverse.
3. State changes flow through core/store.js.

Python package

scq/
├── server.py        HTTP server + arXiv proxy
├── cli.py           subcommand dispatcher (entry point: `scq <cmd>`)
├── __main__.py      `python -m scq <cmd>` entry
├── config/          paths, user, secrets, portable (export/import)
├── db/              init, migrations, merge
├── arxiv/           search, render, email, digest
├── ingest/          process, extract, inbox, mendeley, watch
├── overleaf/sync.py references.bib → Overleaf project
├── search/index.py  full-text search index builder
├── schedule.py      `scq schedule show/update` for the digest cron line
└── migrate.py       `scq migrate-from-legacy` — scraper_config.js → user_config

tools/<x>.py files at the repo root are thin compat shims that import from the matching scq.* module. They're kept because some Claude skill scripts and external docs reference them.

Two seams worth understanding

The page bridge

The HTML boot blocks call extracted module functions as bare globals (e.g. togglePaper(id) from inside an inline onclick). Two patterns:

• Database page: a centralized BRIDGE = {...} in src/ui/database/main.js with Object.assign(window, BRIDGE).
• Scraper page: each module appends globalThis.<name> = <name> at its bottom.

Both have frozen-list specs (src/tests/ui/{database,scraper}/bridge.test.js) that fail on drift — adding a function and forgetting to bridge it produces a clear "Added without updating list" error rather than a confusing runtime undefined.

The config-subscribe rule

Any UI surface that reads merged config either:

1. Lazy-reads on every render (most modules — automatic),
2. Re-fires on bridge ready via bootstrapSearchConfig's onReady callback list (boot-time captures: activeSources, source-toggle DOM, etc.),
3. Subscribes to config:<domain>:changed (when Settings v2 saves, the dormant reload() path now fires this), or
4. Is documented as a boot-time snapshot requiring reload.

The first three are real; the fourth is escape hatch only. Surfaces that don't follow one of them silently ignore user edits until the next page reload.

Configuration model — at a glance

Four storage tiers with intentionally different homes:

Layer	Lives in	Why separate
Bootstrap (paths)	data/user_config/paths.toml	Must be readable BEFORE the DB opens.
Domain (digest, email, citations, ui, ingest, watchlist, privacy, search-sources, auto-tag-rules)	data/user_config/<domain>.json	Hand-editable; safely committable; schema-validated.
Session/UI prefs	settings table inside the SQLite DB	Bound to a specific DB; too noisy for a config file.
Secrets	OS keyring (env-var fallback)	Must not be on disk in plaintext.

docs/configuration.md has the deep dive, including the x-mergeKey schema-aware merge story and the auto-tag-rules / watchlist "reference data vs. settings" distinction.

Testing topology

Layer	Runner	Spec count
src/core/ + src/services/ + src/ui/ (jsdom)	vitest	595
scq/	pytest	325
Cross-language parity	Shared JSON vectors at tests/vectors/	drives both runners
End-to-end	GitHub Actions e2e-smoke job	spins up scq serve, hits 5 endpoints
TypeScript (selective)	tsc --noEmit over // @ts-check files	runs in CI

Adding a vector at tests/vectors/<category>/foo.json adds a parametrized case to both runners — vitest via src/tests/_vectors.js, pytest via tests/conftest.py:vectors_for(). This is how the JS↔Python schema-merge parity is enforced.

Refactor history

The codebase went through a 30+ commit strangler-fig refactor between 2026-04-28 and 2026-05-03. The plan was tracked at plans/architecture-refactor.md (gitignored — local-only working artifact) and is now archived at plans/archive/architecture-refactor.md. Don't add new top-level items to that plan — open new plan files in plans/ for fresh initiatives.

Where to read deeper

• docs/architecture.md — the load-bearing rules, page bridge, config-subscribe rule, dev harness, TypeScript setup.
• docs/configuration.md — config layers, schemas, parity, env-var pattern.
• docs/adding-a-search-source.md — recipe for new journals.
• docs/adding-a-config-key.md — recipe for new settings fields.