Architecture

Architecture overview

A bird's-eye map of how the pieces fit together. For the canonical write-up see /docs/architecture.md.

---

High-level layout

                       ┌──────────────────────┐
                       │   Next.js 14 + TS    │
                       │   shadcn/ui + AG     │
                       │   Grid + Monaco +    │
                       │   ECharts            │
                       └──────────┬───────────┘
                                  │  fetch / JSON
                       ┌──────────▼───────────┐
                       │     FastAPI app      │
                       │  (Pydantic schemas)  │
                       └──────────┬───────────┘
        ┌─────────────┬───────────┼───────────┬─────────────┐
        │             │           │           │             │
   ┌────▼───┐  ┌──────▼───┐ ┌─────▼────┐ ┌───▼─────┐  ┌────▼─────┐
   │ AI     │  │ Connect- │ │ Notebook │ │ Knowledge│  │ Upload + │
   │ provider│  │  ors    │ │  store   │ │  store   │  │ Publish  │
   │  (5)   │  │  (13)   │ │  + git   │ │           │  │  stores   │
   └────────┘  └─────────┘ └──────────┘ └──────────┘  └──────────┘

---

Backend (rednotebook/)

Module	Role
server/	FastAPI app + routers + dependency wiring
auth/	User store, JWT sessions, password hashing, OAuth, API tokens
connectors/	Trino + DuckDB + 11 SQLAlchemy dialects + connection registry
ai/	Provider abstraction (mock / OpenAI / Anthropic / Ollama)
samples/	Bundled starter notebooks lazy-seeded into a fresh user's storage on first list
notebook/	Models, JSON storage, guard-aware runner, publisher (HTML), git_store (autosave history), publish_store (share tokens)
knowledge/	NotebookLM-style internal knowledge layer
visualization/	Chart recommender, ECharts spec builder, infographic HTML/SVG, Playwright PDF/PNG export
uploads/	Per-user file storage (CSV / Parquet / JSON / TSV / JSONL)
profiling/	Per-column stats, PII detector, histograms, mutual-info pairs
security/	sqlglot-backed SQL guard, secret masking
audit/	Append-only audit log
cli/	Typer-driven CLI (rednotebook run, rednotebook check, ...)

---

Data flow: "run a query"

1. User hits ⌘↵ in a SQL cell.
2. Frontend POSTs /api/query/run with {connection, sql, query_id}.
3. Backend's run handler:
   • Builds a connector via build_connector(payload, uploads_dir=...) — the uploads_dir arg lets the DuckDB connector register uploaded-file views before each query.
   • Wraps execution in the query registry (keyed by query_id) so the Stop button can fire a real per-engine cancel.
   • Hands the SQL through the SQL guard (sqlglot-backed) then connector.run_query().
4. Results return as QueryResultPayload with PII-aware coercion in coerce_row_value.
5. Frontend renders via AG Grid + saves to the per-tab cellResultsByTab zustand slice.

---

Data flow: "drop a CSV"

1. User drags customers.csv onto the app.
2. The window-wide drop overlay POSTs /api/files/upload.
3. uploads/store.py writes the file to local_data/uploads/<user-id>/<uuid>.csv and adds a manifest entry with a sanitised table name.
4. The Files panel in the left sidebar re-fetches the manifest.
5. Next time a SQL cell runs on a DuckDB connection, the connector reads the manifest and emits CREATE OR REPLACE VIEW customers AS SELECT * FROM read_csv_auto('/abs/path') before the user's SQL.

---

Data flow: "publish a notebook"

1. User clicks Publish current state in the topbar dialog.
2. Frontend collects all current cell results from cellResultsByTab[activeTab] and POSTs them with the notebook id.
3. Backend's publish handler:
   • Loads the latest saved notebook from disk.
   • Calls render_notebook_html(notebook, results=..., author=...).
   • Saves the HTML under local_data/published/<token>.html and records the token in the manifest.
4. The unauthenticated GET /published/{token} route (registered outside the /api namespace and outside the protected dependency) serves the HTML.

---

Storage paths

Everything user-generated lives under local_data/:

local_data/
  notebooks/<user-id>/<notebook-id>.json   # notebook JSON + .git
  knowledge/<user-id>/...                   # knowledge sources
  uploads/<user-id>/<uuid>.<ext>            # file uploads
                    /manifest.json
  published/<token>.html                    # rendered share pages
            /manifest.json
  auth/                                     # user store
  connections/                              # encrypted connection store
  audit/                                    # append-only audit log
  admin/                                    # runtime config overrides

When AUTH_ENABLED=false (single-user / laptop mode), <user-id> is literally default.

---

Frontend (frontend/)

Directory	Role
app/	Next.js app router — /, /login, /register, /settings/...
components/notebook/	Cell components (SQL, Markdown, AI prompt, Visualization, chart builder)
components/sidebar/	Left sidebar — Files, NotebooksList, MetadataExplorer, ConnectionDialog
components/topbar/	Topbar — Publish, History, autosave indicator, settings
components/panels/	Knowledge panel + Studio dialog + infographic modal
components/ui/	shadcn primitives
hooks/	useAutosave, useAuth, useConnectionMigration
lib/	API client, types, formatters
store/	Zustand stores (notebook, connection, UI) with persist middleware

State design philosophy:

• TanStack Query for server state (notebooks, files, knowledge, history).
• Zustand for local UI state (active tab, cell results in flight, sidebar widths).
• Module-level requestImmediateSave() for the autosave bypass when something imperative (a successful query run, a structural cell op) wants to skip the debounce.

---

Where to start reading

1. rednotebook/server/main.py — full router map.
2. rednotebook/notebook/runner.py — run_sql() is the heart of the query path.
3. rednotebook/connectors/duckdb.py — run_query shows cancellation + uploads + execution in one file.
4. frontend/components/notebook/sql-cell.tsx — the SQL cell pulls together autosave, immediate-save signalling, cancellation, the summarize button, and Monaco.