Claude/csv loading postgres migration qvkl4l

.gitignore

@@ -52,3 +52,6 @@ tfplan
 terraform-provider-*.log
 
 *.zip
+
+# Serena tool workspace
+.serena/

API_INTEGRATION.md

@@ -1,24 +1,33 @@
 # Frontend Integration — CSV Table Hub
 
-Notes on the FastAPI backend in `api/`, which connects the **CSV Table Hub**
-frontend to PostgreSQL.
+Notes on the FastAPI backend in `api/`, which connects the React frontend in
+`frontend/` (originally the standalone **csv-table-hub-main** project, now
+merged into this monorepo) to PostgreSQL.
+
+> **See also** the **Web UI + REST API** section of `README.md` for the
+> user-facing quickstart (two-terminal launch, endpoint table, env vars). This
+> file is the deeper design/integration doc — for how the backend is put
+> together and what's still open.
 
 ---
 
 ## Architecture
 
 ```text
-CSV Table Hub (React)  →  api/ (FastAPI)  →  PostgreSQL
-                                          ├── uploads schema   (dynamic mode)
-                                          └── te_<env> schema  (te mode)
+frontend/ (React 19 + TanStack Start)  →  api/ (FastAPI)  →  PostgreSQL
+                                                          ├── csv_uploads schema  (dynamic mode)
+                                                          └── te_<env>    schema  (te mode)
 ```
 
+Schema names are configurable via `CSV_UPLOADS_SCHEMA` (default `csv_uploads`)
+and `TE_SCHEMA` (default `te_dev`) — see `api/config.py`.
+
 Two upload modes:
 
 | Mode | Destination | Behaviour |
 |---|---|---|
-| `dynamic` | `uploads.csv_<sha256[:16]>` | A typed table per CSV, columns derived from the header |
-| `te` | Fixed T&E schema | Loads into one of the 12 core tables when the columns match |
+| `dynamic` | `csv_uploads.csv_<sha256[:16]>` | A typed table per CSV, columns derived from the header |
+| `te` | Fixed T&E schema (`te_dev.*`) | Loads into one of the 12 core tables when the columns match |
 
 `services/te_loader.match_te_table()` inspects the parsed columns and suggests a
 T&E table, which drives the mode picker in the UI.
@@ -36,12 +45,25 @@ T&E table, which drives the mode picker in the UI.
 
 CSV content is sent as a JSON string, not multipart.
 
+## Authentication
+
+Every endpoint (including `/api/health`) requires an `X-API-Key` header
+matching the `API_KEY` environment variable. If `API_KEY` is unset the check
+is skipped — that's the local-dev default, and `api/main.py` logs a warning
+at startup when it's unset so this isn't silently forgotten in a real
+deployment. Set `API_KEY` (backend) and `VITE_API_KEY` (frontend, same value)
+before deploying anywhere reachable beyond localhost. See `frontend/.env`.
+
+> **Known DX gap** — if `API_KEY` and `VITE_API_KEY` don't match, every request
+> returns 401 with no hint from either process. Tracked as BUG-006 in
+> `BUG_REPORT.md`.
+
 ## What it does well
 
 * **Deduplication at three levels** — filename, whole-file content hash, and
   per-row `_row_hash` with `ON CONFLICT DO NOTHING`. This is what the
   frontend's "no duplicates" promise needs.
-* **Upload registry** — `uploads.csv_files` records filename, hash, table, row
+* **Upload registry** — `csv_uploads.csv_files` records filename, hash, table, row
   count and columns, which is what "Migrated files" renders.
 * **Structured logs** — every upload returns a timestamped `logs[]`, a ready
   foundation for the audit log.
@@ -56,44 +78,24 @@ CSV content is sent as a JSON string, not multipart.
 
 ## Findings
 
-### 1. Package imports — blocks testing (fix required)
-
-`api/` uses bare imports (`from config import settings`, `from routers import
-csv_routes`). These resolve only when the process's working directory is
-`api/`, which is what `scripts/start-api.ps1` arranges with `Set-Location`.
-
-The API runs correctly. But pytest collects from the repository root, so:
+### 1. Package imports — RESOLVED
 
-```text
-$ python -c "import api.main"
-ModuleNotFoundError: No module named 'config'
-```
-
-`tests/test_api.py` therefore cannot be collected, and the API is an **untested
-surface** — invisible even to `scripts/test_report.py`, which can only account
-for tests it can collect.
-
-The fix is mechanical:
+**Previous state:** `api/` used bare imports (`from config import settings`,
+`from routers import csv_routes`). These resolved only when the process's
+working directory was `api/`. `tests/test_api.py` therefore could not be
+collected from the repository root, leaving the API as an untested surface.
 
-1. Add an empty `api/__init__.py`.
-2. Make imports package-relative in every module under `api/`:
+**Resolution:** applied all three steps of the fix.
 
-   ```python
-   from api.config import settings
-   from api.db import Conn
-   from api.services.dynamic_loader import upload_dynamic
-   from api.routers import csv_routes, te_routes
-   ```
+1. `api/__init__.py`, `api/routers/__init__.py`, and `api/services/__init__.py`
+   now exist as empty package markers.
+2. Every module under `api/` uses package-relative imports
+   (`from api.config import settings`, `from api.db import Conn`, etc.).
+3. `scripts/start-api.ps1` does `Set-Location (Join-Path $PSScriptRoot "..")`
+   before invoking `python -m uvicorn api.main:app --reload --port 8000`.
 
-3. Update `scripts/start-api.ps1` to launch from the repository root:
-
-   ```powershell
-   Set-Location $PSScriptRoot\..
-   python -m uvicorn api.main:app --reload --port 8000
-   ```
-
-Verified: with package-relative imports, `from api.main import app` succeeds
-from the root and every endpoint responds.
+Verified: `python -c "from api.main import app"` succeeds from the repo root.
+`pytest tests/test_api.py` collects and runs the full suite.
 
 ### 2. No environment selector
 
@@ -112,6 +114,12 @@ uploads schema; it is not if the API is ever pointed at a shared or production
 database. Consider an allow-list of droppable schemas, or an
 `API_ALLOW_DESTRUCTIVE=1` gate.
 
+Update: the `API_ALLOW_DESTRUCTIVE` gate and an `audit_log` table now exist,
+and every endpoint (including this one) requires the `X-API-Key` header — see
+[Authentication](#authentication). Callers still get no per-request
+confirmation prompt; that remains a UI-level gap if accidental deletes become
+a problem in practice.
+
 ### 4. Two CSV parsers now exist
 
 `api/services/csv_parse.py` parses CSVs in Python. `build/csv/validator.py`
@@ -159,13 +167,13 @@ these were written.
 
 ## Tests
 
-`tests/test_api.py` provides 17 tests:
+`tests/test_api.py` provides 19 tests:
 
 | Group | Count | Needs a database |
 |---|---|---|
 | `unit` — health contract, request validation | 9 | No |
 | `unit` + `security` — table-name guards, AST identifier check | 3 | No |
-| `integration` — upload → list → rows → dedup round trip | 5 | Yes |
+| `integration` — upload → list → rows → dedup round trip | 7 | Yes |
 
 Per the repository's no-skip policy, the integration group **fails** with
 remediation text when the database is unreachable rather than skipping.

ARCHITECTURE.md

@@ -1,26 +1,31 @@
 # ARCHITECTURE — PostgreDataMigrationApp
 
-The project is organised into three categories. Every file in the repo belongs to exactly one of them.
+The project is organised into five categories. Every file in the repo belongs to exactly one of them.
 
 ```text
 PostgreDataMigrationApp/
 |
-+-- build/             <-- production code that gets deployed
++-- build/             <-- production code that gets deployed (schema, adapters, CSV loader)
 +-- tests/             <-- correctness coverage for the production code
 +-- evals/             <-- data-driven black-box scenarios
++-- api/               <-- FastAPI REST layer over the CSV pipeline
++-- frontend/          <-- React 19 + TanStack Start web UI (calls the API only)
 |
++-- scripts/           <-- launcher and dev helpers (start-api.ps1, start-frontend.ps1, ...)
 +-- README.md  LICENSE  ARCHITECTURE.md  .gitignore
 ```
 
 ## Why the split
 
 | Category | Question it answers | Failure means |
 |----------|--------------------|---------------|
-| **build** | "What do we ship?" | The deployed system is broken |
+| **build** | "What do we ship at the DB layer?" | The deployed database is broken |
 | **tests** | "Is the code correct?" | Some function has a bug |
 | **evals** | "Does it handle real-world data correctly end-to-end?" | A whole-system behaviour regressed |
+| **api** | "How does anything outside psql talk to the DB?" | The web UI (and any other client) can't reach the data |
+| **frontend** | "How does a human drive the pipeline?" | The browser UI is broken (backend still works via API) |
 
-The three layers can break independently, so we keep them physically separate. Tests live close to the code they verify; evals stay in their own folder because they're driven by data, not code.
+The five layers can break independently, so we keep them physically separate. Tests live close to the code they verify; evals stay in their own folder because they're driven by data, not code. The API and frontend are new additions from the merge of the earlier `csv-table-hub-main` project — the browser never talks to Postgres directly, only to `api/`.
 
 ## What's in each folder
 
@@ -60,6 +65,38 @@ The three layers can break independently, so we keep them physically separate. T
 | `tests/test_snapshot.py` | golden-file output comparisons (`tests/snapshots/`) |
 | `tests/test_evals_runner.py` | unittest for `evals/runner.py` itself |
 
+### `api/` — FastAPI REST layer
+
+| Path | What it is |
+|------|-----------|
+| `api/main.py` | App entrypoint — CORS, lifespan (pool init/bootstrap/close), health endpoint, optional `require_api_key` global dependency |
+| `api/config.py` | Env-var-driven `Settings` (libpq vars, `CORS_ORIGINS`, `API_KEY`, `MAX_UPLOAD_BYTES`, `API_ALLOW_DESTRUCTIVE`) + `TE_TABLES` whitelist |
+| `api/db.py` | `psycopg2.pool.SimpleConnectionPool` + `Conn` context manager + `bootstrap()` for the `csv_uploads` schema and `csv_files` registry |
+| `api/auth.py` | Optional `X-API-Key` dependency; no-ops when `API_KEY` env var is unset |
+| `api/routers/csv_routes.py` | `POST /api/csv/preview`, `POST /api/csv/upload`, `GET /api/csv/files`, `GET /api/csv/tables/{table_name}/rows`, `DELETE /api/csv/files/{id}` |
+| `api/routers/te_routes.py` | `GET /api/te/tables` — existence + row counts for the 12 fixed T&E tables |
+| `api/services/csv_parse.py` | Pure-Python CSV parser + type inference (mirrors the frontend's original TS logic) |
+| `api/services/dynamic_loader.py` | Creates `csv_uploads.csv_<sha256[:16]>` tables from any CSV, with typed columns + `_id`/`_row_hash`/`_created_at` metadata and `ON CONFLICT DO NOTHING` dedup |
+| `api/services/te_loader.py` | Validates CSV columns are a subset of a fixed T&E table, then inserts row-by-row with `SAVEPOINT`/`ROLLBACK TO SAVEPOINT` |
+| `api/requirements.txt` | `fastapi`, `uvicorn[standard]`, `psycopg2-binary`, `python-multipart`, `pydantic` |
+
+All dynamic SQL uses `psycopg2.sql.Identifier()` / `sql.SQL()` — no f-string interpolation of identifiers (project rule from `CLAUDE.md`).
+
+### `frontend/` — React 19 + TanStack Start UI
+
+| Path | What it is |
+|------|-----------|
+| `frontend/src/routes/__root.tsx` | Root route + `errorComponent` fallback |
+| `frontend/src/routes/_authenticated/route.tsx` | Passthrough layout (auth removed — folder name kept so the generated route tree is unchanged) |
+| `frontend/src/routes/_authenticated/index.tsx` | Main CSV Migrator page |
+| `frontend/src/lib/csv.functions.ts` | `fetch`-based API client (`uploadCsv`, `listCsvFiles`, `previewCsvTable`, `previewCsvContent`, `listTeTables`, `apiHealth`); sends `X-API-Key` when `VITE_API_KEY` is set |
+| `frontend/src/routeTree.gen.ts` | Auto-generated by TanStack Router — do not hand-edit |
+| `frontend/vite.config.ts` | Dev server pinned to port `5173` |
+| `frontend/.env` | `VITE_API_URL=http://localhost:8000`, optional `VITE_API_KEY=` |
+| `frontend/package.json` | React 19, `@tanstack/react-start`, `@tanstack/react-router`, Tailwind v4, shadcn/radix components |
+
+The frontend has **no** Supabase, direct-DB, or `createServerFn` code paths — it only issues `fetch()` calls to the FastAPI backend.
+
 ### `evals/` — data-driven scenarios
 
 | Path | What it is |
@@ -79,25 +116,32 @@ The three layers can break independently, so we keep them physically separate. T
 ## Dependency direction
 
 ```text
-evals/   --reads--->  build/csv/validator.py
-                      build/environments/env_dev.sql
-                      tests/run_all_tests.sql
+frontend/  --reads--->  api/         (HTTP only, via fetch)
+
+api/       --reads--->  PostgreSQL   (psycopg2 pool)
+                        NOTHING in build/, tests/, or evals/
+
+evals/     --reads--->  build/csv/validator.py
+                        build/environments/env_dev.sql
+                        tests/run_all_tests.sql
 
-tests/   --reads--->  build/csv/validator.py
-                      evals/runner.py  (just to verify it imports cleanly)
+tests/     --reads--->  build/csv/validator.py
+                        evals/runner.py  (just to verify it imports cleanly)
 
-build/   --reads--->  nothing in tests/ or evals/
+build/     --reads--->  nothing in tests/, evals/, api/, or frontend/
 ```
 
-`build/` has no dependency on the other two layers. That's the property to defend on every change.
+`build/` has no dependency on any of the other layers. `api/` and `frontend/` are additive — they consume the deployed database created by `build/`, but neither `build/` nor `tests/`/`evals/` depend on them. That's the property to defend on every change.
 
 ## When you add a new file, ask yourself
 
-1. Does this run in production? → `build/<engine-or-folder>/`
+1. Does this run in production DB deployment? → `build/<engine-or-folder>/`
 2. Does this assert that some function is correct? → `tests/`
 3. Does this drive a scenario through the deployed system from outside? → `evals/`
+4. Is this a REST endpoint or a service the API needs? → `api/`
+5. Is this a React component, route, or client-side helper? → `frontend/src/`
 
-If a file would fit two of those, split it — the test belongs in `tests/`, the production code in `build/`.
+If a file would fit two of those, split it — the test belongs in `tests/`, the production code in `build/`, the HTTP handler in `api/`.
 
 ## What's not part of any layer