refactor: consolidate all user-facing HTTP APIs into the api service

[SimplicityGuy]

Overview

Currently, three services expose user-facing HTTP APIs on separate ports. This issue tracks consolidating all user-facing endpoints into the api service, making it the single API gateway for all external clients.

Current State

Services with user-facing HTTP APIs (target for consolidation)

Service	Ports	Endpoints
api	8004 / 8005 (health)	Auth, OAuth, admin config
curator	8010 / 8011 (health)	Sync trigger, sync status
explore	8006 / 8007 (health)	Graph queries, snapshots, user data

Services with health-only HTTP (keep as-is — internal probes only)

Service	Port	Notes
extractor	8000	/health, /metrics, /ready — Rust/Axum, internal only
graphinator	8001	/health — consumed by dashboard
tableinator	8002	/health — consumed by dashboard
dashboard	8003	Monitoring UI + WebSocket; keep separate

Endpoints to Migrate

From curator service → api service

POST /api/sync          # Trigger full collection + wantlist sync (requires JWT)
GET  /api/sync/status   # Sync history for current user (requires JWT)

Current implementation: curator/curator.py — FastAPI app on port 8010 with background task for sync via curator/syncer.py

From explore service → api service

# Public / optionally-authenticated
GET /api/autocomplete                 # Autocomplete search (q, type, limit)
GET /api/explore                      # Center node with category counts
GET /api/expand                       # Expand category (paginated)
GET /api/node/{node_id}              # Full node details
GET /api/trends                       # Time-series release counts
POST /api/snapshot                    # Save graph snapshot
GET  /api/snapshot/{token}            # Restore graph snapshot

# Authenticated (require Bearer token)
GET /api/user/collection              # User's collection from Neo4j
GET /api/user/wantlist                # User's wantlist from Neo4j
GET /api/user/recommendations         # Release recommendations
GET /api/user/collection/stats        # Collection statistics
GET /api/user/status                  # Release in-collection/wantlist check

Current implementation: explore/explore.py with business logic in explore/neo4j_queries.py and explore/user_queries.py

Recommended Approach

True consolidation (preferred over proxy pattern):

Move the HTTP route handlers into the api service. The business logic modules (syncer.py, neo4j_queries.py, user_queries.py, snapshot_store.py, models.py) move to the api package. The curator and explore services become pure background workers with no HTTP API (health endpoints excepted).

This avoids adding network hops and keeps authentication logic co-located with endpoints.

Alternative (proxy/gateway)

The api service forwards authenticated requests to curator and explore internally. Simpler to implement but adds latency and keeps the extra HTTP servers running.

Implementation Plan

1. Restructure the api package

Create a router-based layout to keep the file manageable:

api/
  api.py              # App factory + lifespan (no route changes yet)
  models.py           # Existing models
  routers/
    __init__.py
    auth.py           # Move auth routes from api.py
    oauth.py          # Move OAuth routes from api.py
    admin.py          # Move admin config routes from api.py
    sync.py           # New — migrated from curator/curator.py
    explore.py        # New — migrated from explore/explore.py
    snapshot.py       # New — migrated from explore/explore.py (snapshot endpoints)
    user.py           # New — migrated from explore/explore.py (user endpoints)
  services/
    discogs.py        # Existing
    neo4j.py          # New — Neo4j connection management (from explore)
  queries/
    neo4j_queries.py  # Moved from explore/
    user_queries.py   # Moved from explore/
  snapshot_store.py   # Moved from explore/

2. Update api lifespan

The api service lifespan (api/api.py) currently connects to PostgreSQL and Redis. It needs to also connect to Neo4j (for explore/user queries).

• Add NEO4J_URI, NEO4J_USERNAME, NEO4J_PASSWORD to ApiConfig in common/config.py
• Initialize Neo4j driver in lifespan alongside existing pools
• Pass Neo4j driver to explore/user routers via FastAPI dependency injection

3. Migrate curator endpoints

• Copy sync route logic from curator/curator.py → api/routers/sync.py
• The background task calls curator/syncer.py — either:
  • Option A: Move syncer.py into api/ and call directly (requires Neo4j + Discogs deps in api)
  • Option B: API route publishes a RabbitMQ message; curator runs as a pure consumer (cleaner separation but requires message schema for sync trigger)
• Copy sync status DB query logic into api/routers/sync.py
• curator service: remove FastAPI app entirely, keep only the syncer worker

4. Migrate explore endpoints

• Copy all route handlers from explore/explore.py → api/routers/explore.py, api/routers/snapshot.py, api/routers/user.py
• Move explore/neo4j_queries.py → api/queries/neo4j_queries.py
• Move explore/user_queries.py → api/queries/user_queries.py
• Move explore/snapshot_store.py → api/snapshot_store.py
• Move explore/models.py → api/models.py (merge with existing)
• explore service: remove FastAPI app entirely; the package can be removed or repurposed

5. Update dependencies

• api/pyproject.toml: add neo4j driver (already in explore/pyproject.toml)
• Remove fastapi, uvicorn from curator/pyproject.toml and explore/pyproject.toml

6. Update ports and Docker config

• Remove curator service ports 8010/8011 from docker-compose.yml (keep container for the worker)
• Remove explore service ports 8006/8007 from docker-compose.yml (or remove service entirely)
• Update dashboard/dashboard.py — remove calls to http://explore:8006 if any exist
• Update CLAUDE.md quick reference port table

7. Update the dashboard service

Audit dashboard/dashboard.py for any calls to curator or explore service endpoints and update to call the api service instead (or call data sources directly).

8. Tests

• tests/api/ — add tests for all migrated routes:
  • test_sync.py (POST /api/sync, GET /api/sync/status)
  • test_explore.py (autocomplete, explore, expand, node, trends)
  • test_snapshot.py (POST/GET /api/snapshot/*)
  • test_user.py (collection, wantlist, recommendations, stats, status)
• Remove tests/curator/test_curator.py HTTP route tests (keep syncer unit tests)
• Remove tests/explore/test_explore.py HTTP route tests (keep query unit tests)

Files Changed (estimated)

Modified:

• api/api.py — add routers, add Neo4j lifespan init
• api/models.py — merge with explore models
• common/config.py — add Neo4j fields to ApiConfig
• curator/curator.py — remove FastAPI app, keep worker loop
• docker-compose.yml — remove curator/explore HTTP ports
• CLAUDE.md — update port table

Added:

• api/routers/auth.py
• api/routers/oauth.py
• api/routers/admin.py
• api/routers/sync.py
• api/routers/explore.py
• api/routers/snapshot.py
• api/routers/user.py
• api/queries/neo4j_queries.py
• api/queries/user_queries.py
• api/snapshot_store.py
• api/services/neo4j.py
• tests/api/test_sync.py
• tests/api/test_explore.py
• tests/api/test_snapshot.py
• tests/api/test_user.py

Deleted (or gutted to worker-only):

• explore/explore.py (FastAPI app removed; or entire package)
• explore/models.py (merged into api/models.py)
• explore/snapshot_store.py (moved to api/)
• explore/neo4j_queries.py (moved to api/queries/)
• explore/user_queries.py (moved to api/queries/)

Acceptance Criteria

• All endpoints currently served by curator (8010) and explore (8006) are reachable via the api service (8004)
• curator service runs as a pure background worker (no HTTP API except /health)
• explore service either removed or runs as a pure background worker with no HTTP API
• All existing tests pass; new route tests added for migrated endpoints
• docker-compose.yml no longer exposes ports 8006, 8007, 8010, 8011
• CLAUDE.md port reference table updated
• uv run mypy . passes with no new errors
• uv run pytest passes

Context for Claude Code

When implementing this issue, start by reading:

1. api/api.py — current api service structure (auth, OAuth, admin)
2. curator/curator.py — sync endpoints + background task pattern
3. curator/syncer.py — sync business logic
4. explore/explore.py — all explore endpoints (large file)
5. explore/neo4j_queries.py — Neo4j query helpers
6. explore/user_queries.py — user-specific queries
7. explore/snapshot_store.py — in-memory snapshot cache
8. explore/models.py — Pydantic models
9. common/config.py — configuration classes (add Neo4j fields to ApiConfig)
10. docker-compose.yml — service definitions and port mappings

The api service currently uses module-level globals (_pool, _redis, _config). When refactoring to a router-based layout, these can be passed via FastAPI's dependency injection (Depends) for cleaner testability, or kept as module-level state for minimal diff — choose whichever keeps the PR focused.