Backend Reference

Backend Reference

The backend is a FastAPI application (backend/app/) using async SQLAlchemy 2.0 (PostgreSQL), Motor (MongoDB), Qdrant, and Redis. Entry point: app/main.py, app factory FastAPI(title="Maestro Platform API", version="0.1.0"). Business logic lives in app/services/; route handlers stay thin.

Directory map

app/ root

• main.py — app + lifespan (ensure Mongo indexes, reconcile.startup_reclaim, periodic sweep, trace flusher), CORS, request-context middleware + access log, global exception handler, /health + /health/ready, mounts api_router.

app/agents/ — orchestration (see Agent-Orchestration)

base.py, orchestrator.py, main_agent.py, subagent.py, reviewer.py, registry.py, prompts.py, schemas.py, structured.py, tools.py, validators.py, budget.py, and domains/ (one module per built-in domain).

app/api/

• router.py — aggregates all v1 routers + websocket under /api/v1.
• v1/ — auth.py, users.py, api_keys.py, tasks.py, billing.py, dashboard.py, marketplace.py, agents.py, documents.py.
• websocket.py — live task + architect streaming.

app/core/

• config.py — Settings (pydantic-settings), get_settings(), production-secret guard. See Configuration.
• constants.py — all enums/constants (plans, quotas, rate-limit tiers, agent budgets, providers, model pricing).
• database.py — connection factories (get_db, get_mongo_db, get_qdrant_client, get_redis_client), ensure_indexes, check_readiness, close_connections.
• deps.py — auth dependencies: CurrentUser, ActiveUser, VerifiedUser, CurrentFamily, DbSession.
• security.py — Argon2 hashing, JWT, AES-256-GCM BYOK encryption. See Security.
• observability.py — Sentry init.

app/models/ — SQLAlchemy ORM (see Database-Schema)

base.py, user.py, api_key.py, subscription.py, payment_method.py, usage_record.py, refresh_token.py, recovery_code.py, email_token.py, task_run.py (task_runs + task_checkpoints + task_questions).

app/schemas/ — Pydantic HTTP DTOs

auth.py, user.py, api_key.py, task.py, billing.py, marketplace.py, agent.py, document.py.

app/services/ — business logic

See the service catalog below.

app/utils/

• events.py — in-process pub/sub event_bus (subscribe/publish + control channel for cancel/answer).
• rate_limiter.py — sliding-window limiter (Redis Lua + in-memory fallback), rate_limit(), check_websocket.
• prompt_guard.py — scan_prompt injection heuristics.
• url_guard.py — check_public_url SSRF guard.
• request_context.py — client_ip, user_agent, summarize_user_agent.
• timeseries.py — UTC daily bucketing for dashboard sparklines.
• tracing.py — OTel-shaped span tracer, cost_usd, buffered Mongo flush.

app/scripts/

• purge_deleted_accounts.py — GDPR grace-period purge job (advisory lock, idempotent). Run via cron.
• seed_marketplace.py — idempotent featured-team seeding.

Service catalog (app/services/)

LLM

• llm_service.py — provider-agnostic adapter framework. LLMAdapter + AdapterCapabilities, shared _OpenAICompatAdapter; concrete adapters for 12 chat providers + AnthropicAdapter. Wrappers: FallbackLLMAdapter, TokenMeter, TracedAdapter, AdapterPool. Factory get_adapter, embed_texts. See LLM-Providers-and-BYOK.

Task engine (see Agent-Orchestration)

• task_engine.py — durable ROUTE→EXECUTE→FINALIZE loop, lease heartbeat, control listener, checkpoint replay.
• task_service.py — public task API + Mongo/emit/HITL helpers, RAG gather (_gather_context).
• checkpoint_store.py — is_complete, load, write, token_sum.
• task_run_store.py — authoritative task_runs: create/status/cancel/lease/claim-orphans.
• question_store.py — HITL question rows.
• reconcile.py — crash-recovery sweeps.
• trace_service.py — read side of trace_spans: get_trace, get_trace_summary, get_costs.

Auth / accounts (see Security)

• auth_service.py — refresh-token rotation + reuse-detection: issue_token_pair, rotate_refresh_token, revoke_family, revoke_other_families, list_active_sessions, logout.
• two_factor_service.py — TOTP: begin_setup, enable, disable, verify_login, recovery codes, qr_svg.
• email_service.py — single-use tokens (SHA-256 hash) + fire-and-forget send_verification / send_password_reset / send_deletion_requested / send_deletion_cancelled (never raise).
• email/ — EmailProvider protocol, ConsoleEmailProvider, ResendProvider, registry.get_email_provider, templates.py.
• user_service.py — cross-store erasure purge_user_data (Mongo → Qdrant → Postgres) + GDPR export_user_data, marketplace anonymization.

Billing / quota / usage (see Billing-and-Quota)

• billing_service.py — subscription lifecycle + dashboard aggregations.
• quota_service.py — get_quota_snapshot, enforce_can_start_task, resolve_task_token_budget.
• usage_service.py — authoritative token ledger: used_tokens_this_period, record_task_usage (idempotent per task_id, upsert-max).
• payment/ — PaymentProvider protocol, MockPaymentProvider, card.py (Luhn / brand / last4 / expiry), registry.get_payment_provider.

Memory / documents / marketplace / agents / tools

• memory_service.py — Qdrant RAG (user-filtered): add_memory, retrieve_memories, chunk_text, add_document_chunks, purge_user_vectors. See RAG-and-Memory.
• document_service.py — ingest, list_documents, delete_document.
• marketplace_service.py — list_items, list_showcase, publish (mandatory security scan), install, reviews/rating, purge_user_reviews.
• agent_service.py — custom agents in Mongo agent_configurations: list_agents, list_routable_agents, create/update/delete_agent; prompt scanned on write.
• web_search_service.py — DuckDuckGo (ddgs): search, format_results_block. Best-effort, never fails a task.
• data_fetch_service.py — HTTP GET → readable text: fetch, html_to_text.
• code_execution_service.py — Python-in-Docker sandbox: run_python, is_available. Off by default in prod.

Request middleware and health

main.py installs an HTTP middleware that stamps a server-generated X-Request-ID (inbound header is not trusted), logs one structured line to the maestro.access logger with request_id / method / path / status / duration_ms (excluding /health*), and writes the access line even on unhandled exceptions. The Sentry event is tagged with request_id.

• GET /health — liveness.
• GET /health/ready — readiness; returns 503 when a datastore is degraded.

Conventions (enforced in CI)

• Python 3.11+, mandatory type annotations, Ruff (line length 88, rules E,W,F,I,UP,B), known-first-party = ["app"].
• All endpoints async; Pydantic v2 validation.
• Every HTTP route declares an explicit dependencies=[rate_limit(...)]; tests/test_rate_limiter.py fails the build if a route omits it.
• New LLM/payment/email provider = new adapter class; existing code unchanged.
• Migrations via Alembic only.

See Development-Setup for how to run lint/tests.