Backend Architecture

Canonical source: docs/BACKEND_ARCHITECTURE.md — this page is rendered from it on every publish; edit the source.

ClawAI Backend Architecture & Wiring Audit

Date: 2026-04-07 Scope: Backend services, data flow, queues, persistence, eventing, APIs, adapters, health, runtime

---

1. Current-State Assessment

Service Map (at the audit date)

Service	Port	Database	Status	Key Finding
auth-service	4001	PostgreSQL claw_auth	REAL	JWT + refresh rotation, RBAC (3 roles), user CRUD, password management. Missing: device session tracking
chat-service	4002	PostgreSQL claw_chat	REAL	Thread/message CRUD, context assembly, execution with fallback chain. Missing: token streaming, contextPackIds on thread
connector-service	4003	PostgreSQL claw_connectors	REAL	5 provider adapters (OpenAI, Anthropic, Gemini, Bedrock, DeepSeek), health check, model sync, encrypted config
routing-service	4004	PostgreSQL claw_routing	REAL	7 routing modes, Ollama-assisted AUTO routing, heuristic fallback, decision persistence. Missing: capability matching
memory-service	4005	PostgreSQL claw_memory (pgvector)	PARTIAL	CRUD works. Auto-extraction is a STUB (just logs). Internal endpoints for context retrieval work
file-service	4006	PostgreSQL claw_files	PARTIAL	Upload, validation, chunking (JSON/CSV/MD/text) work. NO vector embeddings, NO semantic search
audit-service	4007	MongoDB claw_audit	REAL	10 event types handled, usage ledger with token/cost tracking, severity levels
ollama-service	4008	PostgreSQL claw_ollama	REAL	Model pull, list, generate, health check, role assignment (ROUTER/FALLBACK/REASONING/CODING). Roles are NOT consumed by routing
health-service	4009	None	REAL	Pings all services, aggregates status (healthy/degraded/unhealthy), used by Docker + frontend
client-logs-service	4010	MongoDB claw_client_logs	REAL	Batched ingestion from frontend, TTL 30 days, structured JSON, searchable
server-logs-service	4011	MongoDB claw_server_logs	REAL	RabbitMQ-driven ingestion from all backend services, Elasticsearch-ready schema

Data Flow (Current)

Frontend → Nginx (4000) → Service endpoints
                        → Auth guard (JWT validation on every request)

Message flow:
  POST /chat-messages → chat-service creates USER message
    → publishes message.created (RabbitMQ)
    → routing-service receives message.created
    → calls Ollama router (AUTO mode) or uses heuristic
    → publishes message.routed
    → chat-service receives message.routed
    → ContextAssemblyManager.assemble():
        - fetches memories from memory-service (HTTP)
        - fetches file chunks from file-service (HTTP)
        - builds prompt with token budgeting
    → ChatExecutionManager.execute():
        - calls Ollama or cloud provider
        - tries fallback chain on failure
    → stores ASSISTANT message
    → emits SSE completion event
    → publishes message.completed (RabbitMQ)
    → audit-service records usage + audit log
    → memory-service receives (but extraction is a STUB)

---

2. Gap Analysis

P0 — Critical Backend Gaps

#	Gap	Impact	Remediation
B1	Memory extraction is a STUB	AI never learns from conversations	Implement extraction in handleMessageCompleted using Ollama
B2	No background job processing	File processing and memory extraction block request threads	Add BullMQ for async jobs
B3	RabbitMQ nack discards messages	Message loss on any handler failure	Add dead-letter exchange + retry queue
B4	No contextPackIds on ChatThread	Context packs can't be attached to threads	Add schema field + migration
B5	Ollama role assignments not consumed by routing	Role system is decorative	Wire routing to query assigned router model

P1 — High Priority Gaps

#	Gap	Impact	Remediation
B6	No vector embeddings for files	Can't do semantic file retrieval, only full-chunk injection	Add pgvector embedding column + embedding generation
B7	No capability-based routing	Can't filter models by supportsVision/supportsTools	Add capability check before route selection
B8	No token-by-token streaming	Users wait for full response	Implement chunked streaming from providers
B9	No device session tracking	Can't manage active sessions per device	Add device fingerprint to Session model
B10	No circuit breaker for provider calls	One failing provider can cascade failures	Add circuit breaker pattern

P2 — Completeness Gaps

#	Gap	Impact
B11	No request correlation from frontend to backend	Can't trace a user action across services
B12	No feature flags / system settings consumed at runtime	Can't toggle features without redeployment
B13	No database backup automation	Data loss risk

---

3. Data Model Assessment

Existing Models (Verified)

auth-service (PostgreSQL claw_auth)

• User — id, email, username, passwordHash, role (ADMIN/OPERATOR/USER), status (ACTIVE/INACTIVE/SUSPENDED), mustChangePassword, languagePreference, appearancePreference
• Session — id, userId, refreshToken, expiresAt, createdAt
• SystemSetting — id, key (unique), value, description

chat-service (PostgreSQL claw_chat)

• ChatThread — id, userId, title, routingMode, lastProvider, lastModel, preferredProvider, preferredModel, isPinned, isArchived, systemPrompt, temperature, maxTokens
• ChatMessage — id, threadId, role (USER/ASSISTANT/SYSTEM/TOOL), content, provider, model, routingMode, routerModel, usedFallback, inputTokens, outputTokens, estimatedCost, latencyMs, feedback, metadata (JSON)
• MessageAttachment — id, messageId, fileId, type

connector-service (PostgreSQL claw_connectors)

• Connector — id, name, provider (OPENAI/ANTHROPIC/GEMINI/AWS_BEDROCK/DEEPSEEK/OLLAMA), status, authType, encryptedConfig, baseUrl, region, isEnabled, defaultModelId
• ConnectorModel — id, connectorId, provider, modelKey, displayName, lifecycle, supportsStreaming, supportsTools, supportsVision, supportsAudio, supportsStructuredOutput, maxContextTokens
• ConnectorHealthEvent — id, connectorId, status, latencyMs, errorMessage, checkedAt
• ModelSyncRun — id, connectorId, status, modelsFound, modelsAdded, modelsRemoved, startedAt, completedAt, errorMessage

routing-service (PostgreSQL claw_routing)

• RoutingDecision — id, messageId, threadId, selectedProvider, selectedModel, routingMode, confidence, reasonTags, privacyClass, costClass, fallbackProvider, fallbackModel
• RoutingPolicy — id, name, routingMode, priority, config (JSON), isActive

memory-service (PostgreSQL claw_memory, pgvector enabled)

• MemoryRecord — id, userId, type (SUMMARY/FACT/PREFERENCE/INSTRUCTION), content, sourceThreadId, sourceMessageId, isEnabled
• ContextPack — id, userId, name, description, scope
• ContextPackItem — id, contextPackId, type, content, fileId, sortOrder

file-service (PostgreSQL claw_files)

• File — id, userId, filename, mimeType, sizeBytes, storagePath, ingestionStatus (PENDING/PROCESSING/COMPLETED/FAILED)
• FileChunk — id, fileId, chunkIndex, content

ollama-service (PostgreSQL claw_ollama)

• LocalModel — id, name, family, parameterSize, quantization, sizeBytes, format, digest
• PullJob — id, modelName, status, progress, error, startedAt, completedAt
• LocalModelRoleAssignment — id, modelId, role (ROUTER/FALLBACK_CHAT/REASONING/CODING), assignedAt

audit-service (MongoDB claw_audit)

• AuditLog — userId, action, entityType, entityId, severity (LOW/MEDIUM/HIGH/CRITICAL), details, ipAddress, createdAt
• UsageLedger — userId, resourceType, action, quantity, unit, metadata, createdAt

client-logs-service (MongoDB claw_client_logs)

• ClientLog — level, message, component, action, userId, route, userAgent, metadata, createdAt (TTL 30 days)

server-logs-service (MongoDB claw_server_logs)

• ServerLog — level, message, serviceName, module, controller, action, requestId, traceId, userId, threadId, messageId, provider, latencyMs, errorCode, errorStack, metadata

Missing Data Model Fields

Model	Missing Field	Purpose
Session	deviceId, deviceName, ipAddress	Device session tracking
ChatThread	contextPackIds String[]	Attach context packs to threads
FileChunk	embedding Vector(384)	Semantic search via pgvector
MemoryRecord	embedding Vector(384)	Semantic memory retrieval

---

4. API Contracts (Current)

Auth Service (4001)

POST   /api/v1/auth/login          — { email, password } → { tokens, user }
POST   /api/v1/auth/refresh         — { refreshToken } → { tokens }
POST   /api/v1/auth/logout          — (authenticated) → 204
GET    /api/v1/auth/me              — (authenticated) → user profile
POST   /api/v1/users               — (admin) create user
GET    /api/v1/users               — (admin) list users
GET    /api/v1/users/:id           — (admin) get user
PATCH  /api/v1/users/:id           — (admin) update user
PATCH  /api/v1/users/me/password   — change own password
PATCH  /api/v1/users/me/preferences — update language/appearance

Chat Service (4002)

POST   /api/v1/chat-threads         — create thread (title, routingMode, systemPrompt, temperature, maxTokens, preferredProvider, preferredModel)
GET    /api/v1/chat-threads         — list threads (paginated, filterable)
GET    /api/v1/chat-threads/:id     — get thread
PATCH  /api/v1/chat-threads/:id     — update thread
DELETE /api/v1/chat-threads/:id     — delete thread (cascades messages)
POST   /api/v1/chat-messages        — send message (threadId, content, routingMode?, provider?, model?, fileIds?)
GET    /api/v1/chat-messages/thread/:threadId — list messages (cursor-paginated: before, limit)
GET    /api/v1/chat-messages/:id    — get message
POST   /api/v1/chat-messages/:id/regenerate — regenerate response
PATCH  /api/v1/chat-messages/:id/feedback — set feedback
SSE    /api/v1/chat-messages/stream/:threadId — stream completion events

Connector Service (4003)

POST   /api/v1/connectors           — create connector
GET    /api/v1/connectors           — list connectors
GET    /api/v1/connectors/:id       — get connector
PATCH  /api/v1/connectors/:id       — update connector
DELETE /api/v1/connectors/:id       — delete connector
POST   /api/v1/connectors/:id/test  — test health
POST   /api/v1/connectors/:id/sync  — sync models
GET    /api/v1/connectors/:id/models — list models
GET    /api/v1/internal/connectors/config?provider=X — (internal) get decrypted config

Routing Service (4004)

POST   /api/v1/routing/evaluate      — evaluate route (routingMode, message, forcedModel?, forcedProvider?)
POST   /api/v1/routing/policies      — create policy
GET    /api/v1/routing/policies      — list policies
PATCH  /api/v1/routing/policies/:id  — update policy
DELETE /api/v1/routing/policies/:id  — delete policy
GET    /api/v1/routing/decisions/:threadId — get routing history for thread

Memory Service (4005)

POST   /api/v1/memories              — create memory
GET    /api/v1/memories              — list memories (paginated, filterable by type)
GET    /api/v1/memories/:id          — get memory
PATCH  /api/v1/memories/:id          — update memory
DELETE /api/v1/memories/:id          — delete memory
PATCH  /api/v1/memories/:id/toggle   — toggle enabled/disabled
POST   /api/v1/context-packs         — create context pack
GET    /api/v1/context-packs         — list packs
GET    /api/v1/context-packs/:id     — get pack with items
PATCH  /api/v1/context-packs/:id     — update pack
DELETE /api/v1/context-packs/:id     — delete pack
POST   /api/v1/context-packs/:id/items — add item
DELETE /api/v1/context-packs/:id/items/:itemId — remove item
GET    /api/v1/internal/memories/for-context?userId=X&limit=N — (internal) get enabled memories
GET    /api/v1/internal/context-packs/:id/items — (internal) get pack items

File Service (4006)

POST   /api/v1/files                 — upload file (base64 content)
GET    /api/v1/files                 — list files
GET    /api/v1/files/:id             — get file metadata
DELETE /api/v1/files/:id             — delete file + chunks
GET    /api/v1/files/:id/chunks      — get file chunks
GET    /api/v1/internal/files/:id/chunks — (internal) get chunks for context assembly

Ollama Service (4008)

GET    /api/v1/ollama/models         — list installed models
POST   /api/v1/ollama/pull           — pull model from registry
POST   /api/v1/ollama/assign-role    — assign model to role
POST   /api/v1/ollama/generate       — (public) generate text
GET    /api/v1/ollama/health         — (public) health check
GET    /api/v1/ollama/runtimes       — list runtimes

Health Service (4009)

GET    /api/v1/health                — (public) aggregated health status

Audit Service (4007)

GET    /api/v1/audits                — (admin) list audit logs
GET    /api/v1/audits/stats          — (admin) audit statistics
GET    /api/v1/usage                 — (admin) usage entries
GET    /api/v1/usage/summary         — (admin) usage summary
GET    /api/v1/usage/cost            — (admin) cost summary
GET    /api/v1/usage/latency         — (admin) latency summary

---

5. Event Bus (RabbitMQ)

Exchange

• Name: claw.events (topic exchange, durable)

Published Events

Event Pattern	Publisher	Subscribers	Payload
message.created	chat-service	routing-service	messageId, threadId, userId, content, routingMode, forcedProvider?, forcedModel?
message.routed	routing-service	chat-service	messageId, threadId, selectedProvider, selectedModel, routingMode, fallbackProvider?, fallbackModel?
message.completed	chat-service	audit-service, memory-service	messageId, threadId, assistantMessageId, provider, model, inputTokens, outputTokens, latencyMs
thread.created	chat-service	—	threadId, userId
user.login	auth-service	audit-service	userId, email
user.logout	auth-service	audit-service	userId
connector.created	connector-service	audit-service	connectorId, provider, name, userId
connector.updated	connector-service	audit-service	connectorId, changes
connector.deleted	connector-service	audit-service	connectorId, userId
connector.synced	connector-service	audit-service, routing-service	connectorId, modelsFound, modelsAdded
connector.health_checked	connector-service	audit-service, routing-service	connectorId, status, latencyMs
routing.decision_made	routing-service	audit-service	threadId, selectedProvider, selectedModel, confidence, reasonTags
memory.extracted	memory-service	audit-service	memoryId, userId, type
file.uploaded	file-service	—	fileId, userId, filename
file.chunked	file-service	—	fileId, chunkCount
log.server	all backend services	server-logs-service	structured log payload

Missing Resilience

• No dead-letter exchange (DLX)
• No retry queue
• Failed messages are nack'd without requeue (discarded)
• No message TTL

---

6. Unwired Backend Smell List

These are things that LOOK complete but are NOT:

Smell	Where	What's Wrong
Memory extraction handler	memory-service handleMessageCompleted	Just logs "Received message.completed" — no extraction logic
Ollama role assignments	ollama-service LocalModelRoleAssignment	Schema exists, CRUD exists, but routing-service never queries roles
Routing policies	routing-service RoutingPolicy	Full CRUD, but evaluateRoute() never reads policies
MessageAttachment model	chat-service schema	Model defined, but never populated — messages don't track file associations
Context pack thread attachment	chat-service	No contextPackIds field on ChatThread — packs can't be linked
SSE streaming	chat-service ChatStreamController	Only emits "done" event, no token-by-token chunks
File embedding	file-service FileChunk	pgvector extension enabled in memory-service but NOT in file-service — no embedding column

---

7. Implementation Plan — Remaining Backend Work

Phase A: Memory Extraction (Priority: P0)

Goal: Make handleMessageCompleted actually extract facts/preferences/instructions from conversations using Ollama.

Changes:

1. memory-service AppConfig: Add OLLAMA_SERVICE_URL
2. memory-service: New MemoryExtractionManager that calls Ollama with extraction prompt
3. memory-service handleMessageCompleted: Call extraction manager, store extracted memories
4. Extraction prompt: Ask Ollama to return JSON array of { type, content } extracted from the assistant response
5. Zod validate Ollama response, discard invalid entries
6. Publish memory.extracted event for each extracted memory

env changes: Add OLLAMA_SERVICE_URL to memory-service AppConfig, .env, .env.example, install scripts

Phase B: Context Pack Thread Attachment (Priority: P0)

Changes:

1. chat-service Prisma: Add contextPackIds String[] to ChatThread
2. chat-service DTOs: Add contextPackIds to create/update thread DTOs
3. chat-service handleMessageRouted: Pass thread.contextPackIds to ContextAssemblyManager
4. Frontend: Add context pack selector in thread settings
5. Frontend types: Add contextPackIds to ChatThread type

Phase C: Wire Ollama Role Assignments into Routing (Priority: P1)

Changes:

1. routing-service AppConfig: Already has OLLAMA_SERVICE_URL
2. routing-service: On startup, fetch ROUTER role model from ollama-service
3. OllamaRouterManager: Use the assigned router model instead of config default
4. Add internal endpoint to ollama-service: GET /api/v1/internal/ollama/router-model

Phase D: RabbitMQ Resilience (Priority: P1)

Changes:

1. shared-rabbitmq: Add DLX (dead-letter exchange) setup on channel creation
2. shared-rabbitmq: Add retry logic with exponential backoff (3 retries, then DLQ)
3. shared-rabbitmq: Add message TTL (24 hours)
4. Add RABBITMQ_RETRY_COUNT and RABBITMQ_RETRY_DELAY_MS to .env

Phase E: Capability-Based Route Validation (Priority: P1)

Changes:

1. routing-service: Before finalizing route, query connector-service for model capabilities
2. If message contains image/file references, filter models without supportsVision
3. If routing to a model that doesn't support the required capability, try next in fallback chain
4. Add internal endpoint: GET /api/v1/internal/connectors/model-capabilities?provider=X&model=Y

---

8. Security / Reliability / Performance

Current Security Posture

• JWT auth with refresh rotation: REAL
• RBAC with 3 roles (ADMIN, OPERATOR, USER): REAL
• Rate limiting (100 req/min configurable): REAL
• Helmet security headers: REAL
• Zod input validation: REAL (all DTOs)
• Prisma ORM (no raw SQL): REAL
• Secret encryption (AES-256-GCM for connector configs): REAL
• Log redaction (passwords, tokens, API keys): REAL
• CORS with configurable origins: REAL

Missing Security Controls

• No HTTPS/TLS termination at Nginx (dev-only HTTP)
• No request signing between services (internal endpoints are @Public)
• No IP allowlisting for internal endpoints
• No audit log for admin actions (user create, role change)

Reliability Concerns

• RabbitMQ message loss on handler failure (nack without requeue)
• No circuit breaker for HTTP calls to providers
• All file processing is synchronous (blocks request thread)
• No database connection pooling configuration visible

Performance Considerations

• 7 PostgreSQL instances (~200MB RAM each, ~1.4GB total)
• Ollama router adds 1-5s latency per AUTO-routed message
• No caching of connector configs (fetched on every cloud provider call)
• No caching of Ollama router model assignment

---

9. Testing Assessment

Service	Test Files	Tests	Coverage
auth-service	8	77	Guards, JWT, DTOs, auth manager, user service
chat-service	5	55	DTOs, messages service, threads service
connector-service	4	46	DTOs, manager, service
routing-service	3	29	Routing manager, routing service
memory-service	4	48	Memory service, context packs service
file-service	3	22	File processing, files service
ollama-service	3	12	Ollama manager (env-dependent failure)
audit-service	4	22	Event manager, services
health-service	1	1	Smoke test only
client-logs-service	0	0	NO TESTS
server-logs-service	0	0	NO TESTS
Total	35	312

Missing Test Types

• Integration tests (cross-service event flow)
• E2E tests (Playwright infrastructure exists but empty)
• API contract tests
• Load/performance tests
• Failure mode tests (provider timeout, RabbitMQ down, DB down)

---

10. Release Gates

Gate	Status	Blocking?
Typecheck passes (0 errors)	PASS	No
Lint passes (0 errors)	PASS	No
Build succeeds	PASS	No
312 unit tests pass	PASS	No
Rate limiting enabled	PASS	No
Helmet security headers	PASS	No
Memory extraction real	FAIL (stub)	YES
Context packs attachable to threads	FAIL (no schema field)	YES
Ollama roles consumed by routing	FAIL (unused)	YES
RabbitMQ retry/DLQ	FAIL (messages discarded)	YES for production
E2E tests	FAIL (none exist)	YES for production
CI/CD pipeline	PASS (GitHub Actions)	No

---

11. Final Verification Checklist

Requirement	Status
Context assembled before execution	REAL — ContextAssemblyManager fetches memories + file chunks
Routing decisions persisted	REAL — RoutingDecision table stores every decision
Every assistant message has provider/model	REAL — stored on ChatMessage record
File ingestion is real	PARTIAL — chunking works, no embeddings
Memory extraction is async and auditable	STUB — handler just logs
Logs emitted on every critical transition	REAL — StructuredLogger used throughout
Fallback paths explicit and safe	REAL — fallback chain with multiple providers
Background jobs for heavy operations	MISSING — all synchronous
Dead letter handling	MISSING — failed messages discarded
Circuit breaker for providers	MISSING — no circuit breaker