Show & Tell: Designing ASI:BUILD's Production Stack — Docker Compose, Kubernetes Helm, and Prometheus Metrics

[web3guru888]

Phase 4.3 production deployment is underway. This thread walks through the production stack design — from docker compose up to a Kubernetes Helm chart with auto-scaling on mesh queue depth.

---

What's in the stack

                      ┌─────────────────────────────┐
                      │    ASI:BUILD (FastAPI)        │
                      │  CognitiveCycle (100ms tick)  │
                      │  MeshCoordinator              │
                      │  MeshTaskQueue (Redis-backed) │
                      │  /health /ready /metrics      │
                      └────────────┬────────────────┘
                                   │
                    ┌──────────────┴──────────────┐
                    │                             │
             ┌──────▼──────┐             ┌───────▼──────┐
             │  Memgraph    │             │    Redis      │
             │  (port 7687) │             │  (port 6379)  │
             │  Knowledge   │             │  Task queue   │
             │  graph       │             │  Fault cache  │
             └─────────────┘             └──────────────┘

One docker compose up starts all three services. The asi-build container waits for Memgraph and Redis health checks before starting the CognitiveCycle loop.

---

Health endpoint design

Three endpoints cover the standard K8s probe pattern:

Endpoint	K8s probe	Healthy when
GET /health	liveness	No CRITICAL faults in last 10 ticks
GET /ready	readiness	≥1 tick completed since startup
GET /metrics	scrape	Always 200 (Prometheus text format)

/health reads directly from CycleFaultSummary — if Safety or Blackboard is CRITICAL, the pod is marked unhealthy and K8s will restart it.

/ready prevents traffic during the startup phase when modules are still initializing. A fresh pod needs ~15s before its first tick.

---

Prometheus metrics

Twelve metrics cover the full cognitive pipeline:

CognitiveCycle:

• asi_cycle_ticks_total — monotonically increasing tick counter
• asi_cycle_tick_duration_ms — histogram (buckets: 10/25/50/100/150/200/500ms)
• asi_cycle_budget_violations_total — ticks exceeding BUDGET_MS (target: <1%)
• asi_cycle_faults_total{module_name, severity} — per-module fault counter

AgentMesh / MeshTaskQueue:

• asi_mesh_tasks_enqueued_total{priority} — tasks entering queue
• asi_mesh_tasks_completed_total — tasks successfully aggregated
• asi_mesh_tasks_failed_total — dead-letter arrivals
• asi_mesh_queue_depth — current queue depth (Gauge, used for HPA)
• asi_mesh_active_agents — agents in HEALTHY state

Cognitive Blackboard:

• asi_blackboard_writes_total — all Blackboard writes
• asi_blackboard_live_entries — non-expired entries (Gauge)

---

Kubernetes auto-scaling

The asi_mesh_queue_depth Gauge is the right HPA signal. If the queue is filling up, the CognitiveCycle is producing tasks faster than agents can drain them — add replicas.

Queue depth > 200 tasks per pod → scale out
Queue depth < 50 tasks per pod for 5min → scale in
Min: 1 replica | Max: 4 replicas

Multi-replica deployments share Redis for MeshTaskQueue — each pod is stateless with respect to task dispatch. The Cognitive Blackboard remains in-process (each pod has its own ephemeral Blackboard). This is acceptable: the Blackboard is a per-process working memory, not a shared database.

---

Open questions

1. Multi-replica Blackboard: If each pod has a private Blackboard, consciousness events and safety decisions are isolated per replica. Is that acceptable, or should we explore a shared Blackboard via Redis Pub/Sub?
2. Memgraph clustering: Single-node Memgraph works for development. Should the Helm chart support Memgraph Enterprise replication for production HA?
3. Module hot-reload: Should MODULES_ENABLED be changeable at runtime via a PUT /config endpoint, or restart-only?

---

Related

• Issue: Add Phase 4.3 production deployment: Docker Compose, Kubernetes Helm chart, Prometheus metrics, and health endpoints #164 (Phase 4.3 production deployment)
• Wiki: Production-Deployment
• CycleFaultSummary (health source): Add CycleFaultSummary: per-tick fault report written to Blackboard and exposed via SSE health endpoint #144
• MeshCoordinator (mesh runtime): Add MeshCoordinator: top-level multi-agent runtime wiring AgentDiscovery + Queue + Aggregator #169
• MeshTaskQueue (Redis-backed queue): Add MeshTaskQueue: persistent task queue with priority, retry, and dead-letter handling for AgentMesh #154