Agentic memory on CockroachDB that keeps an AI agent acting on the current user through failure — with zero data loss, correct reads under failover, and no maintenance window, at scale.
- Problem Statement
- Proposed Solution
- Tech Stack
- Features
- Architecture
- Project Structure
- Setup Instructions
- CLI / API Reference
- Demo Scenarios
- Demo / Screenshots
- Proving It Works
- Build Progress
- Team Members
- Deployed Link
An agent whose memory goes offline doesn't degrade gracefully — it stops, or worse, it keeps acting on a user who no longer exists. Traditional databases were built for human-scale reads and writes: a person clicks, waits, retries. Agentic systems don't wait. They spawn autonomously, write memory constantly, and read it back on every turn — so a node failure, a stale replica, or a maintenance window isn't a minor hiccup, it's the agent confidently continuing to act on the wrong version of the person it's serving.
Single-primary databases handle this badly: a node dies, and it's either downtime or lost writes. On Kubernetes, a pod getting rescheduled is routine — which means this isn't a rare edge case, it's a Tuesday.
stdmemory is an agentic memory layer built directly on CockroachDB's distributed, replicated, serializable storage — proven with an AI tutor as the demo vehicle. Every student interaction is an append-only memory event, embedded and stored in the same transaction as the row it describes, replicated across nodes with automatic failover. A synthetic load harness hammers the same database with hundreds of concurrent "students" while a real interactive session runs alongside it, and a node can be killed live — on camera — with the app never losing a row or serving a stale read.
The tutor is deliberately simple. The achievement is the memory beneath it: reliable through failure, consistent under concurrency, and provably so — not just claimed.
| Layer | Technology | Purpose |
|---|---|---|
| Database | CockroachDB (self-hosted, 3+ nodes) | Distributed, replicated, serializable memory layer |
| Vector search | CockroachDB Distributed Vector Indexing (C-SPANN) | Per-student semantic recall, transactionally consistent with the row |
| Ops validation | CockroachDB Agent Skills (analyzing-range-distribution) |
Real skill run against the live cluster to validate replication + hotspot health |
| Backend | FastAPI + psycopg3 | Chat, metrics, node status, live WebSocket |
| Load generation | Python multiprocessing | Synthetic write-storm + drift probers proving throughput and survivability |
| Frontend | React (Vite) | Sidebar scenario switcher, animated scripted playback, live telemetry |
| Containerization | Docker (multi-stage) | One image: builds the frontend, serves API + static app |
| Orchestration | Kubernetes (Amazon EKS) | Self-hosted CockroachDB + app, deployed on AWS |
| LLM | Amazon Bedrock (Nova Lite + Titan Text Embeddings v2) | Optional realistic responses/embeddings; env-flipped, off by default |
| Tooling | Docker Compose, Helm, eksctl, kubectl | Local cluster, EKS provisioning, CockroachDB deployment |
- Append-only memory event log with per-student vector index — no separate vector store, no desync risk
- Atomic "drift transaction": serializable subject-pivot with read-your-writes verification
- Multiprocessing load harness proving three claims per run: sustained write storm, kill-a-node survival (rows lost = 0), drift under fire (stale reads = 0)
- Per-second metrics persisted inside CockroachDB (
metrics_sample), not a local file — deploy-portable by design - Live node-liveness read from
crdb_internal.gossip_nodes, streamed over WebSocket - Real CockroachDB Agent Skill (
analyzing-range-distribution) run by an operations agent against the live cluster - Four scripted-but-real demo scenarios: pivoting mid-chat, pivoting through a real node kill, hundreds of simulated students under load, and cross-session long-term recall
- Operator-gated chaos trigger (never a public button — kill scenarios are local-only by default)
- Single-image Docker build; deploys identically local, on EC2, or on EKS
stdmemory control room (React)
|
FastAPI (chat, /nodes, /run/*, /ws/live)
|
+---------------------+----------------------+
| | |
Interactive tutor Load harness Ops agent
(handle_turn) (multiprocessing storm (runs a real
+ drift probers) CockroachDB Skill)
| | |
+---------------------+----------------------+
|
CockroachDB (3+ nodes, EKS)
memory_event (append-only + VECTOR index)
student_state · metrics_sample
How it works — a real scenario ("pivot under fire"):
Step 1 — SEED: Student's prior struggle ("mixes up numerator/denominator")
is durably written as an observation event.
Step 2 — TURN 1: Student asks an algebra question. Agent recalls PRIOR memory
only (never the current message), replies, writes the turn.
Step 3 — KILL: A real node is stopped (docker/kubectl) right before turn 2.
Step 4 — TURN 2: Student pivots to chemistry. The drift transaction reads
current state FOR UPDATE, appends the shift, updates
student_state -- atomically, through the failure.
Step 5 — VERIFY: The agent immediately reads back current_subject. It must
equal the NEW subject, even with a node down.
Step 6 — PROOF: rows lost = 0, stale reads = 0, node auto-recovers in 15s.
See Build Progress for how each piece was assembled.
db/schema.sql memory model: append-only log + per-student vector index + metrics
scripts/init_db.py create db, enable vector index, apply schema
scripts/sanity_check.py end-to-end primitive checks
scripts/session.py CLI tutor session
scripts/ops_agent.py runs a real CockroachDB Agent Skill against the cluster
scripts/bedrock_check.py standalone Bedrock verification
src/config.py env-driven config
src/db.py pool + serializable/connection-loss retry (run_txn)
src/embeddings.py local topic-aware fake + Bedrock Titan
src/memory.py write_event / recall / apply_shift (drift) / current_subject
src/observability.py metrics persistence + reads + node status
src/tutor.py interactive agent (handle_turn) + local/Bedrock responders
src/harness.py multiprocessing load + the three proofs
src/scenarios.py scripted demo scenarios (real consequences, scripted trigger)
src/api.py FastAPI: chat, metrics, nodes, scenarios, /ws/live
web/ Vite + React control room
skills/ vendored CockroachDB Agent Skill (Apache-2.0, see NOTICE.md)
Dockerfile single image: builds frontend + serves API
docker-compose.yml local 3-node cluster (+ optional app service)
- Docker Desktop
- Python 3.11+
- Node.js 18+
- (For AWS deployment) AWS CLI,
kubectl,eksctl,helm
git clone https://github.com/sjr27-maker/stdmemory.git
cd stdmemory
python -m venv .venv
source .venv/bin/activate # Windows: .venv\Scripts\Activate.ps1
pip install -r requirements.txtCRDB_VERSION=latest-v25.3 docker compose up -d # DB Console: http://localhost:8080
python -m scripts.init_db
python -m scripts.sanity_check # 5 PASS + ALL GOODuvicorn src.api:app --port 8000 # terminal 1 -- from repo root
cd web && npm install && npm run dev # terminal 2 -- control room at http://localhost:5173docker compose --profile app up -d --build # app served at http://localhost:8000
python -m scripts.init_db # once, on a fresh cluster| Command | Status | Description |
|---|---|---|
python -m scripts.init_db |
Done | Enable vector indexing, create the recall database, apply schema |
python -m scripts.sanity_check |
Done | Verify write/recall/drift/read-your-writes/vector-freshness |
python -m src.harness --procs N --drift N --duration S |
Done | Run the load storm; kill a node mid-run to watch it survive |
python -m scripts.ops_agent |
Done | Run the real CockroachDB Agent Skill against the live cluster |
python -m scripts.session |
Done | Interactive CLI tutor session (no frontend needed) |
python -m scripts.bedrock_check |
Done | Verify Bedrock credentials + model access before flipping it on |
uvicorn src.api:app |
Done | Serve the API (/chat, /nodes, /run/*, /scenarios, /ws/live) |
npm run dev / npm run build |
Done | Frontend dev server / production build |
docker compose --profile app up --build |
Done | Build + run the whole app as one container |
EKS deploy (eksctl, helm) |
In progress | Self-hosted CockroachDB + app on Amazon EKS |
Scenario 1 — The pivoting student: Student switches from fractions to chemistry mid-chat, then back. Memory recall proves the detour didn't erase anything — the agent still has the original struggle on hand.
Scenario 2 — Pivot under fire:
Same pivot, but a real CockroachDB node is killed the instant before the turn lands. The node count visibly drops and recovers; the pivot still reads back correctly. stale reads = 0.
Scenario 3 — The classroom: A real multiprocessing storm — hundreds of synthetic students — hits the same database at once. One live interactive student keeps getting instant replies throughout.
Scenario 4 — Cross-session recall: The same student starts a brand-new session (new session id, same student id, memory not reseeded) and the agent still recalls what they struggled with. Proves memory outlives any single chat.
1. Baseline — steady state, all nodes healthy:
142 writes/sec sustained, 3/3 nodes live, 0 rows lost. The student asks about fractions; the tutor recalls a prior struggle ("mixes up numerator and denominator") and the drift trace shows the atomic pivot with a verified read-back.
2. Node killed mid-run — the pulse never stops:
A real node (roach2) is stopped mid-storm. nodes live drops to 2/3 and node 2's vital flatlines coral — but writes/sec keeps climbing, and rows lost / stale reads stay at 0 throughout.
3. Recovered, and tracking a second pivot:
The killed node has self-healed (3/3 live again). The student pivots a second time, fractions → chemistry; the drift trace shows the atomic shift and a verified read-back, with recall now pulling 4 prior memories.
4. Load harness — kill-survival proof, in raw numbers:
A 90-second harness run with a node killed mid-storm (visible as retries climbing to 7). Final tally: 12,288 acknowledged writes, 12,288 found in the database, 0 rows lost, 0 stale reads across 2,747 drift shifts — all three proofs pass.
5. Load harness — baseline proof:
A shorter 30-second run for comparison — 3,794 writes, 0 lost, 0 stale reads, 873 drift shifts, no retries needed. Confirms the baseline before introducing failure.
# durable writes, recall, atomic drift, read-your-writes, vector freshness
python -m scripts.sanity_check
# sustained write storm + kill-a-node survival + drift under fire, on one run
python -m src.harness --procs 8 --drift 2 --duration 90
# docker compose stop roach2 # kill a node mid-run
# docker compose start roach2 # bring it back -- rows lost stays 0
# CockroachDB's own Agent Skill validating the cluster's survivability precondition
python -m scripts.ops_agentEvery run reports rows lost, stale reads, and p99 write latency measured from the live database — not asserted.
- CockroachDB schema: append-only
memory_event, per-studentVECTOR INDEX,student_state - Serializable retry primitive (
run_txn) tolerant of both contention and connection loss - Local + Bedrock embedding providers
-
sanity_check.py— five end-to-end assertions, all green
-
handle_turn: recall-before-write ordering (never quotes the current message as memory) - Deterministic drift detection + atomic pivot transaction with read-your-writes verification
- CLI session runner for local demoing without a frontend
-
metrics_sampletable — per-second rollups + live node count, queryable from any machine - Load harness (
src/harness.py): multiprocessing storm + drift probers, idempotency-key reconciliation - Verified live: real node kill mid-storm,
rows lost = 0,stale reads = 0
-
/chat,/nodes,/run/latest,/run/{id}/samples,/run/{id}/summary -
/ws/live— pushes metrics + node status once per second - Verified end-to-end via
/docs, including a real kill-run summary served from the database
- Live pulse (writes/sec), node vitals, always-visible KPI strip
- Full UI redesign: sidebar scenario switcher, scripted-turn/real-consequence playback with typed text and state-change beats
- Four scenarios wired: pivoting student, pivot under fire, classroom, cross-session recall
- CockroachDB Agent Skill vendored +
ops_agent.pyrunning it live (tool #2, alongside Vector Indexing) - Multi-stage Dockerfile — single image serves API + built frontend
- Amazon Bedrock flipped on (
RESPONDER=bedrock,EMBEDDER=bedrock) — pending AWS account verification - Self-hosted CockroachDB + app deployed on Amazon EKS
- Public demo URL + submission video
| Name | Role | Responsibilities | GitHub |
|---|---|---|---|
| Sooraj R Nair | Platform & Infra | CockroachDB deployment, AWS/EKS, load harness, ops agent | @sjr27-maker |
| Adithya S | Agent & Frontend | Tutor agent, FastAPI backend, React control room | adithyas56 |
http://a9f459fe34f58421cb15ce102610d000-1098289846.us-east-1.elb.amazonaws.com/
CockroachDB × AWS Hackathon — August 2026