v0.6.0

Fixed

• The default Bedrock region pointed at an endpoint known to throttle. config.py, infra/template.yaml and the docs/DEPLOY.md deploy command all still defaulted to eu-west-1 — contradicted by the 2026-07-07 probe recorded in .env.example. All defaults now align on eu-north-1, and an ADR 008 addendum records the 2026-07-08 probe result (scripts/probe_bedrock.py): every region throttles every model on the first call — Titan "Too many requests", Claude and first-party Nova "Too many tokens per day" — i.e. an account-level dynamic quota clamp that neither model choice, region choice, nor promotional credits routes around; only an AWS Support escalation or account usage history lifts it
• Bedrock clients ran on botocore defaults (60s read timeout, backoff retries) — one throttled/hung call could consume the entire Lambda invocation budget and kill the function mid-step organically, making the recorded demo nondeterministic. Both bedrock-runtime clients now set connect_timeout=5, read_timeout=15, standard-mode retries capped at 2 attempts, and the Lambda Timeout went 30s → 60s with the per-step budget math documented in the template
• The Gradio "Ask via MCP" panel swallowed the real MCP error. The MCP client raises from inside an anyio TaskGroup, so a failure surfaced on the Space as the useless unhandled errors in a TaskGroup (1 sub-exception). The handler now unwraps ExceptionGroups recursively to their leaf errors (e.g. McpError: executing select query: unauthorized) and short-circuits with an actionable setup message when the key/cluster id aren't configured. Exercising the fixed path against the live server also pinned the real failure mode — a service-account key authenticates but every query returns unauthorized until the account holds the Cluster Operator role on the cluster — now documented in docs/DEPLOY.md alongside the previously-undocumented COCKROACH_MCP_CLUSTER_ID Space secret
• Sharpened the TLS guidance across .env.example, docs/DEPLOY.md and infra/template.yaml. sslmode=require stays the documented choice (encrypts, no CA file, works in every fresh container — acceptable for synthetic data, ADR 005), but the why is now empirically grounded: this cluster's cert is actually publicly-trusted (Let's Encrypt, verified 2026-07-08), yet sslmode=verify-full&sslrootcert=system still fails through psycopg/libpq (certificate verify failed, tested on libpq 18) because libpq's system CA store is empty/unresolved on many platforms. The docs now explicitly warn against verify-full&sslrootcert=system and note it's the cause of that exact error, so the guidance doesn't get "corrected" back to a string that doesn't work

Added

• scripts/probe_bedrock.py / make probe-bedrock — one InvokeModel + one Converse per candidate region, retries disabled. Run before recording: both Bedrock paths degrade silently by design (no precedent / precedent-replay), so a fully throttled account looks healthy while never touching Bedrock — the probe tells you which mode you're demoing in
• scripts/demo_run.py --tick --via-lambda — drives an alert tick through the deployed continuum-orchestrator function, making the runbook's "a fresh Lambda invocation starts cold" literal instead of simulated; first real consumer of LAMBDA_FUNCTION_NAME (previously dead config)
• make deploy — sam build --use-container + sam deploy; the container build is required when building on Windows/macOS because psycopg[binary]/pydantic-core ship compiled wheels and a host-platform build crashes on Lambda's Linux runtime with import errors (now documented in docs/DEPLOY.md)
• docs/DEMO_READINESS_CHECKLIST.md — P0/P1 winner-level demo checklist, audited against actual repo state; biggest open blockers called out (no recorded video, Space can't self-trigger an incident, docs/BENCHMARKS.md never populated)
• infra/__init__.py so infra.lambda_handler imports as a regular package rather than relying on namespace-package resolution

Changed

• docs/DEMO_RUNBOOK.md pre-flight now starts with make probe-bedrock, and the recovery beat documents both drivers: restart-and-retick --via-api (what chaos_demo.ps1 does) or --via-lambda for a real cold Lambda invocation

v0.5.0

Fixed

• QueryAgent now speaks the Managed MCP Server's current contract. The server scopes each session to a cluster via an mcp-cluster-id header and requires a database argument per query; without them every call failed (401 Unauthorized unauthenticated, cluster_id not provided once a key was configured). New settings COCKROACH_MCP_CLUSTER_ID / COCKROACH_MCP_DATABASE; constructor args passed as explicit blanks stay blank so unit tests remain hermetic whatever the local .env holds. Verified end to end against the live server (auth, session negotiation, query dispatch)
• .env.example realigned with .env after significant drift: adds the Managed MCP service-account block (endpoint, cluster id, database, API key — with the console steps for creating the key), replaces the now-false "eu-west-1 has full default quota" Bedrock claim with probe-first guidance (2026-07-07: eu-west-1, eu-central-1 and us-east-1 all throttle Titan on this account; eu-north-1 works at a low per-minute rate), and warns that Bedrock retires old model versions (eu.anthropic.claude-3-5-sonnet-20241022-v2:0 — found pinned in a stale .env — now returns "invalid model identifier")

Added

• .mcp.json — project-scoped CockroachDB Cloud MCP server config for Claude Code (streamable HTTP, cluster-scoped via the mcp-cluster-id header, Bearer auth via ${COCKROACH_MCP_API_KEY} environment expansion, so no secret lives in the committed file)
• Lambda deploy runbook in docs/DEPLOY.md
• Bedrock-free seeding so the demo Space can be populated without the throttled Bedrock account (ADR 008): make seed-data-offline / .\scripts\migrate_and_seed.ps1 -Offline seed deterministic synthetic vectors (scripts/synthetic_vectors.py) with no AWS call, and scripts/capture_seed_embeddings.py + seed_memory.py --from-fixture load real Titan vectors captured once where Bedrock is reachable
• Recovery sequence diagram and a typical-vs-Continuum comparison table in docs/ARCHITECTURE.md, making the two-cold-invocation handoff through durable CockroachDB state visible at a glance

Changed

• Sharpened the hero positioning across the README, the Hugging Face Space (ui/app.py), and the DEVPOST elevator pitch to lead with the concrete payoff — resumes the exact step it was killed on, because its memory lives in CockroachDB, not the process — instead of the more abstract earlier tagline
• Repo-wide documentation alignment pass. Synced stale version fields (api/main.py, pyproject.toml → 0.4.0); corrected the integration-test count (2 → 3) and surfaced the real-SIGKILL test_chaos_kill_e2e.py in the README and DEVPOST test-integrity claims; fixed the docs/ARCHITECTURE.md §4.1 vector-index DDL to match infra/schema.sql (always service-prefixed, embedding_model column, service-only filter); standardized the judging criterion name to Production Readiness; aligned the schema.sql recovery-query comment with the actual latest-step query; and removed cross-project references from docs/SUBMISSION.md
• Latency benchmarks (scripts/benchmark.py, make benchmark, docs/BENCHMARKS.md): p50/p95/p99 for recovery read, per-step transaction commits, C-SPANN vector search, and the full cold-resume path — against a live cluster, no Bedrock dependency
• Real-kill chaos integration test (tests/integration/test_chaos_kill_e2e.py): spawns the orchestrator as a uvicorn subprocess, hard-kills it mid-step with scripts/chaos_kill.py (real SIGKILL/TerminateProcess), and asserts a cold restart resumes the interrupted step exactly-once from CockroachDB — the literal process-kill path that test_recovery_e2e.py only simulated with a SQL UPDATE. Shared integration fixtures moved to tests/integration/conftest.py

v0.4.0

Added

• Explicit per-step transaction boundaries (ADR 009). The orchestrator's STEP 3 now goes through MemoryAgent.checkpoint_step_start / checkpoint_step_done, each an explicit with conn.transaction() block (CockroachDB SERIALIZABLE), with the time.sleep execution window between them so a kill still commits executing and nothing else. Makes the "transactional memory" claim literally true, not just per-statement autocommit
• Concurrency-safe exactly-once forward claim (ADR 009). A new step is claimed with INSERT ... ON CONFLICT (incident_id, step_index) DO NOTHING; a racing invocation that loses the claim skips execution instead of double-running the step. checkpoint_step_done guards its transition with AND status = 'executing'. Proven on a real cluster by tests/integration/test_recovery_e2e.py::test_forward_step_claim_is_exactly_once
• Best-effort correlation. Orchestrator STEP 2 (embed + find_similar) is wrapped in try/except — a throttled/misconfigured Bedrock endpoint now degrades to "no precedent" (remediation falls back to page_on_call_engineer) instead of throwing before the incident is durable. Removes the single point of failure that a red Bedrock endpoint (ADR 008's 0-quota risk) posed to the whole recovery demo

Changed

• Aligned docs/claims to the code after auditing the whole repo: corrected "6 ADRs" → 9, unit-test count → 46 (100% measured coverage against the 90% gate), and the ARCHITECTURE.md §3 recovery sequence (it inaccurately showed a killed step being skipped; the code re-runs the interrupted step). docs/DEMO_RUNBOOK.md 0:20 now uses --via-api (a bare --tick runs in-process, leaving nothing for chaos_kill.py to strike) and notes make chaos-demo is POSIX-only

Removed

• MemoryAgent.log_step / set_step_status — dead code once STEP 3 moved to the transactional checkpoints; their docstrings described a mechanism the orchestrator no longer used

v0.3.2

Fixed

• eu-central-1 has a 0, non-adjustable Bedrock quota on this account for every model (Titan Embed V2 and every Claude Sonnet variant, both on-demand and cross-region) — confirmed via aws service-quotas list-service-quotas, not assumed. seed_memory.py's single-transaction seeding was failing silently on the first embedding call and rolling back every record. Added bedrock_region (BEDROCK_REGION, default eu-west-1) used only by the bedrock-runtime clients in correlation_agent.py/remediation_agent.py; aws_region/Lambda/CockroachDB stay in eu-central-1 (ADR 007 untouched, split documented in ADR 008)
• Anthropic models required a one-time, account-wide "use case details" form (agreementAvailability: NOT_AVAILABLE on every Claude version checked, not just 4.5) before any Anthropic model could be invoked — submitted via aws bedrock put-use-case-for-model-access, followed by create-foundation-model-agreement to accept Claude Sonnet 4.5's EULA specifically. Confirmed via get-foundation-model-availability: agreementAvailability now AVAILABLE
• seed_memory.py now retries ThrottlingException with exponential backoff (5s → 160s over 6 attempts) and commits per record instead of one final commit for the whole file, so a mid-run throttle no longer discards already-seeded rows

Known issue (not code — AWS account state)

• Even with the above fixed, this brand-new AWS account still hits sustained ThrottlingException/"Too many tokens per day" on both Titan Embed V2 and Claude Sonnet 4.5 that a 6-retry/~5-minute backoff didn't clear — an automated new-account trust ramp that published Service Quotas values don't reflect. No CLI lever for this; expect it to loosen with time/usage. remediation_agent.py's deterministic precedent-replay fallback keeps the app functional in the meantime

v0.3.1

Fixed

• The 0.3.0 Space still didn't boot: Gradio's launch-time analytics telemetry compares the app's theme against its built-in themes; built-in themes use Font objects for font/font_mono while ours used plain strings, and Gradio's own Font.__eq__ doesn't guard against comparing to a non-Font — crashing with AttributeError: 'str' object has no attribute 'name' whenever analytics is enabled (the Spaces default). Fixed via analytics_enabled=False on the Blocks constructor, which also means this read-only demo doesn't phone telemetry home
• scripts/seed_memory.py, demo_run.py, and chaos_kill.py had the same subdir-import bug as ui/app.py (fixed for that file in 0.3.0, missed here): running python scripts/x.py puts scripts/ — not the repo root — on sys.path, so from agents…/from config…/from observability… raised ModuleNotFoundError. This meant make seed-data, make demo, and make chaos-demo — the actual kill-and-recover sequence docs/DEMO_RUNBOOK.md calls the thing being graded — never worked via their documented entrypoints, only discovered while running the Windows seed script end to end for the first time. All three now bootstrap the repo root the same way ui/app.py does
• seed_memory.py paces its Bedrock embedding calls (1s between records) — a tight back-to-back loop of 40 calls was hitting ThrottlingException immediately

Added

• scripts/migrate_and_seed.ps1 — Windows equivalent of make migrate + make seed-data (no make on Windows), matching the existing chaos_demo.ps1 pattern. Checks COCKROACH_DATABASE_URL and (unless -SkipSeed) AWS credentials up front with a clear message instead of a bare traceback, and checks $LASTEXITCODE after every external python call — $ErrorActionPreference = "Stop" only covers PowerShell cmdlets/terminating errors, not external command exit codes, so a failed step would otherwise print a traceback and the script would carry on and report success anyway (caught by testing the schema step against an unreachable DB)

Changed

• Space pins Python 3.14 (python_version: "3.14" in README frontmatter), matching CI (python-version: "3.14") and local dev — previously unset, so the Space picked whatever Hugging Face's own default was (observed: 3.13) rather than the project's actual target
• CockroachDB Cloud TLS guidance corrected to sslmode=require: sslrootcert=system doesn't work for CockroachDB Cloud, since its clusters use a cluster-specific CA rather than one chained to a public root — verify-full fails there with certificate verify failed even once the root-cert-file-missing error is resolved. require encrypts without needing any CA file, an acceptable trade-off since Continuum only ever stores synthetic data (ADR 005) and this connection only ever reaches CockroachDB Cloud's own endpoint

v0.3.0

Added

• ui/app.py rebuilt into a dark, NOC-style incident console (from a single Dataframe). The resilience story is now visible on screen, not only in the video:
  • Recovery timeline drill-down — pick an incident and replay its remediation_steps log; the step stuck in executing pulses and is flagged "the process died here — the next cold invocation resumes at exactly this step"
  • Resilience banner that reads the live count of in-flight (executing) steps and states the resume guarantee in plain terms
  • KPI tiles (Open · In-flight now · Resolved · Steps committed) using the accessible status palette (icon + label, never colour alone)
  • Incident cards with a per-incident mini step-tracker, and gr.Timer auto-refresh (5s) so the feed stays live during the demo
  • Still read-only: direct psycopg for the feed, query_agent (MCP) for the "Ask via MCP" panel — memory_agent.py remains the only write path
• Static preview of the console for design review (rendered from the app's own functions over synthetic incidents)

Changed

• Space now pins Gradio 6 (sdk_version: 6.19.0 in README frontmatter; requirements.txt floor gradio>=6.0), matching the development environment
• README Screenshots / Demo-UI wording updated to describe the console + recovery timeline rather than a bare incident feed
• docs/DEPLOY.md and docs/DEMO_RUNBOOK.md updated to reflect the new UI (recovery-timeline drill-down as the demo's visual proof) and the Gradio-6 pin

Fixed

• HF Space build was broken in 0.2.0: app_file: ui/app.py is in a subdirectory, so running it as a script put ui/ (not the repo root) on sys.path, and from agents…/from config… failed with ModuleNotFoundError. ui/app.py now bootstraps the repo root onto sys.path before those imports
• Gradio 5→6 API move: css/theme/js relocated from Blocks() to launch(). The stylesheet is now injected as a <style> component (renders on any version) and launch() kwargs are guarded by a signature check, so the app can't crash on a version mismatch

v0.2.0

Added

• Real Amazon Bedrock wiring: Titan Text Embeddings V2 (1024-dim, matching VECTOR(1024) schema) in the Correlation Agent; Claude-on-Bedrock reasoning with deterministic precedent-replay fallback in the Remediation Agent
• Genuinely interruptible remediation loop: each invocation drives one step through proposed → executing → executed with a simulated execution window; a kill mid-step leaves executing durably in CockroachDB and the next cold invocation re-runs that exact step
• Cross-platform chaos_kill.py (psutil) + scripts/chaos_demo.ps1 for Windows; fixed make chaos-demo to actually kill the process doing the work
• Seeded historical incidents now include their remediation step history, so precedent replay has real precedent
• CI quality gate (.github/workflows/ci.yml): Ruff → pytest → coverage → Codecov
• pyproject.toml (Python 3.14, Ruff, pytest/coverage config)
• agents/query_agent.py — a real client of the CockroachDB Cloud Managed MCP Server (official mcp SDK, streamable HTTP), called by the running app itself via GET /api/v1/incidents/open and the Gradio UI's "Ask via MCP" panel, not only by Claude Code during development (ADR 003 amended). This and Distributed Vector Indexing are Continuum's two CockroachDB tools; ccloud CLI was evaluated and cut (ADR 004 resolution, ADR 006)
• Real integration test (tests/integration/test_recovery_e2e.py) exercising the actual kill-and-recover cycle against a live CockroachDB instance — previously a stub that always skipped itself
• CI now provisions an ephemeral single-node CockroachDB container so the integration suite runs on every push, not only when a developer happens to have a cluster handy
• Unit test suite grew from 5 to 42 tests (one file per agent/module: memory, correlation, remediation, query agent, API, observability) — coverage 60% → 100%; CI gate raised 60% → 90%
• .github/workflows/release.yml — tags matching v*.*.* cut a GitHub Release with notes pulled from this file
• docs/adr/007-eu-central-1-region.md — documents the deployment region and cross-region inference profile choice
• Project-level .claude/settings.json — safe read-only + build/test command allowlist, ruff-autofix-on-edit hook

Changed

• Python 3.12 → 3.14; psycopg2 → psycopg 3 (no cp314 wheels for psycopg2); dependencies switched to floor pins that resolve to latest
• Schema: VECTOR(1536) → VECTOR(1024) (Titan v2's max output dimension — 1536 was unsatisfiable)
• README rebuilt to submission grade: logo, CI/coverage badges, brand-colored Mermaid architecture, recovery-timeline table, tech-stack badge table, CI/CD + screenshots sections
• AWS region us-east-1 → eu-central-1; Bedrock reasoning model to eu.anthropic.claude-sonnet-4-5-20250929-v1:0 (EU cross-region inference profile) — kept in sync across .env.example, config.py, infra/template.yaml (ADR 007)
• infra/template.yaml Lambda runtime python3.12 → python3.14, matching the rest of the codebase
• CLAUDE.md expanded (Key Commands, Architecture, Testing Strategy, Deployment, Judging Criteria) to match sibling-repo conventions

Fixed

• config.Settings crashed at import — and thus the whole app failed to start — for anyone with ordinary AWS_ACCESS_KEY_ID/AWS_SECRET_ACCESS_KEY in their environment (pydantic-settings defaults to extra="forbid", and those two vars were never declared fields). Now extra="ignore"
• infra/template.yaml targeted a Lambda runtime (3.12) that couldn't run this codebase's 3.14-only dependencies

Removed

• docs/continuum-DEPLOY.md — an exact duplicate of docs/DEPLOY.md

Earlier (scaffold)

• Initial project scaffold: README, ARCHITECTURE, ADRs 001–006, SUBMISSION checklist, DEMO_RUNBOOK
• CockroachDB schema: incidents, remediation_steps, incident_embeddings with vector index (infra/schema.sql)
• Agent module stubs: orchestrator, correlation, memory, remediation
• Synthetic data generation scripts (stubs)
• Chaos-kill script for resilience demo (stub)
• Hugging Face Spaces deployment: README frontmatter, .github/workflows/sync-to-hf-space.yml, docs/DEPLOY.md — free, cardless public hosting for the Gradio demo UI, replacing the Railway/Vercel/Next.js stack considered and rejected for this project (no frontend framework needed, judged surface is the CockroachDB memory layer)

Planned (before submission)

• Demo video recording per docs/DEMO_RUNBOOK.md
• Deploy orchestrator to AWS Lambda (SAM) and confirm the Hugging Face Space is live
• Complete the docs/SUBMISSION.md checklist end to end