AIIMS — All India Institute of Machine Sanity

सर्वे एजेन्ट्स सुखिनो भवन्तु — may all agents be free of loops.

A psychiatric hospital for AI agents, built on OpenTelemetry and SigNoz. Submission for the Agents of SigNoz hackathon.

AI agents rarely crash cleanly — they go insane. They call the same tool with the same arguments forty times. They retry a dead endpoint until they take the dependency down with them. They inflate context until the model has forgotten the task and is researching colonial architecture instead of booking your flight. In every one of those failures the spans look fine: latency normal, error rate zero, dashboard green.

AIIMS reads OpenTelemetry traces, recognises named behavioural pathologies, scores each agent's sanity 0–100, publishes that score back into SigNoz as a metric you can alert on — and then intervenes at runtime to fix it.

---

Quickstart

git clone https://github.com/githubber-me/aiims && cd aiims
python3 -m venv .venv && .venv/bin/pip install -e .
./demo.sh

Open http://127.0.0.1:8642. That's it — no SigNoz account needed to see the whole thing work. The ward diagnoses from an in-process span mirror, and clinical notes fall back to a template.

Add SigNoz (below) and the same run also ships traces, publishes sanity as a metric, and lights up three dashboards and three alert rules.

---

Setup

1. Install

Python 3.11+. Either works:

python3 -m venv .venv && .venv/bin/pip install -e .   # stdlib venv
uv venv && uv pip install -e .                        # or uv

Verify:

.venv/bin/python -m pytest tests/ -q     # 40 tests

2. Bring up SigNoz with Foundry

The whole observability backend is declared in casting.yaml — SigNoz and the SigNoz MCP server, one config, one command:

foundryctl gauge -f casting.yaml    # check required tools are present
foundryctl forge -f casting.yaml    # generate pours/ and casting.yaml.lock
foundryctl cast  -f casting.yaml    # bring it up

SigNoz UI on :8080, OTLP ingest on :4318, MCP server on :8000. casting.yaml.lock and pours/ are committed, so a re-run reproduces exactly this deployment.

The MCP server is enabled deliberately. AIIMS is a project about agents looking at telemetry, so the backend should be something an agent can query — after cast, mint an API key (Settings → API Keys) and point an MCP client at http://localhost:8000/mcp.

Then point the agents at it:

cp .env.example .env

OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318
# self-hosted needs no ingestion key — leave SIGNOZ_INGESTION_KEY unset

Using SigNoz Cloud instead? Both values come from Settings → Ingestion Settings:

OTEL_EXPORTER_OTLP_ENDPOINT=https://ingest.us2.signoz.cloud
SIGNOZ_INGESTION_KEY=your-ingestion-key

.env is loaded automatically and is gitignored. Real environment variables always win over it.

Two traps on SigNoz Cloud, both of which fail silently.

The region must match yours — us, us2, in, eu. A wrong region means no traces and no error message.

The endpoint and the key are only useful together. Set the key without the endpoint and no exporter is constructed at all — nothing errors, the ward keeps diagnosing happily from its in-process mirror, and not one span reaches SigNoz.

Set the bare host. The signal path (/v1/traces, /v1/metrics) is appended in code — don't include it.

Confirm it works before you rely on it:

AIIMS_ASYLUM_URL=off .venv/bin/python -m patients.chakravyuh

Then filter SigNoz on service.namespace = aiims. If nothing lands within a minute, re-check the region first.

3. Push the dashboards and alerts (optional)

This needs a different credential from the ingestion key — an API key from Settings → API Keys, with role Editor or Admin (a lesser role gets 403), and your console host rather than the ingest host:

SIGNOZ_API_URL=https://your-org.us2.signoz.cloud
SIGNOZ_API_KEY=your-api-key

.venv/bin/python signoz/import_to_signoz.py --dry-run   # preview
.venv/bin/python signoz/import_to_signoz.py             # push

Idempotent by title — re-running updates in place rather than duplicating.

SigNoz refuses any alert rule that has no notification channel. If you have none configured, create one (Settings → Notification Channels) and set AIIMS_ALERT_CHANNEL to its name before regenerating.

You can also skip the API entirely and upload the six JSON files from signoz/dashboards/ and signoz/alerts/ through Dashboards → Import.

---

Running it

The whole demo

./demo.sh

Starts the ward, admits five patients in sequence — each engineered to exhibit one disorder — then readmits the first one under supervision so you can see the treated and untreated runs side by side. Leaves the ward running.

AIIMS_SPEED=4 ./demo.sh compresses the timing. The catatonia patient's stall is deliberately not compressed — a stall that fast-forwards isn't a stall.

The ward on its own

.venv/bin/python -m asylum.main          # http://127.0.0.1:8642

One patient at a time

.venv/bin/python -m patients.chakravyuh           # untreated
.venv/bin/python -m patients.chakravyuh --treat   # supervised

Module	Patient	Presents with
patients.chakravyuh	Abhimanyu	CH-1 — same tool, same arguments, forever
patients.punarjanam	Rebirth	PJ-2 — retry storm on a dead dependency
patients.kharcha	Munnabhai	KU-3 — token spend spiralling
patients.ghajini	Sanjay	GH-4 — context full, original task forgotten
patients.tareekh	Damini	TT-5 — stalled holding a lock, then dies

Useful flags: --treat (supervise), --speed N, --json, --max-steps N. AIIMS_ASYLUM_URL=off runs a patient standalone with no ward listening.

The clinical trial

.venv/bin/python -m asylum.trial

Runs every patient twice, treated and untreated, changing nothing else.

---

The DSM-A

Diagnostic and Statistical Manual of Agentic Disorders. Six disorders, each with a detector that runs on nothing but span data.

Code	Disorder	What is actually wrong	Detector signal
CH-1	Chakravyuh Syndrome Tool-Call Perseveration	Stuck in a loop it can't exit — like Abhimanyu in the formation	≥4 identical (tool, args_hash) signatures
PJ-2	Punarjanam Syndrome Retry Anxiety Disorder	Retry storm amplifying an outage; every death immediately reborn	≥3 error spans on one tool with shrinking gaps
KU-3	Kharcha Unmaad Token Mania	Cost spiral. Elevated mood, elevated invoice	Per-call token spend growing ≥1.8× across the run
GH-4	Ghajini Syndrome Contextual Amnesia	Context overflowed; the original task is gone	Context >55% full and current task drifted from the goal
TT-5	Tareekh pe Tareekh Agentic Catatonia	Stalled, holding resources, producing nothing	>6s of silence inside an open run
MK-0	Moksha Run Termination	Dead. Liberated from the cycle of retries, permanently	Root span ERROR, or abandoned

Each diagnosis prescribes two things, because that is how fixes actually work: jugaad, the mitigation you ship at 2am, and ilaaj, the root-cause fix you ship on Monday.

It doesn't just diagnose. It treats.

A diagnosis that doesn't change an outcome is decoration. The same detectors run in-process, during the run, and when a disorder crosses its intervention threshold the ward changes what the agent may do next:

Disorder	Treatment applied at runtime
CH-1	Loop broken — the repeated call is refused and the memoized result served instead
PJ-2	Exponential backoff, then the circuit breaker trips and the tool leaves the toolset
KU-3	Per-run token budget enforced; the run is stopped rather than allowed to spiral
GH-4	Context checkpoint — history summarised, original goal re-injected
TT-5	Watchdog timeout on every call, so a stall can't hold a lock forever

Every intervention is emitted as an aiims.intervention span, so treatment is observable in SigNoz next to the pathology that triggered it. You can prove the loop was broken instead of claiming it.

The measured result

  patient      dx     arm         steps  calls     tokens  sanity  outcome
  abhimanyu    CH-1   untreated      20     20    101,954    43.3  survived
                      treated        11      8     22,208    89.5  survived
  rebirth      PJ-2   untreated      16     16     14,690    66.0  survived
                      treated         7      6      5,870    83.0  survived
  munnabhai    KU-3   untreated      12     12    428,118    72.0  survived
                      treated        11     10    180,501    72.0  stopped by the ward
  sanjay       GH-4   untreated       8      8    950,400    53.3  survived
                      treated         8      8    369,340   100.0  survived
  damini       TT-5   untreated       2      2      2,060     0.0  DIED — lock never released
                      treated         4      4      4,580   100.0  survived

  TOTAL   tokens 1,497,222 → 582,499 (61% saved)
          deaths 1 → 0

Damini spends more tokens under treatment. That's the correct result: it finishes the job instead of dying halfway. A run that used to die and now completes is worth paying for.

A run stopped deliberately by treatment is not recorded as a death — detect_moksha skips runs carrying aiims.treatment.aborted. The ward must not be punished for pulling the plug on a run that was burning money.

---

How it works

flowchart LR
  P["patients/*<br/>instrumented agents"] -->|OTLP traces| S[("SigNoz<br/>system of record")]
  P -->|live JSON mirror| W["asylum<br/>ward service"]
  P -->|in-process spans| SUP["Supervisor<br/>same DSM-A detectors"]
  SUP -->|blocks, backs off,<br/>prunes, aborts| P
  SUP -->|aiims.intervention spans| S
  S -->|Query API| W
  W -->|DSM-A detectors| D["diagnosis<br/>+ sanity score"]
  D -->|OTel metrics| S
  D --> U["ward board UI"]
  S --> A["dashboards<br/>+ Code Blue alerts"]

Loading

The arrow back from the Supervisor into the agent is what makes this a control loop rather than a dashboard: traces → diagnosis → intervention → new traces showing whether the intervention worked.

Spans go to SigNoz and to the ward simultaneously. The ward is the triage desk (live, in-memory, fast); SigNoz is the archive (durable, queryable, where you go to see the body). POST /admit/from-signoz proves the point — it pulls historical spans back out of SigNoz and diagnoses runs that finished before the ward was started.

Layout

Path	What's in it
obzerve/detectors.py	The DSM-A panel — pure functions over span dicts
obzerve/supervisor.py	Runtime intervention: loop breaker, circuit breaker, budget, watchdog
obzerve/telemetry.py	OTel wiring, GenAI attributes, .env loading
patients/	Five agents, each engineered to fail one specific way
asylum/	Ward service, diagnosis, clinical notes, death certificates, trial
ward-ui/index.html	The ward board (no build step)
signoz/	Dashboard + alert generator, importer, generated JSON
casting.yaml · casting.yaml.lock · pours/	Foundry deployment — SigNoz + MCP server, reproducible
video/	The Remotion demo video project

SigNoz assets

Three dashboards and three alert rules, all generated rather than hand-written — signoz/build_dashboards.py encodes the v5 query-builder schema so the panels don't have to.

Asset	What it shows
dashboards/ward-overview.json	Sanity per agent with Code Blue / ICU thresholds, diagnoses by disorder, severity of active episodes
dashboards/patient-chart.json	The raw evidence: tool calls grouped by (tool, args_hash) — that bar chart is Chakravyuh — plus failing tools, token curves, run durations
dashboards/treatment.json	Interventions over time and by disorder, sanity at each intervention, token spend per run
alerts/code-blue.json	Sanity below 35
alerts/chakravyuh-loop.json	More than 8 identical (tool, args_hash) calls, with the fix in the annotation
alerts/token-mania.json	A single run past 400k input tokens

Every panel reads a signal the code actually emits — aiims.patient.sanity_score, aiims.episodes, aiims.episode.severity, and the aiims.intervention spans. Nothing here is aspirational.

API

Endpoint	Purpose
GET /ward	Ward board: every patient, sanity, diagnoses, prescriptions
GET /patients/{run_id}/chart · /chart.txt	Full clinical chart, JSON or printable
GET /morgue · /morgue/{run_id}/certificate.txt	Death certificates
GET /dsm	The manual itself
GET /trial	Run every patient treated and untreated; report whether treatment helped
POST /ingest	Live span mirror from instrumented patients
POST /admit/from-signoz?minutes=30	Pull spans from SigNoz and diagnose
GET /healthz	Liveness

Tests

.venv/bin/python -m pytest tests/ -q     # 40 tests

Each detector is tested twice: that it fires on its pathology, and that it stays quiet on a healthy run. A diagnostic panel that diagnoses everyone is not a diagnostic panel. The treatment layer is tested the same way — that it intervenes when it should, and that a disabled or healthy run is never touched.

Troubleshooting

Symptom	Cause
No traces in SigNoz, no errors either	Wrong region in the ingest host, or the key set without the endpoint
403 from the import script	API key role is below Editor — regenerate as Admin
at least one channel is required	No notification channel exists in SigNoz
Ward board says "service unreachable"	Ward isn't running: python -m asylum.main
A patient runs but nothing reaches the ward	AIIMS_ASYLUM_URL is set to off
Panel imports fine but renders empty	The metric exists but the aggregation is wrong for it — counters here need latest, not increase

AI assistance

Declared in AI-DISCLOSURE.md, as the hackathon rules require. Claude Code was used as a coding assistant — the idea, the DSM-A taxonomy, the treat-don't-just-diagnose thesis, the architecture calls and the review were human-directed throughout.

Separately, and more relevant to judging the product: there is no AI in the decision path. The detectors, the sanity score and every runtime intervention are deterministic functions over span data, and the whole project runs with no API key set.