Tracely

Production failures become regression tests.

Trace-native CI/CD for AI agents. Tracely grades every agent trace as it lands, clusters the failures into issues, freezes the bad runs into hermetic replayable cases — and blocks the pull request that would ship them again.

production trace  →  failure detection  →  regression test  →  CI gate

Website · Docs · Agent skill · Guided tour · 2-min demo · Design dossier

Self-host the whole stack in one click — API, worker, UI, Postgres, ClickHouse, Redis and MinIO:

Because observability stops at the dashboard. You can see that your agent broke — then what?

Every eval tool asks you to hand-author a dataset: sit down, invent questions, write ideal answers, keep them current as the product changes. That dataset is a guess about what might break.

Production already handed you the real thing: a trace of the exact run that failed, with the exact input, the exact tool calls, the exact model responses.

The recorded run is the test. Tracely freezes that trace into a hermetic regression case and replays it on every PR. Everything else — quality scores, failure clusters, suggested fixes, CI verdicts, trends — is derived from the trace. There are no hand-authored datasets.

The product maps onto four steps. Each one is a page in the app.

Traces arrive over plain OTLP. Agent semantics (agent.id, conversation.id, turn, step) are promoted to first-class indexed columns, so runs group into conversation threads instead of a flat span soup. The waterfall shows agent → tool → thinking → generation, with the failing span in red.

Evaluators are columns on the trace table, not a separate tab — each one grades at conversation, run or span level and writes its verdict into the grid. Scores stream in live over SSE as judges finish, so you watch a run get graded in place.

Online evaluators grade every run as it lands (LLM-as-judge at conversation / run / span level, plus structural checks that need no model at all). Failures then cluster — structurally and semantically — so 31 broken runs become one issue with a count, not 31 rows to read.

One click promotes a failing trace into a hermetic case: recorded input, tool and LLM outputs bundled as fixtures, and a fail-to-pass contract attached — the case must fail on the old code and pass on the fix, or the promotion isn't trusted.

The suite replays in CI against recorded fixtures: deterministic, offline, no API keys and no model spend. tracely gate exits non-zero, posts a commit status, and upserts a PR comment.

Daily failure and gate pass-rates, latency percentiles, token spend, and per-agent meta-analysis (Spearman correlations + z-score outliers, LLM-synthesized).

Prerequisites: Docker + Docker Compose. (For local dev also uv and Node 20+ / pnpm.)

git clone https://github.com/Jwuthri/Tracely && cd Tracely
docker compose --profile demo up -d --build --wait
open http://localhost:3001

That brings up ClickHouse, Postgres, Redis and MinIO, runs every migration, seeds the default project and ingest key (tracely_dev_key), then populates traces, clusters, cases and gates — so the app opens with the screenshots above rather than an empty shell.

docker compose down            # stop  (add -v to wipe data)

Host ports default to web :3001 and backend :8000; remap with TRACELY_WEB_PORT / TRACELY_BACKEND_PORT. backend/worker/frontend run off source volume-mounts, so most edits need only docker compose restart <svc> — except the Celery worker, which doesn't hot-reload.

cp .env.example .env
make infra-up      # clickhouse, postgres, redis, minio
make install       # uv sync + pnpm install
make migrate       # ClickHouse DDL + Alembic (Postgres)
make seed          # default project + ingest key → tracely_dev_key

make backend       # FastAPI  :8000  (OpenAPI at /docs)   ┐
make workers       # Celery ingestion/eval worker          ├ three terminals
make frontend      # Next.js  :3001                        ┘

make demo          # populate the WHOLE product: traces + clusters + cases + gates
make test          # backend unit tests (no infra, ~6s)

One click provisions the whole stack on Railway — API, worker, UI, Postgres (pgvector), ClickHouse, Redis and MinIO, wired together with volumes and private networking. Migrations and seeding run on the first deploy; set SESSION_SECRET and SECRETS_ENCRYPTION_KEY (openssl rand -hex 32 each) when prompted, then open the frontend's domain and create your workspace.

Prefer to wire it yourself, or deploying somewhere else? The manual walkthrough is deploy/railway/README.md (every variable pre-written in .env.railway.example), and the production-hardening runbook — auth guards, backups, worker pool, post-deploy verification — is guides/DEPLOY.md.

pip install "tracely-ai[openai]"     # or [anthropic], [langchain], [all]

Initialize once at startup, then wrap a run — your normal provider calls are captured automatically, with no span code:

import tracely_sdk as tracely

tracely.init(
    endpoint="http://localhost:8000",   # your Tracely API
    api_key="tracely_dev_key",          # an ingest key
    service_name="support-agent",
    env="prod",                         # prod | staging | ci | dev — the gating axis
    instrument="auto",                  # auto-detect openai / anthropic / google / mistral / langchain
)

with tracely.trace(agent="support-agent", conversation="conv-1", user="u_42"):
    client.chat.completions.create(
        model="gpt-4o",
        messages=[{"role": "user", "content": "Where is order ORD-4471?"}],
    )

That produces a GENERATION span with model, messages, tokens, latency, tool calls and cost. Need spans for your own logic? @observe and the manual agent / tool / llm / retriever / guardrail context managers are all there.

Any OTLP/HTTP exporter works too — point it at POST {endpoint}/v1/traces with Authorization: Bearer tracely_dev_key. Tracely reads standard gen_ai.* / OpenInference attributes plus first-class hints: tracely.agent.id (auto-registered), tracely.agent.version, tracely.conversation.id / turn.* / step.*, tracely.observation.type, and tracely.env (prod|staging|ci|dev — the gating axis).

Two optional lines make a conversation self-describing. The agent catalog tells Tracely which agents, tools, prompts and models the conversation has (not just which ones fired) — it fills the Conversation Agents panel and is readable from judge prompts as @LIST_AGENT. State deltas record what each step wrote to your shared state, folded into the Conversation State drawer and the per-message State Δ column:

AGENTS = [{
    "name": "support",
    "description": "front-line agent; routes billing questions",
    "system_prompt": "You are the support agent for Acme…",   # free-form keys kept verbatim
    "model": "gpt-5.2",
    "tools": {"lookup_order": {"name": "lookup_order", "description": "order by id",
                               "parameters": {"type": "object", "properties": {"order_id": {"type": "string"}}}}},
}]

with tracely.trace(agent="support", conversation="conv-1", agents=AGENTS):
    ...
    tracely.set_state({"cart": cart, "last_action": "add_to_cart"})   # inside any span/@observe

LangGraph users get state for free (node outputs are captured as deltas automatically), and non-Python services can push the catalog with POST /api/sessions/{conversation_id}/config.

Full instrumentation guide → doc.tracely-studio.xyz · sdk/README.md

A promoted production failure becomes a regression test that blocks the PR that reintroduces it. All you need is your ingest key (it identifies your workspace) and your Tracely API URL. Three ways to wire it, depending on how your CI can reach your agent.

      - uses: Jwuthri/Tracely/.github/actions/tracely-gate@master
        with:
          api:   https://tracely.your-co.dev
          key:   ${{ secrets.TRACELY_KEY }}
          # agent: planner,support-agent   ← a subset; omit it to gate EVERY agent with scenarios

# .github/workflows/tracely.yml
name: Tracely gate
on: pull_request
permissions:
  contents: read
  statuses: write          # post the blocking commit status
  pull-requests: write     # upsert the results comment
jobs:
  gate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      # → your existing step(s) that run the agent and emit env=ci traces go here ←
      - uses: Jwuthri/Tracely/.github/actions/tracely-gate@master
        with:
          mode:  gate                          # grade the ci traces this workflow emitted
          agent: planner                       # which agent's promoted suite to run
          api:   https://tracely.your-co.dev   # your Tracely backend (TRACELY_API)
          key:   ${{ secrets.TRACELY_KEY }}    # your ingest key = your workspace

      - run: pip install tracely-ai
      - run: tracely replay planner --entrypoint my_pkg.agent:run    # a Python agent
        # …or any language:  tracely replay planner --cmd "node run.js"  (reads $TRACELY_INPUT)
        env:
          TRACELY_API: https://tracely.your-co.dev
          TRACELY_KEY: ${{ secrets.TRACELY_KEY }}

Both commands auto-detect the PR/commit from the Actions context; web-url / TRACELY_WEB_URL is optional and only builds the "view gate run" link in the PR comment.

ℹ️ The repo's own .github/workflows/tracely-gate.yml is Tracely dogfooding itself — it replays the bundled weather_agent example, which is why it uses an in-repo pip install ./sdk. Your integration is one of the three options above, not that file.

Every backend serves an MCP endpoint at /mcp, so a coding agent reads your traces and writes your evaluators without any glue code:

claude mcp add --transport http tracely http://localhost:8000/mcp \
  --header "Authorization: Bearer tracely_dev_key"

Then: "look at the last 20 traces, find what's failing, and add an evaluation column that catches it." Eleven tools over traces, failure clusters, evaluators and trends — scoped to the key's workspace, same as every other call. On hosted Tracely the endpoint is https://api.tracely-studio.xyz/mcp. Docs

MCP gives your agent your data. The Tracely skill gives it the know-how — how to instrument, what to evaluate, and how to wire the gate — so "add Tracely to this agent" is one sentence instead of a docs tab.

npx skills add https://github.com/Jwuthri/Tracely --skill tracely

Works with Claude Code, Cursor, Copilot, Antigravity and anything else the skills CLI supports — add -g for a global install, --agent '*' for every agent on the machine.

Prefer to read it yourself? It's plain Markdown: skills/tracely/.

The write path deliberately mirrors Langfuse's proven design — reimplemented in Python, with agent semantics promoted to first-class indexed columns (Langfuse keeps them as read-time strings):

SDK/OTLP → POST /v1/traces → S3 blob (durable FIRST) → Redis/Celery
  → worker: otel mapping → registry upsert → ClickHouse events
  → evaluate_run_task → scores + structural clustering

One deliberate adaptation: ClickHouse server-side async_insert instead of an in-process write buffer (Celery tasks don't share memory). Why

The core trace → detect → cluster → regression → gate loop is end-to-end:

• Ingest — any OTLP/HTTP source, first-class agent semantics, blob-first durability.
• Evaluate — DB-backed evaluators as table columns: CRUD from the UI, run on every ingest. Multi-output LLM-as-judge (score / number / boolean / text / JSON with custom schema) at conversation / run / span granularity, in basic (context auto-injected) or advanced (@VARIABLE template prompts with live preview + autocomplete) mode. Batch and sequential execution, per-evaluator targeting (agent/env) + deterministic sampling to scope judge spend; advisory evaluators record a verdict without flipping the roll-up.
• Triage — structural + semantic failure clustering, creatable suggested-evaluator drafts, promote-to-case.
• Regression — hermetic fixture bundles, fail-to-pass contracts, CI replay.
• Gate — PR blocking via tracely simulate / replay / gate, GitHub status + comment.
• Insights — daily traces/failures/gate pass-rate Trends + per-agent cross-metric meta-analysis.
• Conversation intelligence — real-time rolling summary (per-turn memory backing the judge's @HISTORY) + a conversation-agents panel.
• Judge calibration — label judge verdicts against human review, get per-evaluator agreement, and catch an over-flagging judge before you let it gate a release.
• MCP — the API doubles as an MCP server (/mcp): a coding agent reads traces, inspects failure clusters and creates evaluators itself, authenticated by an ordinary ingest key.
• Auth — three modes: dev (open), local (email/password + invites, self-host), clerk (hosted). Team management, API keys, invitations, account settings.

Near-term plan: design/part2-tracely/11-prd-next-steps.md · Long-term roadmap: 10-mvp-and-roadmap.md

With no LLM key at all the pipeline still runs — judges, failure intelligence and meta-analysis degrade rather than crash.

Issues and PRs welcome. Before pushing:

uv run pytest -q backend/tests sdk/tests    # what CI runs
uv run ruff check . && uv run ruff format .
cd frontend && pnpm test && pnpm build      # vitest + tsc typecheck + lint

MIT © Julien Wuthrich