Project Updates

Project Updates

What shipped, newest first. Reconstructed from the commit history for June–August 2026; from here on this page is updated with each notable change. SDK releases: PyPI.

August 2026 (second half)

• In-app assistant — a chat widget that operates the product as you: streams tool calls and the answer, reads traces and clusters, creates evaluators / scenarios / cases, backfills evaluations; runs on Tracely's model key with a per-conversation budget while its tools carry your credentials; every turn is recorded as an internal trace.
• Alerts flow builder (in progress) — rule flows on a canvas (@xyflow/react): condition, Slack, email, webhook, LLM prompt and Python-expression steps over the failure's own variables.
• Security pass — closed invite takeover, SSRF through customer-supplied URLs and root ingest keys; rate-limited auth endpoints; bounded Postgres pools (fixes too many clients already).
• Conversation replay + Fleet view — a timeline that replays a conversation step by step, and a pixel-office view of every agent at work; delegate and skill spans as first-class types.
• SDK 0.3.3 → 0.4.1 — hermetic replay for Gemini, Mistral and LiteLLM; delegate/skill spans; tenant support (trace(tenant=…) — one codebase, one agent per customer); releases now by tag via PyPI trusted publishing.
• Docs — CI gate, evaluators, manual instrumentation, troubleshooting and Trends pages; a first-run checklist from signup to first gate; light/dark theme toggle.
• Ops — the deployment watches itself (beat-driven self-check, /health/queue); judge-conversation checkpoints pruned nightly with a grace window; PruneAgentsPanel to clean unused agents.

August 2026 (first half)

• Hosted cloud — organizations above workspaces, 20k traces/month free quota pooled per account, Stripe billing, bring-your-own-LLM-key hard gate, demo data seeded into every new workspace, workspace and organization deletion.
• MCP server at /mcp — eleven tools over traces, clusters, evaluators and trends, scoped by an ordinary ingest key. Works with Claude Code, Cursor, etc.
• Bulk export — GET /api/export + tracely.export_conversations, NDJSON, filterable by time and span metadata (SDK 0.3.1 / 0.3.2).
• Evaluation — sequential grading with chain progress, include_answer, durable judge conversations on LangGraph's Postgres checkpointer, intent classification.
• Cost — every span priced from OpenRouter's live model catalog; sortable Datetime / Duration / Usage columns; every unbounded list paginated.
• OTLP robustness — read the conventions every framework actually emits (OpenInference invocation parameters, RAG retrievals, thinking blocks); fixed a cross-trace tool-result leak; near-identical failures no longer split into duplicate clusters.
• Scenarios — session management for agent endpoints (client-supplied or server-minted), import a recorded conversation as a scenario, W3C traceparent propagation, hardening for launch.
• Marketing + SEO — pricing section, LLM-as-a-judge / LLM evaluation / Langfuse-alternatives pages; README rewritten with screenshots; one-click Railway deploy.

July 2026

• Scenarios — scenario management and a conversation editor; the gate can drive an agent's own HTTP endpoint.
• Shared state — set_state() deltas folded into a Conversation State drawer and a per-message State Δ column; LangGraph node outputs captured automatically.
• Per-workspace OpenRouter key — customers bring their own model key, encrypted at rest.
• Clustering precision — near-duplicate failures merged, one-off noise hidden (PR #32); every surface themed with design tokens.
• Evaluator authoring — scoring + reasoning fields in templates, AddColumnModal and OutputSchemaBuilder redesigned; project data wipe; agent configuration + system prompt panel.
• SDK — agent attribution fixed; the SDK can embed in a host app's existing OTel setup.

June 2026 — v0.1, the spine

• 3 June, first commit — ingest, failure clustering, CI/CD gate, replay, analytics, all in the first two days.
• Evaluators rebuilt — sequential evaluation, JSON-schema judges, CI-gate verdict logic with delta warnings, regression tests for silent-failure scenarios.
• Honest gate + trustworthy evals (PR #2) — a gate that tested nothing can't be green; judge in the gate.
• Automatic tracing — init(instrument="auto") with provider + framework examples; the Nextra docs site.
• Meta-analysis (per-agent Spearman / z-score + LLM synthesis), judge calibration with score annotations, conversation history for multi-turn agents, monitors + cost dashboard (PR #24), account settings, the hermetic replay bridge for @observe(as_type="tool").

---

Want these in your inbox or feed? Watch the repo (Releases only) or follow the GitHub Project board linked from the repo.