Troubleshooting

Troubleshooting

The complete symptom → cause → fix tables live in the agent skill: skills/tracely/references/troubleshooting.md. This page is the short list — most reports are one of these.

Nothing arrives

Symptom	Fix
No trace after a script run	tracely.flush() before exit
No trace, no error	endpoint must be the API (:8000 / https://api.tracely-studio.xyz), not the UI (:3001)
401	The ingest key is the workspace — Settings → API keys
Spans arrive, nothing renders	Hand-written OTLP with hex span ids; OTLP span ids are base64
Traces appear seconds late	Normal: evaluation debounces ~4s for late spans

The tree is wrong

Symptom	Fix
Turns don't group into a conversation	Same conversation= on every turn (raw OTLP: tracely.conversation.id)
Everything under one default agent	Set init(service_name=…) / agent=; framework gen_ai.agent.name is ignored by design
All customers share one agent	trace(tenant=customer_id) — each tenant becomes its own agent
Every LLM call appears twice	Two instrumentors on one call (LangChain + provider). Under "auto" LangChain wins; otherwise name one
Spans from a worker thread float free	tracely.run_in_thread(fn, …) copies the context

Numbers are missing

Symptom	Fix
Streamed OpenAI calls have no tokens/cost	stream_options={"include_usage": True}
Cost blank	Cost is derived from model id + tokens; gateways rewrite model ids
State drawer empty	set_state() needs an active span — pass span= inside framework callbacks

Evaluators don't grade

Symptom	Fix
New column is blank	New evaluators grade traces ingested from now on — backfill from the UI
Judge columns all blank	No workspace OpenRouter key. Structural checks still run
One rubric turns the workspace red	Set advisory: true
Trace count doubled after enabling evaluators	Internal eval runs leaking into counts — that's a bug, please report it

The gate is lying

Symptom	Fix
Tool expectations all SKIP	Forward traceparent into tracely.trace(traceparent=…)
Adversarial suite always passes	Polarity: goal achieved = FAIL. No LLM key → skipped, not passed
An agent silently isn't gated	It has no enabled scenario. Enabled scenarios + no endpoint → NO_COVERAGE (blocks)
Exit code 2	Timeout / unreachable API / server error — not a pass
Replay makes real API calls	--live, or the provider isn't covered by the fixture patch; use call_tool / call_llm

Self-hosted

Symptom	Fix
Worker code changes have no effect	docker compose restart worker — it doesn't hot-reload
Forms refuse to save an OpenRouter key	SECRETS_ENCRYPTION_KEY unset (same value on backend + worker)
Prod refuses to boot	AUTH_MODE=dev or tracely_dev_key in the DB — hard guards, fix rather than bypass
Login loops	NEXT_PUBLIC_AUTH_MODE ≠ backend AUTH_MODE
too many clients already	Bound the Postgres pools; fixed on master 2026-08-21

When nothing fits

1. Reduce to init() → one provider call → flush().
2. Confirm the raw POST {endpoint}/v1/traces returns 2xx — then the problem is downstream.
3. Open one span in the UI's span panel: its attributes show which convention was recognised.
4. Ask the MCP server: list_traces(limit=5) then get_trace(id).

Still stuck → open a bug.