tracelabel

Local-first, lightweight labeling — keyboard-fast, no accounts, no server.

One pip install. One command. Your browser opens on a keyboard-driven labeling UI over your own traces. No sign-up, no cloud, no Node, no database to stand up. It's a single Python wheel that bundles a FastAPI server, a prebuilt React app, and SQLite — one .db file per project.

uvx tracelabel demo

Press j to jump to the first labelable turn, 1 to mark it pass, Enter to commit and advance. That's the whole loop.

Install

pip install tracelabel          # from PyPI
uvx tracelabel demo             # run without installing (via uv)
python -m tracelabel serve …    # module entry point

Requires Python ≥ 3.10; runs on macOS, Linux, and Windows. LLM-assisted prefill (tracelabel suggest) needs the optional extra:

pip install "tracelabel[ai]"

Quickstart

pip install tracelabel
tracelabel serve traces.jsonl     # imports the file + opens http://127.0.0.1:8377
tracelabel export                 # → <task>-annotations.jsonl

Your traces are a UTF-8 JSONL file, one trace per line — see Data formats. No config needed: tracelabel defaults to a turn-level pass/fail task, so you can point it at a file and start labeling. The file you serve is the queue — tracelabel serve week-28.jsonl labels only week 28's traces (see One db, many files).

Data formats

Everything you import is normalized to one internal shape — the tracelabel trace format (full spec: docs/trace-format.md). You rarely need to produce it by hand: --from auto (the default) sniffs the first few lines and routes your data through the right adapter, in priority order:

ctf  →  adk  →  datadog  →  documents  →  loose

Force a specific one with --from ctf|adk|datadog|documents|loose. Input can be a .jsonl file (one JSON value per line), a single JSON object, a top-level JSON array, or — for documents — a folder.

Native traces (JSONL)

One trace per line: an object with an optional id and a required messages array. This is the tracelabel trace format itself — what every other adapter converts into. Roles are system | user | assistant | tool (plus document for single-message document traces). Assistant turns may carry tool_calls; tool turns carry a tool_call_id:

{"id": "demo_001", "metadata": {"model": "gpt-4o", "env": "prod"}, "messages": [
  {"role": "system", "content": "You are Aria, a support agent."},
  {"role": "user", "content": "Status of order #48213?"},
  {"role": "assistant", "content": "", "tool_calls": [
    {"id": "call_1", "type": "function",
     "function": {"name": "lookup_order", "arguments": "{\"order_id\": \"48213\"}"}}]},
  {"role": "tool", "tool_call_id": "call_1", "name": "lookup_order",
   "content": "{\"status\": \"shipped\", \"carrier\": \"UPS\"}"},
  {"role": "assistant", "content": "Order #48213 has shipped via UPS."}
]}

content may be a plain string or a parts array — {"type": "text"|"json"|"html", …} — for mixed text/JSON/HTML turns. A handful of validation rules apply (tool_calls only on assistant, tool_call_id only on tool, empty content allowed only with tool_calls); the importer rejects violations with a fixed example. Full rules: docs/trace-format.md.

Loose inputs (almost-native)

Most people arrive with data that's nearly the native format. The loose adapter accepts common shapes and prints a one-line summary of what it remapped (e.g. interpreted "turns" as "messages" on 412 lines):

You have	tracelabel does
A bare OpenAI messages array per line: [{"role": "user", …}, …]	Wraps it as {"messages": […]}
{"conversation": […]} / {"turns": […]} / {"chat": […]}	Renames the key to messages
Messages using speaker / from instead of role	Renames; maps human→user, ai/bot/agent→assistant
LangSmith-style runs with inputs.messages / outputs	Best-effort maps to messages; extras → raw

Documents mode

Label freeform text/Markdown/HTML/JSON (notes, transcripts, policy pages) instead of agent conversations. Documents label at the trace level (there's nothing to break into turns), and Markdown/HTML render with real formatting in the UI. Two ways in:

A JSONL of documents — each line is a bare string, or an object with a required content:

"A plain document is just a string."
{"content": "# Report\n\nFindings go here.", "content_type": "markdown", "id": "report-1"}

A bare string defaults to content_type: "text". --as-documents forces this adapter on JSONL input even if auto-detection would pick something else.

A folder of files — a non-recursive scan; one document per file:

tracelabel serve ./docs     # every .md / .markdown / .txt / .text / .html / .htm file

The id is the filename, the extension sets content_type, and the real path is stored in metadata.path. Other file types (.json, .jsonl, hidden files, unknown extensions) are skipped with a summary note.

ADK sessions

An exported Google ADK session envelope — {"events": […], "appName"?, "userId"?, "id"?} — maps to one trace. Each event's author becomes a user or assistant turn (with the author name as a chip, so multi-agent sessions stay legible), and function_call / function_response parts become tool_calls + tool turns. See Exporting from ADK for how to produce this file.

Datadog LLM-Observability spans

An exported JSON/JSONL of Datadog LLM-Observability spans — each span carrying trace_id, span_id, start_ns, duration, and a meta object with a kind. Spans are grouped by trace_id and ordered by start_ns into one trace each: llm spans' input/output messages become turns, tool spans become tool_calls + tool turns. See Exporting from Datadog for how to produce this file. (File import only — there is no live Datadog API sync.)

Commands

Command	What it does	When to reach for it
serve [file|dir]	Import + create/open a task + build the labeling queue + open the browser UI. The interactive entry point.	Normal labeling. Point it at your data and go.
import <file|dir>	Load data into the db only — no task, no queue, no server.	Bulk ingest, or when you need format knobs serve doesn't expose. Follow with serve --all.
export	Read the db and write annotations to JSONL/CSV. Pure read — no server needed.	Get labels out for analysis.
suggest [file]	Optional LLM prefill of label suggestions (needs [ai] extra).	Warm-start labeling with a model's guesses.
demo	Copy bundled sample traces to a temp dir and serve them.	Try tracelabel with zero setup.
tasks list	Print a progress table across the whole db.	Check how far along each task is.

import vs serve — both ingest through the same importer, but:

• import loads data and exits. It exposes the full ingest surface: --from auto|ctf|adk|datadog|documents, --on-conflict fail|skip, --skip-invalid (skip malformed lines instead of failing), --as-documents. It does not create a task or start a server.
• serve loads data and opens/creates a task, builds the labeling queue, and starts the web UI. It fixes on-conflict=fail and doesn't expose --from/--skip-invalid — so when your data isn't already in the native format, import it first, then serve --all to label everything in the db.

Useful serve flags: --task NAME, --level turn|trace, --all (label the whole db, not just the file you served), --review-of NAME / --labels-from KEY (review an LLM judge's existing labels — see Reviewing an LLM judge's labels), --port (default 8377), --no-browser, --shuffle/--no-shuffle. The server binds 127.0.0.1 only.

Common workflows

1 · Just try it

tracelabel demo

2 · Label your own traces

tracelabel serve traces.jsonl     # label in the browser
tracelabel export                 # → traces-annotations.jsonl (or <task>-annotations.jsonl)

3 · Ingest a messy/odd format first, then label all of it

tracelabel import dump.jsonl --from adk --skip-invalid
tracelabel serve --all            # queue = every trace in the db

4 · Scoped weekly queues over one shared db

tracelabel serve week-28.jsonl --task empathy   # only week 28; resumes where you left off

5 · LLM-assisted prefill, then review

pip install "tracelabel[ai]"
export OPENAI_API_KEY=…           # or your provider's key
tracelabel suggest traces.jsonl   # writes suggestions; you still confirm each label
tracelabel serve traces.jsonl

6 · Check progress and export for analysis

tracelabel tasks list
tracelabel export --joined --status labeled --out labels.jsonl

Exported data

tracelabel export is a pure db read with a stable column contract — the columns are an API. Default format is JSONL (one row per annotation), with the label values nested as an object; CSV flattens them into value.<field> columns.

Base columns (always present):

task  trace_id  target_type  target_id  turn_index  annotator
status  prefill_model  schema_hash  created_at  updated_at

A default JSONL row:

{"task": "empathy", "trace_id": "conv_1", "target_type": "turn", "target_id": "conv_1#4",
 "turn_index": 4, "annotator": "me", "status": "labeled", "prefill_model": null,
 "schema_hash": "a1b2c3…", "created_at": "2026-07-12T15:04:05Z", "updated_at": "2026-07-12T15:04:05Z",
 "values": {"verdict": "pass", "failure_modes": ["formatting"], "notes": "minor wording nit"}}

--joined folds in the source content so you never join back to the original file: turn-level rows gain role, content, content_type, trace_metadata, source; trace-level rows gain the reconstructed messages array (or content/content_type for document traces). Other flags: --task, --format jsonl|csv, --status labeled|skipped|all, --out PATH (- = stdout).

Load it in three lines:

import pandas as pd
df = pd.read_json("empathy-annotations.jsonl", lines=True)
df.groupby("task")["values"].apply(lambda v: (pd.json_normalize(v)["verdict"] == "pass").mean())

See docs/pandas.md for a groupby recipe per field type (single_select, multi_select, text).

Exporting from ADK

The adk adapter wants the session envelope JSON. ADK Session/Event objects are Pydantic models, so you serialize them with model_dump() / model_dump_json(). Illustrative helper (one session per line — adapt to your session service and ids):

# pip install google-adk
import json

session = await session_service.get_session(app_name=APP, user_id=UID, session_id=SID)
with open("adk-sessions.jsonl", "w") as f:
    f.write(json.dumps(session.model_dump(mode="json")) + "\n")

Then:

tracelabel serve adk-sessions.jsonl        # or: tracelabel import … --from adk

The adapter needs, per session: events[].author, and events[].content.parts[] where a part is {"text": …}, {"function_call": {"name", "args", "id"?}}, or {"function_response": {"name", "response", "id"?}}. Top-level appName / userId / id are optional and land in trace metadata. Because each event's author becomes the assistant name, multi-agent sessions render with a per-agent chip.

Exporting from Datadog

The datadog adapter wants an exported JSON/JSONL of LLM-Observability spans (file import only — no live sync). Pull them from Datadog's Export API and write each span as one JSONL line. Illustrative helper:

curl -s \
  -H "DD-API-KEY: $DD_API_KEY" -H "DD-APPLICATION-KEY: $DD_APP_KEY" \
  "https://api.datadoghq.com/api/v2/llm-obs/v1/spans/events?filter[from]=now-1d&filter[to]=now" \
  | jq -c '.data[].attributes' > datadog-spans.jsonl

tracelabel import datadog-spans.jsonl --from datadog

The adapter needs, per span: trace_id, span_id, start_ns, duration, and a meta object with a kind (llm / tool / workflow); LLM spans carry meta.input.messages / meta.output.messages. Spans are grouped by trace_id into one trace each.

Adjust the host for your Datadog site (e.g. api.datadoghq.eu), and the .data[].attributes jq path if your export nests fields differently — the requirement is only that each output line is a span object with the fields above.

Configuring the task

Drop a config.yaml next to your data (or pass --config). Everything not specified falls back to sensible defaults; unknown keys are hard errors with a pointed message.

name: empathy
level: turn                 # label per-turn (default) or per-trace
label_roles: [assistant]    # which roles are labelable
fields:
  - name: verdict
    type: single_select
    options: [pass, fail]
    required: true
  - name: failure_modes
    type: multi_select
    options: [hallucination, refused, wrong_tool, formatting]
  - name: notes
    type: text

Field types map one-to-one to UI controls and to export columns. Add a field, get a new keyboard target and a new column — no redesign. With a config.yaml present you can run tracelabel serve (no file argument) and it uses the data: path from the config.

Reviewing an LLM judge's labels

When a model has already judged your traces — a pass/fail verdict plus reasoning per trace — review mode lets a human sweep those predictions and approve or correct each one, keyboard-fast. It's serve inverted: instead of stepping through unlabeled targets, it steps through the targets the judge already labeled, seeding the form from the judge's verdict so Enter approves it as-is, 1/2 flip the verdict, and r edits the reasoning.

Put each judge label on its trace line under a judge key (a values dict keyed by your schema fields), and give every labeled line an id so the label can be matched to its trace:

{"id": "t1", "messages": [...], "judge": {"verdict": "pass", "reasoning": "answered correctly"}}
{"id": "t2", "messages": [...], "judge": {"verdict": "pass", "reasoning": "looks fine"}}

tracelabel serve traces.jsonl --review-of gpt-4o   # opens on the first judge label to review
tracelabel export --joined                          # judge + your labels, one row each

The judge is stored as its own annotator (here gpt-4o), and your corrections as a second annotator — so the original prediction is preserved, and export emits one row per annotator per trace. Diff them to measure how often the judge was right:

import pandas as pd
df = pd.read_json("traces-2026-07-12-annotations.jsonl", lines=True)
v = df.assign(verdict=df["values"].str["verdict"]).pivot(
    index="trace_id", columns="annotator", values="verdict")
agree = (v["gpt-4o"] == v["me"]).mean()   # judge accuracy vs. your review

Review mode is trace-level (the pass_fail default) and single judge per run. Flags: --review-of NAME (the judge's annotator name; turns review mode on), --labels-from KEY (the source-line key; default judge), and --annotator NAME for your own name (must differ from the judge). You can also set these under a review: block in config.yaml (review: {of: gpt-4o, labels_from: judge}).

One db, many files

tracelabel stores one shared pool of traces per project (.tracelabel/tracelabel.db) — traces are deduped by id/content hash and accumulate across every file you've ever served or imported. But the file you serve is a lens over that pool, not the pool itself: tracelabel serve week-28.jsonl --task empathy scopes the labeling queue and progress bar to exactly the traces in week-28.jsonl, even if the db already contains traces from week-27.jsonl or other tasks. Re-serving an old file resumes exactly where you left off — nothing is re-scrambled or un-completed by importing something new.

• Each file is imported idempotently, so re-serving the same file (or one with overlapping traces) is always safe.
• tracelabel export and tracelabel tasks list are db-wide — they report on the whole pool, across every file and session, not just the last one served.
• tracelabel serve <file> --all opts back into whole-db behavior: it still imports <file> (idempotent, as always), but the queue is every trace in the db, not just that file's.

Privacy & security

Your traces never leave your machine unless you run suggest.

• Loopback only. The server binds 127.0.0.1; there is no --host flag and no auth, because nothing is ever exposed off your loopback interface.
• No telemetry, ever — not opt-in, not opt-out. The only outbound network call this package can make is a model call you explicitly trigger with tracelabel suggest, using your own API key from your own environment.
• API keys from env only. Putting an api_key: in your config is a hard error; keys are never logged and never written to the database.
• Untrusted HTML is sandboxed. HTML traces render in an iframe with an empty sandbox attribute; there is no dangerouslySetInnerHTML anywhere in the app.
• Strict config. Unknown/typo'd config keys are hard errors.
• Tiny dependency surface. Runtime core is fastapi, uvicorn, pydantic, typer, pyyaml; litellm is an optional [ai] extra; shadcn/ui is vendored, not a dependency.

When to use something else

tracelabel is deliberately small. Reach for a full platform when you need what it doesn't do:

• Label Studio / Argilla — hosted multi-annotator platforms with accounts, projects, review workflows, and rich media (images, audio, bounding boxes). tracelabel is single-player, text/JSON/HTML/Markdown only, and runs on your laptop.
• Use tracelabel when you want to label agent traces right now, keyboard-fast, without standing up infrastructure or sending your data anywhere.

Teams

tracelabel is single-player today — one annotator, one db file. But the schema is already multi-annotator ready (every annotation carries an annotator and a schema_hash), so teams aren't a dead end. The planned answer is:

tracelabel merge alice.db bob.db      # (planned) combine independent annotators' db files

Each person labels locally into their own .db; you merge and compute agreement offline. Nothing about the storage format needs to change to get there.

Contributing

See CONTRIBUTING.md for development setup, frontend build behavior, and test commands.

License

Apache-2.0.