shipit-watcher 1.5.0 — gateway attribution, and a context leak closed

Attribution that survives a proxy — and a fix for a leak that should never have been possible in this library.

Gateway attribution

A LiteLLM proxy sits between your code and the provider, and that is where a lot of teams do cost allocation and prompt governance. existing_trace_id told the gateway which trace to join but nothing about whose call it was, so the gateway was allocating spend it could not attribute.

Attribution and prompt identity now travel with the call:

with wt.bind(company_id="acme", cost_center="support-ops"):
    litellm.completion(model="gpt-4o", messages=[...])   # via your proxy

The gateway receives the cost centre, client, user, session, channel, service and environment — plus prompt_name, prompt_version, prompt_fingerprint and prompt_registered, forwarded verbatim so a gateway in enforce mode can decide on them without a translation table.

Everything comes from the ambient context, never from a lookup afterwards. A cost centre resolved after the fact may since have changed, and an attribution that is only usually right is not one you can bill from.

On by default; wire names are yours via gateway_key_map. generation_owner decides who writes the generation record when a proxy is in the path — leave it app for a proxy you own, set it to gateway when the proxy already writes them, or every call is recorded twice and the cost doubles on paper.

Fixed: one caller's output reaching the next

ContextVar was constructed with default=TraceContext() — one object shared by every context that never set its own. TraceContext is frozen, which protects the fields but not the mutable result dict that set_output writes into:

with bind(user_id="alice") as ctx: ctx.set_output("alice's answer")
with bind(user_id="bob") as ctx:   ctx.result
# {'output': "alice's answer"}

Output recorded while no trace was active stayed in that dict, and the next unbound caller read it back. In a library whose entire purpose is per-tenant attribution and PII masking, that is the worst place in the codebase for a leak.

current_context() now returns a fresh context when nothing is bound. Reproduced before the fix, and three regression tests were checked against the old code to confirm they actually fail on it.

If you attribute per tenant, upgrade. The window is narrow — it needed output written outside a trace — but the failure mode is silent and it is the kind that only shows up in someone else's dashboard.

Housekeeping

• CI on every pull request and push: ruff plus the full suite on 3.11, 3.12 and 3.13, and the package built and twine checked on every change rather than first at the tag.
• Release by tag with Trusted Publishing, gated on green tests, and a check that the tag matches the version in pyproject.toml.
• 350 → 0 lint errors. Most mechanical (List → list, Optional[X] → X | None, sorted imports), the rest by hand — zip(strict=), StrEnum, combined with statements. Tests re-run after every batch.
• A fixture gap fixed in test_datasets: run_experiment links each result to its trace, and a tracer with no sink never opens one, so there was no trace id to link and the test read as a linking bug.
• Workflow comments translated to English.

258 tests, green on 3.11, 3.12 and 3.13.

shipit-watcher 1.4.1

Upgrade if you call flush() while a trace is open — you were losing those traces entirely.

flush() no longer orphans an in-flight trace

Calling flush() mid-trace sent the buffered child spans and cleared the root, so they arrived referencing a parent that had not been sent. Langfuse accepts orphaned children with a 200 and then discards the whole trace rather than render a tree with a missing root — silently, with nothing logged.

The symptom is distinctive and easy to misread: the local ledger records every call perfectly while Langfuse shows nothing at all.

flush() now leaves a trace alone while its root is open, and sends only traces that have already ended — including late events arriving from LiteLLM's callback thread after end_trace, which do have a parent on the server to attach to.

shipit-watcher 1.4.0

Per-agent datasets

wt.agent_dataset_name(agent) → myapp-eval-inbox-manager. One dataset per agent, for the same reason prompts are namespaced per agent: a failing case for one says nothing about another, and a shared dataset makes every run an average of unrelated cases.

Cost is no longer silently zero

Streaming calls through a gateway routinely arrive with every cost field at zero — the response is reassembled from chunks and the proxy's figure never makes it back. Cost is now computed from tokens instead, with two guards:

• The routing prefix is stripped. Pricing knows gemini-2.5-flash; openai/gemini-2.5-flash is a routing instruction that matches no pricing entry.
• Unpriced models report nothing, not a guess. litellm's helpers price models they do not recognise at a generic rate; putting that number on a private deployment is worse than reporting none, because an invented figure looks authoritative.

Going to production

A new README section covering the five things to set up in order — environment, configure-once-at-startup, the ledger migration, seeding the prompt registry without overwriting edits made in Langfuse, and letting datasets fill themselves — plus a copy-pasteable verification block.

290 tests.

shipit-watcher 1.3.0 — datasets

Turn real traffic into a regression suite.

pip install -U shipit-watcher

Capture as it happens

with wt.trace("agent.turn", user_id=user.email) as ctx:
    ...
    if user_reported_it_wrong:
        wt.capture(input=question, metadata={"reported_by": user.email})

Called inside a trace, the origin fills itself in — the row links back to the trace that produced it. The dataset name defaults to WATCHER_DATASET, and the dataset is created on first use.

Replay and compare

wt.run_experiment("regressions", task=..., run_name="prompt-v7", evaluators=[...])

Each item gets its own trace linked under run_name, with evaluator scores attached. The difference between "the new prompt feels better" and "0.82 against 0.71 on the same 40 cases".

An item whose task raises is recorded and the run continues — aborting would discard results already gathered, and a task that fails on one input is itself a finding.

New

• wt.capture, wt.add_item, wt.get_items, wt.create_dataset, wt.run_experiment
• WATCHER_DATASET config

288 tests.

shipit-watcher 1.2.1

stale now means stale.

A prompt nobody has registered is re-served from cache on every request once its TTL expires. That was reported as serving stale prompt, which reads as a Langfuse outage and sends people hunting for a problem that is not happening — it is the ordinary steady state for an unregistered prompt, and the fallback ladder working as designed.

ManagedPrompt.stale now tracks whether the cached value actually came from the registry, and the warning only fires when a real registry value could not be refreshed.

263 tests.

shipit-watcher 1.2.0

Fixes three silent failures found by running a real agent turn end to end rather than trusting a green test suite. If you are on 1.0.x or 1.1.0 with LiteLLM, your LLM calls were not being recorded at all — upgrade.

pip install -U shipit-watcher

The handler was never called

LiteLLM dispatches litellm.callbacks with isinstance(callback, CustomLogger). Ours was duck-typed, so it was accepted into the list and never invoked — no error, no warning, just a trace with tool spans and no LLM call in it. Now a CustomLogger subclass, built lazily so the package still imports without litellm.

Streaming generations detached from their turn

LiteLLM runs the success handler for a streaming call on a plain threading.Thread, which does not inherit contextvars. Entry points are now wrapped to read the context in the caller's thread and carry it with the request.

Cost was zero behind a proxy

The model is an alias the local pricing map has never heard of, so completion_cost computes 0. _hidden_params and standard_logging_object carry the gateway's own figure.

Agent prompt keys changed: agent/<slug> → agent:<slug>

The Langfuse client does not URL-encode the prompt name when fetching, so agent/support returns 404 — while the write succeeds, because the name travels in the request body. A slash gave you a prompt you could publish and never read back. If you published prompts under agent/…, republish them under agent:….

Quieter logs

A prompt simply absent from the registry no longer logs a traceback with the server's entire HTML 404 page attached. That is an expected rung of the fallback ladder, not an error.

261 tests.

shipit-watcher 1.1.0

Reusable in any Python project — the package no longer carries references to the codebase it grew in.

pip install shipit-watcher

Public-ready

• Removed 63 client-specific references across 17 files: product and customer names, a private Langfuse hostname, internal audit finding ids, and a client's accounting vocabulary that had been used as if it were a general concept.
• Package metadata now carries an OSI licence classifier and project URLs, so the project is findable and its licence machine-readable.
• Capability table states what is not built (budgets/alerts, Langfuse datasets) alongside what is.

Since 1.0.0 on the library itself

Nothing changed in behaviour — 1.1.0 is 1.0.0 with honest, reusable documentation.

What 1.0.0 introduced

• Agent graphs. Langfuse renders a graph only for observations carrying a semantic type (agent / tool / retriever / guardrail). Those cannot be sent over the classic ingestion API, and the Langfuse SDK exposes them only from 3.3.1 — which drops client.trace(). LangfuseOTLPSink speaks OTLP directly, so only the server has to be v3. Enable with WATCHER_LANGFUSE_TRANSPORT=otlp.
• Prompt registry: create_prompt() publishes versions, get_agent_prompt() namespaces prompts per agent, use_prompt() binds one ambiently for calls that cannot pass metadata, run_prompt() does fetch → compile → call → trace in one step.
• LLM gateway: traced completions and streaming, governance enforced pre-call, provider-agnostic retries, model-alias routing through a proxy.
• LiteLLM calls are named for the operation (rag.embedding, never litellm-aembedding), and background calls get their own trace instead of being dropped.
• Django ledger accepts any user identifier — the FK resolves only for a real primary key, user_ref always holds the raw value.

246 tests.