shipit-watcher 1.4.1

Upgrade if you call flush() while a trace is open — you were losing those traces entirely.

flush() no longer orphans an in-flight trace

Calling flush() mid-trace sent the buffered child spans and cleared the root, so they arrived referencing a parent that had not been sent. Langfuse accepts orphaned children with a 200 and then discards the whole trace rather than render a tree with a missing root — silently, with nothing logged.

The symptom is distinctive and easy to misread: the local ledger records every call perfectly while Langfuse shows nothing at all.

flush() now leaves a trace alone while its root is open, and sends only traces that have already ended — including late events arriving from LiteLLM's callback thread after end_trace, which do have a parent on the server to attach to.

shipit-watcher 1.4.0

Per-agent datasets

wt.agent_dataset_name(agent) → myapp-eval-inbox-manager. One dataset per agent, for the same reason prompts are namespaced per agent: a failing case for one says nothing about another, and a shared dataset makes every run an average of unrelated cases.

Cost is no longer silently zero

Streaming calls through a gateway routinely arrive with every cost field at zero — the response is reassembled from chunks and the proxy's figure never makes it back. Cost is now computed from tokens instead, with two guards:

• The routing prefix is stripped. Pricing knows gemini-2.5-flash; openai/gemini-2.5-flash is a routing instruction that matches no pricing entry.
• Unpriced models report nothing, not a guess. litellm's helpers price models they do not recognise at a generic rate; putting that number on a private deployment is worse than reporting none, because an invented figure looks authoritative.

Going to production

A new README section covering the five things to set up in order — environment, configure-once-at-startup, the ledger migration, seeding the prompt registry without overwriting edits made in Langfuse, and letting datasets fill themselves — plus a copy-pasteable verification block.

290 tests.

shipit-watcher 1.3.0 — datasets

Turn real traffic into a regression suite.

pip install -U shipit-watcher

Capture as it happens

with wt.trace("agent.turn", user_id=user.email) as ctx:
    ...
    if user_reported_it_wrong:
        wt.capture(input=question, metadata={"reported_by": user.email})

Called inside a trace, the origin fills itself in — the row links back to the trace that produced it. The dataset name defaults to WATCHER_DATASET, and the dataset is created on first use.

Replay and compare

wt.run_experiment("regressions", task=..., run_name="prompt-v7", evaluators=[...])

Each item gets its own trace linked under run_name, with evaluator scores attached. The difference between "the new prompt feels better" and "0.82 against 0.71 on the same 40 cases".

An item whose task raises is recorded and the run continues — aborting would discard results already gathered, and a task that fails on one input is itself a finding.

New

• wt.capture, wt.add_item, wt.get_items, wt.create_dataset, wt.run_experiment
• WATCHER_DATASET config

288 tests.

shipit-watcher 1.2.1

stale now means stale.

A prompt nobody has registered is re-served from cache on every request once its TTL expires. That was reported as serving stale prompt, which reads as a Langfuse outage and sends people hunting for a problem that is not happening — it is the ordinary steady state for an unregistered prompt, and the fallback ladder working as designed.

ManagedPrompt.stale now tracks whether the cached value actually came from the registry, and the warning only fires when a real registry value could not be refreshed.

263 tests.

shipit-watcher 1.2.0

Fixes three silent failures found by running a real agent turn end to end rather than trusting a green test suite. If you are on 1.0.x or 1.1.0 with LiteLLM, your LLM calls were not being recorded at all — upgrade.

pip install -U shipit-watcher

The handler was never called

LiteLLM dispatches litellm.callbacks with isinstance(callback, CustomLogger). Ours was duck-typed, so it was accepted into the list and never invoked — no error, no warning, just a trace with tool spans and no LLM call in it. Now a CustomLogger subclass, built lazily so the package still imports without litellm.

Streaming generations detached from their turn

LiteLLM runs the success handler for a streaming call on a plain threading.Thread, which does not inherit contextvars. Entry points are now wrapped to read the context in the caller's thread and carry it with the request.

Cost was zero behind a proxy

The model is an alias the local pricing map has never heard of, so completion_cost computes 0. _hidden_params and standard_logging_object carry the gateway's own figure.

Agent prompt keys changed: agent/<slug> → agent:<slug>

The Langfuse client does not URL-encode the prompt name when fetching, so agent/support returns 404 — while the write succeeds, because the name travels in the request body. A slash gave you a prompt you could publish and never read back. If you published prompts under agent/…, republish them under agent:….

Quieter logs

A prompt simply absent from the registry no longer logs a traceback with the server's entire HTML 404 page attached. That is an expected rung of the fallback ladder, not an error.

261 tests.

shipit-watcher 1.1.0

Reusable in any Python project — the package no longer carries references to the codebase it grew in.

pip install shipit-watcher

Public-ready

• Removed 63 client-specific references across 17 files: product and customer names, a private Langfuse hostname, internal audit finding ids, and a client's accounting vocabulary that had been used as if it were a general concept.
• Package metadata now carries an OSI licence classifier and project URLs, so the project is findable and its licence machine-readable.
• Capability table states what is not built (budgets/alerts, Langfuse datasets) alongside what is.

Since 1.0.0 on the library itself

Nothing changed in behaviour — 1.1.0 is 1.0.0 with honest, reusable documentation.

What 1.0.0 introduced

• Agent graphs. Langfuse renders a graph only for observations carrying a semantic type (agent / tool / retriever / guardrail). Those cannot be sent over the classic ingestion API, and the Langfuse SDK exposes them only from 3.3.1 — which drops client.trace(). LangfuseOTLPSink speaks OTLP directly, so only the server has to be v3. Enable with WATCHER_LANGFUSE_TRANSPORT=otlp.
• Prompt registry: create_prompt() publishes versions, get_agent_prompt() namespaces prompts per agent, use_prompt() binds one ambiently for calls that cannot pass metadata, run_prompt() does fetch → compile → call → trace in one step.
• LLM gateway: traced completions and streaming, governance enforced pre-call, provider-agnostic retries, model-alias routing through a proxy.
• LiteLLM calls are named for the operation (rag.embedding, never litellm-aembedding), and background calls get their own trace instead of being dropped.
• Django ledger accepts any user identifier — the FK resolves only for a real primary key, user_ref always holds the raw value.

246 tests.