Skip to content

0.8.0

Choose a tag to compare

@amiable-dev amiable-dev released this 01 Oct 12:07
d6f069b

Harness-native telemetry, ADR-014. Claude Code's own per-request records become the source of cost
and tokens, loaded into llm_request. stdtel-install settings now switches Claude Code's own
telemetry on as well as registering the hooks.
Use --no-native for hooks only.
docs/for-developers.md says what that sends and how to stop it. Nothing on
the wire or in the warehouse breaks: every schema change is additive.

Changed

  • The collector enforces privacy on every pipeline, for both harnesses (ADR-014 decision 3, #109).
    It had to be done before any native telemetry is switched on.
    • Identity is removed from spans, events and metric data points. The email becomes
      std.user.hash; account and organisation ids are deleted. Verified live, Claude Code puts these
      on every record rather than on the resource, so the old resource-level rule would not have seen
      them.
    • Content is deleted for both harnesses' native keys: Claude Code's tool_input,
      tool_parameters, full_command, prompt, response and similar; Copilot's
      github.copilot.tool.parameters.command and .file_path, and its GenAI message and tool-call
      keys.
    • Copilot's git remote and branch become std.branch.hash, computed exactly as the capture
      side computes it, and the plain values are deleted. A remote URL can carry a token.
    • All of this is tested against the pinned collector image, under both the default and the
      Langfuse configuration. Before this change, 22 of those 24 checks failed.

Added

  • Third-party skills are named from stdtel's own record (ADR-014 decision 10, #117). Without the
    detailed view, Claude Code names a third-party plugin's skill "third-party". The new view
    llm_request_attributed names such a request from the one skill that stdtel saw run on the same
    prompt and that the harness did not name itself, labelled derived. With more than one candidate
    it stays "third-party". It is read-time, so loader order cannot matter. Removing the tail rule and
    chars/4 load_tokens remains scheduled for one release after native records are verified live.
  • skills/stdtel-setup: interactive setup for both harnesses (ADR-014 decision 12, #114). An
    agent runs it with the user.
    • It installs the hooks and switches on Claude Code's and Copilot's own telemetry without content.
    • It offers Claude Code's detailed view and says plainly what it costs: shell commands, file paths
      and written file contents travel to the collector and are deleted there.
    • It enables the view only on an explicit yes, through the new
      stdtel-install detailed-view on. That command enforces the rest in code: the collector must
      be local or vouched for, and the content probe must pass on the spot. Otherwise it writes nothing.
      off removes the flag.
    • The gate and stdtel-doctor's content check now probe the endpoint Claude Code will send
      to
      , read from its settings files in its own precedence order (project local, then project,
      then user). Until now they probed stdtel's own endpoint. That is usually the same collector, but
      not after a hand edit, a reinstall with another endpoint, or a project override, and then the
      check proved something about a collector Claude Code would never use.
  • stdtel-install switches on both harnesses' own telemetry (ADR-014 decision 2, #113).
    • settings now writes Claude Code's env block alongside the hooks: events on, exported to the
      collector stdtel already uses, with repository attributes on and account uuid off.
    • It never writes OTEL_LOG_TOOL_DETAILS or a metrics exporter. A detailed view the user already
      chose survives a reinstall. --no-native writes the hooks only.
    • New stdtel-install copilot prints, or merges with --vscode-settings, the VS Code
      github.copilot.chat.otel.* settings with captureContent: false, and the Copilot CLI's
      COPILOT_OTEL_* variables. The keys come from the VS Code documentation; they have not yet been
      seen in a live trace (#115).
    • docs/for-developers.md now says what Claude Code sends and how to stop it: STDTEL_DISABLED
      does not. It also drops two stale ticket references left over from ADR-013.
  • One table of model requests, loaded from the harness's own records (ADR-014 decisions 5 and 7,
    #112).
    warehouse/load_requests.py reads Claude Code's api_request events from Loki into
    llm_request: tokens, exact cost, model, and the skill, agent, plugin and MCP server each request
    served. A live capture on Claude Code 2.1.285 confirmed that api_request carries the real skill
    name (probe-echo) even where skill_activated redacts it. That capture, scrubbed, is the
    replayed fixture.
    • llm_request_change_request joins each request to its change request through the stdtel turn
      with the same prompt_id, reusing activation_change_request rather than restating ADR-013's
      rule. Only turns bridge: a sub-agent in its own worktree shares the prompt id but not the
      branch.
    • session_cost_reconciliation compares the harness's session total with the sum of its
      requests. The two are never added together.
    • Cost is stored as unbounded NUMERIC. The harness sends seven decimals and more, which
      NUMERIC(12,4) would have rounded.
    • Loki's query_range stops at its limit silently, so the loader pages until a short page comes
      back. A request with no id fails the run and is counted, not dropped.
    • make load and the compose loader service now run it after the traces load. make load is
      an && chain, so a Loki outage stops the delivery half there, exactly as a Tempo outage
      already did.
  • stdtel-doctor proves the collector drops content (ADR-014 decision 13, #111). A new
    content dropped check sends a probe span and event holding a marker in content-shaped keys and a
    control id, then reads Tempo and Loki back. It passes only when the control arrived and the
    marker did not. "Not found" alone would also pass when nothing arrived, so a missing control
    fails as inconclusive. It runs by default only when OTEL_LOG_TOOL_DETAILS is on;
    stdtel-doctor --content-check runs it on demand and is what stdtel-setup will call before
    offering the detailed view. The trace loader ignores service.name=stdtel-probe.
  • Loki stores harness-native events (ADR-014 decision 4, #110). Claude Code's per-request cost,
    tokens and skill attribution exist only as events, and the logs pipeline used to export them to
    debug and nowhere else. Loki runs on host port 11010, this project's berth extra, and Grafana
    has it as a datasource. Native metrics are dropped at the collector: each carries session.id
    as a label, so it opens a series per session.

Fixed

  • Coding-agent PRs were being discarded as bots; they are now agent work for their harness
    (ADR-014 decision 9, #116).
    The fix was verified with the loader's own gh pr list call on real
    public PRs:

    • the Copilot coding agent is app/copilot-swe-agent and the Claude GitHub app is app/claude,
      both is_bot: true, so the #62 filter dropped every one;
    • both are now kept, including the gh search spellings Copilot and claude[bot];
    • dependency bots are still excluded.

    assisted_by now combines every signal: the arm labels, Anthropic's claude-code-assisted label,
    and an agent author.

    • One harness evidenced is that harness, even under no-ai, because a control an agent touched is
      not a control.
    • Two harnesses is mixed.
    • The absence of evidence is still unknown, never none.
  • A credential in Claude Code's repository URL no longer reaches Loki (#128). With
    OTEL_METRICS_INCLUDE_REPOSITORY on, every native event carries vcs.repository.url.full, and a
    remote such as https://x-access-token:...@github.com/... would have been stored verbatim. The
    collector now strips the userinfo, on the resource and on the record, for every signal. It is
    tested against the real image under both overlays.

  • A session with no repository now has branch_hash NULL, not '' (#122). The capture side
    sends an empty hash on purpose; the loader stored it verbatim, so branch_hash IS NULL
    undercounted sessions that cannot join a change request (ADR-005). schema.sql converts existing
    '' rows, touching nothing else.

  • The compose loader service's traces load had never run (#121). It called
    python -m warehouse.load_traces with no arguments while --tempo and --dsn were required, so
    every cycle ended in a usage error. That happened before a loader_run row could be written, and
    || echo "traces failed" was the only sign. Both flags now fall back to STDTEL_TEMPO and
    STDTEL_DSN, which the service sets, and a test parses the service's own invocation.