Skip to content

feat(claw): add session usage analytics#1969

Merged
Nikhil (shadowfax92) merged 4 commits into
mainfrom
feat/claw-session-usage-analytics
Jul 22, 2026
Merged

feat(claw): add session usage analytics#1969
Nikhil (shadowfax92) merged 4 commits into
mainfrom
feat/claw-session-usage-analytics

Conversation

@shadowfax92

Copy link
Copy Markdown
Contributor

Summary

  • enrich session-end analytics with real tool dispatch counts, harness identity, and observed used-session concurrency
  • emit one privacy-safe aggregate per distinct recognized tool with bounded total and maximum duration
  • enforce the six-event catalog, closed tool names, safe integers, and double allowlisting at the analytics boundary

Design

A session-owned in-memory tracker records each recognized tools/call once at the common completion boundary. Exactly-once teardown snapshots deterministic aggregates, emits sorted per-tool summaries, then emits the enriched terminal session event; no per-dispatch analytics, identifiers, content, or persistence are added.

Test plan

  • cargo fmt --all -- --check
  • cargo test -p claw-server-rust --locked
  • cargo clippy -p claw-server-rust --all-targets --locked -- -D warnings
  • cargo test --workspace --locked
  • bun run test:claw-api-contract

@shadowfax92

Copy link
Copy Markdown
Contributor Author

Claude (@claude) review

@shadowfax92

Copy link
Copy Markdown
Contributor Author

Greptile (@greptileai) review

@claude

claude Bot commented Jul 22, 2026

Copy link
Copy Markdown

Claude finished Nikhil (@shadowfax92)'s task in 2m 53s —— View job


Code Review

  • Gather context and understand the PR
  • Read CLAUDE.md for repo guidelines
  • Review changed files
    • usage.rs (new file — core tracker)
    • events.rs (analytics event schema changes)
    • service.rs (MCP service integration)
    • sessions/manager.rs (session teardown / emit logic)
    • sessions/session.rs
    • analytics/service.rs
    • Supporting / test files
  • Synthesize findings and post review

Overall this is a clean, well-structured implementation. The design is disciplined: the tracker is pure in-memory state with no analytics coupling, the emit boundary is a single point (finish_teardown), and the double-allowlist provides real defense in depth. The test coverage is thorough and the tests are well-named and focused. A few observations below — most are minor.

Strengths

  • Clean separation: UsageTracker lives in the service layer, knows nothing about analytics; finish_teardown is the single bridge. Adding a new metric means only touching the tracker and the emit site.
  • Deterministic ordering: BTreeMap in UsageState gives alphabetical tool ordering for free, which makes the emitted events stable across runs and easier to assert against in tests.
  • Double allowlisting: sanitize at capture time and final_allowlist at the PostHog boundary. Unknown tool names, negative integers, or content-shaped strings can't survive either layer.
  • MAX_SAFE_INTEGER cap: capped_add + capped_u128 prevent precision loss at JavaScript consumers. The test at usage.rs:177 explicitly covers the boundary.
  • Exact-once teardown: The session is removed from the live map before teardown runs, so snapshot() → emit → AGENT_SESSION_ENDED is guaranteed to happen exactly once even under concurrent removes and shutdown.

Observations

1. capped_u128(concurrent_used_sessions as u128) — mild roundtrip (usage.rs:75)

let concurrent_used_sessions = capped_u128(concurrent_used_sessions as u128);

usize is at most 64 bits and MAX_SAFE_INTEGER fits in u64, so casting through u128 is unnecessary — concurrent_used_sessions.min(MAX_SAFE_INTEGER as usize) as u64 would be just as safe and would avoid the silent widening/narrowing. Not a bug in practice but could confuse a reader.

Fix this →

2. name_session string literal in known_tool_names() (events.rs:107)

.chain(std::iter::once("name_session"))

NAME_SESSION_TOOL_NAME in api/mcp/service.rs is the canonical spelling of this tool name. The analytics catalog hardcodes the string literal instead. If the constant changes, this silently decouples — the tool stops being tracked and neither the compiler nor tests catch it. Since NAME_SESSION_TOOL_NAME is private to service.rs, the cleanest fix would be moving it to a shared location (e.g., api/mcp/naming.rs or a constants module); alternatively, a targeted test asserting known_tool_names().contains("name_session") would at least make the coupling explicit.

3. .max(1) after mark_used() is correct but unobvious (service.rs:327)

started.session.mark_used();
let concurrent_used_sessions = self.state.sessions.used_count().await.max(1);

The .max(1) is correct because mark_used has already run, so the current session is counted, making the minimum 1. But a reader who doesn't know the ordering might wonder why .max(1) is needed at all. A one-line comment would make the invariant explicit.

4. HARNESS field absent from AGENT_SESSION_ENDED despite PR description

The PR summary says the implementation "enrich[es] session-end analytics with real tool dispatch counts, harness identity, and observed used-session concurrency", but AGENT_SESSION_ENDED doesn't include a harness property — harness identity is tracked separately via HARNESS_CONNECTED/HARNESS_DISCONNECTED. The client_name field covers the identity for aggregation purposes, so this is likely intentional, but the description may set incorrect expectations. Worth a small doc clarification if the PR description is the canonical spec.

5. dispatch_count includes all recognized tool calls, including cancelled ones

finish_tool_call records usage regardless of whether the result is Ok or Err:

async fn finish_tool_call(..., result: Result<CallToolResult, McpError>) -> ... {
    session.record_tool_usage(tool_name, started_at.elapsed(), concurrent_used_sessions).await;
    result
}

Cancelled dispatches return Err(McpError) and are counted. This is intentional per the design ("at the common completion boundary"), and the test at service.rs:531 covers it. Just worth a note for downstream data consumers that dispatch_count includes cancellations.

6. No boundary test for unknown tools reaching the tracker

In practice, call_tool rejects unknown tools before finish_tool_call is called, so the tracker never receives an unrecognized tool name. But there's no explicit test asserting that dispatch_count in AGENT_SESSION_ENDED equals the sum of per-tool dispatch_count values (which holds precisely because only known-catalog tools reach the tracker). If future code paths ever bypass the gate, the totals would silently diverge. A brief comment near record_tool_usage explaining that callers are responsible for pre-filtering to recognized names would make this contract explicit.


Verdict: No blocking issues. The double-allowlist and exact-once teardown are the two highest-value invariants and both are well-enforced. The test suite is comprehensive. The items above are minor — (#2) is the most worth addressing since it's a silent naming coupling; the rest are clarity nits.

@shadowfax92
Nikhil (shadowfax92) merged commit 8aee56c into main Jul 22, 2026
24 of 25 checks passed
@greptile-apps

greptile-apps Bot commented Jul 22, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

This PR adds aggregate session usage analytics to the Claw server. The main changes are:

  • Session-owned tracking for tool counts, durations, and used-session concurrency.
  • Allowlisted schemas for per-tool summaries and enriched session-end events.
  • Deterministic, bounded aggregates emitted during session teardown.
  • Shutdown ordering that drains session events before stopping analytics.

Confidence Score: 4/5

The session teardown path can emit incomplete usage and should be fixed before merging.

  • Active dispatch cancellation is not awaited before the terminal snapshot.
  • A finishing call can record usage after the only aggregate events have been emitted.
  • The analytics schemas, bounds, and shutdown ordering otherwise appear consistent.

packages/browseros-agent/apps/claw-server-rust/src/services/sessions/manager.rs

Important Files Changed

Filename Overview
packages/browseros-agent/apps/claw-server-rust/src/analytics/events.rs Adds closed schemas and normalization for terminal and per-tool usage events.
packages/browseros-agent/apps/claw-server-rust/src/api/mcp/service.rs Records recognized tool calls at the common completion boundary.
packages/browseros-agent/apps/claw-server-rust/src/services/sessions/usage.rs Adds deterministic, saturating per-session and per-tool aggregates.
packages/browseros-agent/apps/claw-server-rust/src/services/sessions/manager.rs Emits usage during teardown but can snapshot before an active dispatch finishes recording.
packages/browseros-agent/apps/claw-server-rust/src/runtime.rs Drains session teardown before shutting down the analytics client.

Sequence Diagram

%%{init: {'theme': 'neutral'}}%%
sequenceDiagram
    participant Call as Tool call
    participant Usage as Usage tracker
    participant Manager as Session manager
    participant Analytics as Analytics sink
    Call->>Call: Dispatch tool
    Manager->>Call: Request cancellation
    Manager->>Usage: Take usage snapshot
    Manager->>Analytics: Emit tool and session events
    Call->>Usage: Record completed call
    Note over Usage,Analytics: Late record is absent from emitted events
Loading
%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
sequenceDiagram
    participant Call as Tool call
    participant Usage as Usage tracker
    participant Manager as Session manager
    participant Analytics as Analytics sink
    Call->>Call: Dispatch tool
    Manager->>Call: Request cancellation
    Manager->>Usage: Take usage snapshot
    Manager->>Analytics: Emit tool and session events
    Call->>Usage: Record completed call
    Note over Usage,Analytics: Late record is absent from emitted events
Loading
Prompt To Fix All With AI
Fix the following 1 code review issue. Work through them one at a time, proposing concise fixes.

---

### Issue 1 of 1
packages/browseros-agent/apps/claw-server-rust/src/services/sessions/manager.rs:393
**Teardown Snapshots In-Flight Usage**

When a session closes during a tool call, `cancel_active_dispatches()` requests cancellation but does not wait for the dispatch to finish. This snapshot can therefore run before `finish_tool_call()` records the completed or cancelled call, permanently omitting it from both the per-tool summary and the terminal dispatch count.

Reviews (1): Last reviewed commit: "feat(claw): emit session usage summaries" | Re-trigger Greptile

kind: &str,
audit_result: AppResult<()>,
) -> AppResult<()> {
let usage = session.usage_snapshot().await;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Teardown Snapshots In-Flight Usage

When a session closes during a tool call, cancel_active_dispatches() requests cancellation but does not wait for the dispatch to finish. This snapshot can therefore run before finish_tool_call() records the completed or cancelled call, permanently omitting it from both the per-tool summary and the terminal dispatch count.

Prompt To Fix With AI
This is a comment left during a code review.
Path: packages/browseros-agent/apps/claw-server-rust/src/services/sessions/manager.rs
Line: 393

Comment:
**Teardown Snapshots In-Flight Usage**

When a session closes during a tool call, `cancel_active_dispatches()` requests cancellation but does not wait for the dispatch to finish. This snapshot can therefore run before `finish_tool_call()` records the completed or cancelled call, permanently omitting it from both the per-tool summary and the terminal dispatch count.

How can I resolve this? If you propose a fix, please make it concise.

@greptile-apps

greptile-apps Bot commented Jul 22, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

This PR adds aggregate session usage analytics to BrowserClaw. The main changes are:

  • Tracks bounded dispatch counts and durations by recognized tool.
  • Emits sorted tool summaries before the enriched session-end event.
  • Adds client identity and observed used-session concurrency.
  • Enforces closed event schemas and double allowlisting.

Confidence Score: 4/5

The usage lifecycle and concurrency metric need fixes before merging.

  • Teardown can emit a terminal snapshot before an active call records its usage.
  • The concurrency field counts idle sessions that were used earlier, so sequential calls can appear concurrent.
  • Event schemas, allowlisting, numeric bounds, and deterministic tool ordering are consistent.

packages/browseros-agent/apps/claw-server-rust/src/api/mcp/service.rs; packages/browseros-agent/apps/claw-server-rust/src/services/sessions/manager.rs

Important Files Changed

Filename Overview
packages/browseros-agent/apps/claw-server-rust/src/analytics/events.rs Adds the sixth event and closed, bounded schemas for session and tool usage properties.
packages/browseros-agent/apps/claw-server-rust/src/analytics/service.rs Extends final allowlist coverage and tests the aggregate wire payload.
packages/browseros-agent/apps/claw-server-rust/src/api/mcp/service.rs Adds common completion accounting, but samples ever-used sessions instead of active concurrency.
packages/browseros-agent/apps/claw-server-rust/src/services/sessions/session.rs Attaches client identity and the usage tracker to each session.
packages/browseros-agent/apps/claw-server-rust/src/services/sessions/usage.rs Implements bounded, mutex-protected, deterministically ordered usage aggregation.
packages/browseros-agent/apps/claw-server-rust/src/services/sessions/manager.rs Emits aggregate events at teardown, but can snapshot before an active call records completion.

Sequence Diagram

%%{init: {'theme': 'neutral'}}%%
sequenceDiagram
    participant Call as Tool call
    participant Sessions as Session manager
    participant Usage as Usage tracker
    participant Analytics as Analytics sink
    Call->>Usage: mark session used
    Call->>Sessions: count live ever-used sessions
    Sessions-->>Call: sampled count
    Call->>Call: dispatch tool
    par Teardown may start
        Sessions->>Usage: snapshot usage
        Sessions->>Analytics: emit terminal events
    and Call completes
        Call->>Usage: record dispatch and duration
    end
Loading
%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
sequenceDiagram
    participant Call as Tool call
    participant Sessions as Session manager
    participant Usage as Usage tracker
    participant Analytics as Analytics sink
    Call->>Usage: mark session used
    Call->>Sessions: count live ever-used sessions
    Sessions-->>Call: sampled count
    Call->>Call: dispatch tool
    par Teardown may start
        Sessions->>Usage: snapshot usage
        Sessions->>Analytics: emit terminal events
    and Call completes
        Call->>Usage: record dispatch and duration
    end
Loading
Prompt To Fix All With AI
Fix the following 2 code review issues. Work through them one at a time, proposing concise fixes.

---

### Issue 1 of 2
packages/browseros-agent/apps/claw-server-rust/src/services/sessions/manager.rs:393
**Teardown Snapshots Before Call Completion**

When an idle sweep or transport close removes a session during a tool call, teardown can take this snapshot before `finish_tool_call` records the cancelled call. The terminal events then omit that dispatch and its duration, while the later update remains only on the detached session and is never emitted.

### Issue 2 of 2
packages/browseros-agent/apps/claw-server-rust/src/api/mcp/service.rs:327
**Concurrency Counts Idle Used Sessions**

`used_count()` counts every live session that has ever entered a tool call, not sessions with calls currently in flight. Two long-lived sessions making sequential calls therefore record a maximum concurrency of two even though their calls never overlap, inflating the new analytics field.

Reviews (2): Last reviewed commit: "feat(claw): emit session usage summaries" | Re-trigger Greptile

kind: &str,
audit_result: AppResult<()>,
) -> AppResult<()> {
let usage = session.usage_snapshot().await;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Teardown Snapshots Before Call Completion

When an idle sweep or transport close removes a session during a tool call, teardown can take this snapshot before finish_tool_call records the cancelled call. The terminal events then omit that dispatch and its duration, while the later update remains only on the detached session and is never emitted.

Prompt To Fix With AI
This is a comment left during a code review.
Path: packages/browseros-agent/apps/claw-server-rust/src/services/sessions/manager.rs
Line: 393

Comment:
**Teardown Snapshots Before Call Completion**

When an idle sweep or transport close removes a session during a tool call, teardown can take this snapshot before `finish_tool_call` records the cancelled call. The terminal events then omit that dispatch and its duration, while the later update remains only on the detached session and is never emitted.

How can I resolve this? If you propose a fix, please make it concise.

ownership_key,
agent_label: started.agent_label,
started.session.mark_used();
let concurrent_used_sessions = self.state.sessions.used_count().await.max(1);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Concurrency Counts Idle Used Sessions

used_count() counts every live session that has ever entered a tool call, not sessions with calls currently in flight. Two long-lived sessions making sequential calls therefore record a maximum concurrency of two even though their calls never overlap, inflating the new analytics field.

Prompt To Fix With AI
This is a comment left during a code review.
Path: packages/browseros-agent/apps/claw-server-rust/src/api/mcp/service.rs
Line: 327

Comment:
**Concurrency Counts Idle Used Sessions**

`used_count()` counts every live session that has ever entered a tool call, not sessions with calls currently in flight. Two long-lived sessions making sequential calls therefore record a maximum concurrency of two even though their calls never overlap, inflating the new analytics field.

How can I resolve this? If you propose a fix, please make it concise.

@github-actions

Copy link
Copy Markdown
Contributor

✅ Tests passed — 1607/1611

Suite Passed Failed Skipped
agent 338/338 0 0
build 34/34 0 0
claw-app 185/185 0 0
⚠️ claw-mcp 0/0 0 0
⚠️ claw-onboard 0/0 0 0
⚠️ claw-server-rust-quality 0/0 0 0
⚠️ claw-server-rust 0/0 0 0
server-agent 314/314 0 0
server-api 147/147 0 0
server-browser 10/10 0 0
server-integration 10/10 0 0
server-lib 277/278 0 1
server-root 38/41 0 3
server-tools 254/254 0 0

View workflow run

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok ready

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants