Show the user their own rows at the end of hyp init

README.md

@@ -64,7 +64,9 @@ On a TTY this launches the interactive walkthrough:
    persistent daemon (launchd on macOS, systemd `--user` on Linux),
    attaches the selected clients, and starts capturing.
 5. The walkthrough finishes by printing the config path, daemon status,
-   per-client attach results, and a first `hyp query` command to run.
+   per-client attach results, and a first look at what was captured: token
+   volume per model, activity per day, which repos the sessions ran in, and
+   which tools got called. Reprint it any time with `hyp query overview`.
 
 For unattended installs (CI, scripted bootstraps, dotfiles) use the
 non-interactive flags:
@@ -178,6 +180,25 @@ before invoking the CLI or the daemon.
 
 ## Querying captured data
 
+Start with the overview: input, cached and output tokens per provider and
+model, the same per day, which repos the sessions ran in, and which tools
+get called - the same block `hyp init` ends on.
+
+```sh
+hyp query overview            # --json to script it, --sql to print the queries
+hyp query overview --days 90  # widen the window past what fits by default
+```
+
+The block states the period it covers. It picks a window it can summarize
+quickly, so a large cache narrows the period rather than hanging - and the
+line under the title always says which days the numbers describe.
+
+Input is prompt sent fresh and cached is prompt served from (or written
+to) the cache, so `input + cached` is the whole prompt; output is what the
+model generated.
+
+Then query anything directly:
+
 ```sh
 hyp query sql "select count(*) from ai_gateway_messages"
 hyp query sql "select count(*) from traces"

bin/hypaware.js

@@ -38,6 +38,15 @@ if (argv[0] === '__smoke_internal') {
 const { dispatch } = await import('../src/core/cli/dispatch.js')
 const { installObservability } = await import('../src/core/observability/index.js')
 const { flushStream } = await import('../src/core/cli/flush-streams.js')
+const { installStreamErrorHandlers } = await import('../src/core/cli/stream_errors.js')
+
+// Before anything writes: an asynchronous stdout/stderr failure (EPIPE when
+// a reader like `head` walks away mid-write) is delivered as an 'error'
+// event, which bypasses the try/catch below and every one inside the
+// commands. Unlistened, it crashes a run that had already succeeded.
+installStreamErrorHandlers([process.stdout, process.stderr], (message) => {
+  try { process.stderr.write(message) } catch { /* the stream is what failed */ }
+})
 
 const obs = installObservability()
 let exitCode = 1

hypaware-core/plugins-workspace/claude/skills/hypaware-ai-usage-report/SKILL.md

@@ -65,16 +65,32 @@ plain `SUM` over assistant rows is correct with no dedup (the one-carrier rule,
 `input_tokens` is net of cache, so it never double-counts. Report the four types
 separately (cache-read is usually the bulk; output the scarce slice).
 
+**A missing provider field NULLs your arithmetic, it does not zero it.** Not every
+provider emits every usage field - `cache_write_tokens` is Claude-only - and both
+SQL layers turn that into silent loss, not an error:
+
+- *Per row:* `CAST(...cache_read...) + CAST(...cache_write...)` is NULL for every
+  OpenAI row, so `sum()` skips those rows entirely and the provider's whole cache-read
+  total reads 0. COALESCE each term *inside* the addition, not just around the sum.
+- *Per aggregate:* `sum()` over all-NULL returns NULL, so a Codex-scoped slice yields
+  `t_cw: null` and any `t_in + t_cr + t_cw` total is NULL.
+
+Both were measured on a real install: 25,581,312 OpenAI cache-read tokens silently
+became 0. COALESCE every token sum, and every term of every token addition.
+
 ```sql
 SELECT
-  sum(CAST(JSON_EXTRACT(attributes,'$.usage.input_tokens')       AS BIGINT)) t_in,
-  sum(CAST(JSON_EXTRACT(attributes,'$.usage.output_tokens')      AS BIGINT)) t_out,
-  sum(CAST(JSON_EXTRACT(attributes,'$.usage.cache_write_tokens') AS BIGINT)) t_cw,
-  sum(CAST(JSON_EXTRACT(attributes,'$.usage.cache_read_tokens')  AS BIGINT)) t_cr
+  COALESCE(sum(CAST(JSON_EXTRACT(attributes,'$.usage.input_tokens')       AS BIGINT)), 0) t_in,
+  COALESCE(sum(CAST(JSON_EXTRACT(attributes,'$.usage.output_tokens')      AS BIGINT)), 0) t_out,
+  COALESCE(sum(CAST(JSON_EXTRACT(attributes,'$.usage.cache_write_tokens') AS BIGINT)), 0) t_cw,
+  COALESCE(sum(CAST(JSON_EXTRACT(attributes,'$.usage.cache_read_tokens')  AS BIGINT)), 0) t_cr
 FROM ai_gateway_messages
 WHERE date BETWEEN '<start>' AND '<end>'
   AND role='assistant' AND JSON_EXTRACT(attributes,'$.usage') IS NOT NULL;
 -- One carrier row per response (LLP 0035): a plain SUM is correct, no dedup.
+-- COALESCE is NOT decorative: a field a provider never emits (cache_write_tokens
+-- on OpenAI/ChatGPT rows) makes sum() return NULL, and NULL poisons any total
+-- built from it -- t_in + t_cr + t_cw goes NULL and the real cache reads vanish.
 -- Slice by adding gateway_id / model / repo_root / date to SELECT + GROUP BY.
 -- Defensive equivalent: max(...) GROUP BY session_id, message_id -- session_id is the
 -- uniform key; conversation_id is null for Claude and only separates Codex threads.

hypaware-core/plugins-workspace/claude/skills/hypaware-query/SKILL.md

@@ -20,14 +20,17 @@ Use `hyp query` to inspect local HypAware recordings. By default it reads local
 ## Common Commands
 
 ```bash
+hyp query overview                                     # orientation: tokens per model/day/repo/tool (--sql prints its queries, --json for machine output)
 hyp query status
 hyp query schema <table> --format json
 hyp query sql "<sql>" --format json
 hyp query sql "<sql>" --format jsonl --output <file>   # full result, lossless
 hyp query refresh <dataset>
 ```
 
-These are the only subcommands in the installed CLI (`hyp query`: schema, status, sql, refresh, maintain). There are no high-level `catalog`/`logs`/`traces`/`metrics` query commands — answer questions with `hyp query sql`, and discover datasets from the `hyp query status` output.
+**`hyp query overview` totals are windowed, not all-time.** It probes the cache, times that probe to measure this machine, and picks the widest recent window it can summarize quickly, so on a large cache it silently covers a subset. The line under the title always states the period (`2026-07-24 to 2026-07-27 - showing 3 of 31 active days …`); read it before quoting any number, and pass `--days <n>` to widen (that overrides the budget, whatever it costs). Never report its totals as the full history without checking that line.
+
+These are the only subcommands in the installed CLI (`hyp query`: overview, schema, status, sql, refresh, maintain). There are no high-level `catalog`/`logs`/`traces`/`metrics` query commands; answer questions with `hyp query sql`, and discover datasets from the `hyp query status` output.
 
 ## Remote queries (other HypAware hosts)
 
@@ -76,11 +79,11 @@ Recorded AI-gateway traffic is exposed through one dataset: `ai_gateway_messages
 Key columns:
 
 - `session_id`, `conversation_id`, `message_id`, `message_index`, `part_id`, `part_index` — stable identity. `session_id` is the always-present session key (group/scope on it); `conversation_id` is a nullable thread within a session (a Codex thread; null for Claude).
-- `provider`, `model`, `role`, `part_type`, `content_text` — normalized provider/message content fields.
+- `provider`, `model`, `role`, `part_type`, `content_text`: normalized provider/message content fields. `part_type` is HypAware's own vocabulary, NOT the provider's wire name: `text`, `reasoning`, `tool_call`, `tool_result`, `image`, `fallback`. Tool calls are `part_type='tool_call'`: Anthropic's `tool_use` matches no row and returns a silently empty result. `role` is `user` / `assistant` / `tool` / `system` / `developer`.
 - `tool_name`, `tool_call_id`, `tool_args`, `status` — tool-call/result joins and sparse status such as `finish_reason`.
 - `attributes` (JSON) — request settings, usage, propagated `dev_run_id`, and gateway diagnostics under `attributes.gateway`.
 
-**Token counts** live under `attributes.usage` on `role='assistant'` rows (NOT in `raw_frame`): `input_tokens`, `output_tokens`, `cache_read_tokens`, `cache_write_tokens`. Codex (`provider='openai'`) omits `cache_write_tokens` and adds `reasoning_tokens` + `total_tokens`. Extract with `CAST(JSON_EXTRACT(attributes,'$.usage.input_tokens') AS BIGINT)`. Usage rides exactly one row per response (the last assistant part; non-carrier parts are null), so a plain `SUM` over assistant rows is correct with no dedup (the one-carrier rule, LLP 0035). If you prefer a defensive dedup, `max(...) GROUP BY session_id, message_id` returns the same number: key on `session_id` (`conversation_id` is null for Claude, and only separates threads within a Codex session).
+**Token counts** live under `attributes.usage` on `role='assistant'` rows (NOT in `raw_frame`): `input_tokens`, `output_tokens`, `cache_read_tokens`, `cache_write_tokens`. Codex (`provider='openai'`) omits `cache_write_tokens` and adds `reasoning_tokens` + `total_tokens`. Extract with `COALESCE(CAST(JSON_EXTRACT(attributes,'$.usage.input_tokens') AS BIGINT), 0)` - **always COALESCE**: a field the provider never emits is NULL, and NULL propagates instead of zeroing. Per row, `CAST(...cache_read...) + CAST(...cache_write...)` is NULL for every OpenAI row, so `sum()` skips them and that provider's whole cache-read total silently reads 0 (measured: 25,581,312 -> 0). Per aggregate, `sum()` over all-NULL returns NULL, so a Codex-scoped `t_in + t_cr + t_cw` total is NULL. COALESCE each term inside an addition, and each sum. Usage rides exactly one row per response (the last assistant part; non-carrier parts are null), so a plain `SUM` over assistant rows is correct with no dedup (the one-carrier rule, LLP 0035). If you prefer a defensive dedup, `max(...) GROUP BY session_id, message_id` returns the same number: key on `session_id` (`conversation_id` is null for Claude, and only separates threads within a Codex session).
 
 Claude transcript enrichment adds `provider_uuid`, `parent_uuid`, `request_id`, `entrypoint`, `client_version`, `user_type`, `permission_mode`, and `hook_event` when the local Claude Code JSONL transcript can be matched.
 

hypaware-core/plugins-workspace/codex/skills/hypaware-ai-usage-report/SKILL.md

@@ -65,16 +65,32 @@ plain `SUM` over assistant rows is correct with no dedup (the one-carrier rule,
 `input_tokens` is net of cache, so it never double-counts. Report the four types
 separately (cache-read is usually the bulk; output the scarce slice).
 
+**A missing provider field NULLs your arithmetic, it does not zero it.** Not every
+provider emits every usage field - `cache_write_tokens` is Claude-only - and both
+SQL layers turn that into silent loss, not an error:
+
+- *Per row:* `CAST(...cache_read...) + CAST(...cache_write...)` is NULL for every
+  OpenAI row, so `sum()` skips those rows entirely and the provider's whole cache-read
+  total reads 0. COALESCE each term *inside* the addition, not just around the sum.
+- *Per aggregate:* `sum()` over all-NULL returns NULL, so a Codex-scoped slice yields
+  `t_cw: null` and any `t_in + t_cr + t_cw` total is NULL.
+
+Both were measured on a real install: 25,581,312 OpenAI cache-read tokens silently
+became 0. COALESCE every token sum, and every term of every token addition.
+
 ```sql
 SELECT
-  sum(CAST(JSON_EXTRACT(attributes,'$.usage.input_tokens')       AS BIGINT)) t_in,
-  sum(CAST(JSON_EXTRACT(attributes,'$.usage.output_tokens')      AS BIGINT)) t_out,
-  sum(CAST(JSON_EXTRACT(attributes,'$.usage.cache_write_tokens') AS BIGINT)) t_cw,
-  sum(CAST(JSON_EXTRACT(attributes,'$.usage.cache_read_tokens')  AS BIGINT)) t_cr
+  COALESCE(sum(CAST(JSON_EXTRACT(attributes,'$.usage.input_tokens')       AS BIGINT)), 0) t_in,
+  COALESCE(sum(CAST(JSON_EXTRACT(attributes,'$.usage.output_tokens')      AS BIGINT)), 0) t_out,
+  COALESCE(sum(CAST(JSON_EXTRACT(attributes,'$.usage.cache_write_tokens') AS BIGINT)), 0) t_cw,
+  COALESCE(sum(CAST(JSON_EXTRACT(attributes,'$.usage.cache_read_tokens')  AS BIGINT)), 0) t_cr
 FROM ai_gateway_messages
 WHERE date BETWEEN '<start>' AND '<end>'
   AND role='assistant' AND JSON_EXTRACT(attributes,'$.usage') IS NOT NULL;
 -- One carrier row per response (LLP 0035): a plain SUM is correct, no dedup.
+-- COALESCE is NOT decorative: a field a provider never emits (cache_write_tokens
+-- on OpenAI/ChatGPT rows) makes sum() return NULL, and NULL poisons any total
+-- built from it -- t_in + t_cr + t_cw goes NULL and the real cache reads vanish.
 -- Slice by adding gateway_id / model / repo_root / date to SELECT + GROUP BY.
 -- Defensive equivalent: max(...) GROUP BY session_id, message_id -- session_id is the
 -- uniform key; conversation_id is null for Claude and only separates Codex threads.

hypaware-core/plugins-workspace/codex/skills/hypaware-query/SKILL.md

@@ -20,14 +20,17 @@ Use `hyp query` to inspect local HypAware recordings. By default it reads local
 ## Common Commands
 
 ```bash
+hyp query overview                                     # orientation: tokens per model/day/repo/tool (--sql prints its queries, --json for machine output)
 hyp query status
 hyp query schema <table> --format json
 hyp query sql "<sql>" --format json
 hyp query sql "<sql>" --format jsonl --output <file>   # full result, lossless
 hyp query refresh <dataset>
 ```
 
-These are the only subcommands in the installed CLI (`hyp query`: schema, status, sql, refresh, maintain). There are no high-level `catalog`/`logs`/`traces`/`metrics` query commands — answer questions with `hyp query sql`, and discover datasets from the `hyp query status` output.
+**`hyp query overview` totals are windowed, not all-time.** It probes the cache, times that probe to measure this machine, and picks the widest recent window it can summarize quickly, so on a large cache it silently covers a subset. The line under the title always states the period (`2026-07-24 to 2026-07-27 - showing 3 of 31 active days …`); read it before quoting any number, and pass `--days <n>` to widen (that overrides the budget, whatever it costs). Never report its totals as the full history without checking that line.
+
+These are the only subcommands in the installed CLI (`hyp query`: overview, schema, status, sql, refresh, maintain). There are no high-level `catalog`/`logs`/`traces`/`metrics` query commands; answer questions with `hyp query sql`, and discover datasets from the `hyp query status` output.
 
 ## Remote queries (other HypAware hosts)
 
@@ -76,11 +79,11 @@ Recorded AI-gateway traffic is exposed through one dataset: `ai_gateway_messages
 Key columns:
 
 - `session_id`, `conversation_id`, `message_id`, `message_index`, `part_id`, `part_index` — stable identity. `session_id` is the always-present session key (group/scope on it); `conversation_id` is a nullable thread within a session (a Codex thread; null for Claude).
-- `provider`, `model`, `role`, `part_type`, `content_text` — normalized provider/message content fields.
+- `provider`, `model`, `role`, `part_type`, `content_text`: normalized provider/message content fields. `part_type` is HypAware's own vocabulary, NOT the provider's wire name: `text`, `reasoning`, `tool_call`, `tool_result`, `image`, `fallback`. Tool calls are `part_type='tool_call'`: Anthropic's `tool_use` matches no row and returns a silently empty result. `role` is `user` / `assistant` / `tool` / `system` / `developer`.
 - `tool_name`, `tool_call_id`, `tool_args`, `status` — tool-call/result joins and sparse status such as `finish_reason`.
 - `attributes` (JSON) — request settings, usage, propagated `dev_run_id`, and gateway diagnostics under `attributes.gateway`.
 
-**Token counts** live under `attributes.usage` on `role='assistant'` rows (NOT in `raw_frame`): `input_tokens`, `output_tokens`, `cache_read_tokens`, `cache_write_tokens`. Codex (`provider='openai'`) omits `cache_write_tokens` and adds `reasoning_tokens` + `total_tokens`. Extract with `CAST(JSON_EXTRACT(attributes,'$.usage.input_tokens') AS BIGINT)`. Usage rides exactly one row per response (the last assistant part; non-carrier parts are null), so a plain `SUM` over assistant rows is correct with no dedup (the one-carrier rule, LLP 0035). If you prefer a defensive dedup, `max(...) GROUP BY session_id, message_id` returns the same number: key on `session_id` (`conversation_id` is null for Claude, and only separates threads within a Codex session).
+**Token counts** live under `attributes.usage` on `role='assistant'` rows (NOT in `raw_frame`): `input_tokens`, `output_tokens`, `cache_read_tokens`, `cache_write_tokens`. Codex (`provider='openai'`) omits `cache_write_tokens` and adds `reasoning_tokens` + `total_tokens`. Extract with `COALESCE(CAST(JSON_EXTRACT(attributes,'$.usage.input_tokens') AS BIGINT), 0)` - **always COALESCE**: a field the provider never emits is NULL, and NULL propagates instead of zeroing. Per row, `CAST(...cache_read...) + CAST(...cache_write...)` is NULL for every OpenAI row, so `sum()` skips them and that provider's whole cache-read total silently reads 0 (measured: 25,581,312 -> 0). Per aggregate, `sum()` over all-NULL returns NULL, so a Codex-scoped `t_in + t_cr + t_cw` total is NULL. COALESCE each term inside an addition, and each sum. Usage rides exactly one row per response (the last assistant part; non-carrier parts are null), so a plain `SUM` over assistant rows is correct with no dedup (the one-carrier rule, LLP 0035). If you prefer a defensive dedup, `max(...) GROUP BY session_id, message_id` returns the same number: key on `session_id` (`conversation_id` is null for Claude, and only separates threads within a Codex session).
 
 Claude transcript enrichment adds `provider_uuid`, `parent_uuid`, `request_id`, `entrypoint`, `client_version`, `user_type`, `permission_mode`, and `hook_event` when the local Claude Code JSONL transcript can be matched.
 

llp/0035-token-usage-normalization.decision.md

@@ -124,21 +124,37 @@ row, so a plain sum is correct:
 
 ```sql
 SELECT
-  SUM(CAST(JSON_EXTRACT(attributes, '$.usage.input_tokens')      AS BIGINT)) AS input,
-  SUM(CAST(JSON_EXTRACT(attributes, '$.usage.output_tokens')     AS BIGINT)) AS output,
-  SUM(CAST(JSON_EXTRACT(attributes, '$.usage.cache_read_tokens') AS BIGINT)) AS cache_read,
-  SUM(CAST(JSON_EXTRACT(attributes, '$.usage.cache_write_tokens')AS BIGINT)) AS cache_write,
-  SUM(CAST(JSON_EXTRACT(attributes, '$.usage.reasoning_tokens')  AS BIGINT)) AS reasoning
+  COALESCE(SUM(CAST(JSON_EXTRACT(attributes, '$.usage.input_tokens')      AS BIGINT)), 0) AS input,
+  COALESCE(SUM(CAST(JSON_EXTRACT(attributes, '$.usage.output_tokens')     AS BIGINT)), 0) AS output,
+  COALESCE(SUM(CAST(JSON_EXTRACT(attributes, '$.usage.cache_read_tokens') AS BIGINT)), 0) AS cache_read,
+  COALESCE(SUM(CAST(JSON_EXTRACT(attributes, '$.usage.cache_write_tokens')AS BIGINT)), 0) AS cache_write,
+  COALESCE(SUM(CAST(JSON_EXTRACT(attributes, '$.usage.reasoning_tokens')  AS BIGINT)), 0) AS reasoning
 FROM ai_gateway_messages
 WHERE role = 'assistant' AND JSON_EXTRACT(attributes, '$.usage') IS NOT NULL
 ```
 
 A defensive `max()`-per-`COALESCE(raw_frame.message_id, message_id)` rollup also
 remains correct (the non-carrier blocks are null and ignored), so it's safe to
-keep in queries written before this decision. Field union across providers:
-Codex carries `reasoning_tokens` and no `cache_write_tokens`; Claude is the
-reverse; `input_tokens` is net for both (#net-input). `COALESCE(..., 0)` the
-union.
+keep in queries written before this decision.
+
+<a id="null-union"></a>**The field union is a null trap, and the nulls are
+silent.** Codex carries `reasoning_tokens` and no `cache_write_tokens`; Claude
+is the reverse; `input_tokens` is net for both (#net-input). A field the
+provider never emits reads NULL, and NULL propagates rather than zeroing, in
+two distinct places:
+
+- *Inside a row's arithmetic.* `CAST(...cache_read...) + CAST(...cache_write...)`
+  is NULL on every OpenAI row, so `SUM` skips the row entirely and that
+  provider's whole cache-read total collapses to 0. Measured on a real install:
+  25,581,312 cache-read tokens became 0, with no error.
+- *At the aggregate.* `SUM` over all-NULL returns NULL, not 0, so an
+  OpenAI-scoped slice yields `cache_write: null` and any
+  `input + cache_read + cache_write` total built from it is NULL.
+
+So: `COALESCE(..., 0)` **every** token sum, and **every term** of every token
+addition. The rule is the same shape as the net/gross normalization this
+document exists for - a cross-provider query must not silently mean different
+things per provider - one layer down, in null handling.
 
 ### Consequences