Telemetry for context window usage over time

[haakons]

Goal - have visibility into context window usage to help identify when and why a user might run out of context window and need to compact or start a new session.

The current /usage command does a great job of describing what areas are specifically consuming context.

Current context window (60630 of 200k tokens used)
|██████████████|██████████████████████████████████████████████████████████████████ 30.32%

█ Context files: ~120 tokens (0.06%)
█ Tools: ~35490 tokens (17.75%)
█ Q responses: ~2120 tokens (1.06%)
█ Your prompts: ~22910 tokens (11.45%)

Ideally we'd have logging or other instrumentation that gives this information at meaningful events:

1. Session Startup
2. Context overflow
3. After compacting
4. If it is easy to have this information recorded during the session as well, this could also be useful.

If we can log the number of "turns" or other increments it would help us see if tokens get consumed by large single actions, or a collection of actions.

When this is instrumented and segmented we can start to characterize the "common" experiences that people have.

Things like - are there patterns for people who run out of the context window more? Is it b/c tool responses are chewing up context? Or MCP tool descriptions consume too much? Or users have too many context files?