Chronicle cost tips: parity with CLI /chronicle cost-tips#327175
Draft
cwebster-99 wants to merge 2 commits into
Draft
Chronicle cost tips: parity with CLI /chronicle cost-tips#327175cwebster-99 wants to merge 2 commits into
cwebster-99 wants to merge 2 commits into
Conversation
Restructure the Cost Tips workflow in the chronicle skill so the report
is grounded in observed evidence rather than a generic checklist:
- Require a visible `## Top cost drivers` section before any
recommendations, listing 3-5 material patterns most-impactful first.
- Classify each driver's evidence as exact (cloud usage fields),
proxy (local turns/checkpoints/files/message lengths), or mixed.
Only exact evidence may back a specific token or cost claim.
- Cap `## Recommendations` at 3, each tied to a named driver, each
directly actionable in VS Code Chat (model picker, /compact, fresh
chat, narrower framing, subagent delegation, reusable instructions
or custom skills/agents only when the pattern justifies it). No
CLI- or Coding-Agent-only commands.
- Split `Expected impact` from `Confidence` (high/medium/low).
- Add a required `## Data caveats` disclosure covering
backend/source, time window, in-scope agent_name values and session
count, and fallback/precision limits.
- Gate context-growth findings at an inclusive >=50% late-vs-early
input-token increase; treat smaller growth as noise.
- Treat attachments as neutral work artifacts unless there is concrete
evidence of avoidable repeated ingestion.
- Only include percentages when computable against a real period
grand total (never against a truncated top-N sum); otherwise drop.
- Sparse-data behavior: still fill out Data caveats, skip Top cost
drivers or mark it empty, and offer no more than 2-3 clearly
labeled general habits ("General habit - not observed in your
data:"). Do not invent hard thresholds.
Extend sessionStoreSqlTool.spec.ts anchors for the new prompt
contract so the key requirements stay pinned without overfitting the
full prose.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: fb34298e-e0b6-4e13-bd5e-17621261dfdd
Contributor
There was a problem hiding this comment.
Pull request overview
Adds a structured, evidence-based cost-tips report contract to the Chronicle skill.
Changes:
- Defines cost-driver, recommendation, and caveat sections.
- Adds evidence classifications and anti-fabrication guardrails.
- Extends tests with required prompt anchors.
Show a summary per file
| File | Description |
|---|---|
extensions/copilot/assets/prompts/skills/chronicle/SKILL.md |
Defines the new cost-tips report contract. |
extensions/copilot/src/extension/tools/node/test/sessionStoreSqlTool.spec.ts |
Verifies required contract markers remain present. |
Review details
- Files reviewed: 2/2 changed files
- Comments generated: 4
- Review effort level: Medium
|
|
||
| *1. `## Top cost drivers` — MUST appear before any recommendations.* | ||
|
|
||
| List the top 3-5 material patterns you actually observed, most impactful first. For each driver: |
Member
Author
There was a problem hiding this comment.
Fixed in 2ae0bff — dropped the "3-5" minimum. Section now says "up to 5, fewer is fine (do not pad the list with non-material findings just to reach a count)".
| - Classify the evidence as one of **exact**, **proxy**, or **mixed** and label it inline: | ||
| - **exact** — grounded in cloud `events` fields `usage_input_tokens`, `usage_output_tokens`, `usage_model` (from `assistant.usage` rows). | ||
| - **proxy** — inferred from local turns, checkpoints, session_files, tool_requests, or message lengths only. No real token numbers. | ||
| - **mixed** — a report combining exact and proxy drivers. Label each driver individually. |
Member
Author
There was a problem hiding this comment.
Fixed in 2ae0bff — redefined mixed per-driver (a single driver backed by both exact usage rows and reinforcing proxy signals for the same pattern), and explicitly noted that a report containing one exact driver and one proxy driver is not itself "mixed" — each driver is labeled individually.
|
|
||
| - Only include percentages when you can compute them against a real **period grand total** (e.g. sum of `usage_input_tokens + usage_output_tokens` across every in-scope VS Code Chat `assistant.usage` event in the window). Never compute a share against a truncated top-N sum. If the grand total cannot be established, drop the percentage rather than guessing. | ||
| - A context-growth driver ("input tokens keep climbing turn-over-turn") is only material when the late-window vs early-window input tokens grow by **>=50%** (inclusive). Anything smaller is noise; do not call it out. | ||
| - Treat attachments (`attachments` rows, file references, images) as neutral work artifacts. Do not label attachment presence itself as waste. Only flag them when you have concrete evidence of the same attachment / same file being repeatedly ingested across turns or sessions when it did not need to be. |
Member
Author
There was a problem hiding this comment.
Good catch — fixed in 2ae0bff. Removed the attachments reference entirely and reworded in terms of tables that actually exist: "repeated session_files rows for the same path within one session, or repeated large user_message / user_content payloads containing the same content across turns".
Comment on lines
+232
to
+233
| - Skip `## Top cost drivers` or leave it as "No material drivers observed in this window." | ||
| - In place of grounded recommendations, offer **no more than 2-3** clearly labeled **general habits** (prefix each with "General habit — not observed in your data:"). Keep them anchored to VS Code chat features: model picker for routine work, `/compact` or a fresh chat on long sessions, reusable instructions when a workflow starts to repeat. Do not use arbitrary hard thresholds (turn counts, token counts) that the data cannot support. |
Member
Author
There was a problem hiding this comment.
Fixed in 2ae0bff — sparse path now honors the fixed contract: ## Top cost drivers header stays and is filled with the literal placeholder "No material drivers observed in this window."; the general habits are explicitly identified as "the only recommendations allowed to appear without a matching driver, and only under this sparse-data path" so the sparse-data exception is one non-contradictory instruction set.
Tighten self-consistency of the Cost Tips contract based on review: - Drop the "3-5 drivers" minimum. Cap at 5 with an explicit "fewer is fine — do not pad" so the Top cost drivers section cannot force the model to invent findings when only 1-2 material patterns exist. - Redefine `mixed` per-driver (a single driver backed by both exact usage rows and reinforcing proxy signals for the same pattern), not report-wide. Clarify that a report containing one exact driver and one proxy driver is not itself "mixed" — each driver is labeled individually. - Remove the `attachments` table reference from the neutrality clause. The session-store schema exposes only sessions, turns, session_files, session_refs, checkpoints, events, and tool_requests; the CLI-style "attachments rows" phrasing risked driving the model to a guaranteed-invalid query. Reword in terms of session_files rows and repeated large user_message / user_content payloads. - Reconcile the sparse-data path with the fixed three-section contract. All section headers stay mandatory: Top cost drivers is emitted with a literal "No material drivers observed in this window." placeholder, and the general habits are explicitly called out as the sole exception to the "no recommendation without a named driver" rule so the model has one non-contradictory instruction set. Anchors from sessionStoreSqlTool.spec.ts remain satisfied. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: fb34298e-e0b6-4e13-bd5e-17621261dfdd
Contributor
Screenshot ChangesBase: 1 insignificant change(s) omitted (≤20 px, Δ≤2). See CI logs for details. Added (463)
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adapts the highest-value recommendation-quality improvements recently made to the Copilot CLI
/chronicle cost-tipsinto the VS Code chronicle skill. Preserves existing VS Code Chat scoping, cloud/local backend gating, slash-command behavior, security constraints, and telemetry attribution. Does not touch thesessionStoreSqlAPI and does not introduce a deterministic profile API in this change — that's a separate follow-up.What changes
extensions/copilot/assets/prompts/skills/chronicle/SKILL.md— replaces the freeform "Step 3: Provide tips" section under### Cost Tipswith a fixed report contract:## Top cost drivers— required, must appear before any recommendations. 3-5 material patterns, most-impactful first, each with rough numbers and an evidence class:exact— grounded in cloudeventsfieldsusage_input_tokens/usage_output_tokens/usage_modelonassistant.usagerows.proxy— inferred from local turns, checkpoints, session_files, tool_requests, or message lengths only. No real token numbers.mixed— labeled per-driver when a report combines both.Explicit token or cost claims require exact evidence. Proxies stay described as proxies — turn counts, checkpoint gaps, and message lengths never get translated to tokens or dollars.
## Recommendations— at most 3, ordered by expected impact. Each recommendation:/compact(or a fresh chat when compaction isn't available), narrower task framing / smaller working sets, subagent delegation, reusable.github/copilot-instructions.md/.github/skills//.github/agents/entries — only when a repeated pattern justifies it.Expected impact:(quantify only when evidence isexact) andConfidence:(high/medium/low).## Data caveats— required disclosure so the user can weigh the report: backend/source (cloud DuckDB / local SQLite / mixed), time window, in-scopeagent_namevalues + session count, and fallback/precision limits (e.g. "no per-event token data on local", "cloud store empty in window", "percentages omitted — grand total not computable").Additional guardrails baked into the prompt
usage_input_tokens + usage_output_tokens). Never against a truncated top-N sum. If the grand total can't be computed, the percentage is dropped and called out in caveats.>=50%late-vs-early input-token growth, inclusive. Smaller growth is explicitly called noise and must not be listed.Sparse / insufficient data
Hardened so the model doesn't invent findings when the store is thin:
## Data caveats.## Top cost driversor leaves it as "No material drivers observed in this window."General habit — not observed in your data:items, anchored to VS Code chat features (model picker for routine work,/compactor fresh chat on long sessions, reusable instructions when a workflow starts to repeat). No arbitrary hard thresholds.chat.sessionSync.enabledupsell.Preserved behavior
'VS Code Chat'on cloud,'GitHub Copilot Chat'on local, with the mix-check offer to widen toCopilot CLI/Copilot Coding Agent/ subagents on request).eventstable; token-level analysis requires cloud sync).chat.sessionSync.enabledupsell for local users./chronicle cost-tips) and existing Step 1 (Investigate) / Step 2 (Map to features) guidance.sessionStoreSqltool API is untouched. No deterministic profile helper in this change — deferred by design.Tests
extensions/copilot/src/extension/tools/node/test/sessionStoreSqlTool.spec.ts— the existing'chronicle SKILL.md Cost Tips section carries required anchors'test is extended with anchors for the new contract markers so the shape stays pinned without overfitting the whole prose:## Top cost drivers,## Recommendations,## Data caveatsat most 3,Expected impact:,Confidence:**exact**,**proxy**,**mixed**period grand total,>=50%,neutral work artifacts,General habit — not observed in your data:,Never fabricateValidation
precommithygiene hook in the authoring environment: the private npm feed (packagefeedproxy.microsoft.io) is unreachable from this machine so rootnode_modulescannot be populated (event-streammissing whennode build/hygiene.tsruns). Structurally the diff is hygiene-safe by construction —build/filters.tsexcludesextensions/copilot/**fromindentationFilterand excludes**/*.mdfromcopyrightFilter, and neither file's copyright header, tab indentation, or existing code style was modified. CI on this PR should be the real signal.Follow-ups (not in this PR)
### Tips(non-cost) workflow in the chronicle skill.sessionStoreSql(top drivers, grand-total computation, sparse-data flag) that the prompt can consume instead of asking the model to hand-roll aggregates every time.