[AHP/CLI:COGS] Grep Search Tool Output Optimization

[bhavyaus]

The agent grep_search (findTextInFiles) tool was returning verbose, XML‑tagged results that dominated its tool‑call token usage. PRs #321601 and #321938 introduce a compact, grep‑style output format behind an experiment. Early results show the new format roughly halves the tokens returned by the tool with no loss of useful signal — a direct COGS win as we move CLI/AHP to the default agent.

What changed

• New compact output format — A grep-style result (plain path followed by line:text, blank‑line separated) replaces the previous tag‑wrapped output. Gated by the experiment‑based setting github.copilot.chat.tools.grepSearch.outputFormat (tag | grep, default tag).
  • Optimize grep output and enhance configuration #321601 — Optimize grep output and enhance configuration
• Token‑budget‑aware rendering + long‑line handling — Results render through prompt‑tsx, so they participate in token‑budget pruning (earlier files prioritized) and carry editor references instead of a single unprunable blob. Long lines are truncated to a max length with context kept around the match.
  • Fix long line rendering for grep style output #321938 — Fix long line rendering for grep style output
• Smarter result capping — The maxResults budget is distributed proportionally across matching files (largest‑remainder, min one match per file), and paths are returned workspace‑relative.

Experiment

https://exp.microsoft.com/features/30687fdb-eb7c-dbe8-06c4-02c5ece0671a/progressions?workspaceId=dbd12384-5001-4e90-a863-526321eaf233

agent.tool.responselength — treatment enables grepSearch.outputFormat = grep via the github.copilot-chat/agent.tool.responselength flag.

Tokens returned by grep_search (control = tag, treatment = grep)

Group	Events	Total Tokens	Avg	P50	P95	P99
control	363,400	421,529,887	1,160	321.2	5,275.4	13,091.6
treatment	389,097	200,998,321	516.6	137.7	2,123.6	6,380.8

• ~55% fewer tokens per call on average (1,160 → 517); P95 down ~60%, P99 down ~51%.
• Total tokens dropped from 421.5M → 201.0M, despite treatment serving more calls (389K vs 363K events).
• PValue = 0 — statistically significant.

---

Context: Part of the CLI/AHP‑as‑default migration. Tracking given the direct COGS impact of tool response length.

[dbaeumer]

Unfortunately, the score card computation failed. So here are some results directly compute on the telemetry events:

Token saving on the grep_search tool

The token count metric with variance and pValue for the grep_search

The response success metrics with variance and pValue for requests that had at least one grep_search tool call.

[dbaeumer]

And here the cogs impact of the experiment

[dbaeumer]

@bhavyaus since I am new to this what is the usual way to continue?

[dbaeumer]

Stats from today:

[dbaeumer]

Here is the 4 day score card:

However all values are not highly stat sig. The reason for this is that the score card can't be filter by harness (local) and the grep_search tool is only called in the local harness. Hence events may end up in the treatment group although they are not treated. The queries run on the Kusto events directly will not have that problem since they filter the response events accordingly.

[dbaeumer]

Including TTFT in ms

[dbaeumer]

Cancellation rate:

[dbaeumer]

Accepted / Rejected per session

[dbaeumer]

How are these values computed:

• treatment group: all github.copilot-chat/findtextinfilestoolinvoked (this is the grep_search event) requestIds that where in the treatment. Same for control
• looked at all github.copilot-chat/response.success events for requestId in treatment and control. Note that this is not for a single reponse for greap_search, it contains all responses from a prompt request that had at least on greap_search in it. Computed input, cache, output and total tokens for the whole prompt and compared.
• same approach for cancellation rate, accepted and rejected code per session.

Here is query used for the token computation:

let startTimestamp = ago(7d);
let Opportunities = RawEventsVSCodeExt
| where ServerTimestamp > startTimestamp
| where EventName == "github.copilot-chat/findtextinfilestoolinvoked"
| where ExtensionName == "GitHub.copilot-chat"
| extend ABExp = tostring(Properties["abexp.assignmentcontext"])
| extend RequestId = tostring(Properties["requestid"])
| summarize
    Control = make_set_if(RequestId, ABExp has_any ("52ge9494:31538318")),
    Treatment = make_set_if(RequestId, ABExp has_any ("h7dd7680:31538319"));
let ControlOpportunities = toscalar(Opportunities | project Control);
let TreatmentOpportunities = toscalar(Opportunities | project Treatment);
RawEventsVSCodeExt
| where ServerTimestamp > startTimestamp
| where EventName == "github.copilot-chat/response.success"
| extend ABExp = tostring(Properties["abexp.assignmentcontext"])
| extend isControl = ABExp has_any ("52ge9494:31538318")
| extend isTreatment = ABExp has_any ("h7dd7680:31538319")
| where isTreatment or isControl
| extend
    id = tostring(Properties["requestid"])
| where iff(isControl, id in (ControlOpportunities), id in (TreatmentOpportunities))
| extend
    Source = tostring(Properties["source"])
| where Source == "chat.editAgent"
| extend
    Model = tostring(Properties["model"]),
    TotalInputTokens = toreal(Measures["prompttokencount"]),
    CacheTokens = toreal(Measures["promptcachetokencount"]),
    OutputTokens = toreal(Measures["completiontokens"]),
    TotalGenerated = toreal(Measures["tokencount"]),
    TimeToFirstToken = toreal(Measures["timetofirsttoken"])
| extend
	InputTokens = TotalInputTokens - CacheTokens,
    Group = iff(isTreatment, "treatment", "control")
| summarize
    Events = count(),
    DistinctRequests = dcount(id),
    InputTokens = sum(InputTokens),
    AvgInputTokens = avg(InputTokens),
    VarInputTokens = variance(InputTokens),
    TotalCacheTokens = sum(CacheTokens),
    AvgTotalCacheTokens = avg(CacheTokens),
    VarTotalCacheTokens = variance(CacheTokens),
    TotalOutputTokens = sum(OutputTokens),
    AvgTotalOutputTokens = avg(OutputTokens),
    VarTotalOutputTokens = variance(OutputTokens),
    TotalInputTokens = sum(TotalInputTokens),
    AvgTotalInputTokens = avg(TotalInputTokens),
    VarTotalInputTokens = variance(TotalInputTokens),
    TotalGeneratedTokens = sum(TotalGenerated),
    AvgTotalGeneratedTokens = avg(TotalGenerated),
    VarTotalGeneratedTokens = variance(TotalGenerated),
    TotalTimeToFirstToken = sum(TimeToFirstToken),
    AvgTimeToFirstToken = avg(TimeToFirstToken),
    VarTimeToFirstToken = variance(TimeToFirstToken)
    by Group
| summarize
    ControlEvents = maxif(Events, Group == "control"),
    TreatmentEvents = maxif(Events, Group == "treatment"),
    ControlInputTokens = maxif(InputTokens, Group == "control"),
    TreatmentInputTokens = maxif(InputTokens, Group == "treatment"),
    ControlAvgInputTokens = maxif(AvgInputTokens, Group == "control"),
    TreatmentAvgInputTokens = maxif(AvgInputTokens, Group == "treatment"),
    ControlVarInputTokens = maxif(VarInputTokens, Group == "control"),
    TreatmentVarInputTokens = maxif(VarInputTokens, Group == "treatment"),
    ControlTotalCacheTokens = maxif(TotalCacheTokens, Group == "control"),
    TreatmentTotalCacheTokens = maxif(TotalCacheTokens, Group == "treatment"),
    ControlAvgTotalCacheTokens = maxif(AvgTotalCacheTokens, Group == "control"),
    TreatmentAvgTotalCacheTokens = maxif(AvgTotalCacheTokens, Group == "treatment"),
    ControlVarTotalCacheTokens = maxif(VarTotalCacheTokens, Group == "control"),
    TreatmentVarTotalCacheTokens = maxif(VarTotalCacheTokens, Group == "treatment"),
    ControlTotalOutputTokens = maxif(TotalOutputTokens, Group == "control"),
    TreatmentTotalOutputTokens = maxif(TotalOutputTokens, Group == "treatment"),
    ControlAvgTotalOutputTokens = maxif(AvgTotalOutputTokens, Group == "control"),
    TreatmentAvgTotalOutputTokens = maxif(AvgTotalOutputTokens, Group == "treatment"),
    ControlVarTotalOutputTokens = maxif(VarTotalOutputTokens, Group == "control"),
    TreatmentVarTotalOutputTokens = maxif(VarTotalOutputTokens, Group == "treatment"),
    ControlTotalInputTokens = maxif(TotalInputTokens, Group == "control"),
    TreatmentTotalInputTokens = maxif(TotalInputTokens, Group == "treatment"),
    ControlAvgTotalInputTokens = maxif(AvgTotalInputTokens, Group == "control"),
    TreatmentAvgTotalInputTokens = maxif(AvgTotalInputTokens, Group == "treatment"),
    ControlVarTotalInputTokens = maxif(VarTotalInputTokens, Group == "control"),
    TreatmentVarTotalInputTokens = maxif(VarTotalInputTokens, Group == "treatment"),
    ControlTotalGeneratedTokens = maxif(TotalGeneratedTokens, Group == "control"),
    TreatmentTotalGeneratedTokens = maxif(TotalGeneratedTokens, Group == "treatment"),
    ControlAvgTotalGeneratedTokens = maxif(AvgTotalGeneratedTokens, Group == "control"),
    TreatmentAvgTotalGeneratedTokens = maxif(AvgTotalGeneratedTokens, Group == "treatment"),
    ControlVarTotalGeneratedTokens = maxif(VarTotalGeneratedTokens, Group == "control"),
    TreatmentVarTotalGeneratedTokens = maxif(VarTotalGeneratedTokens, Group == "treatment"),
    ControlTotalTimeToFirstToken = maxif(TotalTimeToFirstToken, Group == "control"),
    TreatmentTotalTimeToFirstToken = maxif(TotalTimeToFirstToken, Group == "treatment"),
    ControlAvgTimeToFirstToken = maxif(AvgTimeToFirstToken, Group == "control"),
    TreatmentAvgTimeToFirstToken = maxif(AvgTimeToFirstToken, Group == "treatment"),
    ControlVarTimeToFirstToken = maxif(VarTimeToFirstToken, Group == "control"),
    TreatmentVarTimeToFirstToken = maxif(VarTimeToFirstToken, Group == "treatment")
| extend
    Metric = dynamic(["InputTokens", "TotalCacheTokens", "TotalOutputTokens", "TotalInputTokens", "TotalGeneratedTokens", "AvgTimeToFirstTokenMs"]),
    Control = pack_array(ControlInputTokens, ControlTotalCacheTokens, ControlTotalOutputTokens, ControlTotalInputTokens, ControlTotalGeneratedTokens, ControlAvgTimeToFirstToken),
    Treatment = pack_array(TreatmentInputTokens, TreatmentTotalCacheTokens, TreatmentTotalOutputTokens, TreatmentTotalInputTokens, TreatmentTotalGeneratedTokens, TreatmentAvgTimeToFirstToken),
    ControlMean = pack_array(ControlAvgInputTokens, ControlAvgTotalCacheTokens, ControlAvgTotalOutputTokens, ControlAvgTotalInputTokens, ControlAvgTotalGeneratedTokens, ControlAvgTimeToFirstToken),
    TreatmentMean = pack_array(TreatmentAvgInputTokens, TreatmentAvgTotalCacheTokens, TreatmentAvgTotalOutputTokens, TreatmentAvgTotalInputTokens, TreatmentAvgTotalGeneratedTokens, TreatmentAvgTimeToFirstToken),
    ControlVariance = pack_array(ControlVarInputTokens, ControlVarTotalCacheTokens, ControlVarTotalOutputTokens, ControlVarTotalInputTokens, ControlVarTotalGeneratedTokens, ControlVarTimeToFirstToken),
    TreatmentVariance = pack_array(TreatmentVarInputTokens, TreatmentVarTotalCacheTokens, TreatmentVarTotalOutputTokens, TreatmentVarTotalInputTokens, TreatmentVarTotalGeneratedTokens, TreatmentVarTimeToFirstToken)
| mv-expand
    Metric to typeof(string),
    Control to typeof(real),
    Treatment to typeof(real),
    ControlMean to typeof(real),
    TreatmentMean to typeof(real),
    ControlVariance to typeof(real),
    TreatmentVariance to typeof(real)
| extend PValue = welch_test(TreatmentMean, TreatmentVariance, TreatmentEvents, ControlMean, ControlVariance, ControlEvents)
| project
    Metric,
    Control,
    Treatment,
    PctDiff = round(100.0 * (Treatment - Control) / Control, 1),
    ControlVariance = round(ControlVariance, 1),
    TreatmentVariance = round(TreatmentVariance, 1),
    PValue = round(PValue, 4)

[dbaeumer]

First results including 50%/50% stable progression

General control / treatment size

The response length of the grep_search

Token consumption

Cancellation rate

Accepted / Rejected

[dbaeumer]