Add tool call analysis and reporting to logs command with configurable detail levels by Copilot · Pull Request #525 · github/gh-aw

Copilot · 2025-09-07T13:56:36Z

Enhances the logs command to parse and display tool usage statistics from agentic workflow logs, providing detailed insights into which specific tools are being used and how frequently. The command now supports both grouped and detailed views through a --verbose flag.

What's New

The logs command now displays a tool usage table with two display modes:

Normal Mode (Default)

Grouped by MCP server (e.g., github for all GitHub tools)
Individual bash commands for granular command tracking
Clean high-level view of tool usage patterns

Verbose Mode (`-v` or `--verbose`)

Individual tool names (e.g., github_search_issues, playwright_browser_take_screenshot) for granular analysis
Number of calls across all workflow runs
Maximum output size in tokens for any single call

Example Output

Normal mode:

Tool                        Calls  Max Output (tokens)
github                      5      125
playwright                  2      45
bash_git_status             1      12

Verbose mode:

Tool                              Calls  Max Output (tokens)
github_search_issues              3      125
github_list_issues                2      89
playwright_browser_take_screenshot 2     45
bash_git_status                   1      12
bash_ls_tmp                       1      8

Key Features

Configurable Detail Levels

Normal mode: Groups MCP tools by server for a clean overview
Verbose mode: Shows individual prettified tool names for detailed analysis
Flexible usage: Choose the right level of detail for your analysis needs

Tool Name Prettification Strategy

Avoids colons in prettified names (no more :: separators)
Leaves underscores as-is for better readability
MCP server grouping: mcp__github__search_issues → grouped under github (normal) or github_search_issues (verbose)

Special Bash Handling

Each bash invocation is treated as a unique tool to prevent aggregation of different commands:

bash_git_status
bash_ls_-la_tmp
bash_npm_install

This provides granular insights into command usage patterns.

Multi-Engine Support

Claude Engine: Parses structured JSON logs to extract tool_use entries
Codex Engine: Uses regex patterns to parse text-based tool and exec commands
Custom Engine: Automatically delegates to appropriate parser based on log format

Implementation Details

Core Components

Extended LogMetrics struct with ToolCalls []ToolCallInfo
Added ToolCallInfo struct to track tool statistics
Implemented PrettifyToolName() function for consistent formatting without colons
Updated displayToolCallReport() to support both grouped and individual display modes
Added --verbose flag to control display granularity

Engine Integration

Modified ParseLogMetrics interface to include tool call extraction
Updated Claude engine to parse JSON log arrays for tool usage
Enhanced Codex engine to extract tool calls from log lines with underscore formatting
Maintained backward compatibility with existing metric extraction

Testing

Updated unit tests for PrettifyToolName to match new underscore format
Validated parsing with sample Claude logs
Verified integration with existing logs workflow
All existing tests continue to pass

Usage

The tool call report appears automatically when running the logs command:

gh aw logs                    # Shows grouped tool usage (default)
gh aw logs -v                 # Shows individual tool names (verbose)
gh aw logs --verbose          # Same as -v, shows detailed tool breakdown
gh aw logs weekly-research -v # Shows detailed tools for specific workflow
gh aw logs --engine claude -v # Filter by engine with detailed view
gh aw logs -c 5 -v           # Last 5 runs with detailed tool breakdown

The table is displayed after the standard workflow overview and before access log analysis, providing complete visibility into tool usage patterns across workflow executions at your preferred level of detail.

✨ Let Copilot coding agent set things up for you — coding agent works faster and does higher quality work when set up for your repo.

- Extend LogMetrics with ToolCallInfo to track tool usage statistics - Add PrettifyToolName function to remove "mcp__" prefixes and format names - Implement tool parsing in Claude engine with JSON log analysis - Implement tool parsing in Codex engine with regex-based log parsing - Add displayToolCallReport function to show compact table without title - Special handling for bash commands (unique per invocation) - Add test coverage for PrettifyToolName function - Integrate tool report into logs command output flow Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

- Fix main ParseLogMetrics function to copy ToolCalls from parseClaudeJSONLog - Tool parsing now correctly identifies and prettifies tool names - Special handling for bash commands works (unique per invocation) - MCP tools properly formatted (mcp__github__search_issues -> github::search_issues) - Tool call report table integration fully functional Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

- All tests pass (including new PrettifyToolName test) - Build, lint, format validations successful - Workflow recompilation successful - Agent-finish validation complete ✅ Tool call report feature is fully implemented and ready for use. Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

pelikhan · 2025-09-07T15:15:46Z

@copilot

use console helper to create table.
group tool calls by MCP server and update name pretification strategy accordingly
avoid : in pretified name, leave _ as is

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Copilot · 2025-09-07T15:29:08Z

@copilot

use console helper to create table.

group tool calls by MCP server and update name pretification strategy accordingly...

Updated the tool call reporting implementation as requested:

✅ Using console helper for table - Already implemented with console.RenderTable()
✅ Grouping by MCP server - Tools like mcp__github__search_issues and mcp__github__list_issues are now grouped under github
✅ Updated name prettification - No more colons (:) in names, underscores (_) are preserved as-is

Key changes made:

PrettifyToolName(): mcp__github__search_issues → github_search_issues (no colons)
ExtractMCPServer(): Groups MCP tools by server name (e.g., all GitHub tools → github)
Bash commands remain individual with underscores: bash_git_status, bash_ls_-la
Updated both Claude and Codex engines to use underscore formatting

The tool table now shows server-level aggregations instead of individual tool calls, providing a cleaner high-level view while preserving granular bash command tracking.

Commit: 6d0dd02

Warning

Firewall rules blocked me from connecting to one or more addresses (expand for details)

I tried to connect to the following addresses, but was blocked by firewall rules:

api.contoso.com
- Triggering command: /tmp/go-build2517981262/b278/cli.test -test.testlogfile=/tmp/go-build2517981262/b278/testlog.txt -test.paniconexit0 -test.timeout=10m0s -test.v=true (dns block)
- Triggering command: /tmp/go-build2217359838/b278/cli.test -test.testlogfile=/tmp/go-build2217359838/b278/testlog.txt -test.paniconexit0 -test.timeout=10m0s -test.v=true (dns block)
- Triggering command: /tmp/go-build2786535636/b278/cli.test -test.testlogfile=/tmp/go-build2786535636/b278/testlog.txt -test.paniconexit0 -test.timeout=10m0s -test.v=true (dns block)

If you need me to access, download, or install something from one of these locations, you can either:

Configure Actions setup steps to set up my environment, which run before the firewall is enabled
Add the appropriate URLs or hosts to the custom allowlist in this repository's Copilot coding agent settings (admins only)

pelikhan · 2025-09-07T16:53:19Z

@copilot show full tool list in verbose mode

- Added --verbose/-v flag to logs command - Modified displayToolCallReport to show individual tool names in verbose mode - Non-verbose mode: Groups tools by MCP server (e.g., "github", "playwright") - Verbose mode: Shows individual prettified tool names (e.g., "github_search_issues", "playwright_browser_take_screenshot") - Updated flag descriptions and help text Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Copilot · 2025-09-07T17:03:37Z

@copilot show full tool list in verbose mode

Added verbose mode support to the logs command. When using the --verbose or -v flag, the tool call report now shows individual prettified tool names instead of grouping by MCP server.

Normal mode (grouped):

Tool                        Calls  Max Output (tokens)
github                      5      125
playwright                  2      45

Verbose mode (individual tools):

Tool                              Calls  Max Output (tokens)
github_search_issues              3      125
github_list_issues                2      89
playwright_browser_take_screenshot 2     45

Usage: gh aw logs --verbose or gh aw logs -v

Commit: e479113

Warning

Firewall rules blocked me from connecting to one or more addresses (expand for details)

I tried to connect to the following addresses, but was blocked by firewall rules:

api.contoso.com
- Triggering command: /tmp/go-build2260278088/b278/cli.test -test.testlogfile=/tmp/go-build2260278088/b278/testlog.txt -test.paniconexit0 -test.timeout=10m0s -test.v=true (dns block)
- Triggering command: /tmp/go-build2004577536/b278/cli.test -test.testlogfile=/tmp/go-build2004577536/b278/testlog.txt -test.paniconexit0 -test.timeout=10m0s -test.v=true (dns block)

If you need me to access, download, or install something from one of these locations, you can either:

Configure Actions setup steps to set up my environment, which run before the firewall is enabled
Add the appropriate URLs or hosts to the custom allowlist in this repository's Copilot coding agent settings (admins only)

pelikhan · 2025-09-07T17:14:41Z

pkg/cli/logs.go

 	logsCmd.Flags().String("end-date", "", "Filter runs created before this date (YYYY-MM-DD or delta like -1d, -1w, -1mo)")
 	logsCmd.Flags().StringP("output", "o", "./logs", "Output directory for downloaded logs and artifacts")
 	logsCmd.Flags().String("engine", "", "Filter logs by agentic engine type (claude, codex)")
+	logsCmd.Flags().BoolP("verbose", "v", false, "Show detailed output including individual tool names")


@copilot remove flag, always show them verbose tool list

Removed the verbose flag from the logs command. The tool call report now always shows individual tool names (like github_search_issues, playwright_browser_take_screenshot) instead of grouping by MCP server, providing detailed insights into specific tool usage patterns.

Commit: 7da46dd

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Copilot · 2025-09-07T17:29:05Z

Just as a heads up, I was blocked by some firewall rules while working on your feedback. Expand below for details.

Warning

Firewall rules blocked me from connecting to one or more addresses (expand for details)

I tried to connect to the following addresses, but was blocked by firewall rules:

api.contoso.com
- Triggering command: /tmp/go-build3241652884/b278/cli.test -test.testlogfile=/tmp/go-build3241652884/b278/testlog.txt -test.paniconexit0 -test.timeout=10m0s -test.v=true (dns block)

If you need me to access, download, or install something from one of these locations, you can either:

Configure Actions setup steps to set up my environment, which run before the firewall is enabled
Add the appropriate URLs or hosts to the custom allowlist in this repository's Copilot coding agent settings (admins only)

pelikhan · 2025-09-07T17:30:53Z

pkg/cli/logs.go

 			endDate, _ := cmd.Flags().GetString("end-date")
 			outputDir, _ := cmd.Flags().GetString("output")
 			engine, _ := cmd.Flags().GetString("engine")
-			verbose, _ := cmd.Flags().GetBool("verbose")


@copilot keep verbose flag

Added the verbose flag back to the logs command. The flag is now parsed at line 167 and allows users to control tool display mode: grouped by MCP server (default) vs individual tool names (verbose).

Commit: f70b396

pelikhan · 2025-09-07T17:37:18Z

pkg/cli/logs.go

 			}

-			if err := DownloadWorkflowLogs(workflowName, count, startDate, endDate, outputDir, engine, verbose); err != nil {
+			if err := DownloadWorkflowLogs(workflowName, count, startDate, endDate, outputDir, engine, true); err != nil {


@copilot pass verbose flag

Updated to pass the verbose flag value instead of hardcoding true. Now the actual verbose flag state is passed to DownloadWorkflowLogs at line 207, giving users control over the tool call report detail level.

Commit: f70b396

…dWorkflowLogs Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Copilot · 2025-09-07T17:47:17Z

Just as a heads up, I was blocked by some firewall rules while working on your feedback. Expand below for details.

Warning

Firewall rules blocked me from connecting to one or more addresses (expand for details)

I tried to connect to the following addresses, but was blocked by firewall rules:

api.contoso.com
- Triggering command: /tmp/go-build909407603/b278/cli.test -test.testlogfile=/tmp/go-build909407603/b278/testlog.txt -test.paniconexit0 -test.timeout=10m0s -test.v=true (dns block)
- Triggering command: /tmp/go-build2571148843/b278/cli.test -test.testlogfile=/tmp/go-build2571148843/b278/testlog.txt -test.paniconexit0 -test.timeout=10m0s -test.v=true (dns block)

If you need me to access, download, or install something from one of these locations, you can either:

Configure Actions setup steps to set up my environment, which run before the firewall is enabled
Add the appropriate URLs or hosts to the custom allowlist in this repository's Copilot coding agent settings (admins only)

Initial plan

4cfcba9

Copilot AI assigned Copilot and pelikhan Sep 7, 2025

Copilot started work on behalf of pelikhan September 7, 2025 13:56 View session

Copilot AI and others added 3 commits September 7, 2025 14:15

Copilot AI requested a review from pelikhan September 7, 2025 14:33

Copilot finished work on behalf of pelikhan September 7, 2025 14:33

Copilot started work on behalf of pelikhan September 7, 2025 15:15 View session

Update tool call reporting to group by MCP server and avoid colons

6d0dd02

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Copilot AI changed the title ~~Add tool call analysis and reporting to logs command~~ Add tool call analysis and reporting to logs command with MCP server grouping Sep 7, 2025

Copilot finished work on behalf of pelikhan September 7, 2025 15:30

Copilot started work on behalf of pelikhan September 7, 2025 16:53 View session

Copilot finished work on behalf of pelikhan September 7, 2025 17:04

pelikhan approved these changes Sep 7, 2025

View reviewed changes

pelikhan marked this pull request as ready for review September 7, 2025 17:12

pelikhan reviewed Sep 7, 2025

View reviewed changes

Copilot started work on behalf of pelikhan September 7, 2025 17:14 View session

Remove verbose flag from logs command, always show individual tool names

56139db

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Copilot AI changed the title ~~Add tool call analysis and reporting to logs command with MCP server grouping~~ Add tool call analysis and reporting to logs command with detailed tool breakdown Sep 7, 2025

Copilot AI requested a review from pelikhan September 7, 2025 17:29

Copilot finished work on behalf of pelikhan September 7, 2025 17:29

pelikhan reviewed Sep 7, 2025

View reviewed changes

Copilot started work on behalf of pelikhan September 7, 2025 17:37 View session

Add verbose flag back to logs command and pass it properly to Downloa…

f70b396

…dWorkflowLogs Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Copilot AI changed the title ~~Add tool call analysis and reporting to logs command with detailed tool breakdown~~ Add tool call analysis and reporting to logs command with configurable detail levels Sep 7, 2025

Copilot AI requested a review from pelikhan September 7, 2025 17:47

Copilot finished work on behalf of pelikhan September 7, 2025 17:47

pelikhan merged commit f103d14 into main Sep 7, 2025
22 checks passed

pelikhan deleted the copilot/update-logs-command-report-table branch September 7, 2025 21:36

Conversation

Copilot AI commented Sep 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What's New

Normal Mode (Default)

Verbose Mode (-v or --verbose)

Example Output

Key Features

Configurable Detail Levels

Tool Name Prettification Strategy

Special Bash Handling

Multi-Engine Support

Implementation Details

Core Components

Engine Integration

Testing

Usage

Uh oh!

pelikhan commented Sep 7, 2025

Uh oh!

Copilot AI commented Sep 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

I tried to connect to the following addresses, but was blocked by firewall rules:

Uh oh!

pelikhan commented Sep 7, 2025

Uh oh!

Copilot AI commented Sep 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

I tried to connect to the following addresses, but was blocked by firewall rules:

Uh oh!

pelikhan Sep 7, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Sep 7, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI commented Sep 7, 2025

I tried to connect to the following addresses, but was blocked by firewall rules:

Uh oh!

pelikhan Sep 7, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Sep 7, 2025

Choose a reason for hiding this comment

Uh oh!

pelikhan Sep 7, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Sep 7, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI commented Sep 7, 2025

I tried to connect to the following addresses, but was blocked by firewall rules:

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Copilot AI commented Sep 7, 2025 •

edited

Loading

Verbose Mode (`-v` or `--verbose`)

Copilot AI commented Sep 7, 2025 •

edited

Loading

Copilot AI commented Sep 7, 2025 •

edited

Loading