Skip to content

fix(daily-syntax-error-quality): unblock /tmp/gh-aw compile path, remove unused GitHub toolset#24925

Merged
pelikhan merged 3 commits intomainfrom
copilot/optimize-daily-syntax-error-check
Apr 6, 2026
Merged

fix(daily-syntax-error-quality): unblock /tmp/gh-aw compile path, remove unused GitHub toolset#24925
pelikhan merged 3 commits intomainfrom
copilot/optimize-daily-syntax-error-check

Conversation

Copy link
Copy Markdown
Contributor

Copilot AI commented Apr 6, 2026

The workflow was consuming ~9.3M tokens/run (~130 turns) because every gh aw compile attempt was denied — the agent copies test files to /tmp/gh-aw/syntax-error-tests/ and invokes the compiler with absolute paths, but only the relative glob gh aw compile * was permitted. This triggered a 64-turn source-code reading fallback. Expected cost: ~15–20 turns (~1.1–1.4M tokens).

Changes

  • Add /tmp/gh-aw compile path"gh aw compile /tmp/gh-aw/syntax-error-tests/*.md" added to bash tool allow-list, unblocking the agent's actual compile workflow
  • Remove github toolset — the workflow does purely local file ops + compile; the default toolset schema was loaded into every turn's system prompt for no reason (~130–260K tokens/run)
  • Align cp/cat/mkdir patterns — updated to use the /tmp/gh-aw/syntax-error-tests/ subdirectory (project-standard /tmp/gh-aw/ root) consistently with the Phase 2 prompt instructions; added mkdir -p /tmp/gh-aw/syntax-error-tests as an explicit allowed command
  • Update prompt — all prompt references to the temp directory now point to /tmp/gh-aw/syntax-error-tests/
# Before
tools:
  github:
    toolsets:
      - default
  bash:
    - "gh aw compile *"
    - "cp .github/workflows/*.md /tmp/*.md"
    - "cat /tmp/*.md"

# After
tools:
  bash:
    - "gh aw compile *"
    - "gh aw compile /tmp/gh-aw/syntax-error-tests/*.md"
    - "cp .github/workflows/*.md /tmp/gh-aw/syntax-error-tests/*.md"
    - "cat /tmp/gh-aw/syntax-error-tests/*.md"
    - "mkdir -p /tmp/gh-aw/syntax-error-tests"

…aily-syntax-error-quality

- Add 'gh aw compile /tmp/syntax-error-tests/*.md' bash tool pattern so the
  agent can compile test files at absolute paths (fixes root cause of 84%
  wasted tokens: 64-turn source-code fallback when compile was blocked)
- Remove unused 'github' toolset (saves ~130-260K tokens/run by not loading
  GitHub tool schemas into every turn's system prompt)
- Update cp/cat patterns to use /tmp/syntax-error-tests/ subdirectory,
  consistent with the Phase 2 instructions in the prompt
- Add explicit 'mkdir -p /tmp/syntax-error-tests' allowed bash command

Addresses: https://github.com/github/gh-aw/issues/...

Agent-Logs-Url: https://github.com/github/gh-aw/sessions/45651d47-88d4-4a14-a1e3-1f587a699977

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Copilot AI changed the title [WIP] Fix daily syntax error quality check for token optimization fix(daily-syntax-error-quality): unblock /tmp compile path, remove unused GitHub toolset Apr 6, 2026
Copilot AI requested a review from pelikhan April 6, 2026 16:39
@pelikhan
Copy link
Copy Markdown
Collaborator

pelikhan commented Apr 6, 2026

@copilot use /tmp/gh-aw/... as a root for temp folder . Update prompt.

Change temp directory from /tmp/syntax-error-tests to /tmp/gh-aw/syntax-error-tests
in both the bash tool allow-list patterns and the prompt instructions.

Agent-Logs-Url: https://github.com/github/gh-aw/sessions/0e153d47-75fe-4716-a0da-39512e826acc

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Copy link
Copy Markdown
Contributor Author

Copilot AI commented Apr 6, 2026

@copilot use /tmp/gh-aw/... as a root for temp folder . Update prompt.

Done in bce4f1b. Updated all /tmp/syntax-error-tests references to /tmp/gh-aw/syntax-error-tests in both the bash tool allow-list and the prompt instructions.

Warning

Firewall rules blocked me from connecting to one or more addresses (expand for details)

I tried to connect to the following addresses, but was blocked by firewall rules:

  • https://api.github.com/graphql
    • Triggering command: /usr/bin/gh /usr/bin/gh api graphql -f query=query($owner: String!, $name: String!) { repository(owner: $owner, name: $name) { hasDiscussionsEnabled } } -f owner=github -f name=gh-aw (http block)
    • Triggering command: /usr/bin/gh /usr/bin/gh api graphql -f query=query($owner: String!, $name: String!) { repository(owner: $owner, name: $name) { hasDiscussionsEnabled } } -f owner=github -f name=gh-aw mz9FZDb/Mkyvi_yn-1 (http block)
  • https://api.github.com/repos/astral-sh/setup-uv/git/ref/tags/eac588ad8def6316056a12d4907a9d4d84ff7a3b
    • Triggering command: /usr/bin/gh gh api /repos/astral-sh/setup-uv/git/ref/tags/eac588ad8def6316056a12d4907a9d4d84ff7a3b --jq .object.sha (http block)
  • https://api.github.com/repos/github/gh-aw
    • Triggering command: /usr/bin/gh gh api /repos/github/gh-aw --jq .visibility (http block)
  • https://api.github.com/repos/githubnext/agentics/git/ref/tags/-
    • Triggering command: /usr/bin/gh gh api /repos/githubnext/agentics/git/ref/tags/- --jq .object.sha (http block)

If you need me to access, download, or install something from one of these locations, you can either:

Copilot AI changed the title fix(daily-syntax-error-quality): unblock /tmp compile path, remove unused GitHub toolset fix(daily-syntax-error-quality): unblock /tmp/gh-aw compile path, remove unused GitHub toolset Apr 6, 2026
@pelikhan pelikhan marked this pull request as ready for review April 6, 2026 16:51
Copilot AI review requested due to automatic review settings April 6, 2026 16:51
@pelikhan pelikhan merged commit 0c76243 into main Apr 6, 2026
56 of 57 checks passed
@pelikhan pelikhan deleted the copilot/optimize-daily-syntax-error-check branch April 6, 2026 16:51
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR updates the daily-syntax-error-quality agentic workflow configuration to reduce token/turn cost by unblocking the compiler’s actual /tmp/gh-aw/... usage and removing unnecessary GitHub toolset configuration from the workflow frontmatter.

Changes:

  • Allow gh aw compile against /tmp/gh-aw/syntax-error-tests/*.md and align temp-file operations under /tmp/gh-aw/syntax-error-tests/.
  • Remove the tools.github.toolsets: [default] block from the workflow frontmatter to avoid loading unused GitHub tool schemas.
  • Regenerate the compiled lock workflow to reflect updated allow-listed shell commands.
Show a summary per file
File Description
.github/workflows/daily-syntax-error-quality.md Updates tool allow-list and prompt instructions to use /tmp/gh-aw/syntax-error-tests/ and allow absolute-path compilation.
.github/workflows/daily-syntax-error-quality.lock.yml Regenerates the rendered workflow and updates --allow-tool entries for the new /tmp/gh-aw/... paths.

Copilot's findings

Tip

Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

  • Files reviewed: 2/2 changed files
  • Comments generated: 3

Comment on lines +189 to 190
cd /tmp/gh-aw/syntax-error-tests
gh aw compile test-1.md 2>&1 | tee test-1-output.txt
Copy link

Copilot AI Apr 6, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The instructions tell the agent to run cd /tmp/gh-aw/syntax-error-tests, but cd is not in the allowed bash tool list for this workflow (strict mode). This can cause tool denials if the agent follows the prompt; either add an allow-list entry for cd (e.g., cd /tmp/gh-aw/syntax-error-tests or cd*) or update the prompt to run gh aw compile using absolute paths without cd.

Suggested change
cd /tmp/gh-aw/syntax-error-tests
gh aw compile test-1.md 2>&1 | tee test-1-output.txt
gh aw compile /tmp/gh-aw/syntax-error-tests/test-1.md 2>&1 | tee /tmp/gh-aw/syntax-error-tests/test-1-output.txt

Copilot uses AI. Check for mistakes.
- "head -n * .github/workflows/*.md"
- "cp .github/workflows/*.md /tmp/*.md"
- "cat /tmp/*.md"
- "cp .github/workflows/*.md /tmp/gh-aw/syntax-error-tests/*.md"
Copy link

Copilot AI Apr 6, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The allowed command pattern cp .github/workflows/*.md /tmp/gh-aw/syntax-error-tests/*.md is not a valid copy invocation if executed literally (multiple sources + a glob destination that likely doesn’t expand) and could fail with “target is not a directory”. Consider changing the allowed pattern to copy into the directory (e.g., /tmp/gh-aw/syntax-error-tests/) while still permitting the intended cp ... test-1.md usage.

Suggested change
- "cp .github/workflows/*.md /tmp/gh-aw/syntax-error-tests/*.md"
- "cp .github/workflows/*.md /tmp/gh-aw/syntax-error-tests/"

Copilot uses AI. Check for mistakes.
Comment on lines +657 to 660
# --allow-tool shell(cat /tmp/gh-aw/syntax-error-tests/*.md)
# --allow-tool shell(cat)
# --allow-tool shell(cp .github/workflows/*.md /tmp/*.md)
# --allow-tool shell(cp .github/workflows/*.md /tmp/gh-aw/syntax-error-tests/*.md)
# --allow-tool shell(date)
Copy link

Copilot AI Apr 6, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The --allow-tool entry for cp .github/workflows/*.md /tmp/gh-aw/syntax-error-tests/*.md reflects a command that would fail if run verbatim (glob destination often won’t expand, and cp requires a directory when copying multiple sources). Prefer an allow pattern that is also a valid command (copy into /tmp/gh-aw/syntax-error-tests/) to avoid the agent selecting a broken allowed command.

Copilot uses AI. Check for mistakes.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[copilot-token-optimizer] Daily Syntax Error Quality Check — 9.3M tokens/run from blocked compile tool

3 participants