Skip to content

[cli-tools-test] compile: unknown tool names in frontmatter are silently ignored without warning #24835

@github-actions

Description

@github-actions

Problem Description

When a workflow markdown file contains an unknown/invalid tool name in the tools: frontmatter section, the compiler silently ignores it without emitting any validation error or warning. The user receives no indication that their tool configuration was not applied.

Steps to Reproduce

  1. Create a workflow markdown file with an unknown tool in the tools: section:
---
engine: copilot
on:
  schedule:
    - cron: "0 0 * * *"
tools:
  nonexistent-tool-xyz:
    version: "1.0"
network:
  allowed:
    - github.com
---

This is a test workflow.
```

2. Compile the workflow using `gh aw compile` or the MCP `compile` tool.

### Expected Behavior

The compiler should emit a **validation warning** (or error in strict mode) indicating that `nonexistent-tool-xyz` is not a recognized tool and was ignored. Something like:

```
WARNING: tools.nonexistent-tool-xyz: unknown tool name, will be ignored

Actual Behavior

The workflow compiles successfully with no errors and no warnings. The unknown tool entry is completely silent — it does not appear in the generated .lock.yml file and there is no indication to the user that their configuration was not applied.

Environment

  • Repository: github/gh-aw
  • Run ID: §24019945745
  • Date: 2026-04-06
  • Test: Daily CLI Tools Exploratory Tester

Impact

  • Severity: Medium
  • Frequency: Always (reproducible with any unknown tool name)
  • Workaround: None — users have no way to know their tool configuration is being ignored unless they manually inspect the generated lock file

This is especially problematic because:

  1. Users may spend time debugging why a tool isn't working without realizing the name is wrong
  2. Typos in built-in tool names (e.g., playwrigjht instead of playwright) would also be silently ignored
  3. There is no feedback loop to help users discover valid tool names

Additional Context

Tested with nonexistent-tool-xyz as the tool name. The generated lock file contained no trace of the tool configuration. Strict mode (strict: true) does not catch this case either — the behavior is the same regardless of strict mode setting.

References:

Generated by Daily CLI Tools Exploratory Tester · ● 1.9M ·

  • expires on Apr 13, 2026, 5:34 AM UTC

Metadata

Metadata

Type

No type
No fields configured for issues without a type.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions