Add GitHub tool/toolset validator for allowed tools configuration by Copilot · Pull Request #2770 · github/gh-aw

Copilot · 2025-10-30T04:49:40Z

GitHub Tool/Toolset Validator Implementation ✅

This PR adds validation to ensure that when users specify individual GitHub tools using the allowed field, the corresponding toolsets are enabled in the configuration.

Recent Updates (addressing review feedback)

Changed to use ParsedTools - Now uses workflowData.ParsedTools.GitHub instead of accessing raw Tools map directly for better type safety.

Moved mapping to embedded JSON - The tool-to-toolset mapping is now loaded from an embedded JSON file (data/github_tool_to_toolset.json) instead of being a large map literal in code.

Fixed toolsets field parsing - Now supports both toolset (singular) and toolsets (plural) field names to handle common user configurations.

Problem Solved

When users specify tools like list_workflows in the allowed field but forget to enable the actions toolset, the workflow would compile but fail at runtime. This validator catches the issue at compile time with clear error messages.

Solution Overview

Tool-to-Toolset Mapping - JSON file mapping 50+ GitHub tools to their toolsets
Validation Function - Checks that all allowed tools have their toolsets enabled
Clear Error Messages - Provides actionable fix suggestions
Compiler Integration - Validation runs automatically during workflow compilation

Example Error Message

ERROR: GitHub tools specified in 'allowed' field require toolsets that are not enabled:

  Toolset 'actions' is required by:
    - list_workflows

  Toolset 'discussions' is required by:
    - create_discussion

Suggested fix: Add the missing toolsets to your GitHub tool configuration:

tools:
  github:
    toolsets:
      - default
      - actions
      - discussions

Test Coverage

✅ 17 unit tests covering all scenarios
✅ 3 integration tests (Copilot, Claude, Remote mode)
✅ 3 error message formatting tests
✅ 2 mapping consistency tests
✅ All tests pass
✅ Manual verification complete

Files Added

pkg/workflow/github_tool_to_toolset.go - Validation logic with embedded JSON loader
pkg/workflow/data/github_tool_to_toolset.json - Tool to toolset mapping data
pkg/workflow/github_tool_to_toolset_test.go - Unit tests
pkg/workflow/github_toolset_validation_error.go - Error type
pkg/workflow/github_toolset_validation_integration_test.go - Integration tests

Files Modified

pkg/workflow/compiler.go - Added validation using ParsedTools
pkg/workflow/tools_types.go - Support both toolset/toolsets field names

Quality Checks

✅ Build passes
✅ All linter checks pass
✅ Code formatted with gofmt
✅ Backwards compatible

Original prompt

Add a github tool / toolset validator.

for each allowed tool in the frontmatter, validate that its corresponding toolset is enabled in the current configuration.

Raise error if toolset is missing

💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

pelikhan · 2025-10-30T11:04:53Z


+	// Validate GitHub tools against enabled toolsets
+	log.Printf("Validating GitHub tools against enabled toolsets")
+	if githubTool, hasGitHub := workflowData.Tools["github"]; hasGitHub {


@copilot use hasGithub or parsed tools instead

Updated to use workflowData.ParsedTools.GitHub for type safety. Also fixed parsing to support both toolset and toolsets field names. (67cee68)

pelikhan · 2025-10-30T11:07:31Z

+
+// GitHubToolToToolsetMap maps individual GitHub MCP tools to their respective toolsets
+// This mapping is based on the documentation in .github/instructions/github-mcp-server.instructions.md
+var GitHubToolToToolsetMap = map[string]string{


@copilot move to embedded JSON file.

Moved to embedded JSON file at pkg/workflow/data/github_tool_to_toolset.json. The mapping is now loaded via go:embed directive in the init function. (67cee68)

@pelikhan

- Use workflowData.ParsedTools.GitHub instead of raw Tools map - Support both 'toolset' and 'toolsets' field names in parseGitHubTool - Move GitHub tool-to-toolset mapping to embedded JSON file - Addresses review feedback from @pelikhan Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

github-actions · 2025-10-30T11:24:21Z

Agentic Changeset Generator triggered by this pull request.

Copilot

Pull Request Overview

This PR adds compile-time validation to ensure that when users specify individual GitHub tools via the allowed field, their corresponding toolsets are enabled in the workflow configuration. Previously, misconfigured workflows would compile successfully but fail at runtime.

Key Changes:

Implemented tool-to-toolset validation with clear error messaging
Moved tool-to-toolset mapping to embedded JSON for maintainability
Added support for both toolset and toolsets field names in configurations

Reviewed Changes

Copilot reviewed 9 out of 11 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
`pkg/workflow/compiler.go`	Integrated validation step during workflow compilation using ParsedTools
`pkg/workflow/tools_types.go`	Added parsing support for plural `toolsets` field name
`pkg/workflow/github_tool_to_toolset.go`	Core validation logic with embedded JSON loader
`pkg/workflow/data/github_tool_to_toolset.json`	Tool-to-toolset mapping data (50+ GitHub tools)
`pkg/workflow/github_toolset_validation_error.go`	Custom error type with formatted output and fix suggestions
`pkg/workflow/github_tool_to_toolset_test.go`	Unit tests covering validation scenarios and mapping consistency
`pkg/workflow/github_toolset_validation_integration_test.go`	Integration tests for Copilot, Claude, and remote mode
`pkg/workflow/permissions_validator.go`	Removed unused `containsToolset` helper function
`.github/workflows/technical-doc-writer.lock.yml`	Updated actions/setup-node to newer version

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2025-10-30T11:25:18Z

+		// Check for both "toolset" and "toolsets" (plural is more common in user configs)
+		if toolset, ok := configMap["toolsets"].([]any); ok {
+			config.Toolset = make([]string, 0, len(toolset))
+			for _, item := range toolset {
+				if str, ok := item.(string); ok {
+					config.Toolset = append(config.Toolset, str)
+				}
+			}
+		} else if toolset, ok := configMap["toolset"].([]any); ok {


The code for parsing toolsets (lines 207-212) and toolset (lines 214-218, not shown but identical) is duplicated. Consider extracting this logic into a helper function to reduce duplication and improve maintainability.

Copilot · 2025-10-30T11:25:18Z

+
+  "get_repository": "repos",


[nitpick] The JSON file contains inconsistent blank lines (e.g., line 5) between groupings. While this doesn't affect functionality, removing these blank lines would make the file more consistent and easier to maintain, as JSON doesn't require blank lines for readability.

Copilot AI assigned Copilot and pelikhan Oct 30, 2025

Initial plan

68428d6

Copilot started work on behalf of pelikhan October 30, 2025 04:50 View session

Copilot AI and others added 2 commits October 30, 2025 05:04

Add GitHub tool/toolset validator with comprehensive tests

b84954a

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Fix linter issues - remove unused function and simplify loop

3ca06c2

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Copilot AI changed the title ~~[WIP] Add GitHub toolset validator for allowed tools~~ Add GitHub tool/toolset validator for allowed tools configuration Oct 30, 2025

Copilot finished work on behalf of pelikhan October 30, 2025 05:10

Copilot AI requested a review from pelikhan October 30, 2025 05:10

pelikhan approved these changes Oct 30, 2025

View reviewed changes

Copilot started work on behalf of pelikhan October 30, 2025 11:09 View session

Copilot AI requested a review from pelikhan October 30, 2025 11:21

Copilot finished work on behalf of pelikhan October 30, 2025 11:21

pelikhan marked this pull request as ready for review October 30, 2025 11:24

Copilot AI review requested due to automatic review settings October 30, 2025 11:24

Copilot AI reviewed Oct 30, 2025

View reviewed changes

Add changeset for GitHub tool/toolset validator [skip-ci]

c814ea7

pelikhan approved these changes Oct 30, 2025

View reviewed changes

pelikhan merged commit 26df517 into main Oct 30, 2025
4 checks passed

pelikhan deleted the copilot/add-toolset-validator branch October 30, 2025 12:04

lpcox mentioned this pull request May 10, 2026

Bump firewall version to v0.25.43 #31376

Closed

6 tasks

Copilot AI mentioned this pull request May 10, 2026

chore: bump DefaultFirewallVersion to v0.25.43 #31379

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add GitHub tool/toolset validator for allowed tools configuration#2770

Add GitHub tool/toolset validator for allowed tools configuration#2770
pelikhan merged 5 commits into
mainfrom
copilot/add-toolset-validator

Copilot AI commented Oct 30, 2025 •

edited

Loading

Uh oh!

pelikhan Oct 30, 2025

Uh oh!

Copilot AI Oct 30, 2025

Uh oh!

pelikhan Oct 30, 2025

Uh oh!

Copilot AI Oct 30, 2025

Uh oh!

github-actions Bot commented Oct 30, 2025

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Copilot AI Oct 30, 2025

Uh oh!

Copilot AI Oct 30, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

Copilot AI commented Oct 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

GitHub Tool/Toolset Validator Implementation ✅

Recent Updates (addressing review feedback)

Problem Solved

Solution Overview

Example Error Message

Test Coverage

Files Added

Files Modified

Quality Checks

Uh oh!

pelikhan Oct 30, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Oct 30, 2025

Choose a reason for hiding this comment

Uh oh!

pelikhan Oct 30, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Oct 30, 2025

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Oct 30, 2025

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Uh oh!

Copilot AI Oct 30, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Oct 30, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Copilot AI commented Oct 30, 2025 •

edited

Loading