[safeoutputs] Strengthen anti-probing guidance for add_comment and create_issue tool descriptions

[github-actions]

Summary

Analysis of workflow runs from the last 24 hours identified 2 errors where the Gemini engine called safe-output tools with completely empty arguments ({}) as a discovery/probe attempt, instead of emitting a real write-intent declaration. Both add_comment and create_issue were probed in parallel before the agent made the actual valid calls.

The existing tool descriptions DO contain anti-probing guidance, but it appears late in the description after the purpose statement. The agent recovered (made successful subsequent calls), but the probe-then-retry pattern wastes turns and contradicts the explicit do not probe rule already stated in the safe-outputs system prompt.

Errors Observed

Tool	Workflow	Engine	Run	Arguments Sent
add_comment	Smoke Gemini	gemini	§26125419168	{}
create_issue	Smoke Gemini	gemini	§26125419168	{}

Raw RPC trace from gateway.md

rpc safeoutputs→tools/call add_comment
{"params":{"arguments":{},"name":"add_comment"}}

rpc safeoutputs→tools/call create_issue
{"params":{"arguments":{},"name":"create_issue"}}

rpc safeoutputs←resp ⚠️ calling "tools/call": Invalid arguments: missing or empty 'body'

rpc safeoutputs←resp ⚠️ calling "tools/call": Invalid arguments: missing or empty 'title', 'body'

After the errors, the agent successfully retried both tools with proper arguments and the workflow completed.

Root Cause Analysis

The workflow prompt for Smoke Gemini is correct — it tells the agent exactly what title/body to use. The safe-outputs system block also explicitly states Safe-output calls are write-once declarations for real downstream side effects. Do NOT use them for probing, auth tests, retries with placeholder content, or 'let me see if this works' experiments.

The Gemini agent ignored this and probed anyway. The tool descriptions themselves include the same warning, but it is buried mid-paragraph:

Create a new GitHub issue for tracking bugs, feature requests, or tasks. Use this for actionable work items that need assignment, labeling, and status tracking. This is a write-once declaration for a real intended issue, not a sandbox or probe: do not call it with placeholder titles/bodies or auth experiments. If you are not ready to open the real issue, use noop or report_incomplete instead. ...

Moving the anti-probing line to the very start (before the purpose blurb) increases the chance an agent reading the first sentence respects the constraint.

Recommended Improvements

1. Hoist anti-probing guidance to the start of the description

In pkg/workflow/js/safe_outputs_tools.json, for both create_issue and add_comment, move the warning to the first sentence with a visual marker.

create_issue — current:

Create a new GitHub issue for tracking bugs, feature requests, or tasks. Use this for actionable work items that need assignment, labeling, and status tracking. This is a write-once declaration for a real intended issue, not a sandbox or probe: do not call it with placeholder titles/bodies or auth experiments. ...

create_issue — suggested:

WRITE-ONCE: do NOT call this tool with empty or placeholder arguments to probe or discover its schema — required fields (title, body) are listed in this schema; if you are not ready to open the real issue, call `noop` instead. Creates a new GitHub issue for tracking bugs, feature requests, or tasks. ...

add_comment — current:

Add a comment to an existing GitHub issue, pull request, or discussion. Use this to provide feedback, answer questions, or add information to an existing conversation. This is a write-once declaration for a real intended comment, not a sandbox or probe: do not call it with placeholder bodies or auth experiments. ...

add_comment — suggested:

WRITE-ONCE: do NOT call this tool with empty or placeholder arguments to probe or discover its schema — the required `body` field is listed in this schema; if you are not ready to post a real comment, call `noop` instead. Adds a comment to an existing GitHub issue, pull request, or discussion. ...

2. Consider a distinct error for completely-empty argument objects

The MCP server validation in actions/setup/js/safe_output_validator.cjs could detect a totally empty {} arguments payload and respond with a different message such as:

Empty arguments are not allowed — this tool is write-once, not a discovery probe. To inspect the schema, use the tools/list MCP method. To signal that no action is needed, call noop with a message.

This would give probing agents a clearer signal than the field-by-field missing-required-field message they currently receive.

3. Document the observed probing pattern

Add a short note in pkg/workflow/js/safe_outputs_tools.json near the top of create_issue / add_comment field-level description strings for title and body reminding agents that field values must be the final intended values, not placeholders.

Affected Workflows

• Smoke Gemini — 2 errors in run §26125419168 (recovered)

No other workflow / engine combinations showed this pattern in the last 24h window.

Implementation Checklist

• Update create_issue description in pkg/workflow/js/safe_outputs_tools.json to lead with the anti-probing warning
• Update add_comment description in pkg/workflow/js/safe_outputs_tools.json the same way
• Mirror the changes in actions/setup/js/safe_outputs_tools.json (generated copy)
• Optionally: emit a distinct error message when arguments object is empty ({})
• Run make build and make recompile
• Run make test to ensure no regressions
• Monitor logs for 2-3 days to verify the empty-arguments probe pattern disappears

References

• Tool schema: pkg/workflow/js/safe_outputs_tools.json (lines 3-4 for create_issue, 282-283 for add_comment)
• Generated copy: actions/setup/js/safe_outputs_tools.json
• Validator: actions/setup/js/safe_output_validator.cjs

References:

• §26125419168 — Smoke Gemini run where both probe errors occurred

Generated by ⚡ Daily Safe Output Tool Optimizer · ● 14.9M · ◷

• expires on May 21, 2026, 9:24 PM UTC

[github-actions]

🍪 Issue Monster selected this for Copilot

I've identified this issue as a good candidate for automated resolution and requested assignment to the Copilot coding agent.

If assignment succeeds, the Copilot coding agent will analyze the issue and create a pull request with the fix.

Om nom nom! 🍪

🍪 Om nom nom by Issue Monster · ● 59.3K · ◷