[cli-tools-test] MCP tools return exceptions instead of structured JSON for some invalid inputs

[github-actions]

Overview

During daily exploratory testing of the audit, logs, and compile MCP tools, three cases were found where tools throw MCP protocol errors (-32602/-32603) instead of returning structured JSON responses. This inconsistency can break callers that expect JSON and makes the API behavior unpredictable.

Test Environment

• Run ID: §23885125084
• Date: 2026-04-02
• Workflows tested: 179 workflows, all compiled successfully

Cases Observed

Case 1: logs with non-existent workflow name

Input: logs(workflow_name: "nonexistent-workflow-xyz", count: 3)

Actual: Throws MCP error -32602: calling "tools/call": workflow 'nonexistent-workflow-xyz' not found...

Expected: Returns {"runs": [], "summary": {"total_runs": 0, ...}} with a helpful message

---

Case 2: audit with invalid/non-existent run ID

Input: audit(run_id_or_url: "99999999999")

Actual: Throws MCP error -32603: calling "tools/call": failed to audit workflow run: ✗ failed to fetch run metadata

Expected: Returns JSON with {"error": "Run 99999999999 not found", ...} or at minimum includes the run ID in the error

---

Case 3: compile with actionlint: true when Docker is unavailable

Input: compile(actionlint: true)

Actual: Throws MCP error -32603: calling "tools/call": docker is not available (cannot connect to Docker daemon). actionlint requires Docker...

Expected: Returns [{"workflow": "...", "valid": false, "errors": [{"type": "config_error", "message": "Docker unavailable for actionlint"}], ...}] — consistent with how other compile errors are returned

---

Contrast: Well-handled case

For reference, compile with a non-existent workflow name works correctly and returns structured JSON:

[{"workflow":"daily-cli-tools-exploratory-tester","valid":false,"errors":[{"type":"resolution_error","message":"✗ workflow 'daily-cli-tools-exploratory-tester' not found..."}],"warnings":[]}]

What Works Well (for context)

• ✅ All 179 workflows compile successfully (0 errors, 0 warnings)
• ✅ logs engine filter, date range, and count limiting work correctly
• ✅ audit for successful runs includes comprehensive job timeline, MCP health, firewall analysis
• ✅ audit for failed runs identifies error location (e.g., exit code 22 from firewall 404)
• ✅ audit correctly detects blocked network domains and provides allow-list recommendations

Impact

• Severity: Medium
• Frequency: Always (reproducible)
• Callers affected: Any code wrapping these MCP tools needs to handle both JSON and exception paths, increasing complexity
• Workaround: Validate inputs before calling (e.g., run status first to confirm workflow exists before calling logs)

Recommendations

1. logs: Return empty result set when workflow_name is not found, rather than throwing
2. audit: Include the run ID in the error message; consider returning a JSON error envelope
3. compile: Catch Docker unavailability early and return structured validation errors per-workflow rather than a top-level exception

Generated by Daily CLI Tools Exploratory Tester · ◷

• expires on Apr 9, 2026, 5:25 AM UTC