[cli-tools-test] Daily CLI Tools Testing Report - 2026-05-20

[github-actions[bot]]

Testing Session Overview

Date: 2026-05-20
Run ID: §26145567529
Tools Tested: audit, logs, compile (agentic-workflows MCP server)
Total Tests: 12 | Passed: 11 | Failed: 1
Overall Health: 🟢 Healthy (91% success rate)

---

Test Coverage Summary

Phase 1: Environment Setup ✅

• ✅ MCP server availability verified
• ✅ 231 workflows discovered and enumerated
• ✅ Representative test workflows selected

Phase 2: Logs Command (6 tests)

• ✅ Basic log download (last 24 hours): 25.9s
• ❌ Filter by workflow name: FAILED (issue created)
• ✅ Filter by engine (copilot): 14.6s
• ✅ Filter by date (2026-05-19): 11.5s
• ✅ Edge case - future date: handled gracefully (0.6s)
• ✅ Log file structure validated

Phase 3: Audit Command (3 tests)

• ✅ Audit successful run: comprehensive report in 56ms
• ✅ Audit with URL format: works identically to numeric ID (57ms)
• ✅ Edge case - invalid run ID: graceful error with suggestions (377ms)

Phase 4: Compile Command (2 tests)

• ✅ Compile single workflow: successful (1.8s)
• ✅ Compile multiple workflows: successful with graceful handling of missing workflow (1.8s)

---

Performance Benchmarks

All performance metrics are within expected ranges:

Tool	Operation	Time	Status
logs	Basic download (3 runs)	25.9s	✅ Expected
logs	Engine filter (3 runs)	14.6s	✅ Good
logs	Date filter (3 runs)	11.5s	✅ Good
audit	Successful run	56ms	✅ Excellent
audit	URL-based	57ms	✅ Excellent
audit	Invalid ID	377ms	✅ Good
compile	Single workflow	1.8s	✅ Acceptable
compile	Multiple workflows	1.8s	✅ Acceptable

---

Issues Found

1. Workflow Name Filter Failure (Medium Severity) ❌

Tool: logs
Description: The --workflow-name filter fails to find workflows that exist in the repository
Impact: Users cannot filter logs by workflow name, requiring manual filtering of results
Workaround: Use other filters (engine, date) and manually filter results
Issue: Created (see related issues)

---

Positive Findings ✨

1. Excellent Error Handling: All edge cases (future dates, invalid run IDs, missing workflows) are handled gracefully with helpful error messages and suggestions

2. Strong Performance: Audit command is exceptionally fast (<100ms), logs command performs within expected ranges

3. Comprehensive Audit Reports: Audit output includes all expected sections:

   • Overview (run metadata, duration, status)
   • Metrics (action minutes, error/warning counts)
   • Jobs (execution timeline)
   • Key findings (categorized insights)
   • Recommendations (actionable next steps)
   • Observability insights (execution patterns)

4. Robust Compilation: Compile command validates workflows, provides clear error messages, and generates valid .lock.yml files

5. Flexible Input Formats: Audit accepts both numeric run IDs and full GitHub URLs

6. Well-Structured Log Storage: Downloaded logs are organized by run ID with clear directory structure

---

Recommendations

1. Fix workflow name filtering: Investigate and resolve the workflow name resolution mismatch in the logs tool (issue created)

2. Documentation: Consider documenting the expected time ranges for logs downloads (10-25s is normal)

3. Caching: The logs tool caches results effectively - this could be highlighted in documentation as a feature

---

Conclusion

The agentic-workflows MCP server tools are functioning well overall with only one medium-severity issue identified. The audit and compile commands work flawlessly with excellent performance. The logs command has robust filtering capabilities except for the workflow-name filter. Error handling is exemplary across all tools.

Next Steps:

• Address the workflow name filter issue
• Continue daily monitoring of tool performance
• Document normal performance ranges for user expectations

Generated by 🧪 Daily CLI Tools Exploratory Tester · ● 12M · ◷

• expires on May 23, 2026, 6:40 AM UTC