Release v1.1.0 - Test Infrastructure Overhaul

tag v1.1.0
Tagger: aaddrick aaddrick@gmail.com

Release v1.1.0 - Test Infrastructure Overhaul
Release v1.1.0 - Test Infrastructure Overhaul

Major performance and reliability improvements to gh-cli-search testing infrastructure.

Test Infrastructure Redesign

Replaced 3-tier agent hierarchy with Python orchestrator:

• Old: test-orchestrator → test-group-leader → test-validator (167 context loads)
• New: run-all-tests.py (80 context loads)
• Performance: 60s+ timeouts → ~6 seconds per test
• Total execution: ~8 minutes for 80 tests

Added Python test orchestrator (testing/scripts/run-all-tests.py):

• Parses test scenarios from markdown files
• Executes 80 tests sequentially with clean isolation
• Generates 3-level reports (master, group, individual)
• Tracks execution time and pass/fail statistics
• Enhanced command extraction for setup commands
• Validates skill usage (gh search vs gh subcommands)

Added test-reviewer agent (agents/test-reviewer.md):

• Post-test analysis for comprehensive result review
• Identifies failure patterns across all tests
• Performs root cause analysis (skill/test/agent/infrastructure)
• Creates REVIEWER-NOTES.md with prioritized recommendations
• Integrated as optional workflow step 3

Critical Fixes

Permission mode bug in run-single-test.sh:

• Changed from invalid 'acceptAll' to 'bypassPermissions'
• Resolved 100% test failure rate

Test timeout issues:

• Disabled episodic memory searches (major speedup)
• Increased timeout from 60s to 120s
• Added concise output mode

Command extraction improvements:

• Enhanced regex for various command formats
• Added support for setup/diagnostic commands
• Better handling of inline vs code block commands

Test prompt improvements:

• "How do I..." → "What command do I run..."
• More explicit requests for code block output

Results

• Pass rate: 70% → 73.8% (59/80 tests)
• Test speed: 60s+ timeouts → ~6s per test
• Timeout rate: 30% → 0%
• Token efficiency: 167 context loads → 80

Skill Updates

testing-gh-skills:

• Updated to reflect Python orchestrator approach
• Marked as SLASH COMMAND ONLY (never auto-invoke)
• Added test-reviewer integration instructions
• Added performance metrics and history

using-gh-cli-search:

• Enhanced with mandatory directive style
• Stronger language to ensure skill usage

Infrastructure

• Added root .gitignore for Python/IDE/OS files
• Added testing/reports/.gitignore for generated reports
• Removed 5 unused slash commands (kept only /test-gh-skills)
• Removed old agent files (test-orchestrator, test-group-leader, test-validator)

Version Bump

• Updated plugin.json to v1.1.0
• Comprehensive CHANGELOG entry for all changes

See CHANGELOG.md for detailed breakdown of all improvements.