feat: add red-blue correlation engine and learning system for investigation coverage

[l50]

Key Changes:

• Introduced a red-blue correlation engine to assess detection coverage and gaps
• Added investigation result persistence and learning system with SQLite backend
• Implemented learning tools for querying past investigations and query effectiveness
• Enhanced blue agent workflow with query rate limiting, improved evidence handling, and robust timeouts

Added:

• Red-Blue Correlation Engine: New src/ares/core/correlation.py parses red and blue reports, correlates activities to detections, generates gap/coverage reports, and outputs markdown
• Investigation Persistence: src/ares/core/persistence.py provides SQLite-backed storage for investigation results, query effectiveness stats, and similarity lookup
• Learning Tools: src/ares/tools/blue/learning.py exposes tools for querying historical investigations, effective queries, and false positive patterns for agent learning
• Query Resilience: src/ares/core/query_resilience.py adds automatic retry, time range reduction, and chunking for large log queries
• Remote Command Execution: src/ares/core/remote.py enables AWS SSM-based remote execution for red team tools, with robust SSO credential validation
• Comprehensive red-blue correlation and learning tests: tests/test_correlation.py, tests/test_persistence.py, tests/test_learning.py, tests/test_query_resilience.py
• Query template tools: src/ares/tools/blue/query_templates.py provides pre-built LogQL queries mapped to MITRE techniques

Changed:

• Blue agent now enforces strict query rate limiting (default 5 per investigation), duplicate query detection, and improved evidence extraction
• Investigation orchestrator adds watchdog thread for hard timeouts and generates partial reports on timeout
• Taskfile and documentation updated for new log/coverage workflow, reduced default max steps, and log management commands
• Agent system instructions and investigation prompt templates improved for IOC extraction and anti-loop guidance
• Red team agent tools now execute on remote Kali via SSM, with robust error handling and output parsing
• Blue agent now records executed queries and integrates query effectiveness into persistence/learning
• Added boto3 as a required dependency

Removed:

• Legacy local subprocess usage for red team tools; all execution now via remote SSM executor
• Unused aiobotocore/aioitertools dependencies from lockfile to resolve S3 compatibility

[linear]

CAP-822 Add AWS SSM Remote Execution & Enhance Blue/Red Tooling

Description:
Implement AWS SSM-based remote command execution, expand blue team Grafana query templates, and improve both blue and red team tools. This upgrade aims to streamline SOC investigations, strengthen network scanning for red teams, and clarify system configuration defaults.

---

Objective:

Enable remote command execution on EC2 instances via AWS SSM, enhance blue team investigation capabilities with new query templates and SOC tools, improve red team network scanning, and update configuration defaults and documentation for better usability.

---

Scope of Work:

• Implement remote.py module to support AWS SSM command execution on remote EC2 (Kali attack box)
• Create and integrate Grafana query templates in query_templates.py for blue team use
• Enhance soc_investigator.py to improve SOC investigation workflows
• Update actions.py with new or improved blue team investigation actions
• Improve red team network scanning capabilities in network.py
• Update default configuration (max-steps) and document timeout behavior in README, Taskfile.yaml, and docs

---

Dependencies:

• AWS IAM permissions for SSM access
• EC2 instances registered with AWS Systems Manager
• Grafana environment for query template integration
• None identified beyond above

---

Acceptance Criteria:

1. remote.py enables successful execution of remote shell commands on registered EC2 instances via AWS SSM.
2. Blue team Grafana query templates are present, well-documented, and usable within investigations.
3. SOC investigator tool improvements are functional and demonstrably enhance investigation workflows.
4. Red team network scanning tool updates are implemented and tested.
5. Configuration defaults (max-steps) and timeout behaviors are clearly updated in documentation.
6. All changes are reflected in the relevant source files and documentation.

---

Additional Notes:

• Ensure all AWS credentials and permissions are handled securely.
• Reference AWS SSM documentation for implementation: https://docs.aws.amazon.com/systems-manager/latest/userguide/execute-remote-commands.html
• Include code samples in documentation where relevant for new features.
• Coordinate with DevOps for deployment/testing on EC2 instances.

---