feat: introduce alert correlation, lateral movement analysis, and investigation enhancements

[l50]

Key Changes:

• Added alert correlation engine for clustering and contextualizing related alerts
• Introduced lateral movement analysis with host connection graph and pivot suggestions
• Enhanced evidence extraction and validation with auto-IOC detection from queries
• Increased investigation query limits and improved investigation workflow for scope analysis

Added:

• Alert correlation engine for grouping related alerts by hosts, users, IPs, and techniques, providing context for investigations (alert_correlation.py)
• Lateral movement analysis module with host-to-host connection graph, detection of lateral movement patterns, and pivot suggestions for investigation expansion (lateral_analyzer.py)
• Automatic evidence extraction from query results, including classification and confidence boosting for high-quality IOCs
• Investigation tools for analyzing lateral movement, recording host connections, and retrieving correlated alert context
• Extensive tests for alert correlation and lateral movement analysis to ensure robust clustering and pivot logic

Changed:

• Investigation orchestrator and state to include correlation context and lateral movement graph, enabling richer investigation context and scope tracking
• Investigation agent factory logic to:
  • Increase query and tool call limits for deeper investigations
  • Adaptively grant more queries as productive evidence is found
  • Auto-extract IOCs from query results and add them to the evidence pool, reducing reliance on LLM extraction alone
• Query resilience executor to start with smaller time ranges and faster timeouts for more reliable querying (especially with mcp-grafana's 10s timeout)
• Investigation completion tools to generate a more comprehensive and structured fallback synopsis, summarizing alert context, MITRE techniques, evidence by pyramid level, lateral movement, and confidence
• Query templates to use a 4-hour default lookback window instead of 24 hours for more focused detection
• System instructions for agents to enforce a mandatory lateral analysis workflow, including correlation, pivoting, and attack path mapping

[linear]

CAP-833 Enhance Blue Team with Lateral Movement Analysis

Description:
Expand ARES blue team investigation capabilities by introducing advanced lateral movement analysis, alert correlation, and evidence validation improvements. This will streamline attack path identification, improve evidence confidence, and provide actionable pivot suggestions for SOC investigators.

---

Objective:

Enable blue team analysts to rapidly detect, correlate, and investigate lateral movement within ARES by leveraging new analysis modules, improved evidence validation, and enhanced investigation tools.

---

Scope of Work:

• Integrate Alert Correlation Engine (alert_correlation.py) for clustering related alerts using similarity scoring
• Implement Lateral Movement Analyzer (lateral_analyzer.py) for graph-based attack path and pivot analysis
• Expand evidence validation in evidence_validation.py:
  • Add IOC pattern extraction (IPs, hostnames, users, hashes, processes, services, computer names)
  • Adjust confidence penalties and boosts based on evidence quality
  • Add auto_extract_evidence_from_query() for automated IOC extraction
• Add new investigation tools in investigation.py:
  • analyze_lateral_movement() for pattern analysis and pivot suggestions
  • record_lateral_connection() for host-to-host tracking
  • track_host_investigation() for investigation scope management
• Update blue team actions and query templates for lateral movement detection
• Improve SOC investigator agent integration and system guidance
• Update factories and system instructions for blue team components
• Expand test_query_resilience.py with relevant test cases

---

Dependencies:

• Existing blue team investigation and evidence validation modules in ARES
• Access to system instructions and agent integration components
• No external dependencies identified

---

Acceptance Criteria:

1. Related alerts are clustered accurately based on shared characteristics with similarity scores.
2. Lateral movement analyzer generates graph-based attack paths and actionable pivot suggestions.
3. Evidence validation extracts IOCs and applies new confidence scoring logic.
4. Investigation tools allow marking and tracking of host investigations and connections.
5. SOC investigator agent utilizes updated instructions and new investigation capabilities.
6. All new features are covered by unit and integration tests, including updates in test_query_resilience.py.
7. Documentation and system instructions are updated to reflect new blue team workflows.

---

Additional Notes:

• Reference MITRE ATT&CK for lateral movement patterns and evidence mapping.
• Ensure all new modules are compatible with existing ARES blue team architecture.
• Consider performance impact of graph-based analysis on large environments.

---