feat: add Grafana detection rule creation and improve LogQL guidance

[l50]

Key Changes:

• Introduced create_detection_rule tool for automated Grafana alert rule creation
• Updated agent instructions to include advanced alert rule creation workflow
• Enforced stricter use of specific LogQL label selectors in templates and docs
• Improved query planning and performance optimization guidance for LogQL
• Auto-rewrite broad selectors to prevent timeouts

Added:

• create_detection_rule async method to GrafanaTools for creating alert rules based
  on investigation findings, with validation for label selectors, severity, and folder
  management
• _ensure_alert_folder internal helper to automatically provision the required
  Grafana folder for new alert rules
• _is_mitre_technique_description to skip validation for MITRE technique IDs (they're
  classifications, not raw log data)
• Auto-rewriting of broad LogQL selectors ({deployment=~".+"}, etc.) to {job="eventlog"}
  to prevent query timeouts
• New documentation in agent system instructions for alert rule creation, including
  guidance on when and how to use create_detection_rule, with examples

Changed:

• Enhanced instructions in system_instructions.md.jinja to require label discovery
  (using list_loki_label_names and list_loki_label_values) before running queries,
  and to use only specific labels like {job="eventlog"} to avoid timeouts
• Expanded the LogQL query optimization section with stricter requirements and practical
  examples for contains (|=) vs regex (|~) usage, combining queries, and avoiding
  anti-patterns
• Updated default label selector in QueryTemplateTools to {job="eventlog"} instead
  of broad patterns, and clarified override instructions
• Strengthened query planning workflow, instructing agents to plan and combine queries
  before execution

Removed:

• Deprecated or redundant broad LogQL label selectors from default values and examples
  in query templates and agent instructions
• Outdated anti-patterns in agent workflow that allowed repeated broad queries without
  evidence recording or label discovery

[linear]

CAP-835 Optimize Blue Agent Loki Querying and Evidence Validation

Description:
Enhance the Blue Agent's interaction with Loki by refining query selectors, improving evidence validation, reducing redundant queries, and optimizing filter usage. These improvements aim to decrease timeouts, improve accuracy, and boost overall agent performance.

---

Objective:

Reduce query timeouts and duplication while improving evidence validation accuracy and query efficiency in the Blue Agent's Loki integration.

---

Scope of Work:

• Replace broad Loki selectors (e.g., {deployment=~".+"}) with specific labels based on discovered label names.
• Implement logic for the agent to retrieve and utilize available Loki label names via list_loki_label_names.
• Update evidence validation to either exclude MITRE technique descriptions or enhance extraction to match technique IDs in query results.
• Refactor query planning to minimize duplicate or redundant Loki queries.
• Update agent/system prompts to prefer literal filters (|=) over regex filters (|~) for exact string matching.
• Research and outline approach for the agent to create alert rules based on observed patterns or evidence.

---

Dependencies:

• Access to Loki API and permissions to use list_loki_label_names.
• Knowledge of existing agent query planning and evidence validation logic.
• Existing prompt templates for system/agent interaction.
• None identified beyond these.

---

Acceptance Criteria:

1. Agent no longer uses broad selectors such as {deployment=~".+"}; queries utilize specific, discovered labels.
2. Evidence validation either excludes MITRE technique descriptions or successfully matches technique IDs in query results.
3. The number of redundant or duplicate queries issued per investigation is reduced by at least 50%.
4. Queries use literal (|=) filters for exact string matching in at least 90% of applicable cases.
5. Documentation or implementation plan exists for agent alert rule creation based on identified patterns.

---

Additional Notes:

• See Loki documentation for best practices on selectors and filtering: https://grafana.com/docs/loki/latest/logql/selectors/
• Consider logging changes for future analysis of agent performance improvements.
• For MITRE technique extraction, reference the ATT&CK framework: https://attack.mitre.org/
• Alert rule creation may depend on integration with external systems or APIs.

---