feat: correlation rule improvements and platform enhancements#70
Merged
TerrifiedBug merged 25 commits intomainfrom Jan 30, 2026
Merged
feat: correlation rule improvements and platform enhancements#70TerrifiedBug merged 25 commits intomainfrom
TerrifiedBug merged 25 commits intomainfrom
Conversation
Navigate to edit mode instead of list after creating new correlation rule. Matches existing behavior for regular rule editor.
- Block creation of exact duplicate exceptions (same field+value+operator) - Warn when new exception overlaps with existing pattern - Add warning field to RuleExceptionResponse schema - Fix test database enum type creation in conftest.py - Fix auth test to expect 401 instead of 403 Closes: Platform Improvements v3 - Task 2
- Add optional alert_id to exception create request - Auto-set alert status to false_positive when exception created from it - Store exception reference on alert document in OpenSearch - Show exception badge with tooltip in alert detail view - Reload alert after exception creation to show updated status
- New AlertComment model with soft delete support - API endpoints: list, create, delete (admin only) - Comments section in alert detail page - Immutable for users, admins can delete
- Assign/unassign endpoints for alert ownership - Owner filter in alerts list (use 'me' for current user) - Take Ownership/Release buttons in alert detail - 'Assigned to me' filter checkbox on alerts page - 'My Alerts' quick link in sidebar navigation
- Fix ownership endpoints to use correct OpenSearch document ID (hit["_id"]) - Add owner_id, owner_username, owned_at to AlertResponse schema - Add exception_created and ti_enrichment to AlertResponse schema - Remove "My Alerts" nav link from Header (use filter on Alerts page instead) - Add Owner column to alerts table showing "Unassigned" for unassigned alerts
Dynamic mapping creates text+keyword multifield for strings. Term queries need to use .keyword suffix for exact matching.
Implement alert clustering feature that groups related alerts by rule_id and entity (host, user, IP) within a configurable time window. This reduces alert fatigue by consolidating repetitive alerts. Backend changes: - Add cluster_alerts() function with time-window based grouping - Add extract_entity_value() helper for entity field extraction - Add GET/PUT /settings/alert-clustering endpoints - Update alerts list endpoint with clustering support - Add AlertCluster and ClusteredAlertListResponse schemas Frontend changes: - Add Alerts tab in Settings with clustering configuration - Update Alerts page with expandable cluster rows - Add cluster selection and bulk actions support - Display cluster count badges and time ranges Tests: - Add 20 test cases covering clustering logic and edge cases
Adds PDF export functionality to the ATT&CK Matrix page: Backend (reports.py): - New POST /api/reports/attack-coverage endpoint - Generates PDF report with: - Summary statistics (total techniques, covered, coverage %) - Coverage breakdown by tactic - Top 10 detection gaps (uncovered techniques) - Top 10 best covered techniques - Uses landscape letter format with styled tables Frontend (AttackMatrix.tsx): - Export PDF button in header with loading state - Downloads PDF file with date-stamped filename - Toast notifications for success/error feedback Tests (test_attack_export.py): - Tests for PDF generation (magic bytes verification) - Authentication requirement test - Test with actual technique/rule data - Content-Disposition header verification - Default format test
Remove entity_fields configuration from alert clustering feature. Alerts are now clustered solely by rule_id within the time window, making the feature simpler to configure and understand. Changes: - Remove entity_fields from AlertClusteringSettings schema - Simplify cluster_alerts() to group by rule_id only - Remove extract_entity_value() helper function (no longer needed) - Update Settings UI to show only enable toggle and time window - Update tests to reflect simplified grouping logic
Add flag_modified() call when updating Setting.value to explicitly mark the JSONB column as modified. Without this, SQLAlchemy may not detect changes to JSON columns and skip the UPDATE statement.
The /{key} catch-all route was matching /alert-clustering requests
before the specific endpoints could handle them, causing saves to
return {"success": true} instead of the proper response and not
using the correct schema validation.
Update alert clustering to include complete alert objects in the response, not just IDs. This allows the frontend to display full alert details (rule title, severity, status, owner, tags, timestamp) when expanding a cluster, matching the non-clustered view. Changes: - Add alerts field to AlertCluster schema (backend and frontend) - Include full alert data in _create_cluster function - Update Alerts page to render full details in expanded cluster rows - Add test assertion for alerts field in cluster output
When clustering is enabled, override the client's page size (typically 25) and fetch up to 1000 alerts. This provides a much better clustering view since pagination doesn't make sense with clustering - we need to see the full picture to properly group alerts by time window.
- Strip .keyword and .text OpenSearch field type suffixes when extracting entity values from log documents (these are query hints, not document paths) - Fix correlation alert payload to include all correlation data (first_alert_id, second_alert_id, rule IDs, timestamps) instead of missing source_alerts key - Fix SQLAlchemy query to use scalars().first() instead of first() to get model object instead of Row - Rewrite correlation tests to properly test the state machine logic with mocked field resolution
- Add clickable "View Alert" link to Discord/Slack webhook notifications - Configure nginx to only log 4xx/5xx responses (reduce log noise) - Wrap Health page Indexes section in Card for design consistency - Persist "Assigned to Me" checkbox preference via localStorage - Add "Take Ownership" bulk action button to Alerts table
Docker/GHCR requires repository names to be lowercase, but github.repository may contain uppercase letters (e.g., TerrifiedBug/chad). Add a step in the changes job to lowercase the repository name and expose it as job outputs. All subsequent jobs now reference these lowercased outputs instead of the direct env var. Fixes build failures on main branch merges.
- Only show time range for grouped alerts when start/end differ by >1min - Add INFO-level logging to correlation service for debugging: - Log when correlation rules are found for a rule - Log entity field resolution results - Log when correlation state is stored (waiting for paired rule)
Previously correlation rules could only use Sigma fields (fields used in rule detection logic) for entity correlation. This limited real-world use cases where users need to correlate on context fields like user.name, source.ip, or host.name. Changes: - Add entity_field_type column to correlation_rules and versions tables - Support "sigma" (default) and "direct" field types - Direct mode bypasses Sigma field mapping, uses field path directly - Add GET /correlation-rules/common-log-fields endpoint for field discovery - Update CorrelationRuleEditor with field type toggle (Sigma Field / Log Field) - Log Field mode shows searchable dropdown of fields common to both rules' indexes
- Add GET /correlation-rules/common-sigma-fields endpoint that returns Sigma fields with valid mappings for both rules' index patterns - Verify mapped target fields actually exist in OpenSearch indexes - Replace rule detection field intersection with field mapping lookup - Make Sigma field dropdown use same searchable Popover UI as Log Field - Fix page refresh issue by removing formData.entity_field from useEffect deps - Add e.preventDefault() to field type toggle buttons for safety
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
|
|
||
|
|
||
| # revision identifiers, used by Alembic. | ||
| revision: str = '09kr_add_entity_field_type' |
Check notice
Code scanning / CodeQL
Unused global variable Note
Copilot Autofix
AI about 1 month ago
Copilot could not generate an autofix suggestion
Copilot could not generate an autofix suggestion for this alert. Try pushing a new commit or if the problem persists contact support.
|
|
||
| # revision identifiers, used by Alembic. | ||
| revision: str = '09kr_add_entity_field_type' | ||
| down_revision: Union[str, None] = 'f42b74859833' |
Check notice
Code scanning / CodeQL
Unused global variable Note
Copilot Autofix
AI about 1 month ago
Copilot could not generate an autofix suggestion
Copilot could not generate an autofix suggestion for this alert. Try pushing a new commit or if the problem persists contact support.
| # revision identifiers, used by Alembic. | ||
| revision: str = '09kr_add_entity_field_type' | ||
| down_revision: Union[str, None] = 'f42b74859833' | ||
| branch_labels: Union[str, Sequence[str], None] = None |
Check notice
Code scanning / CodeQL
Unused global variable Note
Show autofix suggestion
Hide autofix suggestion
Copilot Autofix
AI about 1 month ago
To fix the problem, remove the unused global variable branch_labels since it is not referenced in the file and not clearly needed for Alembic in this simple migration. This aligns with the recommendation to delete unused assignments when they are not intentional documentation artifacts.
Concretely, in backend/alembic/versions/09kr_add_entity_field_type.py, delete the line that defines branch_labels: Union[str, Sequence[str], None] = None and leave the other Alembic identifier variables (revision, down_revision, depends_on) untouched. No additional imports or helper functions are required.
Suggested changeset
1
backend/alembic/versions/09kr_add_entity_field_type.py
| @@ -14,7 +14,6 @@ | ||
| # revision identifiers, used by Alembic. | ||
| revision: str = '09kr_add_entity_field_type' | ||
| down_revision: Union[str, None] = 'f42b74859833' | ||
| branch_labels: Union[str, Sequence[str], None] = None | ||
| depends_on: Union[str, Sequence[str], None] = None | ||
|
|
||
|
|
Copilot is powered by AI and may make mistakes. Always verify output.
| revision: str = '09kr_add_entity_field_type' | ||
| down_revision: Union[str, None] = 'f42b74859833' | ||
| branch_labels: Union[str, Sequence[str], None] = None | ||
| depends_on: Union[str, Sequence[str], None] = None |
Check notice
Code scanning / CodeQL
Unused global variable Note
Show autofix suggestion
Hide autofix suggestion
Copilot Autofix
AI about 1 month ago
In general, an unused global variable that has no side effects should be removed, unless it is intentionally unused and named to indicate that, or it is required by an external framework. Here, Alembic only needs depends_on when there is an actual dependency; setting it to None is equivalent to omitting it. The safest fix that does not change functionality is to delete the depends_on assignment line entirely.
Concretely, in backend/alembic/versions/09kr_add_entity_field_type.py, remove line 18:
depends_on: Union[str, Sequence[str], None] = NoneNo other changes are required—revision, down_revision, and branch_labels remain as they are, and no imports or additional definitions are needed.
Suggested changeset
1
backend/alembic/versions/09kr_add_entity_field_type.py
| @@ -15,7 +15,6 @@ | ||
| revision: str = '09kr_add_entity_field_type' | ||
| down_revision: Union[str, None] = 'f42b74859833' | ||
| branch_labels: Union[str, Sequence[str], None] = None | ||
| depends_on: Union[str, Sequence[str], None] = None | ||
|
|
||
|
|
||
| def upgrade() -> None: |
Copilot is powered by AI and may make mistakes. Always verify output.
Add sanitize_log_value() to escape newlines and control characters in user-controlled values before logging. Also disable postgres checkpoint logs by default.
- alerts.py: Don't expose exception details to users, log internally - Health.tsx: Set isLoading=true at start of loadHealth for spinner
User-facing latency displays now show seconds (e.g., "2.5s" instead of "2500ms") for consistency with the settings UI which accepts seconds. Backend still stores values in ms internally.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
user.name,source.ip) in addition to Sigma fieldsChanges
Correlation Rules
entity_field_typecolumn to support "sigma" (default) and "direct" field typesGET /correlation-rules/common-log-fieldsendpoint for discovering common fields between index patternsGET /correlation-rules/common-sigma-fieldsendpoint for finding Sigma fields with valid mappings for both indexesFrontend
Other
Test plan