Skip to content

December Schema Refresh #5420

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
shashank-elastic merged 6 commits into from
Dec 8, 2025
Merged

December Schema Refresh #5420

shashank-elastic merged 6 commits into from
Dec 8, 2025

Conversation

@shashank-elastic
Copy link
Contributor

@shashank-elastic shashank-elastic commented Dec 8, 2025 •
edited
Loading

Pull Request

Issue link(s): As part of release https://github.com/elastic/ia-trade-team/issues/766

Summary - What I changed

  • Refreshed ECS and Beats Schema
  • Refreshed API Schemas
Refreshed Integration Manifests and Schemas 
python -m detection_rules dev integrations build-schemas
Loaded config file: /Users/shashankks/elastic_workspace/detection-rules/.detection-rules-cfg.json

█▀▀▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄   ▄      █▀▀▄ ▄  ▄ ▄   ▄▄▄ ▄▄▄
█  █ █▄▄  █  █▄▄ █    █   █  █ █ █▀▄ █      █▄▄▀ █  █ █   █▄▄ █▄▄
█▄▄▀ █▄▄  █  █▄▄ █▄▄  █  ▄█▄ █▄█ █ ▀▄█      █ ▀▄ █▄▄█ █▄▄ █▄▄ ▄▄█

Building integration schemas...
processing google_workspace
processing endpoint
processing aws_bedrock
processing crowdstrike
processing beaconing
processing ded
processing fim
processing apm
processing github
processing problemchild
processing cyberarkpas
processing ti_rapid7_threat_command
processing system
processing windows
processing jamf_protect
processing sentinel_one_cloud_funnel
processing lmd
processing panw
processing o365
processing aws
processing gcp
processing azure
processing auditd_manager
processing dga
processing kubernetes
processing okta
processing m365_defender
processing cloud_defend
processing network_traffic
processing pad
processing sysmon_linux
processing elastic_security
processing google_secops
processing microsoft_sentinel
processing splunk
processing sentinel_one
processing azure_openai
processing fortinet_fortigate
processing suricata
processing nginx
processing apache
processing apache_tomcat
processing iis
final integrations manifests dumped: /Users/shashankks/elastic_workspace/detection-rules/detection_rules/etc/integration-schemas.json.gz
Time taken to generate schemas: 2.22 minutes
Refresh MITRE ATT&CK mappings 
❯ python -m detection_rules dev attack refresh-data
Loaded config file: /Users/shashankks/elastic_workspace/detection-rules/.detection-rules-cfg.json

█▀▀▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄   ▄      █▀▀▄ ▄  ▄ ▄   ▄▄▄ ▄▄▄
█  █ █▄▄  █  █▄▄ █    █   █  █ █ █▀▄ █      █▄▄▀ █  █ █   █▄▄ █▄▄
█▄▄▀ █▄▄  █  █▄▄ █▄▄  █  ▄█▄ █▄█ █ ▀▄█      █ ▀▄ █▄▄█ █▄▄ █▄▄ ▄▄█

Replaced file: /Users/shashankks/elastic_workspace/detection-rules/detection_rules/etc/attack-v18.0.0.json.gz with /Users/shashankks/elastic_workspace/detection-rules/detection_rules/etc/attack-v18.1.0.json.gz

detection-rules on  dec_schema_refresh [$✘!+?] is 📦 v1.5.20 via 🐍 v3.12.8 (.venv) on ☁️  shashank.suryanarayana@elastic.co took 9s 
❯ python -m detection_rules dev attack refresh-redirect-mappings
Loaded config file: /Users/shashankks/elastic_workspace/detection-rules/.detection-rules-cfg.json

█▀▀▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄   ▄      █▀▀▄ ▄  ▄ ▄   ▄▄▄ ▄▄▄
█  █ █▄▄  █  █▄▄ █    █   █  █ █ █▀▄ █      █▄▄▀ █  █ █   █▄▄ █▄▄
█▄▄▀ █▄▄  █  █▄▄ █▄▄  █  ▄█▄ █▄█ █ ▀▄█      █ ▀▄ █▄▄█ █▄▄ █▄▄ ▄▄█

refreshing data in attack_technique_redirects.json
refreshed mapping file: /Users/shashankks/elastic_workspace/detection-rules/detection_rules/etc/attack-technique-redirects.json

detection-rules on  dec_schema_refresh [$✘!+?] is 📦 v1.5.20 via 🐍 v3.12.8 (.venv) on ☁️  shashank.suryanarayana@elastic.co took 18s 
❯ python -m detection_rules dev attack update-rules             
Loaded config file: /Users/shashankks/elastic_workspace/detection-rules/.detection-rules-cfg.json

█▀▀▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄   ▄      █▀▀▄ ▄  ▄ ▄   ▄▄▄ ▄▄▄
█  █ █▄▄  █  █▄▄ █    █   █  █ █ █▀▄ █      █▄▄▀ █  █ █   █▄▄ █▄▄
█▄▄▀ █▄▄  █  █▄▄ █▄▄  █  ▄█▄ █▄█ █ ▀▄█      █ ▀▄ █▄▄█ █▄▄ █▄▄ ▄▄█

No rule changes needed
  • Kibana issue updated for MITRE Refresh - Check

How To Test

Checklist

  • Added a label for the type of pr: bug, enhancement, schema, maintenance, Rule: New, Rule: Deprecation, Rule: Tuning, Hunt: New, or Hunt: Tuning so guidelines can be generated
  • Added the meta:rapid-merge label if planning to merge within 24 hours
  • Secret and sensitive material has been managed correctly
  • Automated testing was updated or added to match the most common scenarios
  • Documentation and comments were added for features that require explanation

Contributor checklist

@shashank-elastic shashank-elastic self-assigned this Dec 8, 2025
@botelastic botelastic bot added the schema label Dec 8, 2025
@github-actions
Copy link
Contributor

github-actions bot commented Dec 8, 2025

Enhancement - Guidelines

These guidelines serve as a reminder set of considerations when addressing adding a new schema feature to the code.

Documentation and Context

  • Describe the feature enhancement in detail (alternative solutions, description of the solution, etc.) if not already documented in an issue.
  • Include additional context or screenshots.
  • Ensure the enhancement includes necessary updates to the documentation and versioning.

Code Standards and Practices

  • Code follows established design patterns within the repo and avoids duplication.
  • Ensure that the code is modular and reusable where applicable.

Testing

  • New unit tests have been added to cover the enhancement.
  • Existing unit tests have been updated to reflect the changes.
  • Provide evidence of testing and validating the enhancement (e.g., test logs, screenshots).
  • Validate that any rules affected by the enhancement are correctly updated.
  • Ensure that performance is not negatively impacted by the changes.
  • Verify that any release artifacts are properly generated and tested.
  • Conducted system testing, including fleet, import, and create APIs (e.g., run make test-cli, make test-remote-cli, make test-hunting-cli)

Additional Schema Related Checks

  • Verify that the enhancement works across all relevant environments (e.g., different OS versions).
  • Link to the relevant Kibana PR or issue provided
  • Test export/import flow:
    • Exported detection rule(s) from Kibana to showcase the feature(s)
    • Converted the exported ndjson file(s) to toml in the detection-rules repo
    • Re-exported the toml rule(s) to ndjson and re-imported into Kibana
  • Updated necessary unit tests to accommodate the feature
  • Incorporated a comprehensive test rule in unit tests for full schema coverage
  • Applied min_compat restrictions to limit the feature to a specified minimum stack version
  • Executed all unit tests locally with a test toml rule to confirm passing
  • Included Kibana PR implementer as an optional reviewer for insights on the feature
  • Implemented requisite downgrade functionality
  • Cross-referenced the feature with product documentation for consistency
  • Confirm that the proper version label is applied to the PR patch, minor, major.

@botelastic botelastic bot added Domain: Cloud Domain: Endpoint Integration: Azure azure related rules ML machine learning related rule OS: Linux labels Dec 8, 2025
@shashank-elastic shashank-elastic added patch and removed Integration: Azure azure related rules OS: Linux ML machine learning related rule Domain: Endpoint Domain: Cloud labels Dec 8, 2025
@tradebot-elastic
Copy link

tradebot-elastic commented Dec 8, 2025 •
edited
Loading

⛔️ Test failed

Results
  • ❌ Initial Access via File Upload Followed by GET Request (eql)
    • coverage_issue: no_rta
    • stack_validation_failed: no_rta
  • ❌ Spike in Azure Activity Logs Failed Messages (-)
    • coverage_issue: no_rta
    • stack_validation_failed: no_rta
  • ❌ Web Server Potential Remote File Inclusion Activity (esql)
    • coverage_issue: no_rta
    • stack_validation_failed: no_rta
  • ❌ Unusual Web Server Command Execution (kuery)
    • coverage_issue: no_rta
    • stack_validation_failed: no_rta
  • ❌ Unusual Country for an Azure Activity Logs Event (-)
    • coverage_issue: no_rta
    • stack_validation_failed: no_rta
  • ❌ Unusual Azure Activity Logs Event for a User (-)
    • coverage_issue: no_rta
    • stack_validation_failed: no_rta
  • ❌ Potential HTTP Downgrade Attack (kuery)
    • coverage_issue: no_rta
    • stack_validation_failed: no_rta
  • ❌ Rare Azure Activity Logs Event Failures (-)
    • coverage_issue: no_rta
    • stack_validation_failed: no_rta
  • ❌ Pod or Container Creation with Suspicious Command-Line (eql)
    • coverage_issue: no_rta
    • stack_validation_failed: no_rta
  • ❌ Unusual City for an Azure Activity Logs Event (-)
    • coverage_issue: no_rta
    • stack_validation_failed: no_rta

@botelastic botelastic bot added Domain: Cloud Domain: Endpoint Integration: Azure azure related rules ML machine learning related rule OS: Linux labels Dec 8, 2025
@tradebot-elastic
Copy link

tradebot-elastic commented Dec 8, 2025 •
edited
Loading

⛔️ Test failed

Results
  • ❌ Initial Access via File Upload Followed by GET Request (eql)
    • coverage_issue: no_rta
    • stack_validation_failed: no_rta
  • ❌ Spike in Azure Activity Logs Failed Messages (-)
    • coverage_issue: no_rta
    • stack_validation_failed: no_rta
  • ❌ Web Server Potential Remote File Inclusion Activity (esql)
    • coverage_issue: no_rta
    • stack_validation_failed: no_rta
  • ❌ Unusual Web Server Command Execution (kuery)
    • coverage_issue: no_rta
    • stack_validation_failed: no_rta
  • ❌ Unusual Country for an Azure Activity Logs Event (-)
    • coverage_issue: no_rta
    • stack_validation_failed: no_rta
  • ❌ Unusual Azure Activity Logs Event for a User (-)
    • coverage_issue: no_rta
    • stack_validation_failed: no_rta
  • ❌ Potential HTTP Downgrade Attack (kuery)
    • coverage_issue: no_rta
    • stack_validation_failed: no_rta
  • ❌ Rare Azure Activity Logs Event Failures (-)
    • coverage_issue: no_rta
    • stack_validation_failed: no_rta
  • ❌ Pod or Container Creation with Suspicious Command-Line (eql)
    • coverage_issue: no_rta
    • stack_validation_failed: no_rta
  • ❌ Unusual City for an Azure Activity Logs Event (-)
    • coverage_issue: no_rta
    • stack_validation_failed: no_rta

@tradebot-elastic
Copy link

tradebot-elastic commented Dec 8, 2025 •
edited
Loading

⛔️ Test failed

Results
  • ❌ Initial Access via File Upload Followed by GET Request (eql)
    • coverage_issue: no_rta
    • stack_validation_failed: no_rta
  • ❌ Spike in Azure Activity Logs Failed Messages (-)
    • coverage_issue: no_rta
    • stack_validation_failed: no_rta
  • ❌ Web Server Potential Remote File Inclusion Activity (esql)
    • coverage_issue: no_rta
    • stack_validation_failed: no_rta
  • ❌ Unusual Web Server Command Execution (kuery)
    • coverage_issue: no_rta
    • stack_validation_failed: no_rta
  • ❌ Unusual Country for an Azure Activity Logs Event (-)
    • coverage_issue: no_rta
    • stack_validation_failed: no_rta
  • ❌ Unusual Azure Activity Logs Event for a User (-)
    • coverage_issue: no_rta
    • stack_validation_failed: no_rta
  • ❌ Web Server Local File Inclusion Activity (esql)
    • coverage_issue: no_rta
    • stack_validation_failed: no_rta
  • ❌ Potential HTTP Downgrade Attack (kuery)
    • coverage_issue: no_rta
    • stack_validation_failed: no_rta
  • ❌ Rare Azure Activity Logs Event Failures (-)
    • coverage_issue: no_rta
    • stack_validation_failed: no_rta
  • ❌ Pod or Container Creation with Suspicious Command-Line (eql)
    • coverage_issue: no_rta
    • stack_validation_failed: no_rta
  • ❌ Unusual City for an Azure Activity Logs Event (-)
    • coverage_issue: no_rta
    • stack_validation_failed: no_rta

@tradebot-elastic
Copy link

tradebot-elastic commented Dec 8, 2025 •
edited
Loading

⛔️ Test failed

Results
  • ❌ Initial Access via File Upload Followed by GET Request (eql)
    • coverage_issue: no_rta
    • stack_validation_failed: no_rta
  • ❌ Spike in Azure Activity Logs Failed Messages (-)
    • coverage_issue: no_rta
    • stack_validation_failed: no_rta
  • ❌ Web Server Potential Remote File Inclusion Activity (esql)
    • coverage_issue: no_rta
    • stack_validation_failed: no_rta
  • ❌ Unusual Web Server Command Execution (kuery)
    • coverage_issue: no_rta
    • stack_validation_failed: no_rta
  • ❌ Unusual Country for an Azure Activity Logs Event (-)
    • coverage_issue: no_rta
    • stack_validation_failed: no_rta
  • ❌ Unusual Azure Activity Logs Event for a User (-)
    • coverage_issue: no_rta
    • stack_validation_failed: no_rta
  • ❌ Web Server Local File Inclusion Activity (esql)
    • coverage_issue: no_rta
    • stack_validation_failed: no_rta
  • ❌ Potential HTTP Downgrade Attack (kuery)
    • coverage_issue: no_rta
    • stack_validation_failed: no_rta
  • ❌ Rare Azure Activity Logs Event Failures (-)
    • coverage_issue: no_rta
    • stack_validation_failed: no_rta
  • ❌ Pod or Container Creation with Suspicious Command-Line (eql)
    • coverage_issue: no_rta
    • stack_validation_failed: no_rta
  • ❌ Unusual City for an Azure Activity Logs Event (-)
    • coverage_issue: no_rta
    • stack_validation_failed: no_rta

eric-forte-elastic
eric-forte-elastic approved these changes Dec 8, 2025
View reviewed changes
Copy link
Contributor

@eric-forte-elastic eric-forte-elastic left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Manual review, looks good to me! 👍

@tradebot-elastic
Copy link

tradebot-elastic commented Dec 8, 2025 •
edited
Loading

⛔️ Test failed

Results
  • ❌ Initial Access via File Upload Followed by GET Request (eql)
    • coverage_issue: no_rta
    • stack_validation_failed: no_rta
  • ❌ Spike in Azure Activity Logs Failed Messages (-)
    • coverage_issue: no_rta
    • stack_validation_failed: no_rta
  • ❌ Web Server Potential Remote File Inclusion Activity (esql)
    • coverage_issue: no_rta
    • stack_validation_failed: no_rta
  • ❌ Unusual Web Server Command Execution (kuery)
    • coverage_issue: no_rta
    • stack_validation_failed: no_rta
  • ❌ Unusual Country for an Azure Activity Logs Event (-)
    • coverage_issue: no_rta
    • stack_validation_failed: no_rta
  • ❌ Unusual Azure Activity Logs Event for a User (-)
    • coverage_issue: no_rta
    • stack_validation_failed: no_rta
  • ❌ Web Server Local File Inclusion Activity (esql)
    • coverage_issue: no_rta
    • stack_validation_failed: no_rta
  • ❌ Potential HTTP Downgrade Attack (kuery)
    • coverage_issue: no_rta
    • stack_validation_failed: no_rta
  • ❌ Rare Azure Activity Logs Event Failures (-)
    • coverage_issue: no_rta
    • stack_validation_failed: no_rta
  • ❌ Pod or Container Creation with Suspicious Command-Line (eql)
    • coverage_issue: no_rta
    • stack_validation_failed: no_rta
  • ❌ Unusual City for an Azure Activity Logs Event (-)
    • coverage_issue: no_rta
    • stack_validation_failed: no_rta

@shashank-elastic
Copy link
Contributor Author

shashank-elastic commented Dec 8, 2025

During the Merge of PR

❯ python missing_guides.py          
Loaded config file: /Users/shashankks/elastic_workspace/detection-rules/.detection-rules-cfg.json
All rules have the Investigation Guide

@shashank-elastic shashank-elastic merged commit 58a5143 into main Dec 8, 2025
19 of 21 checks passed
@shashank-elastic shashank-elastic deleted the dec_schema_refresh branch December 8, 2025 16:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@traut traut traut approved these changes

@Mikaayenson Mikaayenson Mikaayenson approved these changes

@eric-forte-elastic eric-forte-elastic eric-forte-elastic approved these changes

Assignees

@shashank-elastic shashank-elastic

Labels

backport: auto Domain: Cloud Domain: Endpoint Integration: Azure azure related rules ML machine learning related rule OS: Linux patch schema

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@shashank-elastic @tradebot-elastic @traut @Mikaayenson @eric-forte-elastic