Skip to content

Update security ML jobs to include MITRE ATT&CK framework tactics and technique metdata #19220

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
ymao1 wants to merge 5 commits into
base: main
Choose a base branch
from
Draft

Update security ML jobs to include MITRE ATT&CK framework tactics and technique metdata #19220

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
d83420f
Adding threat tactics and techniques to ML jobs custom_settings
ymao1 May 26, 2026
c080cf7
Updating changelog
ymao1 May 26, 2026
b3bcf42
Updating manifests
ymao1 May 26, 2026
493e224
Fixing whitespace
ymao1 May 26, 2026
25d94e4
Fixing transforms
ymao1 May 26, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions packages/ded/changelog.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,3 +1,8 @@
- version: "3.1.1"
changes:
- description: Add threat_tactics and threat_techniques to ML job custom_settings.
type: enhancement
link: https://github.com/elastic/integrations/pull/19220
- version: "3.1.0"
changes:
- description: Update default index pattern for ML jobs to use the transform-generated index directly
Expand Down
6 changes: 3 additions & 3 deletions packages/ded/elasticsearch/transform/pivot_transform_ea/transform.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,12 +1,12 @@

dest:
index: ml_network_ded_ea-3.1.0
index: ml_network_ded_ea-3.1.1
aliases:
- alias: ml_network_ded_ea.latest
move_on_creation: true
- alias: ml_network_ded_ea.all
move_on_creation: false
pipeline: 3.1.0-ml_ded_ingest_pipeline
pipeline: 3.1.1-ml_ded_ingest_pipeline
description: This transform runs every 30 minutes and collects network logs to detect data exfiltration in your environment for the past month up to the runtime.
frequency: 30m
pivot:
Expand Down Expand Up @@ -103,5 +103,5 @@ sync:
delay: 120s
field: "@timestamp"
_meta:
fleet_transform_version: 3.1.0
fleet_transform_version: 3.1.1
run_as_kibana_system: false
61 changes: 54 additions & 7 deletions packages/ded/kibana/ml_module/ded-ml.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -87,7 +87,13 @@
"time_format": "epoch_ms"
},
"custom_settings": {
"created_by": "ml-module-ded"
"created_by": "ml-module-ded",
"threat_tactics": [
"TA0010"
],
"threat_techniques": [
"T1041"
]
}
}
},
Expand Down Expand Up @@ -127,7 +133,13 @@
"time_format": "epoch_ms"
},
"custom_settings": {
"created_by": "ml-module-ded"
"created_by": "ml-module-ded",
"threat_tactics": [
"TA0010"
],
"threat_techniques": [
"T1041"
]
}
}
},
Expand Down Expand Up @@ -168,7 +180,15 @@
"time_format": "epoch_ms"
},
"custom_settings": {
"created_by": "ml-module-ded"
"created_by": "ml-module-ded",
"threat_tactics": [
"TA0010",
"TA0011"
],
"threat_techniques": [
"T1041",
"T1571"
]
}
}
},
Expand Down Expand Up @@ -210,7 +230,13 @@
"time_format": "epoch_ms"
},
"custom_settings": {
"created_by": "ml-module-ded"
"created_by": "ml-module-ded",
"threat_tactics": [
"TA0010"
],
"threat_techniques": [
"T1041"
]
}
}
},
Expand Down Expand Up @@ -251,7 +277,14 @@
"time_format": "epoch_ms"
},
"custom_settings": {
"created_by": "ml-module-ded"
"created_by": "ml-module-ded",
"threat_tactics": [
"TA0010"
],
"threat_techniques": [
"T1052",
"T1052.001"
]
}
}
},
Expand Down Expand Up @@ -291,7 +324,14 @@
"time_format": "epoch_ms"
},
"custom_settings": {
"created_by": "ml-module-ded"
"created_by": "ml-module-ded",
"threat_tactics": [
"TA0010"
],
"threat_techniques": [
"T1052",
"T1052.001"
]
}
}
},
Expand Down Expand Up @@ -331,7 +371,14 @@
"time_format": "epoch_ms"
},
"custom_settings": {
"created_by": "ml-module-ded"
"created_by": "ml-module-ded",
"threat_tactics": [
"TA0010"
],
"threat_techniques": [
"T1011",
"T1011.001"
]
}
}
}
Expand Down
2 changes: 1 addition & 1 deletion packages/ded/manifest.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
format_version: 3.0.0
name: ded
title: "Data Exfiltration Detection"
version: 3.1.0
version: 3.1.1
source:
license: "Elastic-2.0"
description: "ML package to detect data exfiltration in your network and file data."
Expand Down
5 changes: 5 additions & 0 deletions packages/dga/changelog.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,9 @@
# newer versions go on top
- version: "3.0.1"
changes:
- description: Add threat_tactics and threat_techniques to ML job custom_settings.
type: enhancement
link: https://github.com/elastic/integrations/pull/19220
- version: "3.0.0"
changes:
- description: Adds new fields for entity resolution with Entity Analytics in Elastic Stack 9.4. Includes a new ML job.
Expand Down
11 changes: 10 additions & 1 deletion packages/dga/kibana/ml_module/dga-ml.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -57,7 +57,16 @@
"time_format": "epoch_ms"
},
"custom_settings": {
"created_by": "ml-module-dga"
"created_by": "ml-module-dga",
"threat_tactics": [
"TA0011"
],
"threat_techniques": [
"T1071",
"T1071.004",
"T1568",
"T1568.002"
]
}
}
}
Expand Down
2 changes: 1 addition & 1 deletion packages/dga/manifest.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
format_version: 3.0.4
name: dga
title: "Domain Generation Algorithm Detection"
version: 3.0.0
version: 3.0.1
source:
license: "Elastic-2.0"
description: "ML solution package to detect domain generation algorithm (DGA) activity in your network data."
Expand Down
5 changes: 5 additions & 0 deletions packages/lmd/changelog.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,9 @@
# newer versions go on top
- version: "3.1.2"
changes:
- description: Add threat_tactics and threat_techniques to ML job custom_settings.
type: enhancement
link: https://github.com/elastic/integrations/pull/19220
- version: "3.1.1"
changes:
- description: Update documentation on ML jobs requirements
Expand Down
2 changes: 1 addition & 1 deletion packages/lmd/elasticsearch/transform/pivot_transform_ea/transform.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -86,5 +86,5 @@ sync:
delay: 60s
field: '@timestamp'
_meta:
fleet_transform_version: 3.1.1
fleet_transform_version: 3.1.2
run_as_kibana_system: false
108 changes: 97 additions & 11 deletions packages/lmd/kibana/ml_module/lmd-ml.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -72,7 +72,14 @@
"time_format": "epoch_ms"
},
"custom_settings": {
"created_by": "ml-module-lmd"
"created_by": "ml-module-lmd",
"threat_tactics": [
"TA0008"
],
"threat_techniques": [
"T1210",
"T1570"
]
}
}
},
Expand Down Expand Up @@ -110,7 +117,16 @@
"time_format": "epoch_ms"
},
"custom_settings": {
"created_by": "ml-module-lmd"
"created_by": "ml-module-lmd",
"threat_tactics": [
"TA0008",
"TA0009"
],
"threat_techniques": [
"T1210",
"T1570",
"T1039"
]
}
}
},
Expand Down Expand Up @@ -148,7 +164,14 @@
"time_format": "epoch_ms"
},
"custom_settings": {
"created_by": "ml-module-lmd"
"created_by": "ml-module-lmd",
"threat_tactics": [
"TA0008"
],
"threat_techniques": [
"T1210",
"T1570"
]
}
}
},
Expand Down Expand Up @@ -186,7 +209,14 @@
"time_format": "epoch_ms"
},
"custom_settings": {
"created_by": "ml-module-lmd"
"created_by": "ml-module-lmd",
"threat_tactics": [
"TA0008"
],
"threat_techniques": [
"T1210",
"T1570"
]
}
}
},
Expand Down Expand Up @@ -232,7 +262,15 @@
"time_format": "epoch_ms"
},
"custom_settings": {
"created_by": "ml-module-lmd"
"created_by": "ml-module-lmd",
"threat_tactics": [
"TA0008"
],
"threat_techniques": [
"T1021",
"T1021.001",
"T1210"
]
}
}
},
Expand Down Expand Up @@ -278,7 +316,15 @@
"time_format": "epoch_ms"
},
"custom_settings": {
"created_by": "ml-module-lmd"
"created_by": "ml-module-lmd",
"threat_tactics": [
"TA0008"
],
"threat_techniques": [
"T1021",
"T1021.001",
"T1210"
]
}
}
},
Expand Down Expand Up @@ -324,7 +370,15 @@
"time_format": "epoch_ms"
},
"custom_settings": {
"created_by": "ml-module-lmd"
"created_by": "ml-module-lmd",
"threat_tactics": [
"TA0008"
],
"threat_techniques": [
"T1021",
"T1021.001",
"T1210"
]
}
}
},
Expand Down Expand Up @@ -361,7 +415,15 @@
"time_format": "epoch_ms"
},
"custom_settings": {
"created_by": "ml-module-lmd"
"created_by": "ml-module-lmd",
"threat_tactics": [
"TA0008"
],
"threat_techniques": [
"T1021",
"T1021.001",
"T1210"
]
}
}
},
Expand Down Expand Up @@ -399,7 +461,15 @@
"time_format": "epoch_ms"
},
"custom_settings": {
"created_by": "ml-module-lmd"
"created_by": "ml-module-lmd",
"threat_tactics": [
"TA0008"
],
"threat_techniques": [
"T1021",
"T1021.001",
"T1210"
]
}
}
},
Expand Down Expand Up @@ -437,7 +507,15 @@
"time_format": "epoch_ms"
},
"custom_settings": {
"created_by": "ml-module-lmd"
"created_by": "ml-module-lmd",
"threat_tactics": [
"TA0008"
],
"threat_techniques": [
"T1021",
"T1021.001",
"T1210"
]
}
}
},
Expand Down Expand Up @@ -483,7 +561,15 @@
"time_format": "epoch_ms"
},
"custom_settings": {
"created_by": "ml-module-lmd"
"created_by": "ml-module-lmd",
"threat_tactics": [
"TA0008"
],
"threat_techniques": [
"T1021",
"T1021.001",
"T1210"
]
}
}
}
Expand Down
2 changes: 1 addition & 1 deletion packages/lmd/manifest.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
format_version: 3.0.0
name: lmd
title: "Lateral Movement Detection"
version: 3.1.1
version: 3.1.2
source:
license: "Elastic-2.0"
description: "ML package to detect lateral movement based on file transfer activity and Windows RDP events."
Expand Down
Loading
Loading