[Security Solution] Add new Timeline Templates #125172

DefSecSentinel · 2022-02-09T23:24:50Z

part of elastic/security-team/issues/2091

Summary

Adds additional comprehensive timelines for process, file, network, and registry events.

Investigating in these timelines will bring up the alert related events and additional events leading up to the alert for a more comprehensive view than the prior generic timeline templates. See each of the below images for the various event types and rows included in each timeline.

Successfully imported each timeline.

Timeline fields

Comprehensive Process Timeline

Comprehensive File Timeline

Comprehensive Network Timeline

Comprehensive Registry Timeline

For maintainers

This was checked for breaking API changes and was labeled appropriately

brokensound77 · 2022-02-10T05:43:10Z

@elasticmachine merge upstream

brokensound77

Looks good to me if these tested fine on an 8.1 stack. Thanks @DefSecSentinel

angorayc · 2022-02-10T17:45:55Z

...ugins/security_solution/server/lib/detection_engine/rules/prepackaged_timelines/file_ex.json

@@ -0,0 +1 @@
+{"savedObjectId":null,"version":null,"columns":[{"columnHeaderType":"not-filtered","id":"@timestamp","type":"number"},{"aggregatable":false,"description":"For log events the message field contains the log message, optimized for viewing in a log viewer. For structured logs without an original message field, other fields can be concatenated to form a human-readable summary of the event. If multiple messages exist, they can be combined into one message.","columnHeaderType":"not-filtered","id":"message","category":"base","type":"string","example":"Hello World"},{"aggregatable":true,"description":"Process name. Sometimes called program name or similar.","columnHeaderType":"not-filtered","id":"process.name","category":"process","type":"string","example":"ssh"},{"aggregatable":true,"description":"The action captured by the event. This describes the information in the event. It is more specific than `event.category`. Examples are `group-add`, `process-started`, `file-created`. The value is normally defined by the implementer.","columnHeaderType":"not-filtered","id":"event.action","category":"event","type":"string","example":"user-password-change"},{"aggregatable":true,"description":"Name of the file including the extension, without the directory.","columnHeaderType":"not-filtered","id":"file.name","category":"file","type":"string","example":"example.png"},{"columnHeaderType":"not-filtered","id":"file.size"},{"columnHeaderType":"not-filtered","id":"file.path"},{"columnHeaderType":"not-filtered","id":"host.name"},{"columnHeaderType":"not-filtered","id":"user.name"}],"dataProviders":[{"excluded":false,"and":[],"kqlQuery":"","name":"file","queryMatch":{"displayValue":null,"field":"event.category","displayField":null,"value":"file","operator":":"},"id":"timeline-1-fd6cfcf0-cfbd-4a42-b58e-9efccca7ecdd","type":"default","enabled":true},{"excluded":false,"and":[],"kqlQuery":"","name":"{signal.group.id}","queryMatch":{"displayValue":null,"field":"signal.group.id","displayField":null,"value":"{signal.group.id}","operator":":"},"id":"timeline-1-072c4726-d198-41c5-a3dc-561062c454a9","type":"template","enabled":true},{"excluded":false,"and":[],"kqlQuery":"","name":"{signal.original_event.id}","queryMatch":{"field":"signal.original_event.id","value":"{signal.original_event.id}","operator":":"},"id":"timeline-1-dba415d2-9968-4961-8b0f-a381c3d28c87","type":"template","enabled":true}],"description":"","eventType":"all","filters":[],"kqlMode":"filter","timelineType":"template","kqlQuery":{"filterQuery":null},"title":"Comprehensive File Timeline","sort":[{"columnType":"number","sortDirection":"desc","columnId":"@timestamp"}],"templateTimelineId":"4d4c0b59-ea83-483f-b8c1-8c360ee53c5c","templateTimelineVersion":1,"created":1618433758898,"createdBy":"1674059739","updated":1618500709024,"updatedBy":"elastic","dateRange":{"start":"2021-04-13T20:55:34.156Z","end":"2021-04-14T20:55:34.157Z"},"indexNames":[".siem-signals-default","apm-*-transaction*","logs-*"],"eqlOptions":{"tiebreakerField":"","size":100,"query":"","eventCategoryField":"event.category","timestampField":"@timestamp"},"savedQueryId":null,"eventNotes":[],"globalNotes":[],"pinnedEventIds":[],"status":"immutable"}


Please update indexNames to [] (to avoid incorrect data view on timeline) and bump templateTimelineVersion to 2 at the same time.

Please do this change for all the json file you added.

Go to cd x-pack/plugins/security_solution/server/lib/detection_engine/scripts,
run sh ./timelines/regen_prepackage_timelines_index.sh

Please check x-pack/plugins/security_solution/server/lib/detection_engine/rules/prepackaged_timelines/index.ndjson is generated with the updated value.

Go to templates' tab and check if they are displayed properly.

More info: https://github.com/elastic/kibana/blob/main/x-pack/plugins/security_solution/server/lib/detection_engine/rules/prepackaged_timelines/README.md#how-to-update-an-existing-prepackage-timeline

Before updating the indexNames - the timeline data view is set incorrectly:

After updating the indexNames - it should be correctly set to Security Default Data View

Made changes and committed them. Thank you for catching this.

DefSecSentinel · 2022-02-10T21:53:59Z

@elasticmachine merge upstream

angorayc

Good to merge once all the builds pass, thank you making the changes.

DefSecSentinel · 2022-02-11T15:18:43Z

@elasticmachine merge upstream

FrankHassanabad · 2022-02-11T21:11:00Z

...ugins/security_solution/server/lib/detection_engine/rules/prepackaged_timelines/index.ndjson

-{"savedObjectId":null,"version":null,"columns":[{"indexes":null,"name":null,"columnHeaderType":"not-filtered","id":"@timestamp","searchable":null},{"indexes":null,"name":null,"columnHeaderType":"not-filtered","id":"kibana.alert.rule.description","searchable":null},{"indexes":null,"aggregatable":true,"name":null,"description":"The action captured by the event.\n\nThis describes the information in the event. It is more specific than `event.category`.\nExamples are `group-add`, `process-started`, `file-created`. The value is\nnormally defined by the implementer.","columnHeaderType":"not-filtered","id":"event.action","category":"event","type":"string","searchable":null,"example":"user-password-change"},{"indexes":null,"aggregatable":true,"name":null,"description":"Array of process arguments, starting with the absolute path to\nthe executable.\n\nMay be filtered to protect sensitive information.","columnHeaderType":"not-filtered","id":"process.args","category":"process","type":"string","searchable":null,"example":"[\"/usr/bin/ssh\",\"-l\",\"user\",\"10.0.0.16\"]"},{"indexes":null,"name":null,"columnHeaderType":"not-filtered","id":"process.pid","searchable":null},{"indexes":null,"aggregatable":true,"name":null,"description":"IP address of the source (IPv4 or IPv6).","columnHeaderType":"not-filtered","id":"source.ip","category":"source","type":"ip","searchable":null},{"indexes":null,"aggregatable":true,"name":null,"description":"Port of the source.","columnHeaderType":"not-filtered","id":"source.port","category":"source","type":"number","searchable":null},{"indexes":null,"aggregatable":true,"name":null,"description":"IP address of the destination (IPv4 or IPv6).","columnHeaderType":"not-filtered","id":"destination.ip","category":"destination","type":"ip","searchable":null},{"indexes":null,"name":null,"columnHeaderType":"not-filtered","id":"destination.port","searchable":null},{"indexes":null,"aggregatable":true,"name":null,"description":"Short name or login of the user.","columnHeaderType":"not-filtered","id":"user.name","category":"user","type":"string","searchable":null,"example":"albert"},{"indexes":null,"name":null,"columnHeaderType":"not-filtered","id":"host.name","searchable":null}],"dataProviders":[{"and":[{"enabled":true,"excluded":false,"id":"timeline-1-e37e37c5-a6e7-4338-af30-47bfbc3c0e1e","kqlQuery":"","name":"{destination.ip}","queryMatch":{"displayField":"destination.ip","displayValue":"{destination.ip}","field":"destination.ip","operator":":","value":"{destination.ip}"},"type":"template"}],"enabled":true,"excluded":false,"id":"timeline-1-ec778f01-1802-40f0-9dfb-ed8de1f656cb","kqlQuery":"","name":"{source.ip}","queryMatch":{"displayField":"source.ip","displayValue":"{source.ip}","field":"source.ip","operator":":","value":"{source.ip}"},"type":"template"}],"description":"","eventType":"all","filters":[],"kqlMode":"filter","kqlQuery":{"filterQuery":{"kuery":{"kind":"kuery","expression":""},"serializedQuery":""}},"title":"Generic Network Timeline","timelineType":"template","templateTimelineVersion":2,"templateTimelineId":"91832785-286d-4ebe-b884-1a208d111a70","dateRange":{"start":1588255858373,"end":1588256218373},"savedQueryId":null,"sort":{"columnId":"@timestamp","sortDirection":"desc"},"created":1594735573866,"createdBy":"Elastic","updated":1611609960850,"updatedBy":"Elastic","eventNotes":[],"globalNotes":[],"pinnedEventIds":[],"status":"immutable"}
-{"savedObjectId":null,"version":null,"columns":[{"indexes":null,"name":null,"columnHeaderType":"not-filtered","id":"@timestamp","searchable":null},{"indexes":null,"name":null,"columnHeaderType":"not-filtered","id":"kibana.alert.rule.description","searchable":null},{"indexes":null,"name":null,"columnHeaderType":"not-filtered","id":"event.action","searchable":null},{"indexes":null,"name":null,"columnHeaderType":"not-filtered","id":"process.name","searchable":null},{"indexes":null,"aggregatable":true,"name":null,"description":"The working directory of the process.","columnHeaderType":"not-filtered","id":"process.working_directory","category":"process","type":"string","searchable":null,"example":"/home/alice"},{"indexes":null,"aggregatable":true,"name":null,"description":"Array of process arguments, starting with the absolute path to\nthe executable.\n\nMay be filtered to protect sensitive information.","columnHeaderType":"not-filtered","id":"process.args","category":"process","type":"string","searchable":null,"example":"[\"/usr/bin/ssh\",\"-l\",\"user\",\"10.0.0.16\"]"},{"indexes":null,"name":null,"columnHeaderType":"not-filtered","id":"process.pid","searchable":null},{"indexes":null,"aggregatable":true,"name":null,"description":"Absolute path to the process executable.","columnHeaderType":"not-filtered","id":"process.parent.executable","category":"process","type":"string","searchable":null,"example":"/usr/bin/ssh"},{"indexes":null,"aggregatable":true,"name":null,"description":"Array of process arguments.\n\nMay be filtered to protect sensitive information.","columnHeaderType":"not-filtered","id":"process.parent.args","category":"process","type":"string","searchable":null,"example":"[\"ssh\",\"-l\",\"user\",\"10.0.0.16\"]"},{"indexes":null,"aggregatable":true,"name":null,"description":"Process id.","columnHeaderType":"not-filtered","id":"process.parent.pid","category":"process","type":"number","searchable":null,"example":"4242"},{"indexes":null,"aggregatable":true,"name":null,"description":"Short name or login of the user.","columnHeaderType":"not-filtered","id":"user.name","category":"user","type":"string","searchable":null,"example":"albert"},{"indexes":null,"aggregatable":true,"name":null,"description":"Name of the host.\n\nIt can contain what `hostname` returns on Unix systems, the fully qualified\ndomain name, or a name specified by the user. The sender decides which value\nto use.","columnHeaderType":"not-filtered","id":"host.name","category":"host","type":"string","searchable":null}],"dataProviders":[{"excluded":false,"and":[],"kqlQuery":"","name":"{process.name}","queryMatch":{"displayValue":null,"field":"process.name","displayField":null,"value":"{process.name}","operator":":"},"id":"timeline-1-8622010a-61fb-490d-b162-beac9c36a853","type":"template","enabled":true}],"description":"","eventType":"all","filters":[],"kqlMode":"filter","kqlQuery":{"filterQuery":{"kuery":{"kind":"kuery","expression":""},"serializedQuery":""}},"title":"Generic Process Timeline","timelineType":"template","templateTimelineVersion":2,"templateTimelineId":"76e52245-7519-4251-91ab-262fb1a1728c","dateRange":{"start":1588161020848,"end":1588162280848},"savedQueryId":null,"sort":{"columnId":"@timestamp","sortDirection":"desc"},"created":1594735629389,"createdBy":"Elastic","updated":1611609848602,"updatedBy":"Elastic","eventNotes":[],"globalNotes":[],"pinnedEventIds":[],"status":"immutable"}
-{"savedObjectId":null,"version":null,"columns":[{"columnHeaderType":"not-filtered","id":"@timestamp"},{"columnHeaderType":"not-filtered","id":"kibana.alert.rule.description"},{"aggregatable":true,"description":"The action captured by the event.\n\nThis describes the information in the event. It is more specific than `event.category`.\nExamples are `group-add`, `process-started`, `file-created`. The value is\nnormally defined by the implementer.","columnHeaderType":"not-filtered","id":"event.action","category":"event","type":"string","example":"user-password-change"},{"aggregatable":true,"description":"Array of process arguments, starting with the absolute path to\nthe executable.\n\nMay be filtered to protect sensitive information.","columnHeaderType":"not-filtered","id":"process.args","category":"process","type":"string","example":"[\"/usr/bin/ssh\",\"-l\",\"user\",\"10.0.0.16\"]"},{"columnHeaderType":"not-filtered","id":"process.pid"},{"aggregatable":true,"description":"IP address of the source (IPv4 or IPv6).","columnHeaderType":"not-filtered","id":"source.ip","category":"source","type":"ip"},{"aggregatable":true,"description":"Port of the source.","columnHeaderType":"not-filtered","id":"source.port","category":"source","type":"number"},{"aggregatable":true,"description":"IP address of the destination (IPv4 or IPv6).","columnHeaderType":"not-filtered","id":"destination.ip","category":"destination","type":"ip"},{"columnHeaderType":"not-filtered","id":"destination.port"},{"aggregatable":true,"description":"Short name or login of the user.","columnHeaderType":"not-filtered","id":"user.name","category":"user","type":"string","example":"albert"},{"columnHeaderType":"not-filtered","id":"host.name"}],"dataProviders":[{"excluded":false,"and":[{"excluded":false,"kqlQuery":"","name":"{threat.enrichments.matched.type}","queryMatch":{"displayValue":null,"field":"threat.enrichments.matched.type","displayField":null,"value":"{threat.enrichments.matched.type}","operator":":"},"id":"timeline-1-ae18ef4b-f690-4122-a24d-e13b6818fba8","type":"template","enabled":true},{"excluded":false,"kqlQuery":"","name":"{threat.enrichments.matched.field}","queryMatch":{"displayValue":null,"field":"threat.enrichments.matched.field","displayField":null,"value":"{threat.enrichments.matched.field}","operator":":"},"id":"timeline-1-7b4cf27e-6788-4d8e-9188-7687f0eba0f2","type":"template","enabled":true}],"kqlQuery":"","name":"{threat.enrichments.matched.atomic}","queryMatch":{"displayValue":null,"field":"threat.enrichments.matched.atomic","displayField":null,"value":"{threat.enrichments.matched.atomic}","operator":":"},"id":"timeline-1-7db7d278-a80a-4853-971a-904319c50777","type":"template","enabled":true}],"description":"This Timeline template is for alerts generated by Indicator Match detection rules.","eqlOptions":{"eventCategoryField":"event.category","tiebreakerField":"","timestampField":"@timestamp","query":"","size":100},"eventType":"alert","filters":[],"kqlMode":"filter","kqlQuery":{"filterQuery":{"kuery":{"kind":"kuery","expression":""},"serializedQuery":""}},"dataViewId": "security-solution","indexNames":[],"title":"Generic Threat Match Timeline","timelineType":"template","templateTimelineVersion":3,"templateTimelineId":"495ad7a7-316e-4544-8a0f-9c098daee76e","dateRange":{"start":1588161020848,"end":1588162280848},"savedQueryId":null,"sort":[{"sortDirection":"desc","columnId":"@timestamp"}],"created":1616696609311,"createdBy":"elastic","updated":1616788372794,"updatedBy":"elastic","eventNotes":[],"globalNotes":[],"pinnedEventIds":[],"status":"immutable"}
+{"savedObjectId":null,"version":null,"columns":[{"columnHeaderType":"not-filtered","id":"@timestamp","type":"number"},{"aggregatable":false,"description":"For log events the message field contains the log message, optimized for viewing in a log viewer. For structured logs without an original message field, other fields can be concatenated to form a human-readable summary of the event. If multiple messages exist, they can be combined into one message.","columnHeaderType":"not-filtered","id":"message","category":"base","type":"string","example":"Hello World"},{"aggregatable":true,"description":"Process name. Sometimes called program name or similar.","columnHeaderType":"not-filtered","id":"process.name","category":"process","type":"string","example":"ssh"},{"aggregatable":true,"description":"The action captured by the event. This describes the information in the event. It is more specific than `event.category`. Examples are `group-add`, `process-started`, `file-created`. The value is normally defined by the implementer.","columnHeaderType":"not-filtered","id":"event.action","category":"event","type":"string","example":"user-password-change"},{"aggregatable":true,"description":"Name of the file including the extension, without the directory.","columnHeaderType":"not-filtered","id":"file.name","category":"file","type":"string","example":"example.png"},{"columnHeaderType":"not-filtered","id":"file.size"},{"columnHeaderType":"not-filtered","id":"file.path"},{"columnHeaderType":"not-filtered","id":"host.name"},{"columnHeaderType":"not-filtered","id":"user.name"}],"dataProviders":[{"excluded":false,"and":[],"kqlQuery":"","name":"file","queryMatch":{"displayValue":null,"field":"event.category","displayField":null,"value":"file","operator":":"},"id":"timeline-1-fd6cfcf0-cfbd-4a42-b58e-9efccca7ecdd","type":"default","enabled":true},{"excluded":false,"and":[],"kqlQuery":"","name":"{signal.group.id}","queryMatch":{"displayValue":null,"field":"signal.group.id","displayField":null,"value":"{signal.group.id}","operator":":"},"id":"timeline-1-072c4726-d198-41c5-a3dc-561062c454a9","type":"template","enabled":true},{"excluded":false,"and":[],"kqlQuery":"","name":"{signal.original_event.id}","queryMatch":{"field":"signal.original_event.id","value":"{signal.original_event.id}","operator":":"},"id":"timeline-1-dba415d2-9968-4961-8b0f-a381c3d28c87","type":"template","enabled":true}],"description":"","eventType":"all","filters":[],"kqlMode":"filter","timelineType":"template","kqlQuery":{"filterQuery":null},"title":"Comprehensive File Timeline","sort":[{"columnType":"number","sortDirection":"desc","columnId":"@timestamp"}],"templateTimelineId":"4d4c0b59-ea83-483f-b8c1-8c360ee53c5c","templateTimelineVersion":2,"created":1618433758898,"createdBy":"1674059739","updated":1618500709024,"updatedBy":"elastic","dateRange":{"start":"2021-04-13T20:55:34.156Z","end":"2021-04-14T20:55:34.157Z"},"indexNames":[],"eqlOptions":{"tiebreakerField":"","size":100,"query":"","eventCategoryField":"event.category","timestampField":"@timestamp"},"savedQueryId":null,"eventNotes":[],"globalNotes":[],"pinnedEventIds":[],"status":"immutable"}{"savedObjectId":null,"version":null,"columns":[{"indexes":null,"name":null,"columnHeaderType":"not-filtered","id":"@timestamp","searchable":null},{"indexes":null,"name":null,"columnHeaderType":"not-filtered","id":"kibana.alert.rule.description","searchable":null},{"indexes":null,"aggregatable":true,"name":null,"description":"The action captured by the event.\n\nThis describes the information in the event. It is more specific than `event.category`.\nExamples are `group-add`, `process-started`, `file-created`. The value is\nnormally defined by the implementer.","columnHeaderType":"not-filtered","id":"event.action","category":"event","type":"string","searchable":null,"example":"user-password-change"},{"indexes":null,"aggregatable":true,"name":null,"description":"Array of process arguments, starting with the absolute path to\nthe executable.\n\nMay be filtered to protect sensitive information.","columnHeaderType":"not-filtered","id":"process.args","category":"process","type":"string","searchable":null,"example":"[\"/usr/bin/ssh\",\"-l\",\"user\",\"10.0.0.16\"]"},{"indexes":null,"name":null,"columnHeaderType":"not-filtered","id":"process.pid","searchable":null},{"indexes":null,"aggregatable":true,"name":null,"description":"IP address of the source (IPv4 or IPv6).","columnHeaderType":"not-filtered","id":"source.ip","category":"source","type":"ip","searchable":null},{"indexes":null,"aggregatable":true,"name":null,"description":"Port of the source.","columnHeaderType":"not-filtered","id":"source.port","category":"source","type":"number","searchable":null},{"indexes":null,"aggregatable":true,"name":null,"description":"IP address of the destination (IPv4 or IPv6).","columnHeaderType":"not-filtered","id":"destination.ip","category":"destination","type":"ip","searchable":null},{"indexes":null,"name":null,"columnHeaderType":"not-filtered","id":"destination.port","searchable":null},{"indexes":null,"aggregatable":true,"name":null,"description":"Short name or login of the user.","columnHeaderType":"not-filtered","id":"user.name","category":"user","type":"string","searchable":null,"example":"albert"},{"indexes":null,"name":null,"columnHeaderType":"not-filtered","id":"host.name","searchable":null}],"dataProviders":[{"and":[{"enabled":true,"excluded":false,"id":"timeline-1-e37e37c5-a6e7-4338-af30-47bfbc3c0e1e","kqlQuery":"","name":"{destination.ip}","queryMatch":{"displayField":"destination.ip","displayValue":"{destination.ip}","field":"destination.ip","operator":":","value":"{destination.ip}"},"type":"template"}],"enabled":true,"excluded":false,"id":"timeline-1-ec778f01-1802-40f0-9dfb-ed8de1f656cb","kqlQuery":"","name":"{source.ip}","queryMatch":{"displayField":"source.ip","displayValue":"{source.ip}","field":"source.ip","operator":":","value":"{source.ip}"},"type":"template"}],"description":"","eventType":"all","filters":[],"kqlMode":"filter","kqlQuery":{"filterQuery":{"kuery":{"kind":"kuery","expression":""},"serializedQuery":""}},"title":"Generic Network Timeline","timelineType":"template","templateTimelineVersion":2,"templateTimelineId":"91832785-286d-4ebe-b884-1a208d111a70","dateRange":{"start":1588255858373,"end":1588256218373},"savedQueryId":null,"sort":{"columnId":"@timestamp","sortDirection":"desc"},"created":1594735573866,"createdBy":"Elastic","updated":1611609960850,"updatedBy":"Elastic","eventNotes":[],"globalNotes":[],"pinnedEventIds":[],"status":"immutable"}


You sure these are ndjson meaning they are 1 and only 1 valid JSON document per line?

I took one of them and tried to format it and got an error about their being multiple JSON roots in my IDE.

DefSecSentinel · 2022-02-16T21:26:11Z

@elasticmachine merge upstream

brokensound77 · 2022-02-17T05:01:51Z

@elasticmachine merge upstream

kibanamachine · 2022-03-03T18:59:41Z

⚪ Backport skipped

The pull request was not backported as there were no branches to backport to. If this is a mistake, please apply the desired version labels or run the backport tool manually.

Manual backport

To create the backport manually run:

node scripts/backport --pr 125172