[New Rule] Threat intel indicator match rule #1133

peasead · 2021-04-20T20:16:02Z

Issues

Resolves #1065
Resolves https://github.com/elastic/protections-team/issues/368

Summary

This adds an Indicator Match rule for the Detection Engine. The indicators are provided by the Threat Intel Filebeat module from the following sources:

Abuse Malware
Abuse URL
Anomali Limo
AlienVault OTX
Malware Bazaar
MISP

Of note, currently, the Indicator Match rule type isn't in the detection_rules module, so this rule was exported from the Detection Engine and converted to TOML manually. This will likely need some massaging to pass unit tests.

Contributor checklist

Have you signed the contributor license agreement? Yes
Have you followed the contributor guidelines? Yes

brokensound77 · 2021-04-21T22:47:12Z

👋

Add threat_match rule type #1138 adds support for the rule type

Some other dependencies before this will pass

need to add support for wildcards in field names (Add wildcard field support to KQL #1139)
need to refresh to latest beats (and ECS) schemas (Refresh beats and ecs schemas #1140)

A separate PR will be added for CLI support of the rule type

rules/cross-platform/indicator-match.toml

Co-authored-by: Ross Wolf <31489089+rw-access@users.noreply.github.com>

brokensound77 · 2021-04-22T20:14:16Z

rules/cross-platform/threat_intel_module_match.toml

+"""
+
+
+[[rule.threat_filters]]


I had to manually lint/format this @rw-access . There appears to be an issue with the MarshmallowDataclassMixin.to_dict method on schema.dump, but I think it is only happening on threat_match rules, so it may not be rendering the schema properly. As of now, it strips a lot of fields

rw-access · 2021-04-22T20:47:29Z

detection_rules/beats.py

@@ -174,6 +174,9 @@ def get_beats_sub_schema(schema: dict, beat: str, module: str, *datasets: str):
        dataset_dir = module_dir.get("folders", {}).get(dataset, {})
        flattened.extend(get_field_schema(dataset_dir, prefix=module + ".", include_common=True))

+    # we also need to capture (beta?) fields which are directly within the module _meta.files.fields
+    flattened.extend(get_field_schema(module_dir, include_common=True))


hmm... is this the right call?
IIRC, this blew up the schema significantly before and added more fields that shouldn't be there.
do you have a sense what new fields and how many were added?

not for every module, but threatintel.indicator* (and some other threatintel.* specifically was entirely embedded under this. Not sure if this is actually where beta fields are stuffed, but it seems consistent. Without it, validation will fail on threatintel.indicator*

yeah, i'm following that this is the only way for this rule to be validated.
we should double check the impact on the schema and validation in general. i think for other beats and modules, this brings in more fields than is correct

Here is a full dump of what's included per beats/module in the base directory only:

beats-module-base-fields.txt

rules/cross-platform/threat_intel_module_match.toml

rw-access · 2021-04-22T20:49:31Z

rules/cross-platform/threat_intel_module_match.toml

+tags = ["Elastic", "Windows", "Elastic Endgame", "Network", "Continuous Monitoring", "SecOps", "Monitoring"]
+type = "threat_match"
+
+threat_index = [ "filebeat-*"]


it's kinda too bad that this all get flattened, but that's more of a kibana API thing
i think it would be nicer if the fields were all namespaced by type instead of jammed together

detection_rules/main.py

Co-authored-by: Ross Wolf <31489089+rw-access@users.noreply.github.com>

brokensound77

Successfully exported and imported into Kibana

If all of the rule data looks accurate to you @peasead, then LGTM to merge 👍

peasead · 2021-04-22T22:35:03Z

Solid.

The indicator match timeline template should be in 7.13, would that be a timeline UUID I could add now?

brokensound77 · 2021-04-22T22:35:38Z

Solid.

The indicator match timeline template should be in 7.13, would that be a timeline UUID I could add now?

yep, just find the guid and timeline title (and test 🙂 )

peasead · 2021-04-23T16:32:43Z

Here's the GUID (495ad7a7-316e-4544-8a0f-9c098daee76e), but how would I test that?

https://raw.githubusercontent.com/elastic/kibana/master/x-pack/plugins/security_solution/server/lib/detection_engine/rules/prepackaged_timelines/threat.json

brokensound77 · 2021-04-23T17:21:57Z

add this to the rule and upload it to a stack that has that template

timeline_id = "495ad7a7-316e-4544-8a0f-9c098daee76e"
timeline_title = "Generic Threat Match Timeline"

Unit tests/validation may need to be updated to pass

brokensound77 · 2021-04-23T20:00:46Z

92562bc

FYSA @rw-access in a separate PR we can start removing irrelevant tests and moving others to within the marshmallow schema validation under TOMLRuleContents.

rules/cross-platform/threat_intel_module_match.toml

Co-authored-by: Justin Ibarra <brokensound77@users.noreply.github.com>

[New Rule] Threat intel indicator match rule (elastic#1133)

peasead added 2 commits April 20, 2021 15:09

add indicator match rule

3d5f7df

formatting a bit

b7a7b6a

peasead added OS: Linux OS: macOS Domain: Network Rule: New Proposal for new rule OS: Windows windows related rules Domain: Endpoint labels Apr 20, 2021

peasead requested review from brokensound77, rw-access and bm11100 April 20, 2021 20:16

peasead self-assigned this Apr 20, 2021

rw-access mentioned this pull request Apr 21, 2021

Separate out query validation from the class hierarchy #1136

Merged

brokensound77 mentioned this pull request Apr 21, 2021

Add threat_match rule type #1138

Merged

rw-access mentioned this pull request Apr 22, 2021

Add wildcard field support to KQL #1139

Merged

Merge branch 'main' into issue-1065

80aceab

rw-access reviewed Apr 22, 2021

View reviewed changes

rules/cross-platform/indicator-match.toml Outdated Show resolved Hide resolved

rw-access reviewed Apr 22, 2021

View reviewed changes

rules/cross-platform/indicator-match.toml Outdated Show resolved Hide resolved

rw-access reviewed Apr 22, 2021

View reviewed changes

rules/cross-platform/indicator-match.toml Outdated Show resolved Hide resolved

brokensound77 and others added 6 commits April 22, 2021 12:49

Merge branch 'main' into issue-1065

e3e0d24

Update rules/cross-platform/indicator-match.toml

9428971

Co-authored-by: Ross Wolf <31489089+rw-access@users.noreply.github.com>

Update rules/cross-platform/indicator-match.toml

398a7e5

Co-authored-by: Ross Wolf <31489089+rw-access@users.noreply.github.com>

Update rules/cross-platform/indicator-match.toml

14f919f

Co-authored-by: Ross Wolf <31489089+rw-access@users.noreply.github.com>

update beats module builder to include beta fields

b357be3

reformat rule and remove extraneous fields

896cf83

brokensound77 changed the title ~~[New Rule] Add Indicator Match Rule~~ [New Rule] Threat intel indicator match rule Apr 22, 2021

lint and format threat intel rule

d1e3a2d

brokensound77 force-pushed the issue-1065 branch from 0c7b27f to d1e3a2d Compare April 22, 2021 20:09

brokensound77 reviewed Apr 22, 2021

View reviewed changes

rw-access reviewed Apr 22, 2021

View reviewed changes

rules/cross-platform/threat_intel_module_match.toml Outdated Show resolved Hide resolved

rw-access reviewed Apr 22, 2021

View reviewed changes

rw-access approved these changes Apr 22, 2021

View reviewed changes

rw-access reviewed Apr 22, 2021

View reviewed changes

detection_rules/main.py Outdated Show resolved Hide resolved

brokensound77 and others added 2 commits April 22, 2021 17:23

tweak help option and note

61bfcef

Co-authored-by: Ross Wolf <31489089+rw-access@users.noreply.github.com>

minor cli fixes to allow export to test

a5889bb

brokensound77 approved these changes Apr 22, 2021

View reviewed changes

brokensound77 mentioned this pull request Apr 22, 2021

[Bug] Look into possible bug in MarshmallowDataclassMixin.to_dict method #1141

Closed

brokensound77 added the v7.13.0 7.13 rules release package label Apr 23, 2021

update definitions and tests with new threat intel timeline template

92562bc

added indicator match timeline template

dfd0dc6

brokensound77 reviewed Apr 23, 2021

View reviewed changes

rules/cross-platform/threat_intel_module_match.toml Outdated Show resolved Hide resolved

peasead and others added 2 commits April 23, 2021 18:02

Update rules/cross-platform/threat_intel_module_match.toml

3740350

Co-authored-by: Justin Ibarra <brokensound77@users.noreply.github.com>

Merge branch 'main' into issue-1065

729a28b

peasead merged commit 92eaa5b into main Apr 26, 2021

peasead deleted the issue-1065 branch April 26, 2021 12:07

austinsonger added a commit to austinsonger/detection-rules that referenced this pull request Apr 26, 2021

Merge pull request #2 from elastic/main

76344b7

[New Rule] Threat intel indicator match rule (elastic#1133)

w0rk3r mentioned this pull request Mar 12, 2024

[Rule Tuning] Replace KQL exceptions for Query DSL Exceptions #3505

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[New Rule] Threat intel indicator match rule #1133

[New Rule] Threat intel indicator match rule #1133

peasead commented Apr 20, 2021 •

edited

Loading

brokensound77 commented Apr 21, 2021 •

edited

Loading

brokensound77 Apr 22, 2021 •

edited

Loading

rw-access Apr 22, 2021

brokensound77 Apr 22, 2021

rw-access Apr 22, 2021

brokensound77 Apr 22, 2021 •

edited

Loading

rw-access Apr 22, 2021

rw-access Apr 22, 2021

brokensound77 left a comment

peasead commented Apr 22, 2021

brokensound77 commented Apr 22, 2021

peasead commented Apr 23, 2021

brokensound77 commented Apr 23, 2021

brokensound77 commented Apr 23, 2021

		"""


		[[rule.threat_filters]]

[New Rule] Threat intel indicator match rule #1133

[New Rule] Threat intel indicator match rule #1133

Conversation

peasead commented Apr 20, 2021 • edited Loading

Issues

Summary

Contributor checklist

brokensound77 commented Apr 21, 2021 • edited Loading

brokensound77 Apr 22, 2021 • edited Loading

Choose a reason for hiding this comment

rw-access Apr 22, 2021

Choose a reason for hiding this comment

brokensound77 Apr 22, 2021

Choose a reason for hiding this comment

rw-access Apr 22, 2021

Choose a reason for hiding this comment

brokensound77 Apr 22, 2021 • edited Loading

Choose a reason for hiding this comment

rw-access Apr 22, 2021

Choose a reason for hiding this comment

rw-access Apr 22, 2021

Choose a reason for hiding this comment

brokensound77 left a comment

Choose a reason for hiding this comment

peasead commented Apr 22, 2021

brokensound77 commented Apr 22, 2021

peasead commented Apr 23, 2021

brokensound77 commented Apr 23, 2021

brokensound77 commented Apr 23, 2021

peasead commented Apr 20, 2021 •

edited

Loading

brokensound77 commented Apr 21, 2021 •

edited

Loading

brokensound77 Apr 22, 2021 •

edited

Loading

brokensound77 Apr 22, 2021 •

edited

Loading