Add malware_signature for Memory protection alert #155

gogochan · 2021-06-30T15:07:54Z

Issue: https://github.com/elastic/security-team/issues/1214

Adding additional fields that are part of yara signature. The fields are used in FileScore alert and MemoryScan Alert.

gabriellandau · 2021-06-30T17:53:40Z

custom_subsets/elastic_endpoint/alerts/malware_event.yaml

@@ -469,6 +469,8 @@ fields:
                exceptionable: true
              status:
                exceptionable: true
+          malware_signature:
+            fields: "*"


Everything else here looks like this:

Suggested change

fields: "*"

fields: {}

We should update the docs on "*" vs {} because I always forget haha. {} only works when the field is a leaf aka if we only wanted malware_signature.all_names in the mapping we could write it like:

malware_signature: fields: all_names: {}

Since we want everything under malware_signature and fields is an array we have to use the "*" syntax like Chan has here.

gabriellandau · 2021-06-30T17:56:33Z

custom_subsets/elastic_endpoint/alerts/malware_event.yaml

@@ -469,6 +469,8 @@ fields:
                exceptionable: true
              status:
                exceptionable: true
+          malware_signature:


Should this go into memory_protection_event, since Memory Scan is presented to the user as part of Memory Protection? https://github.com/elastic/endpoint-package/blob/c23a052ef81ea39ab2934e2e1ea5b32bff102196/custom_subsets/elastic_endpoint/alerts/memory_protection_event.yaml

This should stay where it is at the very least for malware (FileScore) events. We could duplicate it to another location as well for use by memory alerts though I'd vote against that.

I take it back. I now see you're commenting on process.Ext.malware_signature not file.Exe.malware_signature.

Currently Linux only sends file activity based malware (FileScore) alerts (open, write, etc) so wouldn't ever need this location. But Windows and macOS can send process based malware alerts (process start). So I still think this is needed for Windows and macOS signature based FileScore alerts (and wonder why testing has been OK without this so far).

But that's only whether or not this update is beneficial to malware alerts. My presumption would be that a standardized key name for yara signature matches regardless of the protection type would be better than it going in different places for different alert types. But I don't feel strongly, if my presumption is wrong I'll stay silent.

Per our discussion, I will remove this line from this PR. We will add it back in 7.15

jonathan-buttner

Thanks for the PR Chan! Couple questions and changes:

From the example that Kevin linked (memory_signature_agentTesla.json.zip) it looks like memory_region is included in malware_signature:

"memory_region": {
                    "allocation_base": 4194304,
                    "allocation_protection": "RWX",
                    "allocation_size": 245760,
                    "allocation_type": "PRIVATE",
                    "bytes_address": 4194304,
                    "bytes_allocation_offset": 0,

do we need to update https://github.com/elastic/endpoint-package/blob/master/custom_schemas/custom_memory_region.yml#L11 to include it within this schema as well?

jonathan-buttner · 2021-06-30T18:06:36Z

custom_subsets/elastic_endpoint/alerts/malware_event.yaml

@@ -469,6 +469,8 @@ fields:
                exceptionable: true
              status:
                exceptionable: true
+          malware_signature:
+            fields: "*"


We should update the docs on "*" vs {} because I always forget haha. {} only works when the field is a leaf aka if we only wanted malware_signature.all_names in the mapping we could write it like:

malware_signature: fields: all_names: {}

Since we want everything under malware_signature and fields is an array we have to use the "*" syntax like Chan has here.

custom_schemas/custom_malware_signature.yml

jonathan-buttner · 2021-06-30T18:25:33Z

custom_schemas/custom_malware_signature.yml

+
+    - name: secondary
+      level: custom
+      type: array


In Elasticsearch there isn't an array type. Everything can be an array out of the box: https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-types.html#types-array-handling

Could we change this to the structure and type that will be contained within secondary? If secondary will be the exact same as primary, you could abstract it out to a new file and have it overlaid as both primary and secondary using the ECS reuse mechanism.

secondary will be a list of object that are subset of the primary. Should I just say object here then?

If we use object then users won't be able to search for it beyond just using a search based on its existence. If we don't expect users to search for the secondary fields then you could exclude it from the definition here. The only reason we want to define it in the endpoint package is so that users can search for it. If you want to include it just as a description of a field that could exist in an alert, we can add enabled: false (e.g. https://github.com/elastic/endpoint-package/blob/master/custom_schemas/custom_endpoint.yml#L28) that way Elasticsearch will not index it.

gogochan · 2021-06-30T20:54:41Z

do we need to update https://github.com/elastic/endpoint-package/blob/master/custom_schemas/custom_memory_region.yml#L11 to include it within this schema as well?

I'm not sure if we need to include them. IMO creating an Exception based on those values would be too much of information.

jonathan-buttner · 2021-07-01T13:32:02Z

do we need to update https://github.com/elastic/endpoint-package/blob/master/custom_schemas/custom_memory_region.yml#L11 to include it within this schema as well?

I'm not sure if we need to include them. IMO creating an Exception based on those values would be too much of information.

I guess what I mean is, based on the example that Kevin gave I'm assuming the endpoint will send those memory region values. If we expect the user to want to search for alerts based on malware_signature.memory_region.* values then we'd need to include those in the mapping here. If we don't expect the user to search for them then we can leave them out.

@kevinlog @pzl could you take a look at the mappings when you get a chance.

custom_schemas/custom_malware_signature.yml

kevinlog · 2021-07-01T13:43:38Z

@jonathan-buttner thanks for bubbling this up.

I guess what I mean is, based on the example that Kevin gave I'm assuming the endpoint will send those memory region values. If we expect the user to want to search for alerts based on malware_signature.memory_region.* values then we'd need to include those in the mapping here. If we don't expect the user to search for them then we can leave them out.

There are two questions to answer here:

Does the user need to search for alerts based on this field
Should the user be able to add Exceptions based on this field for this Alert
- Implies we need it in the Index pattern

I don't think we need to index this field for searching.

However, we may want it included in the Index pattern so that we can write exceptions for it.

@gogochan @gabriellandau @ferullo any thoughts on writing Exceptions?

If we DO want to write Exceptions, we can enabled: false so that we don't index the field and include it in searches.

ferullo · 2021-07-01T14:14:08Z

@joe-desimone has thoughts on what signature match fields should be available for exceptions.

joe-desimone · 2021-07-01T17:17:09Z

I believe users should have the ability to search for and create exceptions from the following fields: malware_signature.all_names, malware_signature.primary.signature.name, malware_signature.primary.signature.id, malware_signature.memory_region.allocation_protection, malware_signature.memory_region.allocation_size, malware_signature.memory_region.allocation_type, malware_signature.memory_region.region_protection, malware_signature.memory_region.region_size, malware_signature.memory_region.region_state, malware_signature.memory_region.memory_pe.imphash, malware_signature.memory_region.bytes_allocation_offset

@dstepanic17 can you double check my work

Also side note, the memory_region stuff is landing in production at 7.15 at the earliest. Not applicable to linux file yara scanning which is landing in 7.14.

dstepanic · 2021-07-01T17:46:02Z

@joe-desimone Here, are a couple fields I did not see on your list. I believe the region_base maps to allocation_base, but might be confusing for users to not let them search or create exceptions on it. I believe the other field bytes_allocation_offset can bring in unique data so might be useful for searching/exceptions. Everything else looks good.

malware_signature.memory_region.region_base
malware_signature.memory_region.bytes_allocation_offset

joe-desimone · 2021-07-01T18:05:32Z

@joe-desimone Here, are a couple fields I did not see on your list. I believe the region_base maps to allocation_base, but might be confusing for users to not let them search or create exceptions on it. I believe the other field bytes_allocation_offset can bring in unique data so might be useful for searching/exceptions. Everything else looks good.

malware_signature.memory_region.region_base

malware_signature.memory_region.bytes_allocation_offset

Thank you. I actually had allocation_base on there by mistake, and left off region_base intentionally. My thinking was these are going to be generally random so not too helpful from an exceptionlist point of view (but useful for response). allocation_offset I waffled on. In some situations it could be useful so I added it now.

Also, we dont have it yet but we should add malware_signature.memory_region.memory_pe.imphash to match shellcode alert schema more closely. We would want to search+exception on that as well so I added it to my original list.

gogochan · 2021-07-16T14:47:20Z

I believe users should have the ability to search for and create exceptions from the following fields: malware_signature.all_names, malware_signature.primary.signature.name, malware_signature.primary.signature.id, malware_signature.memory_region.allocation_protection, malware_signature.memory_region.allocation_size, malware_signature.memory_region.allocation_type, malware_signature.memory_region.region_protection, malware_signature.memory_region.region_size, malware_signature.memory_region.region_state, malware_signature.memory_region.memory_pe.imphash, malware_signature.memory_region.bytes_allocation_offset

Just want to point out that the current workflow does exception list checking before reading process memory, thus it lacks memory_signature.* fields.

https://github.com/elastic/endpoint-dev/blob/master/Elastic/Plugins/MemoryProtection/Lib/MemoryScan.cpp#L472-L479

We can alter the behavior, but it means that we are checking the exception list after scanning process memory.

gogochan · 2021-07-16T15:24:49Z

do we need to update https://github.com/elastic/endpoint-package/blob/master/custom_schemas/custom_memory_region.yml#L11 to include it within this schema as well?

@jonathan-buttner We will make a separate PR to address this. I've adjusted the PR per comments provided. Let me know if this is good.

gabriellandau · 2021-07-16T15:51:46Z

@joe-desimone per our recent conversations, do you think we should put malware_signature under memory_region instead of vice-versa?

jonathan-buttner

Left a few more comments about changes I think we need

custom_schemas/custom_malware_signature.yml

custom_subsets/elastic_endpoint/alerts/malware_event.yaml

gogochan · 2021-07-16T16:48:40Z

Thanks @jonathan-buttner for the suggestion. It looks better now.

ferullo · 2021-07-16T19:07:06Z

We can alter the behavior, but it means that we are checking the exception list after scanning process memory.

Endpoint should definitely check the exception list after it scans memory and can generate the alert but before it takes action. Checking exceptions earlier than this is a good optimization but this final check is always needed for all protections.

jonathan-buttner

Looks good! Just one question.

jonathan-buttner · 2021-07-16T19:18:39Z

custom_schemas/custom_malware_signature.yml

+    top_level: false
+    expected:
+      - process.Ext
+      - file.Ext


Just a heads up it doesn't look like malware_signature is being placed under file.Ext 🤔 that might be because the file subset file doesn't include it 🤷‍♂️ is that ok?

I updated the title. All fields added here are for Memory Protection plugin.

joe-desimone · 2021-07-17T13:08:01Z

We can alter the behavior, but it means that we are checking the exception list after scanning process memory.

Endpoint should definitely check the exception list after it scans memory and can generate the alert but before it takes action. Checking exceptions earlier than this is a good optimization but this final check is always needed for all protections.

Agree here. I had the understanding that we do some early exceptionlist tests which would allow us better performance like https://github.com/elastic/global-security-exceptionlist/pull/257/files. Might be a good place for unit test.

joe-desimone · 2021-07-17T13:09:22Z

@joe-desimone per our recent conversations, do you think we should put malware_signature under memory_region instead of vice-versa?

Yes which would help makes things more consistent between shellcode alerts and memory scan alerts

joe-desimone · 2021-07-19T14:23:19Z

@joe-desimone per our recent conversations, do you think we should put malware_signature under memory_region instead of vice-versa?

Yes which would help makes things more consistent between shellcode alerts and memory scan alerts

Created an issue to specifically discuss this proposal https://github.com/elastic/security-team/issues/1496

gogochan · 2021-07-19T14:40:22Z

This may require further modification based on https://github.com/elastic/security-team/issues/1496

gogochan · 2021-07-19T20:45:51Z

Added changes proposed in https://github.com/elastic/security-team/issues/1496
basically malware_signature goes under memory_region

gabriellandau · 2021-07-19T21:30:25Z

custom_schemas/custom_malware_signature.yml

+    order: 1
+    top_level: false
+    expected:
+      - memory_region


custom_subsets/elastic_endpoint/alerts/memory_protection_event.yaml

jonathan-buttner · 2021-07-26T14:30:09Z

custom_schemas/custom_memory_region.yml

@@ -9,6 +9,7 @@
    order: 1
    top_level: false
    expected:
+      - process.Ext


Just curious, should we be calling it start_address_details within process.Ext too like in process.thread.Ext?

custom_subsets/elastic_endpoint/alerts/memory_protection_event.yaml

…Names

gabriellandau

I pushed changes that addressed my concerns. LGTM.

jonathan-buttner

Question about renaming start_address_details

jonathan-buttner · 2021-07-28T17:00:06Z

custom_subsets/elastic_endpoint/alerts/memory_protection_event.yaml

@@ -262,27 +262,6 @@ fields:
              start_address_bytes: {}
              start_address_bytes_disasm: {}
              start_address_bytes_disasm_hash: {}
-              start_address_details:


This will result in a breaking change since we're renaming these fields. Has the endpoint already started populating these fields in 7.14 or prior? If it has, do any of our detection rules use these fields? If they do we'll be causing them to no longer work. Do you think it's likely that users have custom detection rules triggering based on these fields?

@joe-desimone @rw-access @gogochan @gabriellandau

Has the endpoint already started populating these fields in 7.14 or prior?

Only for diagnostic. The shellcode feature isn't going live until 7.15.

If it has, do any of our detection rules use these fields?

Negative. Not until 7.15.

Ok perfect! Thanks Gabe!

gogochan added 3 commits June 30, 2021 11:07

added malware_siganture

017c304

additional changes

2121b1d

additional modification

c23a052

gogochan changed the title ~~added malware_signature~~ Add malware_signature Jun 30, 2021

also adding malware_signature under file.Ext

8da9102

gabriellandau reviewed Jun 30, 2021

View reviewed changes

jonathan-buttner requested changes Jun 30, 2021

View reviewed changes

gogochan changed the title ~~Add malware_signature~~ Add malware_signature for FileScore alert Jun 30, 2021

updating malware_signature

333a5cb

jonathan-buttner reviewed Jul 1, 2021

View reviewed changes

custom_schemas/custom_malware_signature.yml Outdated Show resolved Hide resolved

jonathan-buttner requested review from kevinlog and pzl July 1, 2021 13:33

adjustment for PR feedback

4341b40

jonathan-buttner requested changes Jul 16, 2021

View reviewed changes

custom_schemas/custom_malware_signature.yml Outdated Show resolved Hide resolved

custom_schemas/custom_malware_signature.yml Outdated Show resolved Hide resolved

custom_subsets/elastic_endpoint/alerts/malware_event.yaml Outdated Show resolved Hide resolved

schema reuse

6d9e686

jonathan-buttner approved these changes Jul 16, 2021

View reviewed changes

gogochan changed the title ~~Add malware_signature for FileScore alert~~ Add malware_signature for Memory protection alert Jul 16, 2021

memory protection schema consolidation

de52a3a

gabriellandau requested changes Jul 19, 2021

View reviewed changes

gogochan requested a review from jonathan-buttner July 20, 2021 14:41

jonathan-buttner reviewed Jul 26, 2021

View reviewed changes

gabriellandau added 4 commits July 28, 2021 11:40

Merge branch 'master' of github.com:elastic/endpoint-package into all…

1872d93

…Names

Shellcode schema refactor

218e216

rebuild

1f7d793

Adjustments for shellcode schema refactor

239b174

gabriellandau approved these changes Jul 28, 2021

View reviewed changes

gogochan requested a review from jonathan-buttner July 28, 2021 16:46

jonathan-buttner requested changes Jul 28, 2021

View reviewed changes

jonathan-buttner approved these changes Jul 28, 2021

View reviewed changes

gogochan merged commit 9f0692b into master Jul 28, 2021

gogochan deleted the allNames branch July 28, 2021 17:47

gabriellandau mentioned this pull request Aug 12, 2021

Update security telemetry sender for latest memory protection schema elastic/kibana#108462

Merged

1 task

gogochan mentioned this pull request Nov 5, 2021

set malware_signature.primary.matches as unindexed keyword #212

Merged

3 tasks

Add malware_signature for Memory protection alert #155

Add malware_signature for Memory protection alert #155

Conversation

gogochan commented Jun 30, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ferullo Jun 30, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jonathan-buttner left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gogochan commented Jun 30, 2021

jonathan-buttner commented Jul 1, 2021

kevinlog commented Jul 1, 2021

ferullo commented Jul 1, 2021

joe-desimone commented Jul 1, 2021 • edited Loading

dstepanic commented Jul 1, 2021

joe-desimone commented Jul 1, 2021

gogochan commented Jul 16, 2021

gogochan commented Jul 16, 2021

gabriellandau commented Jul 16, 2021

jonathan-buttner left a comment

Choose a reason for hiding this comment

gogochan commented Jul 16, 2021

ferullo commented Jul 16, 2021

jonathan-buttner left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

joe-desimone commented Jul 17, 2021

joe-desimone commented Jul 17, 2021

joe-desimone commented Jul 19, 2021

gogochan commented Jul 19, 2021

gogochan commented Jul 19, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gabriellandau left a comment

Choose a reason for hiding this comment

jonathan-buttner left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gogochan commented Jun 30, 2021 •

edited

Loading

ferullo Jun 30, 2021 •

edited

Loading

joe-desimone commented Jul 1, 2021 •

edited

Loading

gogochan commented Jul 19, 2021 •

edited

Loading