ECS normalization #62

andrewstucki · 2020-05-13T20:36:06Z

So, unfortunately, the normalization file is pretty hard to review due to moving things around for the sake of ergonomics (i.e. ordering by similarly categorized events rather than alphabetically). Aside from the reorg and addition of ECS categorization fields--there is:

a bug fix with the way we were unmarshaling yaml that was breaking yaml anchor node behavior.
extending some of the normalization code, especially for compound events so that we have the associated syscall normalization information RE ECS categorization.
removing the following normalization rule, which AFAIK doesn't actually do anything due to us stripping the AUDIT_ prefixes in our GetAuditMessageType code in the auparse subpackage.

  object:
    what: mac-config
  record_types:
  - AUDIT_DEV_ALLOC
  - AUDIT_DEV_DEALLOC
  - AUDIT_FS_RELABEL
  - AUDIT_USER_MAC_POLICY_LOAD
  - AUDIT_USER_MAC_CONFIG_CHANGE

andrewstucki · 2020-05-14T14:03:37Z

aucoalesce/normalize.go

@@ -52,7 +52,7 @@ type Strings struct {
 func (s *Strings) UnmarshalYAML(unmarshal func(interface{}) error) error {
 	var singleValue string
 	if err := unmarshal(&singleValue); err == nil {
-		s.Values = append(s.Values, singleValue)
+		s.Values = []string{singleValue}


This was causing bad anchor merge behavior. Basically, before this change, array sub-elements of an anchor node would get merged rather than overwritten when used, so:

macros: - &ecs-file category: file type: info normalizations: - action: test ecs: <<: *ecs-file type: test

Would previously result in:

{ ... "normalizations": [ "action": "test", "ecs": { "category": ["file"], "type": ["info", "test"] } ] }

rather than:

{ ... "normalizations": [ "action": "test", "ecs": { "category": ["file"], "type": ["test"] } ] }

which is the intended (and to spec) result.

andrewstucki · 2020-05-14T15:06:08Z

aucoalesce/coalesce.go

+		event.ECS.Event.Category = append(event.ECS.Event.Category, syscallNorm.ECS.Category.Values...)
+		event.ECS.Event.Type = append(event.ECS.Event.Type, syscallNorm.ECS.Type.Values...)
+		if event.Result == "fail" {
+			event.ECS.Event.Outcome = "failure"


we're only explicit if something was a failure otherwise, unknown/success are implied based off of the context of the category/type

aucoalesce/coalesce.go

andrewstucki · 2020-05-14T15:13:48Z

aucoalesce/normalizations.yaml

+  # AUDIT_ANOM_CRYPTO_FAIL - Crypto system test failure
+  # AUDIT_ANOM_MK_EXE - Make an executable
+  # AUDIT_ANOM_ACCESS_FS - Access of file or dir
+  # AUDIT_ANOM_ADD_ACCT - Adding an acct


So, we still have a bunch of ECS normalization we can add for some of these audit types, but I figured I could do that in a subsequent PR since this is already a ton of changes

andrewstucki · 2020-05-14T15:15:31Z

aucoalesce/normalizations.yaml

-  - USER_TTY
+  - ecs: *ecs-process
+    syscalls:
+      - '*' # this is a catch all


The thought is that only a process can actually make syscalls anyway--so if we have a syscall where we haven't normalized it to a behavior (i.e. create a file)--then just say it's "process info" with ECS: event.category: process, event.type: info

andrewstucki · 2020-05-14T15:17:56Z

I know that the yaml manifest is pretty hard to review--but would be appreciative if you took a look @webmat -- it's not as straightforward as last round in beats itself--but it should implement all the stuff we talked about in your former review + some.

andrewstucki · 2020-05-14T15:28:08Z

aucoalesce/normalizations.yaml

+  - action: read-file
+    object:
+      what: file
+    syscalls:
+      - read
+    ecs: *ecs-file
+  - action: wrote-to-file
+    object:
+      what: file
+    syscalls:
+      - write
+    ecs:
+      <<: *ecs-file
+      type: change


These two syscall normalizations were added

leehinman

Nice.

couple of non-blocking questions

leehinman · 2020-05-14T15:30:38Z

aucoalesce/coalesce.go

@@ -34,6 +34,13 @@ import (
 // package does not have a constant defined for this.
 const modeBlockDevice = 060000

+// ECSEvent contains ECS-specific categorization fields
+type ECSEvent struct {


Non blocking
Would event.kind be held somewhere else? or maybe the caller of the library is responsible for that?

What do you think about having something about ECS categorization in the struct type? That might provide clues to the type of fields in the struct.

my initial thought is that it could be in the caller--namely because I'm pretty sure we're working only with event kinds--but then again would "anomaly" events actually be alert kinds? If so, or if we just want to be explicit, I can add kind in a follow-up PR.

For the second question, I'm not entirely sure I follow--do you meaning adding constant enums for the categorization values themselves? If so I'd almost prefer to do that in the ECS go generated code from the ECS repo itself (which ideally we'd be using here too--but we still need to add tags and formalize on things like omitempty, pointer usage, etc. before doing that).

I'm good with it being the callers responsibility, I just wanted to double check that we weren't missing one of the ECS categorization fields.

I was thinking the name of the type might be better as ECSCategorization or ECSCat or something like that, since it really just holds the categorization info, nothing else about ECS.

Enums would be awesome and I agree it would be good to do that work once in ECS repo.

leehinman · 2020-05-14T15:51:46Z

aucoalesce/coalesce.go

+		event.ECS.Event.Category = append(event.ECS.Event.Category, syscallNorm.ECS.Category.Values...)
+		event.ECS.Event.Type = append(event.ECS.Event.Type, syscallNorm.ECS.Type.Values...)
+		if event.Result == "fail" {
+			event.ECS.Event.Outcome = "failure"


non blocking

because "failure" is one of 3 constant values, it might be nice to use a constant for this.

ideally this would also be in an enum in the ECS generated go structures

andrewkroh

Wow I forgot just how many different events there can be. Nice work.

Add a changelog.
Drop a line into the README saying this provides Elastic Common Schema (ECS) 1.5 categorization (with link to docs) for the most prominent events (my vague way of saying not all events will be covered).
Update the example events on the README if you can (not sure if where those raw events came from).

aucoalesce/coalesce.go

aucoalesce/normalizations.yaml

andrewstucki marked this pull request as draft May 13, 2020 20:36

Add normalization stuff

b421380

andrewstucki force-pushed the ecs-normalization branch from e0682df to b421380 Compare May 14, 2020 13:14

andrewstucki added 2 commits May 14, 2020 09:24

Switch apparmor/selinux ordering because it matters

e661278

Fix custom unmarshaler

d853608

andrewstucki commented May 14, 2020

View reviewed changes

andrewstucki added 3 commits May 14, 2020 10:29

Add support for compound events

54e16a6

Fix AUDIT_LOGIN

8ff6108

more default updates

ddba1ce

andrewstucki commented May 14, 2020

View reviewed changes

aucoalesce/coalesce.go Show resolved Hide resolved

andrewstucki marked this pull request as ready for review May 14, 2020 15:07

andrewstucki requested a review from andrewkroh May 14, 2020 15:10

andrewstucki commented May 14, 2020

View reviewed changes

andrewstucki requested a review from leehinman May 14, 2020 15:15

andrewstucki added 2 commits May 14, 2020 11:21

Fix process start v creation

c1d127f

Add write syscall since I added read

39525d6

andrewstucki commented May 14, 2020

View reviewed changes

leehinman approved these changes May 14, 2020

View reviewed changes

andrewstucki added 2 commits May 14, 2020 12:39

Add syscall descriptions and a missing syscall comment

b6f7bcd

remove additional select macros from comment list

8c22c7a

andrewkroh approved these changes May 15, 2020

View reviewed changes

aucoalesce/coalesce.go Show resolved Hide resolved

aucoalesce/normalizations.yaml Outdated Show resolved Hide resolved

Update with comments, types changed, README update, and CHANGELOG entry

2a64f9c

andrewstucki merged commit 006fb63 into elastic:master May 15, 2020

andrewstucki deleted the ecs-normalization branch May 15, 2020 19:02

This was referenced May 15, 2020

[Auditbeat] Update auditbeat ECS mappings elastic/beats#18596

Merged

Cherry-pick #18596 to 7.x: [Auditbeat] Update auditbeat ECS mappings elastic/beats#18598

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ECS normalization #62

ECS normalization #62

andrewstucki commented May 13, 2020 •

edited

andrewstucki May 14, 2020

andrewstucki May 14, 2020

andrewstucki May 14, 2020

andrewstucki May 14, 2020

andrewstucki commented May 14, 2020

andrewstucki May 14, 2020

leehinman left a comment

leehinman May 14, 2020

andrewstucki May 14, 2020

leehinman May 14, 2020

leehinman May 14, 2020

andrewstucki May 14, 2020

andrewkroh left a comment •

edited

ECS normalization #62

ECS normalization #62

Conversation

andrewstucki commented May 13, 2020 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

andrewstucki commented May 14, 2020

Choose a reason for hiding this comment

leehinman left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

andrewkroh left a comment • edited

Choose a reason for hiding this comment

andrewstucki commented May 13, 2020 •

edited

andrewkroh left a comment •

edited