Add ECS 1.8 session and map user and groups #86

adriansr · 2021-01-20T17:26:25Z

Add ECS user and group fields to aucoalesce.Event:

Update normalizations to enable population of ECS user.* and group.*
fields from event data (entity., object., uids and raw data).
Refactors the user/group ID lookup as sometimes it's necessary to
lookup a user/group ID from a name.

Adds the new ECS session categorization to USER_START and USER_END messages.

elasticmachine · 2021-01-20T17:28:41Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Build Cause: Pull request #86 updated
- Start Time: 2021-02-02T15:20:51.129+0000
Duration: 2 min 2 sec
Commit: 9113b6a

Test stats 🧪

Test	Results
Failed	0
Passed	365
Skipped	40
Total	405

Add ECS user and group fields to aucoalesce.Event. Update normalizations to enable population of ECS user.* and group.* fields from event data (entity.*, object.*, uids and raw data). Also refactors the user/group ID lookup as sometimes it's necessary to lookup a user/group ID from a name.

andrewstucki

I'll work through a bit more of a detailed review in a bit, but general approach looks good to me, I did have a question about other effective/target users though, a quick pass through the normalization file and I'm wondering if the following should also be handled?

AUDIT_LOGIN
AUDIT_CHGRP_ID
AUDIT_CHUSER_ID
syscalls: chown, fchown, fchownat, lchown, setuid, seteuid, setfsuid, setreuid, setgid, setegid, setfsgid, setregid, setresuid, setresgid

WDYT?

andrewkroh

Nice progress! A few things to cover before closing this out:

Update examples in README with the new fields
Update the ECS version in the README
Add a changelog

aucoalesce/testdata/random-internet.json.golden

Nit: Fixes an implicit rune to string conversion warning emitted by vet. There was no bug, but the implicit conversion is deprecated starting in Go 1.15. > rule/rule.go:367:29: conversion from untyped int to string yields a string of one rune, not a string of digits (did you mean fmt.Sprint(x)?)

adriansr · 2021-01-25T15:37:27Z

Been discussing @andrewstucki requests via Slack. The current status of this PR is that everything that we can support is supported:

* `AUDIT_LOGIN`

This one was (erroneously) ignored by Auditbeat. I've added support for it and will also update Auditbeat in elastic/beats#23594.

* `AUDIT_CHGRP_ID`
* `AUDIT_CHUSER_ID`

This ones don't contain any interesting information for ECS multiuser enrichment, only generic *id fields which were already supported.

* syscalls: `chown`, `fchown`, `fchownat`, `lchown`

We decided to leave file ownership syscalls for now, as it's not clear how/if they fit into ECS 1.8 multiuser at all.

setuid, seteuid, setfsuid, setreuid, setgid, setegid, setfsgid, setregid, setresuid, setresgid

The set*id syscalls are already supported through the generic *id fields.

adriansr · 2021-01-25T18:41:33Z

aucoalesce/normalizations.yaml

@@ -1001,7 +1072,9 @@ normalizations:
  - <<: *macro-user-session
    record_types: USER_END
    action: ended-session
-    ecs: *ecs-auth


This changes USER_START/USER_END messages from event.category: authentication to session.

I wonder if I should keep the former, event.category: [authentication, session]

WDYT @andrewstucki ?

I'm leaning toward only session since there are separate events for authentication. If we had session earlier I assume that's the way we would have done it. It's a minor breaking change though.

I'm fine with that change. Also, just in the interest of spreading the knowledge around, main place I could find a reference to AUDIT_USER_LOGIN is when login from shadow-utils returns successfully.

https://github.com/shadow-maint/shadow/blob/ae169c40468bc9d016016880613c01aca6050b60/src/login.c#L837

AUDIT_USER_START is predominantly an artifact of PAM and is called in pam_open_session:

https://github.com/linux-pam/linux-pam/blob/491e5500b6b3913f531574208274358a2df88659/libpam/pam_audit.c#L148-L151

Both work hand-in-hand, as you can see the invocation of pam_open_session here:

https://github.com/shadow-maint/shadow/blob/ae169c40468bc9d016016880613c01aca6050b60/src/login.c#L885

Seems to mirror exactly what's described in the audit framework docs

The next event can be AUDIT_USER_ROLE_CHANGE depending on whether a MAC framework that has the notion of roles is in use. It records what role has been assigned to the login session. This event is followed by AUDIT_USER_LOGIN to signify that the session that is being created is interactive. This is followed by AUDIT_USER_START to denote that the session has started.

marc-gr · 2021-01-29T15:13:10Z

README.md

+    target: {}
+    changes: {}
+  group:
+    effective: {}


I think the only one having effective etc is the user field https://www.elastic.co/guide/en/ecs/master/ecs-user-usage.html.

For the effective group would be something like user.effective.group.*

Good catch. group.effective was unused, but group.target is used for group creation/deletion/changes.

I think the best for now is to set group.id in those cases? Because it's mapping actions like User A creates Group B. Another option is user.target.group? But that doesn't make a lot of sense without filling the user fields in user.target.

/cc @andrewstucki

I think as far as user is modifying/creating a group does does not belong to a different user, is probably fine having both user and group at the root. Maybe @ebeahan can bring some guideline

Yes, I agree with @marc-gr that using the top-level user.* and group.* is most appropriate in this case.

group.target is not defined by ECS.

andrewstucki · 2021-02-02T19:24:31Z

aucoalesce/testdata/ubuntu-16.10-linux-4.8.0.json.golden

+        "user": {
+          "id": "unset",
+          "effective": {
+            "name": "(invalid user)"


does it make sense to actually enrich these if they're invalid/unset?

This is a failed ssh login attempt using an invalid user, I think it's worth to report that back to the user instead of an empty user.

adriansr force-pushed the ecs_18_norms branch from 2edc0b1 to 91952da Compare January 20, 2021 17:39

Fix mistake

1cb7f34

andrewstucki reviewed Jan 20, 2021

View reviewed changes

adriansr added 3 commits January 21, 2021 17:10

Multi-user support for AUDIT_LOGIN

6cff2ac

Support AUDIT_LOGIN from kernels <3.14

2507d6f

Rename actor.* to subject.* to align with normalizations yml

117b2ab

andrewkroh reviewed Jan 22, 2021

View reviewed changes

aucoalesce/testdata/random-internet.json.golden Show resolved Hide resolved

adriansr added 4 commits January 24, 2021 12:03

Fix ecs struct encoding and update README samples

bb3682b

Update ECS reference in README to v1.8

897c3d4

Add changelog entry

2b8c647

Update test files

e5c1ed2

adriansr force-pushed the ecs_18_norms branch from 704845e to e5c1ed2 Compare January 25, 2021 11:21

Support ECS session category

61b4ffa

adriansr force-pushed the ecs_18_norms branch from fd9ea90 to 61b4ffa Compare January 25, 2021 11:41

adriansr marked this pull request as ready for review January 25, 2021 14:03

adriansr changed the title ~~aucoalesce: ECS user and group mapping~~ Add ECS session and user, group mapping Jan 25, 2021

Update changelog

8e0720d

adriansr changed the title ~~Add ECS session and user, group mapping~~ Add ECS 1.8 session and map user and groups Jan 25, 2021

adriansr requested review from andrewstucki and andrewkroh January 25, 2021 15:18

adriansr mentioned this pull request Jan 25, 2021

Update Auditbeat auditd module to ECS 1.8 elastic/beats#23594

Merged

adriansr added the review label Jan 25, 2021

adriansr commented Jan 25, 2021

View reviewed changes

Add AUDIT_SOFTWARE_UPDATE from filebeat auditd/log

11ee0db

marc-gr reviewed Jan 29, 2021

View reviewed changes

Map to group.* instead of group.target.*

f3af093

group.target is not defined by ECS.

Update README.md

9113b6a

marc-gr approved these changes Feb 2, 2021

View reviewed changes

andrewstucki reviewed Feb 2, 2021

View reviewed changes

adriansr merged commit dde11ff into elastic:master Feb 3, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add ECS 1.8 session and map user and groups #86

Add ECS 1.8 session and map user and groups #86

adriansr commented Jan 20, 2021 •

edited

elasticmachine commented Jan 20, 2021 •

edited

Build stats

Test stats 🧪

andrewstucki left a comment

andrewkroh left a comment •

edited by adriansr

adriansr commented Jan 25, 2021

adriansr Jan 25, 2021

andrewkroh Jan 26, 2021

andrewstucki Jan 27, 2021

marc-gr Jan 29, 2021

adriansr Jan 31, 2021

marc-gr Feb 1, 2021

ebeahan Feb 2, 2021

andrewstucki Feb 2, 2021

adriansr Feb 3, 2021

Add ECS 1.8 session and map user and groups #86

Add ECS 1.8 session and map user and groups #86

Conversation

adriansr commented Jan 20, 2021 • edited

elasticmachine commented Jan 20, 2021 • edited

💚 Build Succeeded

Build stats

Test stats 🧪

andrewstucki left a comment

Choose a reason for hiding this comment

andrewkroh left a comment • edited by adriansr

Choose a reason for hiding this comment

adriansr commented Jan 25, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

adriansr commented Jan 20, 2021 •

edited

elasticmachine commented Jan 20, 2021 •

edited

andrewkroh left a comment •

edited by adriansr