Add fileset for parsing linux auditd logs #3750

andrewkroh · 2017-03-15T18:13:58Z

Update: Module renamed in #3941.

The PR adds and audit fileset to the system module for parsing logs from the auditd daemon. There is a sample dashboard for viewing this data.

Features

Parses audit event type, unix epoch time, audit event counter, and the arbitrary key/value pairs that follow.
Applies geoip lookup to the system.audit.addr field which is present for some remote login events.
Decodes hex encoded ascii values that are sometimes used for the system.audit.exe and system.audit.cmd fields.
Remove key/value pairs where the value is ?.

Missing Features

Decoder for system.audit.saddr field present in SOCKADDR audit events. A recipe is described here and could probably be ported to painless. Sample value:
type=SOCKADDR msg=audit(1481078693.491:873): saddr=01002F7661722F72756E2F6E7363642F736F636B657400008983B330BE7F0000000000000000000070830130BE7F000004000000000000001B00000000000000D128B7ED000000008B8BB330BE7F00003B00000000000000403E4BF7FD7F0000447F0130BE7F000080150230BE7F
Syscall number to name conversion. This would require an architecture specific mapping table.
Join data from events with the same auditd counter value. For example there are separate log lines for SYSCALL, CWD, and PATH audit types that all correspond to the same overall action.

ruflin · 2017-03-17T08:21:23Z

filebeat/fileset/modules.go

 			fcfg, err = applyOverrides(fcfg, mcfg.Module, filesetName, overrides)
 			if err != nil {
 				return nil, fmt.Errorf("Error applying overrides on fileset %s/%s: %v", mcfg.Module, filesetName, err)
 			}

+			if fcfg.Enabled != nil && (*fcfg.Enabled) == false {


Is there a specific reason this was moved?

This applies the overrides first, which allows an override to disable a fileset. It allows a CLI config like:

-M system.audit.enabled=false

ruflin · 2017-03-17T08:23:24Z

filebeat/module/system/audit/ingest/pipeline.json

+        {
+            "script": {
+                "lang": "painless",
+                "inline": "    String trimQuotes(def v) {\n        if (v.startsWith(\"'\") || v.startsWith('\"')) {\n          v = v.substring(1, v.length());\n        }\n        if (v.endsWith(\"'\") || v.endsWith('\"')) {\n          v = v.substring(0, v.length()-1);\n        }  \n        return v;\n    }\n    \n    boolean isHexAscii(String v) {\n      def len = v.length();\n      if (len == 0 || len % 2 != 0) {\n        return false; \n      }\n  \n      for (int i = 0 ; i < len ; i++) {\n        if (Character.digit(v.charAt(i), 16) == -1) {\n          return false;\n        }\n      }\n\n      return true;\n    }\n    \n    String convertHexToString(String hex) {\n\t    StringBuilder sb = new StringBuilder();\n\n      for (int i=0; i < hex.length() - 1; i+=2) {\n          String output = hex.substring(i, (i + 2));\n          int decimal = Integer.parseInt(output, 16);\n          sb.append((char)decimal);\n      }\n\n      return sb.toString();\n    }\n    \n    def possibleHexKeys = ['exe', 'cmd'];\n    \n    def audit = ctx.system.get(\"audit\");\n    Iterator entries = audit.entrySet().iterator();\n    while (entries.hasNext()) {\n      def e = entries.next();\n      def k = e.getKey();\n      def v = e.getValue(); \n\n      // Remove entries whose value is ?\n      if (v == \"?\") {\n        entries.remove();\n        continue;\n      }\n      \n      // Convert hex values to ASCII.\n      if (possibleHexKeys.contains(k) && isHexAscii(v)) {\n        v = convertHexToString(v);\n        audit.put(k, v);\n      }\n      \n      // Trim quotes.\n      if (v instanceof String) {\n        v = trimQuotes(v);\n        audit.put(k, v);\n      }\n      \n      // Convert arch.\n      if (k == \"arch\" && v == \"c000003e\") {\n        audit.put(k, \"x86_64\");\n      }\n    }\n    \n    // Fix inner msg parsing.\n    def msg = audit.get(\"msg\");\n    if (msg != null) {\n      def i = msg.indexOf(\"=\");\n      if (i >= 0) {\n        def newKey = trimQuotes(msg.substring(0, i));\n        def newValue = trimQuotes(msg.substring(i+1, msg.length()));\n        audit.remove(\"msg\");\n        audit.put(newKey, newValue);\n      }\n    }"


That is quite a script. I'm curious if that has a major impact on ingestion speed.

I haven't measured in any way the impact on ingestion rates.

I was able to remove a small part of the script by changing the grok pattern (just pushed it), but I think it is as simple as I can make it.

The PR adds and audit fileset to the system module for parsing logs from the auditd daemon. There is a sample dashboard for viewing this data. Features - Parses audit event type, unix epoch time, audit event counter, and the arbitrary key/value pairs that follow. - Applies geoip lookup to the `system.audit.addr` field which is present for some remote login events. - Decodes hex encoded ascii values that are sometimes used for the `system.audit.exe` and `system.audit.cmd` fields. - Remove key/value pairs where the value is `?`. Missing Features - Decoder for `system.audit.saddr` field present in SOCKADDR audit events. A recipe is described [here](https://unix.stackexchange.com/questions/102926/how-to-interpret-the-saddr-field-of-an-audit-log) and could probably be ported to painless. Sample value: `type=SOCKADDR msg=audit(1481078693.491:873): saddr=01002F7661722F72756E2F6E7363642F736F636B657400008983B330BE7F0000000000000000000070830130BE7F000004000000000000001B00000000000000D128B7ED000000008B8BB330BE7F00003B00000000000000403E4BF7FD7F0000447F0130BE7F000080150230BE7F`

Disable other filesets in a module while testing.

ruflin · 2017-03-20T09:47:13Z

LGTM. Waiting for @tsg to have a look.

tsg · 2017-03-23T10:34:14Z

filebeat/tests/system/test_modules.py

+                assert "error" not in obj
+
+            if module != "system" and fileset != "audit":
+                # There are dynamic fields in audit logs that are not documented.


Are the dynamic fields grouped under a sub-directory? Then we could document it as a dict type?

All of the fields are written under system.audit.* with no inner dictionaries. Is there anything I should do in this case?

Would it be possible / make sense to group them under a sub-dict? I think we don't have a precedent in any Beat so far that dynamic and static fields are mixed together.

It's possible, but I don't think using a sub-dict in this case provides any additional context to the user about the fields. We would be using sub-dict here due to the implementation details (like the fact that it uses a kv processor instead of a specific grok pattern for each audit record type). From a user perspective I don't think a sub-dict would add value (at least not with what I'm thinking -- system.audit.kv.*), but I don't like to be the one breaking precedent.

The main thing I worry is that a dynamic field could overwrite a static field and conflict on the type. Could we perhaps just have one entry with all the fields as a string? Or use the raw namespace? Also not sure yet how to deal with it.

If there was an exclude_keys option in the KV processor then it would easy to prevent the dynamic fields from overwriting the well-known fields from the header. The Logstash KV filter has this option so I'll ask about getting this in ES. I could enhance this module to use that once it's available.

There are only a few keys in the audit header that could be overwritten (record_type and sequence) by KV data and I haven't observed these being used in any audit events. I think the risk is minimal.

tsg · 2017-03-23T10:34:45Z

filebeat/tests/system/test_modules.py

+                module, obj["fileset"]["module"])
+
+            if not (module == "mysql" and fileset == "slowlog") and not (module == "system" and fileset == "auth"):
+                # TODO: There are errors parsing the test logs from these modules.


Yeah, this is not ideal, but I don't have an easy solution for now.

I agree, definitely not idea, but I really wanted to enable error checking. Maybe as part of the manifest or some file in the test directory the fileset could self-describe its testing parameters (e.g. allow_errors: true, has_dynamic_fields: true).

Yep, we probably need something like that. I'm good with the current solution for the time being.

tsg · 2017-03-23T10:36:04Z

filebeat/module/system/_meta/docs.asciidoc

 === Compatibility

-This module was tested with logs from OSes like Ubuntu 12.04, Centos 7, macOS Sierra, and others.
+This module was tested with logs from OSes like Ubuntu 12.04, Centos 7, macOS
+Sierra, and others.


The "and others" might come back to us as an issue in support, maybe it's better to leave it out.

I agree, probably a good idea to drop that. Will do.

tsg · 2017-03-23T10:38:51Z

filebeat/module/system/audit/ingest/pipeline.json

+                "field": "system.audit.sub_kv",
+                "ignore_failure": true
+            }
+        },


In the other filesets so far we remove the original message key, to avoid duplication. For consistency, we might want to do it here as well, unless there's a lot of value in keeping it?

One reason I like having the message field is because it gets analyzed and make searching simpler when I don't know exactly what I'm looking for (helps with _all being deprecated). WDYT?

I was always on the fence (for each new module) about keeping the message in, but thought I'd try it without it first, since it's easier to add than remove. My understanding is that by default Kibana will still search in all fields, even without having an explicit _all.

What you think if we leave it out here again for consistency, but we reconsider before 6.0 if we want to add it everywhere?

I like to not have the raw message in by default. But it can become useful in some case, for example for debugging. We could introduce a config option which can be set per module to turn it on. Problem is that this would mean a change in the processor which should not change based on config options.

Removed the message field in ececd94.

tsg · 2017-03-23T10:39:53Z

filebeat/module/system/audit/ingest/pipeline.json

+        },
+        {
+            "convert": {
+                "field" : "system.audit.item",


Interesting, are these conversions needed? In the other places I have assumed that if we set it as a number in the mapping template, that's enough for ES to do the conversion itself.

You're right, I don't think these are strictly necessary. ES should do the conversion.

The only difference that I can possibly think of here is how errors would be handled. If the conversion fails in the ingest processor the event is still written to ES, but has an error field and is not completely enriched. If a failure occurs during automatic conversion in ES then you get an mapping exception and the whole document gets dropped.

Good point, I agree it's worth keeping them in this case.

tsg · 2017-03-23T10:45:36Z

filebeat/module/system/audit/ingest/pipeline.json

+                    "AUDIT_KEY_VALUES": "%{WORD}=%{GREEDYDATA}"
+                },
+                "patterns": [
+                    "%{AUDIT_PREFIX} %{AUDIT_KEY_VALUES:system.audit.kv} old auid=%{NUMBER:system.audit.old_auid} new auid=%{NUMBER:system.audit.new_auid} old ses=%{NUMBER:system.audit.old_ses} new ses=%{NUMBER:system.audit.new_ses}",


Might be worth documenting old_auid, old_ses, etc?

I'll add those.

Added in 7cd7d52

tsg

This is great, sorry for the late review. I left a few comments. This would also be worth a quick blog post like the one I did on the auth logs.

@timestamp

- Renamed system.audit.type to system.audit.record_type - Renamed system.audit.counter to system.audit.sequence - Dropped system.audit.epoch after parsing it into @timestamp - Dropped message field. - Removed “and others” from the supported OSes listed in docs. - Updated dashboards after renaming and deleting fields. - Updated script to drop fields where value was `(null)`.

… (elastic#3941) (elastic#3962) The PR adds an auditd module for parsing logs from the auditd daemon. There is a sample dashboard for viewing this data. Features - Parses audit event type, unix epoch time, audit event counter, and the arbitrary key/value pairs that follow. - Applies geoip lookup to the `auditd.log.addr` field which is present for some remote login events. - Decodes hex encoded ascii values that are sometimes used for the `auditd.log.exe` and `auditd.log.cmd` fields. - Remove key/value pairs where the value is `?`. Missing Features - Decoder for `auditd.log.saddr` field present in SOCKADDR audit events. A recipe is described [here](https://unix.stackexchange.com/questions/102926/how-to-interpret-the-saddr-field-of-an-audit-log) and could probably be ported to painless. Sample value: `type=SOCKADDR msg=audit(1481078693.491:873): saddr=01002F7661722F72756E2F6E7363642F736F636B657400008983B330BE7F0000000000000000000070830130BE7F000004000000000000001B00000000000000D128B7ED000000008B8BB330BE7F00003B00000000000000403E4BF7FD7F0000447F0130BE7F000080150230BE7F`

) (#3975) The PR adds an auditd module for parsing logs from the auditd daemon. There is a sample dashboard for viewing this data. Features - Parses audit event type, unix epoch time, audit event counter, and the arbitrary key/value pairs that follow. - Applies geoip lookup to the `auditd.log.addr` field which is present for some remote login events. - Decodes hex encoded ascii values that are sometimes used for the `auditd.log.exe` and `auditd.log.cmd` fields. - Remove key/value pairs where the value is `?`. Missing Features - Decoder for `auditd.log.saddr` field present in SOCKADDR audit events. A recipe is described [here](https://unix.stackexchange.com/questions/102926/how-to-interpret-the-saddr-field-of-an-audit-log) and could probably be ported to painless. Sample value: `type=SOCKADDR msg=audit(1481078693.491:873): saddr=01002F7661722F72756E2F6E7363642F736F636B657400008983B330BE7F0000000000000000000070830130BE7F000004000000000000001B00000000000000D128B7ED000000008B8BB330BE7F00003B00000000000000403E4BF7FD7F0000447F0130BE7F000080150230BE7F`

andrewkroh added Filebeat Filebeat in progress Pull request is currently in progress. labels Mar 15, 2017

andrewkroh force-pushed the feature/filebeat-audit-fileset branch 2 times, most recently from 3e91ac1 to 8794516 Compare March 16, 2017 21:19

andrewkroh added review and removed in progress Pull request is currently in progress. labels Mar 16, 2017

andrewkroh force-pushed the feature/filebeat-audit-fileset branch from 8794516 to a5ef7ae Compare March 16, 2017 21:25

andrewkroh requested a review from tsg March 16, 2017 21:25

ruflin approved these changes Mar 17, 2017

View reviewed changes

andrewkroh added 2 commits March 17, 2017 12:37

Apply fileset overrides before checking enabled

c945fb7

Disable other filesets in a module while testing.

andrewkroh force-pushed the feature/filebeat-audit-fileset branch from a5ef7ae to c945fb7 Compare March 17, 2017 16:38

tsg reviewed Mar 23, 2017

View reviewed changes

tsg approved these changes Mar 23, 2017

View reviewed changes

monicasarbu added v6.0.0-alpha1 in progress Pull request is currently in progress. labels Mar 30, 2017

andrewkroh added 2 commits March 31, 2017 13:17

Document old_auid, new_auid, old_ses, new_ses

7cd7d52

andrewkroh removed the in progress Pull request is currently in progress. label Mar 31, 2017

tsg merged commit e5f5bef into elastic:master Apr 3, 2017

tsg added needs_backport PR is waiting to be backported to other branches. v5.4.0 labels Apr 5, 2017

andrewkroh mentioned this pull request Apr 10, 2017

Cherry-pick #3750 to 5.x: Add fileset for parsing linux auditd logs #3975

Merged

andrewkroh removed the needs_backport PR is waiting to be backported to other branches. label Apr 10, 2017

andrewkroh deleted the feature/filebeat-audit-fileset branch July 5, 2017 22:04

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add fileset for parsing linux auditd logs #3750

Add fileset for parsing linux auditd logs #3750

andrewkroh commented Mar 15, 2017 •

edited

ruflin Mar 17, 2017

andrewkroh Mar 17, 2017

ruflin Mar 17, 2017

andrewkroh Mar 17, 2017

andrewkroh Mar 17, 2017 •

edited

ruflin commented Mar 20, 2017

tsg Mar 23, 2017

andrewkroh Mar 23, 2017

tsg Mar 23, 2017

andrewkroh Mar 23, 2017

ruflin Mar 25, 2017

andrewkroh Mar 31, 2017

tsg Mar 23, 2017

andrewkroh Mar 23, 2017

tsg Mar 23, 2017

tsg Mar 23, 2017

andrewkroh Mar 23, 2017

tsg Mar 23, 2017

andrewkroh Mar 23, 2017

tsg Mar 23, 2017

andrewkroh Mar 23, 2017

ruflin Mar 25, 2017

andrewkroh Mar 31, 2017

tsg Mar 23, 2017 •

edited

andrewkroh Mar 23, 2017

tsg Mar 23, 2017

tsg Mar 23, 2017

andrewkroh Mar 23, 2017

andrewkroh Mar 31, 2017

tsg left a comment

Add fileset for parsing linux auditd logs #3750

Add fileset for parsing linux auditd logs #3750

Conversation

andrewkroh commented Mar 15, 2017 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

andrewkroh Mar 17, 2017 • edited

Choose a reason for hiding this comment

ruflin commented Mar 20, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tsg Mar 23, 2017 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tsg left a comment

Choose a reason for hiding this comment

andrewkroh commented Mar 15, 2017 •

edited

andrewkroh Mar 17, 2017 •

edited

tsg Mar 23, 2017 •

edited