Logstash - grok parser error

[ghost]

FYSA: Although the sysmon logs are coming in and most are being parsed correctly - thus visible after ingest - in the "Syslog" and "Sysmon" dashboards. The problem is that there is also a high number of grok parser errors as shown in the totally awesome "stats" dashboard. Example event below:

Stats event shows:

tags:syslogng, syslog, _grokparsefailure, process_creation, internal_source syslog-priority:notice logstash_time:0.016 message:2017 Dec 14 00:53:35 .... then onto truncated message.....

Raw message:

2017 Dec 14 00:53:35 (VICTIM02) 10.0.1.102->WinEvtLog 2017 Dec 13 16:53:35 WinEvtLog: Microsoft-Windows-Sysmon/Operational: INFORMATION(1): Microsoft-Windows-Sysmon: SYSTEM: NT AUTHORITY: VICTIM02.looty.xyz: Process Create: UtcTime: 2017-12-14 00:53:35.541 ProcessGuid: {0ED6B270-CB8F-5A31-0000-0010A7F79E00} ProcessId: 6076 Image: C:\Windows\System32\AppHostRegistrationVerifier.exe CommandLine: C:\Windows\system32\AppHostRegistrationVerifier.exe CurrentDirectory: C:\Windows\system32\ User: VICTIM02\cuckoo LogonGuid: {0ED6B270-C7AF-5A31-0000-0020A3295000} LogonId: 0x5029A3 TerminalSessionId: 2 IntegrityLevel: Medium Hashes: MD5=349A75331035C5266DA27BC3AFC552CC,SHA256=3D134E850306AE1DFBCC84F01503A8B63438CE72AAC1C62B9DF420DB28100114 ParentProcessGuid: {0ED6B270-5CD1-5A31-0000-001001C30000} ParentProcessId: 988 ParentImage: C:\Windows\System32\svchost.exe ParentCommandLine: C:\Windows\system32\svchost.exe -k netsvcs

[weslambert]

I think this is due to some spacing inconsistencies.

The following should work:

"rest_of_msg", "Microsoft-Windows-Sysmon/Operational: INFORMATION(%{INT:event_id}): Microsoft-Windows-Sysmon: SYSTEM: NT AUTHORITY: %{DATA:hostname}: %{DATA:event_type}: UtcTime: %{DATA:sysmon_timestamp} ProcessGuid: {%{DATA:process_guid}} ProcessId: %{INT:process_id} Image: %{DATA:image_path} CommandLine: %{DATA:process_name} %{DATA:process_arguments}CurrentDirectory: %{DATA:current_directory} User: %{DATA:user} LogonGuid: {%{DATA:logon_guid}} LogonId: %{DATA:logon_id} TerminalSessionId: %{INT:terminal_id} IntegrityLevel: %{DATA:integrity_level} Hashes: MD5=%{DATA:md5},SHA256=%{DATA:sha256} ParentProcessGuid: {%{DATA:parent_process_guid}} ParentProcessId: %{NONNEGINT:parent_process_id} ParentImage: %{DATA:parent_image_path} ParentCommandLine: %{GREEDYDATA:parent_process_name}",

Inserted after the following line:
https://github.com/dougburks/elastic-test/blob/master/configfiles/6501_ossec_sysmon.conf#L25

in /etc/logstash/conf.d/6501_ossec_sysmon.conf

Thanks,
Wes

[dougburks]

Merged dougburks/securityonion-elastic#146 and added this to Beta 3 list:
#1172

[ghost]

Thanks, Gents!

[ghost]

Testing beta3 ISO now, just had to add this manually..

grok failures were showing and when I checked https://github.com/dougburks/elastic-test/blob/master/configfiles/6501_ossec_sysmon.conf#L25
the line was not there..

[ghost]

found another spacing issue and just added another line containing this: (space needed at "CurrentDirectory")

"rest_of_msg", "Microsoft-Windows-Sysmon/Operational: INFORMATION(%{INT:event_id}): Microsoft-Windows-Sysmon: SYSTEM: NT AUTHORITY: %{DATA:hostname}: %{DATA:event_type}: UtcTime: %{DATA:sysmon_timestamp} ProcessGuid: {%{DATA:process_guid}} ProcessId: %{INT:process_id} Image: %{DATA:image_path} CommandLine: %{DATA:process_name} %{DATA:process_arguments} CurrentDirectory: %{DATA:current_directory} User: %{DATA:user} LogonGuid: {%{DATA:logon_guid}} LogonId: %{DATA:logon_id} TerminalSessionId: %{INT:terminal_id} IntegrityLevel: %{DATA:integrity_level} Hashes: MD5=%{DATA:md5},SHA256=%{DATA:sha256} ParentProcessGuid: {%{DATA:parent_process_guid}} ParentProcessId: %{NONNEGINT:parent_process_id} ParentImage: %{DATA:parent_image_path} ParentCommandLine: %{GREEDYDATA:parent_process_name}",

[weslambert]

Have you confirmed the above mentioned (1st) line is present in the config file in /opt/elastic/src/configfiles/?

To confirm, are you implying that there are two different patterns that aren't being matched correctly?

Would you be able to provide the raw log(s) for each so that we can confirm each is working on our side?

Thanks,
Wes

[ghost]

Wes,

Your parser template string above was completely missing from the new beta3 ISO that I just built from. I added it like you have it above after the line you specified. What I found was that the grok errors persisted, so I looked deeper and found that there was no space at

%{DATA:process_arguments}CurrentDirectory: %

So, I created a new parser string and added it which contains the space. I guess I can comment out the line without the space and see if errors are thrown again.

Here is what the grok pattern for event 1 is in my .conf now:

if [event_id] == 1 {
grok {
match => ["rest_of_msg", "Microsoft-Windows-Sysmon: SYSTEM: NT AUTHORITY: %{DATA:hostname}: %{DATA:event_type}: UtcTime: %{DATA:sysmon_timestamp} ProcessGuid: {%{DATA:process_guid}} ProcessId: %{INT:process_id} Image: %{DATA:image_path} CommandLine: %{DATA:process_name} %{DATA:process_arguments} CurrentDirectory: %{DATA:current_directory} User: %{DATA:user} LogonGuid: {%{DATA:logon_guid}} LogonId: %{DATA:logon_id} TerminalSessionId: %{INT:terminal_id} IntegrityLevel: %{DATA:integrity_level} Hashes: MD5=%{DATA:md5},SHA256=%{DATA:sha256} ParentProcessGuid: {%{DATA:parent_process_guid}} ParentProcessId: %{NONNEGINT:parent_process_id} ParentImage: %{DATA:parent_image_path} ParentCommandLine: %{GREEDYDATA:parent_process_name}",
"rest_of_msg", "Microsoft-Windows-Sysmon/Operational: INFORMATION(%{INT:event_id}): Microsoft-Windows-Sysmon: SYSTEM: NT AUTHORITY: %{DATA:hostname}: %{DATA:event_type}: UtcTime: %{DATA:sysmon_timestamp} ProcessGuid: {%{DATA:process_guid}} ProcessId: %{INT:process_id} Image: %{DATA:image_path} CommandLine: %{DATA:process_name} %{DATA:process_arguments}CurrentDirectory: %{DATA:current_directory} User: %{DATA:user} LogonGuid: {%{DATA:logon_guid}} LogonId: %{DATA:logon_id} TerminalSessionId: %{INT:terminal_id} IntegrityLevel: %{DATA:integrity_level} Hashes: MD5=%{DATA:md5},SHA256=%{DATA:sha256} ParentProcessGuid: {%{DATA:parent_process_guid}} ParentProcessId: %{NONNEGINT:parent_process_id} ParentImage: %{DATA:parent_image_path} ParentCommandLine: %{GREEDYDATA:parent_process_name}",
"rest_of_msg", "Microsoft-Windows-Sysmon/Operational: INFORMATION(%{INT:event_id}): Microsoft-Windows-Sysmon: SYSTEM: NT AUTHORITY: %{DATA:hostname}: %{DATA:event_type}: UtcTime: %{DATA:sysmon_timestamp} ProcessGuid: {%{DATA:process_guid}} ProcessId: %{INT:process_id} Image: %{DATA:image_path} CommandLine: %{DATA:process_name} %{DATA:process_arguments} CurrentDirectory: %{DATA:current_directory} User: %{DATA:user} LogonGuid: {%{DATA:logon_guid}} LogonId: %{DATA:logon_id} TerminalSessionId: %{INT:terminal_id} IntegrityLevel: %{DATA:integrity_level} Hashes: MD5=%{DATA:md5},SHA256=%{DATA:sha256} ParentProcessGuid: {%{DATA:parent_process_guid}} ParentProcessId: %{NONNEGINT:parent_process_id} ParentImage: %{DATA:parent_image_path} ParentCommandLine: %{GREEDYDATA:parent_process_name}",
"rest_of_msg", 'Microsoft-Windows-Sysmon: SYSTEM: NT AUTHORITY: %{DATA:hostname}: %{DATA:event_type}: UtcTime: %{DATA:sysmon_timestamp} ProcessGuid: {%{DATA:process_guid}} ProcessId: %{INT:process_id} Image: %{DATA:image_path} CommandLine: "%{DATA:process_name}" %{DATA:process_arguments} CurrentDirectory: %{DATA:current_directory} User: %{DATA:user} LogonGuid: {%{DATA:logon_guid}} LogonId: %{DATA:logon_id} TerminalSessionId: %{INT:terminal_id} IntegrityLevel:%{DATA:integrity_level}',
"rest_of_msg", "Microsoft-Windows-Sysmon/Operational: INFORMATION(%{INT:event_id}): Microsoft-Windows-Sysmon: SYSTEM: NT AUTHORITY: %{DATA:hostname}: %{DATA:event_type}: UtcTime: %{DATA:sysmon_timestamp} ProcessGuid: {%{DATA:process_guid}} ProcessId: %{INT:process_id} Image: %{DATA:image_path} CommandLine: %{DATA:process_name} %{DATA:process_arguments}CurrentDirectory: %{DATA:current_directory} User: %{DATA:user} LogonGuid: {%{DATA:logon_guid}} LogonId: %{DATA:logon_id} TerminalSessionId: %{INT:terminal_id} IntegrityLevel: %{DATA:integrity_level} Hashes: MD5=%{DATA:md5},SHA256=%{DATA:sha256} ParentProcessGuid: {%{DATA:parent_process_guid}} ParentProcessId: %{NONNEGINT:parent_process_id} ParentImage: %{DATA:parent_image_path} ParentCommandLine: %{GREEDYDATA:parent_process_name}"]
}
mutate {

[ghost]

Found another error and it all came down to more spaces..
Here is what I added spaces to - as a final result.. (Not sure why so many spacing differences)

"rest_of_msg", "Microsoft-Windows-Sysmon/Operational: INFORMATION(%{INT:event_id}): Microsoft-Windows-Sysmon: SYSTEM: NT AUTHORITY: %{DATA:hostname}: %{DATA:event_type}: UtcTime: %{DATA:sysmon_timestamp} ProcessGuid: {%{DATA:process_guid}} ProcessId: %{INT:process_id} Image: %{DATA:image_path} CommandLine: %{DATA:process_name} %{DATA:process_arguments} CurrentDirectory: %{DATA:current_directory} User: %{DATA:user} LogonGuid: {%{DATA:logon_guid}} LogonId: %{DATA:logon_id} TerminalSessionId: %{INT:terminal_id} IntegrityLevel: %{DATA:integrity_level} Hashes: MD5=%{DATA:md5},SHA256=%{DATA:sha256} ParentProcessGuid: {%{DATA:parent_process_guid}} ParentProcessId: %{NONNEGINT:parent_process_id} ParentImage: %{DATA:parent_image_path} ParentCommandLine: %{GREEDYDATA:parent_process_name}",

This one basically need two spaces between everything... /-:

[weslambert]

The above grok pattern in regard to CurrentDirectory (single space -- to clarify, I did not modify it, but left it with no space) works for me if I run it through Grok Debugger -- that log in particular has no "process arguments" value. I'm wondering if this event is coming with both no spaces, and one and/or two spaces in between some/all terms.

In regard to the second one involving two spaces, are you referring to this log coming in with both one and two spaces between terms?

Not to defer from improving the OSSEC/Sysmon config files, but you may get better results with Winlogbeat shipping those Sysmon logs to Logstash.

Thanks,
Wes

[weslambert]

I think the best approach for the existing OSSEC/Sysmon config may be to use %{SPACE} to match on 0 or more spaces.

[ghost]

Wes, you are correct I am inferring that there are two formats. I was wondering if there is a "wildcard' I could use. Thanks!

I'll have to test the difference between the winbeats and ossec event forwarding. Great idea. Thanks again!

[ghost]

This seems to have halted all the errors:

"rest_of_msg", "Microsoft-Windows-Sysmon/Operational:%{SPACE}INFORMATION(%{INT:event_id}):%{SPACE}Microsoft-Windows-Sysmon:%{SPACE}SYSTEM:%{SPACE}NT AUTHORITY:%{SPACE}%{DATA:hostname}:%{SPACE}%{DATA:event_type}:%{SPACE}UtcTime:%{SPACE}%{DATA:sysmon_timestamp}%{SPACE}ProcessGuid:%{SPACE}{%{DATA:process_guid}}%{SPACE}ProcessId: %{INT:process_id}%{SPACE}Image:%{SPACE}%{DATA:image_path}%{SPACE}CommandLine:%{SPACE}%{DATA:process_name}%{SPACE}%{DATA:process_arguments}%{SPACE}CurrentDirectory:%{SPACE}%{DATA:current_directory}%{SPACE}User: %{DATA:user}%{SPACE}LogonGuid:%{SPACE}{%{DATA:logon_guid}}%{SPACE}LogonId:%{SPACE}%{DATA:logon_id}%{SPACE}TerminalSessionId: %{INT:terminal_id}%{SPACE}IntegrityLevel:%{SPACE}%{DATA:integrity_level}%{SPACE}Hashes: MD5=%{DATA:md5},SHA256=%{DATA:sha256}%{SPACE}ParentProcessGuid:%{SPACE}{%{DATA:parent_process_guid}}%{SPACE}ParentProcessId: %{NONNEGINT:parent_process_id}%{SPACE}ParentImage:%{SPACE}%{DATA:parent_image_path}%{SPACE}ParentCommandLine: %{GREEDYDATA:parent_process_name}",

[weslambert]

Thanks for confirming. I'll work on adding %{SPACE} for space matching in the config and getting it committed soon.

Thanks,
Wes