Missing IIS file format

[funkwhatyouheard]

I was attempting to ingest some logs with an updated version of the sof-elk VM with the following format:
#Fields: date time s-sitename s-computername s-ip cs-method cs-uri-stem cs-uri-query s-port cs-username c-ip cs-version cs(User-Agent) cs(Referer) cs-host sc-status sc-substatus sc-win32-status sc-bytes cs-bytes time-taken X-Forwarded-For X-Forwarded-Proto x-SSL-Ver Crypt-Protocol

Currently, the logs ingest, but all the timestamps are set to ingest time and none of the fields are mapping correctly so I think that the parsers just aren't sure how to handle the format.

I think I have a functional grok expression to handle them... but doesn't seem to parse the logs I have when adding it to the 6100=httpd.conf on the VM. Very likely that user error is involved here... new to ELK and grok. What I believe to be a good grok expression for the aforementioned format below.

%{TIMESTAMP_ISO8601:timestamp} %{NOTSPACE:service_name} %{NOTSPACE:server_hostname} %{IPORHOST:destination_ip} %{WORD:request_method} %{URIPATH:request} %{NOTSPACE:query_string} %{NUMBER:destination_port} %{NOTSPACE:ident} %{IPORHOST:source_ip} %{NOTSPACE:version} %{NOTSPACE:useragent} %{NOTSPACE:referrer} %{NOTSPACE:hostname} %{NUMBER:response_code} %{NUMBER:response_sub} %{NUMBER:win_status} %{NUMBER:destination_bytes} %{NUMBER:source_bytes} %{NUMBER:response_time} %{NOTSPACE:x_source_ip} %{NOTSPACE:x_protocol} %{NOTSPACE:ssl_version} %{NOTSPACE:crypt_protocol}

[philhagen]

quite likely a Grok bug. have you run that Grok statement in the debugger within Kibana? I usually test new Grok parsers one field at a time to find which one breaks, then adjust as needed.

[philhagen]

if you can post a few lines (sanitized is ok), I can try to take a look show you the debug process here as well

[funkwhatyouheard]

That's neat, didn't know that existed. I'd tested with something similar but news to me that there was a debugger built in to Kibana. From what I can see, the Grok seems to function as expected.

I neutered a decent chunk of data for testing and it seems consistently successful.

2022-08-29 17:22:55 W3SVC2 SERVERNAME1 127.0.0.1 POST /placeholder.aspx - 80 - 127.0.0.1 HTTP/1.1 - - localhost 200 0 0 378 590 7545 - - - -
2022-08-29 17:22:55 W3SVC2 SERVERNAME1 127.0.0.1 POST /placeholder.aspx - 80 - 127.0.0.1 HTTP/1.1 - - localhost 200 0 0 521 570 33 - - - -
2022-08-29 17:25:06 W3SVC2 SERVERNAME1 ::1 GET /placeholder.aspx transaction=wakeup 80 - ::1 HTTP/1.1 - - localhost 200 0 0 378 90 2331 - - - -
2022-08-29 17:25:06 W3SVC2 SERVERNAME1 ::1 GET /filler/filler.dll action=ping 80 - ::1 HTTP/1.1 - - localhost 200 0 0 215 74 23 - - - -
2022-08-29 17:29:01 W3SVC2 SERVERNAME1 ::1 GET / - 443 - ::1 HTTP/2 Mozilla/5.0+(Windows+NT+10.0;+Win64;+x64)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/104.0.5112.102+Safari/537.36+Edg/104.0.1293.70 - localhost 200 0 0 5186 688 793 - - - 400
2022-08-29 17:29:01 W3SVC2 SERVERNAME1 ::1 GET /js/dist/smartbanner.min.css v=13 443 - ::1 HTTP/2 Mozilla/5.0+(Windows+NT+10.0;+Win64;+x64)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/104.0.5112.102+Safari/537.36+Edg/104.0.1293.70 https://localhost/ localhost 200 0 0 1615 780 4 - - - 400
2022-08-29 17:29:01 W3SVC2 SERVERNAME1 ::1 GET /templates/buttonlinkbase.css v=13 443 - ::1 HTTP/2 Mozilla/5.0+(Windows+NT+10.0;+Win64;+x64)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/104.0.5112.102+Safari/537.36+Edg/104.0.1293.70 https://localhost/ localhost 200 0 0 11100 781 10 - - - 400
2022-08-29 17:29:01 W3SVC2 SERVERNAME1 ::1 GET /templates/stylesheet_placeholder.css v=13 443 - ::1 HTTP/2 Mozilla/5.0+(Windows+NT+10.0;+Win64;+x64)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/104.0.5112.102+Safari/537.36+Edg/104.0.1293.70 https://localhost/ localhost 200 0 0 12710 789 12 - - - 400

[philhagen]

just pushed a change to the latest public branch in ce4fea7. I think the ingest timestamps you were seeing may have been from an older data load because the (admittedly oddball) problem with your grok was that the X-Forwarded-For IP address being a - was not being handled and it caused an error in downstream processing that expects that to be a proper IPv4 or IPv6 address. A different Grok parser in the same file names that xff_ip, which is already handled downstream (i.e. if it's a -, it is converted to 0.0.0.0 so type constraints can be met.)

That said, this version should handle your format for you!

[funkwhatyouheard]

Thank you! Really appreciate the assistance.

I just updated and gave it a go, but doesn't seem to ingest at all now. Tested with a sanitized version so I could share. Worst case I can stand up a fresh instance and give it another go but I haven't made any changes to the VM other than adding data and updating sof-elk itself. Any ideas?

[philhagen]

what version of the VM are you using? (it's shown in the pre-authentication screen.) the changes were pushed to the latest public version (2023-04-19). I generally push updates just to the newest version but this is very small so I could merge the update to a different one if needed. either way, the result from the update command shows that the VM is not getting the updated parser.

[funkwhatyouheard]

Sorry, that was the second time I ran the update so it doesn't appear to have done anything but is in fact up to date. I was just trying to provide evidence (I confirmed the additional grok expression was in the httpd config file). I just grabbed a fresh version of the VM the other day so it's revision 2023-04-19.

[philhagen]

Roger - thanks for clarifying! I've run the sanitized data from your earlier post through and it worked so I'm not sure what the problem may be. Are you putting the input data into a new file or overwriting an existing one? (The latter generally won't trip the filebeat prospectors to get the new content.). If you do something like cd /logstash/httpd/ ; cp -a oldfile.txt newfile.txt it would pick that up though. (Even mv may not work here... odd way filebeat monitors by inode.)

[funkwhatyouheard]

You bet! And I figured you had :) Seems even with the fresh copy in /logstash/httpd/ it still isn't picking anything up. At this point I'm tempted to spin up a new instance and try from the start as it seems like something particular to my instance.

[philhagen]

what about permissions? can the logstash account read the file (and read+traverse any parent directories between /logstash/httpd/ itself and the file)?

[funkwhatyouheard]

I had the files directly in /logstash/httpd and permissions looked good. Tried another instance I had setup for a class a few weeks ago and couldn't get ingest there either. I'm probably making a small mistake somewhere.

[philhagen]

ugh - well this is exceptionally frustrating. Sorry to hear it's persisting. Could you try the load on a clean instance (ideally freshly-extracted and updated), and send the following log files via email?

• /var/log/logstash/logstash-plain.log*
• /var/log/messages
• /var/log/elasticsearch/elasticsearch*.log*

Also, let me know the approximate UTC time that you added the file to load.

[funkwhatyouheard]

Sorry for the delay. Just sent you over an email with the requested log files. Thanks for all the help through this!

[philhagen]

Thanks! Closing the loop in the thread that this should be all fixed up now. Corrected the embarrassingly missing comma in 761535b.

[funkwhatyouheard]

Thanks for all the help! Confirmed ingest is functioning as expected.