-
Notifications
You must be signed in to change notification settings - Fork 3.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Windows EventLog support #1395
Comments
This would instantly make Loki viable in my environment - fluentd requires ruby, which is no-go, but having a single go executable would be perfect. Alternatively, being able to accept data from winlogbeat (or beats in general!) and run it through the pipline would be amazing! |
+1 I have many Windows systems in my environment. |
This would be great! I'm currently using InfluxDB/Telegraf as Syslog receiver with NXlog (https://nxlog.co/) to convert Windows Event logs to Syslog, using the im_msvistalog module. I see that Promtail can be used as Syslog target (https://github.com/grafana/loki/blob/master/docs/clients/promtail/scraping.md#syslog-target), so maybe something like that would be a temporary solution until this is implemented? |
Another alternative is using Fluentd's Windows EventLog plugin. Fluentd ecosystem has fluent-plugin-windows-eventlog's in_windows_eventlog2 plugin which can consume .evtx format Windows EventLog. |
If you convert Windows events to a log, then it can already be sent to loki via promtail |
Just wanted to let interested parties know - winlogbeat (and all the elastic beats) can be configured to output to rolling files instead of logstash, so you can scrape them with promtail! |
We're going to add logstash soon, this is nice to know for windows users. |
I am currently working on this one (i.e a golang prototype to get windows event logs directly to promtail). So far the executable size is ~2.5MB. Your feedback is appreciated for the following:
Kindly provide example production workload numbers. Your feedback is appreciated 😄 |
That's awesome, thanks so much for working on this! As for feedback:
I think this will need to be able to support arbitrary log names, just try all the configured ones and ignore ones that fail. It's pretty standard practice for enterprise applications to create their own logs that need to be monitored, even ignoring all of the other Microsoft ones. Here are a just a few of the ones I would need to monitor:
You can see the list of ones available in powershell by running: Get-WinEvent -ListLog * My desktop has 510 logs registered!
As for tailing, I would think that using
This would need to be rotated and cleaned up - my personal favorite method to rotate logs that are being consumed is to include the date in the file name, such as always writing to
This would need to be configurable. The executable will generally be in the program files directory which will require admin access to write to, and for some of our servers I would want to move it to another drive for space considerations - the logs easily reach a couple of gigs a month, and we need to retain them for regulatory reasons. Thanks again, and I'm happy to provide feedback! |
From what I understand you're listening to event and writing them to a temporary files. This is nice, although alternatively later we could read directly from promtail with a new windows target. Really thank you for contributing this is awesome ! and @azawawi join #loki-dev if you need anything. |
@randomchance Thanks for the useful info and feedback. I really appreciate it. @cyriltovena Yes that sums it up. So from I understand, I need to add a new windows-only build target (i.e. |
@azawawi If it's possible, I agree with @cyriltovena that it would be better to send directly to promtail, technically you can already use Winlogbeat to write eventlogs directly to files. Thanks again! |
@azawawi You asked for example production workload numbers, so I got some for you! I just checked one of our installations were we batch logs for retention.
... and I just realized this is an aggregate number so not super applicable, but maybe if you divide it by the 9 servers? |
No that’s the idea you got it right, we can help you of course along the way. |
Since the eventlog API supports xpath queries, I think that would be a good low hanging fruit for any solution. @azawawi I did some digging into how DotNet handles persisting the last read location in the EventLog stream. DotNet has an This means that if you use an XPath query, you can filter it with something like: Where 83005 is the RecordID of the last record stored. This query gets records after record 83005 and older than 86400000 milliseconds Or if you want to get the specific event you left off on: I think it would be smart to look for the single event first, and if it's no longer there you can assume the the log has dropped it and just start at the beginning, or also store the timestamp and fall back to it. I don't know if that's helpful, but I was looking into writing something similar in DotNet Core and this was stumping me for a while. You can test the filters/queries in PowerShell pretty easily: $query = "Event[System[EventRecordID > 83005 and TimeCreated[timediff(@SystemTime) <= 86400000]]]"
Get-WinEvent -FilterXPath $query -LogName System |
I hope to hear more about this, I am currently using winlogbeat with elasticsearch, but if the use of loki for these event logs is possible I could remove the elasticsearch instance and save cpu resources. |
Why not using fluentbin input winlogevent plugin https://docs.fluentbit.io/manual/pipeline/inputs/windows-event-log ? I have both winlogbeat (which is amazing) and the above config with fluent bit and seems to work, only caveat is that I have yet to include eventlog level information (info,error...) in the loki labels. |
fluent-bit winlogevent plugin does not support to retrieve eventlog's description which should be supported with |
|
@Jacq Can you share how you have built the Loki FluentD-bit plugin? |
I'm also in need of a Loki compliant log shipper for Windows Event Logs. I'm currently trying to get Fluentbit working, since it has a winlog plugin. I'm not sure if I can build a Loki plugin using /cmd/fluent-bit. It just creates a file for *nix usage, not a Windows DLL/library. Is there a way to target a different arch for the file output? |
I have to check, I think I tried to build also but finally grab the binary from the online repo. |
As far as I can tell, all of the fluent options only support the older style logs, not the newer "channels" such as Right now I'm using winlogbeat => logstash => loki and while I like winlogbeat, I really dislike running logstash on windows. |
AFAIK, the newer Windows EventLog should be retrieved with |
I have found the online repo @JacoboDominguez is referencing. Further I have opened an issue to supply the fluent-bit loki plugin for windows as binary or add a make command for this. #2563 |
@azawawi are you able to share the code for the work you are doing? We have many Windows servers I can potentially test this on. |
@randomchance what kind of mutations are you doing to add labels for loki and drop high cardinality ones? I'm configuring this rn, but the output from winlogbeat is massive... |
@Ulfy sorry for the delay - I'm really only doing a couple of things, and taking advantage of the fact that top level fields with multiple values are dropped. A quick summary:
The guidelines for loki stress how important it is to not add a ton of labels, and point out that the regex filtering is very powerful.
It's possible that sending logs to promtail first and generating metrics there will help address some of the issues I've met, but I'm probably going to need to add some indication of the log channel as a label, though I'm not sure what that will look like yet. |
@randomchance would you be willing to share your config file for Logstash to Loki? I got winlogbeat to talk to Logstash, now I am a bit lost in actually formatting the data to get into Loki. |
@calebcoverdale Here is a excerpt from a lessons learned KB I put together for my team:
Here is full example config, I tried to add explanatory comments:
|
This thread was useful getting a working fluentd->loki setup going for Windows EventLog using in_windows_eventlog2 as @cosmo0920 suggested. This supports ?new? channels. This is a ruby-based solution but @randomchance this does allow you to specify channels like When specifying channels in the fluentd config the key is no quoting or escaping which tricked me out initially:
|
@fifofonix That's awesome! The docs do not support that, so I opened an issue to get them updated. For anyone curious, the documentation says/implies that the standard four logs are the entire set of possible options.
I'm already pretty invested in my current config, but I'll definitely try out a fluentd configuration if I get a chance. Having more options is definitely better. Now that promtail supports syslog input, using a log shipper that outputs syslog is also an option. |
...and Telegraf now has an input plugin for Windows Event Log https://github.com/influxdata/telegraf/blob/v1.16.0/plugins/inputs/win_eventlog/README.md |
Hey @randomchance, thanks a lot for your logstash example, I appreciate it. I'm currently setting up something similar for my company and I had a question about the My understanding is Loki tags must be key, value pairs, and the values cannot be nested. When you add a new field such as I'm fairly new to Logstash so I'm probably misinterpreting this entirely. I'd appreciate your guidance. Thanks! Edit: Nevermind -- I had some time today to go through the entire config and I see it gets amended to message. |
@danfoxley Does Telegraf have a Loki plugin to output w/ labels? I'm still looking for something to ship windows event logs to Loki... |
Having the logs be represented in JSON would potentially be better than XML as Loki 2.0 has the ability to do parsing at query time for high cardinality data, using JSON and Regex, but not XML. |
* Use enums to represent the clients (grafana#1395) Signed-off-by: ChinYing-Li <chinying.li@mail.utoronto.ca> Update PR number in CHANGELOG.md * Use the StorageType enums in the case statements (grafana#1395) Signed-off-by: ChinYing-Li <chinying.li@mail.utoronto.ca>
🔔 Hear ye, hear ye! 🔔 Sorry for commenting on the graveyard. This is not required Ruby setup. We can use it by only deploying Fluent Bit executable. |
Hiyo! I might have missed it, but, am I correct in that Promtail will not convert the XML to json, nor has an XML processor been added to Loki, meaning folks essentially just struggle by and parse the message field with regex when working with Windows event logs? Cheers! |
https://grafana.com/docs/agent/latest/static/set-up/install/install-agent-on-windows/ the grafana agent has config for events. |
To clarify, I have no issues at all processing and sending Windows events. This question is about the format of the data. Promtail (which I am testing already, and which grafana agent embeds, afaik, so using it would not change this) does not process the actual event data, and simply sends a string of XML for the event_data. This is... not helpful. I will open another issue if this is the case, but looking to confirm I am not missing anything as I've only spent an afternoon looking at this. An example of the data in Grafana Cloud:
So! Windows events derive significant value from the structured data, in this case, under event_data. For example, if these were Active Directory logs, that's where you would find who did what to what principal, among other essential data. That this data is (1) not processed into JSON, and (2) not processable via some XML processor in Grafana, is a significant gap in functionality. I could absolutely be mistaken, but it appears that folks are resorting to regex parsing the meant-for-human-eyes With that said, before I open an issue that boils down to "this is a terrible experience and for windows event log users, please consider these alternatives," I want to make sure I'm not missing something obvious. Cheers! |
@RamblingCookieMonster |
That probably would help! To be honest though, there's such a wide variety to what could be included in a field that it's probably not viable outside of one-off solutions to use a parser like that post-ingest, at least, IMHO. Here's a synopsis of what I've found, keeping in mind this is mostly superficial level time/effort, so take with a grain of salt: What I want: (1) structured windows event data, (2) with parsed event data field names, (3) in a format loki can process
Ultimately, this bit of processing, at least superficially, gets telegraf output into a format Loki / logfmt will be happy with. Haven't tested it much, there might be other characters/sequences that break logfmt, but so far so good: [[processors.strings]]
# Duct taping OSS that isn't designed for Windows, so... escape something that
# will later become an escape character and confuse logfmt (for Loki queries)
[[processors.strings.replace]]
field = "*"
old = '\'
new = '\\'
# Duct taping OSS that isn't designed for Windows, so... handle the many cases
# where a field will have a double quote (command line, script content, cron/task definition, etc.)
# Telegraf sends key="value", and key="value with "quotes"" is not valid for logfmt (for Loki queries)
[[processors.strings.replace]]
field = "*"
old = '"'
new = '\"'
# Duct taping OSS that isn't designed for Windows, so... handle the few event data field names
# that will have spaces in them, as logfmt (for Loki queries) will be quite confused without this.
[[processors.strings.replace]]
field_key = "*"
old = ' '
new = '_'
[[processors.strings.tagpass]]
__name = 'win_eventlog' It's not as batteries-included as something like elastic or splunk agents, but, it appears that this will be viable. I do think it would be valuable for promtail to be able to meet the needs I mentioned, IMHO it's absolute-bare-minimum functionality for logging in a windows environment, but I can see why it's not a thing yet (if ever). Cheers! |
@wardbekker please, for Windows Event Log where EventData comes in as XML
Is there consideration to using Go encoding/xml package to parse XML? Considering parsing the ingested XML in LOKI is not available today, what are your thoughts / comments .. use pattern parser? Other? |
@RamblingCookieMonster Filebeat, I guess, can't go straight to Loki. How about using Filebeat to ?? (file, logstash...) then Loki? |
Is your feature request related to a problem? Please describe.
Windows logs are stored in Event Log (
.evtx
files), which currently not possible to scrape it via currently available promtail methods.Describe the solution you'd like
Since we do have systemd journal support for Linux, it would be nice to have support for Event Log on Windows in a similar matter.
Describe alternatives you've considered
Key part of the solution is actually being able to parse the logs. If I haven't missed anything, it seems that there are currently two Golang modules that can do that: github.com/0xrawsec/golang-evtx and github.com/elastic/beats/winlogbeat/eventlog.
The text was updated successfully, but these errors were encountered: