Child Decoders for WinEvtLog are Broken

[sempervictus]

Wazuh version	Component	Action type
4.3.7	Decoders	Error

Description

Decoders added to local_decoders.xml are ignored when they specify the <parent>windows</parent> flow.
Asked about this in slack and received confirmation that this workflow is known to be broken: it's not possible to write child decoders for WinEvtLog logs as they are decoded internally.
I'm a bit at a loss as to what "internally" means. The processing which occurs in the log reader stage of the pipeline is specific to dealing with the charming flavors of character encoding used by MS platforms and streaming data out of an API vs a file source (i modeled the journald reader after it) but actual decoding should be happening higher up in the stack as with all other data streams - logcollectors aren't privy to the logic and contents of decoders...

Service/Product/Module

WinEvtLog decoding is broken

Errors/Improvements

Current results

Field extractors from

<decoder name="windows_firewall_blocked_fields">
  <type>windows</type>
  <parent>windows</parent>
  <prematch>The Windows Filtering Platform blocked a packet.|The Windows Filtering Platform has blocked a connection.</prematch>
  <regex offset="after_prematch"> Application Information: Process ID: (\d+) Application Name: (\S+)\.*Source Address: (\S+) Source Port: (\d+) Destination Address: (\S+)  Destination Port: (\d+) Protocol: (\d+)</regex>
  <order>win.eventdata.processId,win.eventdata.imagePath,srcip,srcport,dstip,dstport,win.eventdata.protocol</order>
</decoder>

are ignored

Expected results

All field extractors work, not just the ones used by the Wazuh team.

Resources

Log source / integration

WinEvtLog

Log reference

2022 Sep 08 09:28:57 WinEvtLog: Security: AUDIT_FAILURE(5152): Microsoft-Windows-Security-Auditing: (no user): no domain: hostname.fq.dn: The Windows Filtering Platform blocked a packet. Application Information: Process ID: 4784 Application Name: \device\harddiskvolume3\windows\system32\dashost.exe Network Information: Direction: %%14592 Source Address: 172.20.10.10 Source Port: 64039 Destination Address: 239.255.255.250 Destination Port: 3702 Protocol: 17 Filter Information: Filter Run-Time ID: 367646 Layer Name: %%14610 Layer Run-Time ID: 44

should produce field extractions for relevant terms

[sempervictus]

This appears to be a result of Wazuh having written a dedicated EventChannel decoder into analysisd hard-coding the step into a bunch of static C. I am guessing that this was done for some performance oriented reason since upstream handles these events same as any others.
Given the (unfortunate) prevalence of Windows out there, this change in behavior is pretty heavily breaking for anyone migrating from OSSEC or other forks. However, backing out such a major piece of work seems unwise; so i'd like to propose a "compromise" by way of configuration:

1. Add an <evtdecoderlegacy/> option to the <localfile/> section or use <log_format>eventchannelfullparse</log_format> to "tag" data sourced from a specific event channel for processing by the full (xml-driven) pipeline
2. Teach the reader to understand the tagging from step 1 and include in the final event sent to the queue for analysisd to work through.
3. Allow analysisd to "fast-path" the incoming data by default through the Wazuh C processor code or go into the full/flexible pipeline if the flag from step 2 is accordingly set, permitting use of local_decoders.xml for EventChannel data
4. Document the mechanism for anyone trying to write local decoders for evtchan data.

[sempervictus]

Thanks for checking in on this @juliancnn. Any thoughts within the team about my proposed approach? You guys put in some hefty effort to write that decoder (kudos) right in C, but enterprise use-cases effectively require the ability to parse-out fields from event channel streams as enterprise applications running on windows log there and require context-specific decoders for field extraction. That said, same enterprise workloads generate a metric tonne of EPS, most of which is OS-native stuff that can be handled by your optimized implementation. So i figure that if we handle the majority with the optimized code and the domain-specific stuff in classic pipeline, it should permit the EPS to stay quite high but avoid turning people away because they just can't use Wazuh to extract the critical data they need for their context.

[juliancnn]

Hi @sempervictus,

Thank you for your contributions to the Wazuh community.
We know that embedded decoders in C and the predecoder are often a limitation. This is why we are developing a proof of concept #11334 of a new log analysis engine (currently under development, which, perhaps, in the future could replace analisysd).

The new engine tries to handle json, when it receives any type of log, it adds it to a json field. This event enters in the chain of operations that will be defined by the 'Assets' (they would be like filters, decoders and rules). These assets will be defined by the user in YMl documents and will perform the task of enriching and normalizing the event.
The engine aims to be as transparent as possible, receiving the events as they are picked up by the agent, and inserting them into a chain of operations that are easy to define, read and manipulate by the users. We want there to be no "hidden" or unavoidable manipulations, such as the predecoder or embedded decoders of Wazuh-Analisisd. In the case of windows events, we are looking to receive them as XML, and transform them to JSON once inside the user-defined operation chain. This will give great potential and open new opportunities for windows event handling.

I encourage you to take a look at the epic and share your thoughts with us!