x-pack/filebeat/input/etw: New input (#36915)

* First version of ETW input

* Minor fixes for ETW input

* More fixes and requested changes for ETW input

* Include ETW in the default input list for Windows

* Tests for config input

* Sync input close calls

* Update config file and docs

* Fix some tabs in reference file

* Add metadata to ETW events

* Fix PR checks (docs and tests)

* Fix lint error in input

* Improve docs with supported providers and platforms

* Fix requested changes for ETW input

* Add ETW input to changelog

* Rename GetHandler to AttachToExistingSession in ETW input

* Fix NewSession unit test

* Add tests for input helpers

* Fix linting error in input_test.go

* Fix some unit tests

* Unit tests for ETW input

* Fix CloseSession call in tests

* Fix building of event and some refactors

* Add field mapping to ETW input

* Added files after make update

* Export fields mapping to docs

* Fix timestamp and GUID for buildEvent tests

* Adjust ETW mapping to fit ECS

* Update fields built files

* Address review comments

* filebeat/docs - rebuild with field changes

* fix tests

---------

Co-authored-by: Andrew Kroh <andrew.kroh@elastic.co>
Co-authored-by: Mariana Dima <mariana@elastic.co>

.github/CODEOWNERS

@@ -112,6 +112,7 @@ CHANGELOG*
 /x-pack/filebeat/input/cel/ @elastic/security-service-integrations
 /x-pack/filebeat/input/cometd/ @elastic/obs-infraobs-integrations
 /x-pack/filebeat/input/entityanalytics/ @elastic/security-service-integrations
+/x-pack/filebeat/input/etw/ @elastic/sec-windows-platform
 /x-pack/filebeat/input/gcppubsub/ @elastic/security-service-integrations
 /x-pack/filebeat/input/gcs/ @elastic/security-service-integrations
 /x-pack/filebeat/input/http_endpoint/ @elastic/security-service-integrations

CHANGELOG.next.asciidoc

@@ -179,6 +179,7 @@ Setting environmental variable ELASTIC_NETINFO:false in Elastic Agent pod will d
 - Prevent complete loss of long request trace data. {issue}37826[37826] {pull}37836[37836]
 - Added experimental version of the Websocket Input. {pull}37774[37774]
 - Add support for PEM-based Okta auth in CEL. {pull}37813[37813]
+- Add ETW input. {pull}36915[36915]
 - Update CEL mito extensions to v1.9.0 to add keys/values helper. {pull}37971[37971]
 
 *Auditbeat*

filebeat/docs/fields.asciidoc

@@ -92,6 +92,7 @@ grouped in the following categories:
 * <<exported-fields-threatintel>>
 * <<exported-fields-tomcat>>
 * <<exported-fields-traefik>>
+* <<exported-fields-winlog>>
 * <<exported-fields-zeek>>
 * <<exported-fields-zookeeper>>
 * <<exported-fields-zoom>>
@@ -158960,6 +158961,200 @@ alias to: source.geo.region_iso_code
 
 --
 
+[[exported-fields-winlog]]
+== Windows ETW fields
+
+Fields from the ETW input (Event Tracing for Windows).
+
+
+
+[float]
+=== winlog
+
+All fields specific to the Windows Event Tracing are defined here.
+
+
+
+*`winlog.activity_id`*::
++
+--
+A globally unique identifier that identifies the current activity. The events that are published with this identifier are part of the same activity.
+
+
+type: keyword
+
+required: False
+
+--
+
+*`winlog.channel`*::
++
+--
+Used to enable special event processing. Channel values below 16 are reserved for use by Microsoft to enable special treatment by the ETW runtime. Channel values 16 and above will be ignored by the ETW runtime (treated the same as channel 0) and can be given user-defined semantics.
+
+
+type: keyword
+
+required: False
+
+--
+
+*`winlog.event_data`*::
++
+--
+The event-specific data. The content of this object is specific to any provider and event.
+
+
+type: object
+
+required: False
+
+--
+
+*`winlog.flags`*::
++
+--
+Flags that provide information about the event such as the type of session it was logged to and if the event contains extended data.
+
+
+type: keyword
+
+required: False
+
+--
+
+*`winlog.keywords`*::
++
+--
+The keywords are used to indicate an event's membership in a set of event categories.
+
+
+type: keyword
+
+required: False
+
+--
+
+*`winlog.level`*::
++
+--
+Level of severity. Level values 0 through 5 are defined by Microsoft. Level values 6 through 15 are reserved. Level values 16 through 255 can be defined by the event provider.
+
+
+type: keyword
+
+required: False
+
+--
+
+*`winlog.opcode`*::
++
+--
+The opcode defined in the event. Task and opcode are typically used to identify the location in the application from where the event was logged.
+
+
+type: keyword
+
+required: False
+
+--
+
+*`winlog.process_id`*::
++
+--
+Identifies the process that generated the event.
+
+
+type: keyword
+
+required: False
+
+--
+
+*`winlog.provider_guid`*::
++
+--
+A globally unique identifier that identifies the provider that logged the event.
+
+
+type: keyword
+
+required: False
+
+--
+
+*`winlog.provider_name`*::
++
+--
+The source of the event log record (the application or service that logged the record).
+
+
+type: keyword
+
+required: False
+
+--
+
+*`winlog.session`*::
++
+--
+Configured session to forward ETW events from providers to consumers.
+
+
+type: keyword
+
+required: False
+
+--
+
+*`winlog.severity`*::
++
+--
+Human-readable level of severity.
+
+
+type: keyword
+
+required: False
+
+--
+
+*`winlog.task`*::
++
+--
+The task defined in the event. Task and opcode are typically used to identify the location in the application from where the event was logged.
+
+
+type: keyword
+
+required: False
+
+--
+
+*`winlog.thread_id`*::
++
+--
+Identifies the thread that generated the event.
+
+
+type: keyword
+
+required: False
+
+--
+
+*`winlog.version`*::
++
+--
+Specify the version of a manifest-based event.
+
+
+type: long
+
+required: False
+
+--
+
 [[exported-fields-zeek]]
 == Zeek fields
 

filebeat/docs/filebeat-options.asciidoc

@@ -75,8 +75,10 @@ You can configure {beatname_uc} to use the following inputs:
 * <<{beatname_lc}-input-cometd>>
 * <<{beatname_lc}-input-container>>
 * <<{beatname_lc}-input-entity-analytics>>
+* <<{beatname_lc}-input-etw>>
 * <<{beatname_lc}-input-filestream>>
 * <<{beatname_lc}-input-gcp-pubsub>>
+* <<{beatname_lc}-input-gcs>>
 * <<{beatname_lc}-input-http_endpoint>>
 * <<{beatname_lc}-input-httpjson>>
 * <<{beatname_lc}-input-journald>>
@@ -90,7 +92,6 @@ You can configure {beatname_uc} to use the following inputs:
 * <<{beatname_lc}-input-syslog>>
 * <<{beatname_lc}-input-tcp>>
 * <<{beatname_lc}-input-udp>>
-* <<{beatname_lc}-input-gcs>>
 * <<{beatname_lc}-input-websocket>>
 
 include::multiline.asciidoc[]
@@ -113,10 +114,14 @@ include::inputs/input-container.asciidoc[]
 
 include::../../x-pack/filebeat/docs/inputs/input-entity-analytics.asciidoc[]
 
+include::../../x-pack/filebeat/docs/inputs/input-etw.asciidoc[]
+
 include::inputs/input-filestream.asciidoc[]
 
 include::../../x-pack/filebeat/docs/inputs/input-gcp-pubsub.asciidoc[]
 
+include::../../x-pack/filebeat/docs/inputs/input-gcs.asciidoc[]
+
 include::../../x-pack/filebeat/docs/inputs/input-http-endpoint.asciidoc[]
 
 include::../../x-pack/filebeat/docs/inputs/input-httpjson.asciidoc[]
@@ -145,6 +150,4 @@ include::inputs/input-udp.asciidoc[]
 
 include::inputs/input-unix.asciidoc[]
 
-include::../../x-pack/filebeat/docs/inputs/input-gcs.asciidoc[]
-
 include::../../x-pack/filebeat/docs/inputs/input-websocket.asciidoc[]

x-pack/filebeat/_meta/config/filebeat.inputs.reference.xpack.yml.tmpl

@@ -180,3 +180,44 @@
   # This is used to shift collection start time and end time back in order to
   # collect logs when there is a delay in CloudWatch.
   #latency: 1m
+
+#------------------------------ ETW input --------------------------------
+# Beta: Config options for ETW (Event Trace for Windows) input (Only available for Windows)
+#- type: etw
+  #enabled: false
+  #id: etw-dnsserver
+
+  # Path to an .etl file to read from.
+  #file: "C:\Windows\System32\Winevt\Logs\Logfile.etl"
+
+  # GUID of an ETW provider.
+  # Run 'logman query providers' to list the available providers.
+  #provider.guid: {EB79061A-A566-4698-9119-3ED2807060E7}
+
+  # Name of an ETW provider.
+  # Run 'logman query providers' to list the available providers.
+  #provider.name: Microsoft-Windows-DNSServer
+
+  # Tag to identify created sessions.
+  # If missing, its default value is the provider ID prefixed by 'Elastic-'.
+  #session_name: DNSServer-Analytical-Trace
+
+  # Filter collected events with a level value that is less than or equal to this level.
+  # Allowed values are critical, error, warning, informational, and verbose.
+  #trace_level: verbose
+
+  # 8-byte bitmask that enables the filtering of events from specific provider subcomponents.
+  # The provider will write a particular event if the event's keyword bits match any of the bits
+  # in this bitmask.
+  # Run 'logman query providers "<provider.name>"' to list available keywords.
+  #match_any_keyword: 0x8000000000000000
+
+  # 8-byte bitmask that enables the filtering of events from
+  # specific provider subcomponents. The provider will write a particular
+  # event if the event's keyword bits match all of the bits in this bitmask.
+  # Run 'logman query providers "<provider.name>"' to list available keywords.
+  #match_all_keyword: 0
+
+  # An existing session to read from.
+  # Run 'logman query -ets' to list existing sessions.
+  #session: UAL_Usermode_Provider