ETW exporter

[maxgolov]

This project is created in response to Community Bridge Q3-Q4. We are looking for interns to work on this project.

open-telemetry/community#468

Project

Add exporter for ETW (Event Tracing for Windows) as one of the optional recommended exporters that languages may decide to implement as part of SDK.

Logical Reasoning

Application logs, service logs, even Docker-on-Windows logs can be directed to ETW (ref. https://docs.docker.com/config/containers/logging/etwlogs/ ). Many products running on Windows (most Microsoft products, Intel products, Docker, etc.) - rely on ETW channel.

Adding support for ETW for Windows platforms in SDK would allow any cross-platform codebase to be instrumented with OpenTelemetry instead of using Win32 API directly, e.g. replacing Docker etwlogs driver with structured ETW traces emitted using OpenTelemetry C++ SDK. Final product demo could be the Docker using OpenTelemetry C++ SDK for traces, and Docker on Windows - using OpenTelemetry C++ SDK with ETW exporter.

Extra bonus points

Two 'extra' pieces that may be covered in addition to the main exporter piece:

• writing a projection layer for other languages (e.g. Java or node on Windows), that allows to dispatch traces from OT Java to OT C++ SDK with ETW exporter. That flow would allow Java and node services / servers running on Windows hosts to route their telemetry to ETW listener agent.

• separate effort that implements ETW Provider listener (Agent), that subsequently uses OpenTelemetry .NET SDK to dispatch all telemetry collected to OTLP exporter (or any other exporter supported by .NET SDK). Very nice and simple example how to implement such a listener: https://github.com/fireeye/SilkETW

Two alternate implementations of the ETW message packing

• First approach is to utilize MsgPack-encoded event payload. This approach allows to send structured objects / nested objects over ETW to the Agent. This approach has no external dependencies, but is incompatible with some existing ETW tracing tools. Work on this track can be started right away.

• Second approach is to use standard ETW "dynamic manifest" events. This approach allows to express an event as key-value property bag, flat list. I am working with MSFT CELA to contribute Microsoft TraceLoggingDynamic C++ implementation in OSS. Main benefit of this approach is it will be compatible with already existing ETW tracing tools (such as Windows Performance Analyzer, xperf, etc.). ETA for OSS is by the end of this year.

[shivam-51]

Hi @maxgolov I would love to take on this project at Community Bridge. I am comfortable in C, C++ and Java and have contributed to various FOSS. Can you suggest me some material I should go through to get a better insight?

[maxgolov]

@shivam-51 - sounds good. Let's sync-up on Gitter: https://gitter.im/maxgolov

There's some initial read / implementation you can take a look here:
https://github.com/maxgolov/opentelemetry-cpp/tree/master/examples/ETWTracer

This depends presently on a proprietary header. But there's a minor mod we can make to utilize MsgPack serialization instead of TraceLoggingDynamic.h. Eventually the header I'm talking about will be opensourced this year.. Semantic difference is that MsgPack-encoded ETW events would have to be decoded by the listener, whereas dynamic manifest events can be easily processed by existing .NET Framework / .NET Standard code on Windows.

Would you be able to join our weekly community meetings on Mondays?

[shivam-51]

Hi, @maxgolov . Thanks for the initial read material.
I had messaged you on gitter but looks like you missed the notification :)
Today is the last date for applying so am following up here.
I understand that the project is essentially to create a ETW exporter which the cross-platform applications can access instead of accessing the WIN32 API. I went through your ETWTracer repo but could not create the .exe executable file as TraceLoggingDynamic.h is not included yet(which you had mentioned earlier too). I am not very familiar with using MsgPack serialization for the data exchange here. Could you guide me a bit on that ?

Would you be able to join our weekly community meetings on Mondays?

Yes, I will be able to join all the weekly community meetings. My time zone is +5:30

[maxgolov]

@shivam-51 - these are the additional materials:

• take a look at json.hpp library here: https://github.com/nlohmann/json - specifically the MsgPack encoder here: https://github.com/nlohmann/json#binary-formats-bson-cbor-messagepack-and-ubjson - write a simple program that populates JSON and transforms it into MessagePack binary serialized buffer.
• next stage is going to be to provide an exporter that forward events in MessagePack serialized format using this Win32 API: https://docs.microsoft.com/en-us/windows/win32/api/evntprov/nf-evntprov-eventwritestring - note that the API accepts PCWSTR - wide UTF-16 string, but in fact as long as you provide the necessary padding at the end, you can send any arbitrary binary payload. Some pointers: https://stackoverflow.com/questions/2134384/how-to-use-etw-from-a-c-windows-client

Above approach - EventWriteString of an event encoded with MessagePack using json.hpp library would allow us to send any structured event to ETW listener on the other end.

[maxgolov]

@shivam-51 @mishal23 @theadityasam - I'm gonna send a PR with two parts draft implementation:

• demo of how to use EventWriteString function with MsgPack-serialized buffer.
• and a test listener tool in C# that is accepting incoming events at the receiving end of the pipe (kinda mini-Agent for test and validation).

Subsequently we need to figure out the API and protocol details:

• logging vs. tracing
• ETW payload contents
• encoding and format, plus structure (do we pack OTLP protocol binary events instead of MsgPack, for example); and concrete contents of common fields populated by SDK

Don't worry about your Community Bridge applications. We are sorting it out this week.

[EngAlaaEmad]

Hi @maxgolov i love to work in this project at community bridge, i am comfortable with C ,C++ and C#

[prondubuisi]

Hello @maxgolov I got your well-written email. I applied for the Open Telemetry PHP project, but I will be following up on this one too for the fun of trying out my c++ skills.

[The4kings]

Hi @maxgolov i would like to contribute in this project, as i have experience in C/C++ programming and web development.

[maxgolov]

This is the draft implementation that should be sufficient to get started with the project. The goal is to shape it into ETW exporter that follows common principles of other OpenTelemetry C++ SDK exporters.

I did the initial very crude draft implementation that you should be able to compile right now:
https://github.com/maxgolov/opentelemetry-cpp/tree/maxgolov/etw_MsgPack

It's in this directory:
https://github.com/maxgolov/opentelemetry-cpp/tree/maxgolov/etw_MsgPack/examples/ETWTracer

You can open it and Visual Studio and compile. Directory above contains Visual Studio solution. What the code is doing:

• initializes OpenTelemetry
• obtains ETW tracer using GetTracer API call
• by default the code assumes that it will transform all Spans into ETW event that contains MessagePack-encoded payload (there is no more dependency on any proprietary headers)
• the other end may listen and decode the event. We can consider various alternatives, but I'll share a separate ETW listener tool. We can also explore an option of packaging OTLP protocol-encoded buffers inside ETW event.

[AdhitiS]

Hey @maxgolov I would love to be a part of this project. I have prior experience in C , C++ and Web development. Can you please share some more insights on how to proceed with this project?

[maxgolov]

@AdhitiS - Hello! :) Let's sync-up on Gitter. You can try to clone the code above to get started, then feel free to ping me on Gitter if you have questions or get stuck with this.

[maxgolov]

Partially implemented in #376. Working to address comments from review in a separate PR.