Setup support for managed runtime events #87785

LakshanF · 2023-06-19T21:49:53Z

Initial work to enable managed runtime events in EventPipe. The work includes

Enable threadpool events in EventPipe and aligns to the merged PR related to Contention, Add managed entry points for raising Contention events #87087. However, additional work needs to be done to have all the relevant events fired. PR Expose a Lock type in preview mode #87672, and issue Enable relevant thread pool events in Windows #87020 need to be completed to get a larger set of events hooked.
Enabled managed exceptions (FireEtwExceptionThrown_V1) to be fired by EventPipe. Issue DotnetRuntime provider needs to handle WCHAR strings #87978 tracks to handle WCHAR strings in EventPipe
Wire up additional native GC events that will be fired in Windows. [NativeAOT] GC events don't get logged on Linux #87445 needs to be completed for these to be seen from Linux.
Enabled additional support for managed side to enable EventPipe. Issue .net8 prev 5 unassociated AOT processes crash when the debugger starts on the system. #87739 runs into NYI RhEventPipeInternal_WaitForSessionSignal, and hopefully should be unblocked with these changes.
Build the full Tracing test subdirectory for NativeAOT

…Events

ghost · 2023-06-19T21:50:07Z

Tagging subscribers to this area: @agocke, @MichalStrehovsky, @jkotas
See info in area-owners.md if you want to be subscribed.

Issue Details

Initial work to enable managed runtime events in EventPipe.

Author:	LakshanF
Assignees:	LakshanF
Labels:	`area-NativeAOT-coreclr`
Milestone:	8.0.0

elinor-fung · 2023-06-28T19:31:46Z

Initial work to enable managed runtime events in EventPipe.

Is the plan to also fire these via ETW (on Windows)?

Wire up additional native GC events that will be fired in Windows. #87445 needs to be completed for these to be seen from Linux.

I'd expect only the events that go through ETW::GCLog to not be fired on non-Windows and direct usage of the FireEtw*/EventPipeWriteEvent* functions should fire through EventPipe on all platforms. Are you seeing that event the FireEtw*/EventPipeWriteEvent* ones firing aren't on Linux?

LakshanF · 2023-06-28T21:11:36Z

I'd expect only the events that go through ETW::GCLog to not be fired on non-Windows and direct usage of the FireEtw*/EventPipeWriteEvent* functions should fire through EventPipe on all platforms. Are you seeing that event the FireEtw*/EventPipeWriteEvent* ones firing aren't on Linux?

I can see events GCGlobalHeapHistory and GCHeapStats getting fired in Linux.

Opened #88162 for adding these events to ETW.

src/coreclr/nativeaot/System.Private.CoreLib/src/System/Runtime/RuntimeImports.cs

src/coreclr/nativeaot/Runtime/gctoclreventsink.cpp

src/coreclr/nativeaot/Runtime/disabledeventpipeinternal.cpp

src/coreclr/nativeaot/System.Private.CoreLib/src/System/Runtime/RuntimeImports.cs

src/coreclr/nativeaot/Runtime/eventpipeinternal.cpp

src/coreclr/nativeaot/Runtime/clretwallmain.h

src/coreclr/nativeaot/Runtime/eventpipeinternal.cpp

src/tests/tracing/eventpipe/simpleprovidervalidation/simpleprovidervalidation.cs

Co-authored-by: Elinor Fung <elfung@microsoft.com>

src/coreclr/nativeaot/Runtime/gctoclreventsink.cpp

src/coreclr/nativeaot/Runtime/eventpipeinternal.cpp

src/coreclr/nativeaot/Runtime/runtimeeventinternal.cpp

src/coreclr/nativeaot/System.Private.CoreLib/src/System/Runtime/RuntimeImports.cs

.../src/System/Diagnostics/Eventing/NativeRuntimeEventSource.Threading.NativeSinks.NativeAot.cs

src/coreclr/nativeaot/Runtime/runtimeeventinternal.cpp

elinor-fung

I'm assuming the dotnetruntime.cpp functions added for writing the events is lifted from the auto-generated ones for coreclr, so I didn't look at them much.

cc @davmason - if you are interested in the tracing/eventpipe test changes

elinor-fung · 2023-07-05T21:48:20Z

src/coreclr/nativeaot/Runtime/disabledruntimeeventinternal.cpp

+
+#ifdef FEATURE_PERFTRACING
+
+// We will do a no-op for events in the disabled EventPipe This is similar to the way eventpipe checks if the provider and an event is enabled before firting the event, and no-op otherwise.


Can we not make the managed side conditionally call the runtime import based on event source support and let trimming do its thing? Or are we trying to keep the event source feature switch and disabled/enabled eventing native library independent from one another?

Just thinking that the disabled and enabled definitions might be getting a bit unwieldy and we could presumably remove the disabled ones only called from managed (maybe also a really small size decrease).

I updated the issue #82231 with this as well.

.../src/System/Diagnostics/Eventing/NativeRuntimeEventSource.Threading.NativeSinks.NativeAot.cs

src/tests/tracing/eventpipe/simpleruntimeeventvalidation/simpleruntimeeventvalidation.cs

src/tests/tracing/eventpipe/providervalidation/providervalidation.cs

elinor-fung · 2023-07-05T22:32:52Z

src/tests/tracing/eventpipe/providervalidation/providervalidation.cs

        {
            { "MyEventSource", new ExpectedEventCount(100_000, 0.30f) },
            { "Microsoft-Windows-DotNETRuntimeRundown", -1 },
            { "Microsoft-DotNETCore-SampleProfiler", -1 }
        };

+        private static Dictionary<string, ExpectedEventCount> _expectedEventCountsNativeAOT = new Dictionary<string, ExpectedEventCount>()
+        {
+            { "MyEventSource", 100_000 },


Does coreclr need the ExpectedEventCount error, but nativeaot does not?

I believe coreclr error range tolerance is an artifact of earlier times when EventPipewas dropping events. NativeAOT doesn't support rundown and sampleprofiler and followed a policy of not touching coreclr testing.

`tracing/eventpipe/simpleruntimeeventvalidation`: ``` Generated app bundle at /root/helix/work/workitem/e/tracing/eventpipe/simpleruntimeeventvalidation/simpleruntimeeventvalidation/WasmApp/ Incoming arguments: --run simpleruntimeeventvalidation.dll Application arguments: --run simpleruntimeeventvalidation.dll console.info: Initializing dotnet version 8.0.0-ci commit hash f47b553f129cfa7f006cb1a2f2088112c5ca0112 0.0s: ==TEST STARTING== 0.1s: System.PlatformNotSupportedException: System.Diagnostics.Process is not supported on this platform. at System.Diagnostics.Process.GetCurrentProcess() at Tracing.Tests.Common.IpcTraceTest.Validate(Boolean enableRundownProvider) at Tracing.Tests.Common.IpcTraceTest.RunAndValidateEventCounts(Dictionary`2 expectedEventCounts, Action eventGeneratingAction, List`1 providers, Int32 circularBufferMB, Func`2 optionalTraceValidator, Boolean enableRundownProvider) 0.1s: ==TEST FINISHED: FAILED!== test-main.js exiting simpleruntimeeventvalidation.dll with result -1 console.info: WASM EXIT -1 ``` This was added in dotnet#87785, but got merged on red.

`tracing/eventpipe/simpleruntimeeventvalidation`: ``` Generated app bundle at /root/helix/work/workitem/e/tracing/eventpipe/simpleruntimeeventvalidation/simpleruntimeeventvalidation/WasmApp/ Incoming arguments: --run simpleruntimeeventvalidation.dll Application arguments: --run simpleruntimeeventvalidation.dll console.info: Initializing dotnet version 8.0.0-ci commit hash f47b553f129cfa7f006cb1a2f2088112c5ca0112 0.0s: ==TEST STARTING== 0.1s: System.PlatformNotSupportedException: System.Diagnostics.Process is not supported on this platform. at System.Diagnostics.Process.GetCurrentProcess() at Tracing.Tests.Common.IpcTraceTest.Validate(Boolean enableRundownProvider) at Tracing.Tests.Common.IpcTraceTest.RunAndValidateEventCounts(Dictionary`2 expectedEventCounts, Action eventGeneratingAction, List`1 providers, Int32 circularBufferMB, Func`2 optionalTraceValidator, Boolean enableRundownProvider) 0.1s: ==TEST FINISHED: FAILED!== test-main.js exiting simpleruntimeeventvalidation.dll with result -1 console.info: WASM EXIT -1 ``` This was added in #87785, but got merged on red.

LakshanF added 12 commits June 1, 2023 08:55

preliminery managed runtime event support

0b82d34

Linux fixes 1

c780753

full portable threadpool support

974159f

checkpoint before python script

5e0cfff

Additional GC event support

21c63f8

NativeAOT to use sink threadpool file

95e2149

Merge branch 'main' of https://github.com/dotnet/runtime into Runtime…

5356bca

…Events

CI check

dcda64e

Enable GC runtime tests

4d0c67d

Merge branch 'main' of https://github.com/dotnet/runtime into Runtime…

df1050c

…Events

Post main merge fixups

aab2ab0

change tests that run in CI

41f0644

LakshanF added the area-NativeAOT-coreclr label Jun 19, 2023

LakshanF added this to the 8.0.0 milestone Jun 19, 2023

LakshanF requested a review from MichalStrehovsky as a code owner June 19, 2023 21:49

LakshanF self-assigned this Jun 19, 2023

LakshanF marked this pull request as draft June 19, 2023 21:50

runfoapp bot mentioned this pull request Jun 20, 2023

Infra improvements for Helix #68176

Closed

Support native sinks for threading events

6f5de9d

LakshanF mentioned this pull request Jun 22, 2023

.net8 prev 5 unassociated AOT processes crash when the debugger starts on the system. #87739

Closed

LakshanF added 3 commits June 23, 2023 14:12

Enable managed exceptions via eventpipe

8168918

Move Exception event to EventPipe

99c62bd

Add an empty exception event to disabled EventPipe

4e13d99

runfoapp bot mentioned this pull request Jun 26, 2023

Long Running Test: Interop/MonoAPI/MonoMono/PInvokeDetach/PInvokeDetach.sh #73040

Closed

make events do no-op in disabled eventpipe

c2f65c1

build-analysis bot mentioned this pull request Jun 26, 2023

JIT/jit64/opt/rngchk/RngchkStress2.cs failing to build with error CS8078: An expression is too long or complex to compile #87879

Closed

LakshanF marked this pull request as ready for review June 27, 2023 09:39

LakshanF requested a review from elinor-fung June 27, 2023 09:39

build-analysis bot mentioned this pull request Jun 27, 2023

[browser][MT] The WebSocket is not connected. #88084

Closed

LakshanF mentioned this pull request Jun 28, 2023

Consider matching ETW to fire the new events added to EventPipe #88162

Open

elinor-fung reviewed Jun 29, 2023

View reviewed changes

LakshanF and others added 2 commits June 30, 2023 04:29

Update src/coreclr/nativeaot/Runtime/clretwallmain.h

d13003b

Co-authored-by: Elinor Fung <elfung@microsoft.com>

FB first stage

df3a3d5

LakshanF mentioned this pull request Jun 30, 2023

Remove EventPipe adapter from NativeAOT implementation #88262

Closed

Test change

1fb838a

build-analysis bot mentioned this pull request Jun 30, 2023

Tracking issue for CI build timeouts #76454

Closed

FB

60ebb00

elinor-fung reviewed Jul 3, 2023

View reviewed changes

src/coreclr/nativeaot/Runtime/gctoclreventsink.cpp Outdated Show resolved Hide resolved

elinor-fung reviewed Jul 4, 2023

View reviewed changes

elinor-fung mentioned this pull request Jul 4, 2023

Fix writing of WCHAR strings to event pipe buffer in nativeaot #88365

Closed

LakshanF added 2 commits July 4, 2023 07:27

latest FB

e4926c8

Pulled PR 88365

fb14519

elinor-fung approved these changes Jul 5, 2023

View reviewed changes

FB

07721f6

LakshanF merged commit b78345e into dotnet:main Jul 6, 2023
163 of 171 checks passed

LakshanF deleted the RuntimeEvents branch July 6, 2023 12:29

LakshanF mentioned this pull request Jul 6, 2023

DotnetRuntime provider needs to handle WCHAR strings #87978

Closed

radical mentioned this pull request Jul 6, 2023

[wasm] Disable runtime test dependent on creating a process #88498

Merged

ericstj mentioned this pull request Jul 6, 2023

simpleruntimeeventvalidation test failing in CI #88499

Closed

sbomer mentioned this pull request Jul 28, 2023

Use latest ILC to AOT-compile ILC #89655

Merged

This was referenced Jul 31, 2023

[.NET Tutorial]: dot net 8 web api aot compilation error dotnet/core#8649

Closed

IndexOutOfRangeException in tracing infrastructure when build is running ILC while ETW is enabled #88081

Closed

ghost locked as resolved and limited conversation to collaborators Aug 5, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Setup support for managed runtime events #87785

Setup support for managed runtime events #87785

LakshanF commented Jun 19, 2023 •

edited

Loading

ghost commented Jun 19, 2023

elinor-fung commented Jun 28, 2023

LakshanF commented Jun 28, 2023

elinor-fung left a comment

elinor-fung Jul 5, 2023

LakshanF Jul 6, 2023

elinor-fung Jul 5, 2023

LakshanF Jul 6, 2023


		#ifdef FEATURE_PERFTRACING

		// We will do a no-op for events in the disabled EventPipe This is similar to the way eventpipe checks if the provider and an event is enabled before firting the event, and no-op otherwise.

Setup support for managed runtime events #87785

Setup support for managed runtime events #87785

Conversation

LakshanF commented Jun 19, 2023 • edited Loading

ghost commented Jun 19, 2023

elinor-fung commented Jun 28, 2023

LakshanF commented Jun 28, 2023

elinor-fung left a comment

Choose a reason for hiding this comment

elinor-fung Jul 5, 2023

Choose a reason for hiding this comment

LakshanF Jul 6, 2023

Choose a reason for hiding this comment

elinor-fung Jul 5, 2023

Choose a reason for hiding this comment

LakshanF Jul 6, 2023

Choose a reason for hiding this comment

LakshanF commented Jun 19, 2023 •

edited

Loading