tracing/runtimeeventsource/nativeruntimeeventsource/nativeruntimeeventsource failing in CI

[ericstj]

Build Information

Build: https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_build/results?buildId=374041
Build error leg or test failing: tracing/runtimeeventsource/nativeruntimeeventsource/nativeruntimeeventsource.sh
Pull request: #90553

Error Message

Fill the error message using step by step known issues guidance.

{
  "ErrorMessage": "System.Exception: Condition 'listener.EventCount > 0' is not true",
  "ErrorPattern": "",
  "BuildRetry": false,
  "ExcludeConsoleLog": false
}

Failure:

Unhandled Exception:
      System.Exception: Condition 'listener.EventCount > 0' is not true
         at Tracing.Tests.NativeRuntimeEventSourceTest.Assert2.True(String name, Boolean condition)
         at Tracing.Tests.NativeRuntimeEventSourceTest.Main()
      [ERROR] FATAL UNHANDLED EXCEPTION: System.Exception: Condition 'listener.EventCount > 0' is not true
         at Tracing.Tests.NativeRuntimeEventSourceTest.Assert2.True(String name, Boolean condition)
         at Tracing.Tests.NativeRuntimeEventSourceTest.Main()
      
      Return code:      1
      Raw output file:      /root/helix/work/workitem/uploads/Reports/tracing.runtimeeventsource/nativeruntimeeventsource/nativeruntimeeventsource.output.txt
      Raw output:
      BEGIN EXECUTION
      /root/helix/work/correlation/corerun -p System.Reflection.Metadata.MetadataUpdater.IsSupported=false -p System.Runtime.Serialization.EnableUnsafeBinaryFormatterSerialization=true nativeruntimeeventsource.dll ''

https://helixre8s23ayyeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-pull-90553-merge-02a796bd6a9f40108e/PayloadGroup0/1/console.cd59d68c.log?helixlogtype=result

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=374041
Error message validated: System.Exception: Condition 'listener.EventCount > 0' is not true
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 8/15/2023 4:15:41 PM UTC

Report

Build	Definition	Test	Pull Request
711703	dotnet/runtime	tracing/runtimeeventsource/nativeruntimeeventsource/nativeruntimeeventsource.sh	#103647
707058	dotnet/runtime	PayloadGroup0.WorkItemExecution
702836	dotnet/runtime	tracing/runtimeeventsource/nativeruntimeeventsource/nativeruntimeeventsource.sh
699880	dotnet/runtime	tracing\runtimeeventsource\nativeruntimeeventsource\nativeruntimeeventsource.cmd	#103139
684463	dotnet/runtime	tracing/runtimeeventsource/nativeruntimeeventsource/nativeruntimeeventsource.cmd	#102563
696739	dotnet/runtime	tracing/runtimeeventsource/nativeruntimeeventsource/nativeruntimeeventsource.sh
690534	dotnet/runtime	tracing/runtimeeventsource/nativeruntimeeventsource/nativeruntimeeventsource.sh
689458	dotnet/runtime	PayloadGroup0.WorkItemExecution	#102748
686411	dotnet/runtime	tracing/runtimeeventsource/nativeruntimeeventsource/nativeruntimeeventsource.sh

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
0	1	9

[AaronRobinsonMSFT]

This is actually failing in mono, not coreclr. The leg is

mono linux arm64 Release @ (Ubuntu.1804.Arm64.Open)Ubuntu.1804.Armarch.Open@mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm64v8

[AaronRobinsonMSFT]

@tommcdon Since this looks like a mono issue I am marking it untriaged and removing @davmason. Apologies if I have overstepped.

[AaronRobinsonMSFT]

/cc @marek-safar @lambdageek

[carlossanlop]

This failure is pretty impactful in the release/8.0-rc1 and release/8.0 branches.

Note that the 8.0-rc1 branch is closed now, so if there is a fix, it will have to go to RC2 (release/8.0 branch).

[carlossanlop]

Another hit in release/8.0:

• Affected PR: [release/8.0] Update dependencies from dotnet/roslyn #90816
• Queue: linux-arm64 Release AllSubsets_Mono_Minijit_RuntimeTests minijit
• Build: https://dev.azure.com/dnceng-public/public/_build/results?buildId=383187&view=logs&j=f979b8ec-ebd4-511f-1076-61f8eccd76f8&t=492e1b7b-2815-5b61-9ca1-e91e53779a38&l=28
• Log: https://helixre8s23ayyeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-pull-90816-merge-88f063107fd94ee2a9/PayloadGroup0/1/console.0088424c.log?helixlogtype=result
• Output:

 Starting:    tracing.runtimeeventsource.XUnitWrapper (parallel test collections = on, max threads = 2)
    tracing/runtimeeventsource/nativeruntimeeventsource/nativeruntimeeventsource.sh [FAIL]
      
      Unhandled Exception:
      System.Exception: Condition 'listener.EventCount > 0' is not true
         at Tracing.Tests.NativeRuntimeEventSourceTest.Assert2.True(String name, Boolean condition)
         at Tracing.Tests.NativeRuntimeEventSourceTest.Main()
      [ERROR] FATAL UNHANDLED EXCEPTION: System.Exception: Condition 'listener.EventCount > 0' is not true
         at Tracing.Tests.NativeRuntimeEventSourceTest.Assert2.True(String name, Boolean condition)
         at Tracing.Tests.NativeRuntimeEventSourceTest.Main()
      
      Return code:      1
      Raw output file:      /root/helix/work/workitem/uploads/Reports/tracing.runtimeeventsource/nativeruntimeeventsource/nativeruntimeeventsource.output.txt
      Raw output:
      BEGIN EXECUTION
      /root/helix/work/correlation/corerun -p System.Reflection.Metadata.MetadataUpdater.IsSupported=false -p System.Runtime.Serialization.EnableUnsafeBinaryFormatterSerialization=true nativeruntimeeventsource.dll ''
      buffer length=16
      (Listener) [63, (Failed to listen, Was not heard or didn't fire)]
      (Listener) [64, (Failed to listen, Was not heard or didn't fire)]
      (Listener) [65, (Failed to listen, Was not heard or didn't fire)]
      

      Expected: 100
      Actual: 1
      END EXECUTION - FAILED
      Test failed. Trying to see if dump file was created in /home/helixbot/dotnetbuild/dumps since 8/23/2023 5:36:48 AM
      Test Harness Exitcode is : 1
      To run the test:
      > set CORE_ROOT=/root/helix/work/correlation
      > /root/helix/work/workitem/e/tracing/runtimeeventsource/nativeruntimeeventsource/nativeruntimeeventsource.sh
      Expected: True
      Actual:   False
      Stack Trace:
           at tracing_runtimeeventsource._nativeruntimeeventsource_nativeruntimeeventsource_._nativeruntimeeventsource_nativeruntimeeventsource_sh()
           at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)
           at System.Reflection.MethodBaseInvoker.InvokeWithNoArgs(Object obj, BindingFlags invokeAttr)
      Output:
        
        Unhandled Exception:
        System.Exception: Condition 'listener.EventCount > 0' is not true
           at Tracing.Tests.NativeRuntimeEventSourceTest.Assert2.True(String name, Boolean condition)
           at Tracing.Tests.NativeRuntimeEventSourceTest.Main()
        [ERROR] FATAL UNHANDLED EXCEPTION: System.Exception: Condition 'listener.EventCount > 0' is not true
           at Tracing.Tests.NativeRuntimeEventSourceTest.Assert2.True(String name, Boolean condition)
           at Tracing.Tests.NativeRuntimeEventSourceTest.Main()
        
        Return code:      1
        Raw output file:      /root/helix/work/workitem/uploads/Reports/tracing.runtimeeventsource/nativeruntimeeventsource/nativeruntimeeventsource.output.txt
        Raw output:
        BEGIN EXECUTION
        /root/helix/work/correlation/corerun -p System.Reflection.Metadata.MetadataUpdater.IsSupported=false -p System.Runtime.Serialization.EnableUnsafeBinaryFormatterSerialization=true nativeruntimeeventsource.dll ''
        buffer length=16
        (Listener) [63, (Failed to listen, Was not heard or didn't fire)]
        (Listener) [64, (Failed to listen, Was not heard or didn't fire)]
        (Listener) [65, (Failed to listen, Was not heard or didn't fire)]
        

        Expected: 100
        Actual: 1
        END EXECUTION - FAILED
        Test failed. Trying to see if dump file was created in /home/helixbot/dotnetbuild/dumps since 8/23/2023 5:36:48 AM
        Test Harness Exitcode is : 1

[SamMonoRT]

@steveisok - @fanyang-mono brought this to my attention. A PR to disable the test in main and release\8.0 will be stood up soon to enable cleaner CI, and we can work on identifying the root cause of the failures.