Perf is not working for JITed code in DotNet

[shivanirmishra]

@adamsitnik
Profiling the DotNet Benchmarks (perf report) is not working for annotating the symbols of JITed code.
mono/mono#21646

[adamsitnik]

@shivanirmishra we provide a script that does all the heavy lifting related to perf: https://github.com/microsoft/perfview/blob/main/src/perfcollect/perfcollect

the docs: https://learn.microsoft.com/en-us/dotnet/core/diagnostics/trace-perfcollect-lttng

cc @brianrob

[brianrob]

The first thing to look at here is whether or not the /tmp/perf-%pid.map file or /tmp/jit-%pid.dump file is present when you're running the workload. If it is not there, then this must be resolved, since that's the source of symbol information for managed code.

[shivanirmishra]

Thank you @adamsitnik @brianrob
@adamsitnik,
Actually, we tried the "perf-collect" script before as well, and seems like it requires "Lttng" and that is not supported for dotnet with mono runtime. (Also mentioned in [https://learn.microsoft.com/en-us/dotnet/core/diagnostics/trace-perfcollect-lttng] - Used to capture event data emitted at run time by CoreCLR)

@brianrob,
In our case, /tmp/perf-%pid.map and /tmp/jit-%pid.dump files are getting generated but still, we are not able to see the annotations and it was failing to say objdump: /tmp/perf-800728.map: File format not recognized
We are using MONO_ENV_OPTIONS="--jitmap --jitdump perf record to generate .map and .dump files in /tmp/ folder.
So, we also want to make sure that --jitdump is supported or not in dotnet with mono runtime?

[adamsitnik]

So, we also want to make sure that --jitdump is supported or not in dotnet with mono runtime?

@lambdageek @vargaz who would be the best person to answer this question?

[lambdageek]

@fanyang-mono worked on the jitdump stuff in dotnet/runtime mono most recently, I think...

It looks like jitdump support shoudl be enabled on the linux versions of mono:

if(HOST_LINUX AND HAVE_SYS_MMAN_H AND HAVE_ELF_H AND HAVE_SYS_SYSCALL_H)
  set(ENABLE_JIT_DUMP 1)
endif()

---

@shivanirmishra

Can you say what linux distro you're on, and what version of dotnet you're using? Also do you have a small sample project and instructions to demonstrate the issue?

[shivanirmishra]

Thank you @lambdageek for your response
I am using RHEL 8.7 for both the Power and x86 system with DotNet 7.

We are using https://github.com/paulchapmanibm/myMLApp
This is a text-based sentiment analysis model.
It's taking around only 3sec runtime in both Power and X86.

Steps to collect the profile using perf:
Path: myMLApp/myMLApp/

1) MONO_ENV_OPTIONS=--jitmap perf record dotnet run
2) perf report -i perf.data

This environment variable will create a .map file in /tmp/ folder to resolve function names for JITed code sections in the profile. But we are not able to access the annotations inside those functions. For that, I think we need a .dump file as well.

So, we also tried using --jitdump along with --jitmap in the same environment variable which generated some .dump files within /tmp/ but those are not recognized during annotation.

[janani66]

May be worth trying the fix in dotnet/runtime#85107

[shivanirmishra]

Thanks @janani66,
After pulling the above changes also, we are seeing the same issue.

The steps we tried are:

1) dotnet build
2) MONO_ENV_OPTIONS="--jitmap --jitdump" perf record -k 1 dotnet run
3) perf inject --jit -i perf.data -o perf.jit.data
4) perf report -i perf.jit.data

Error:
objdump: /tmp/perf-886285.map: File format not recognized

[brianrob]

The objdump error is likely a red herring. The steps that you tried look right to me. If you open the perf-%pid.map file in a text editor does it look right? You should see lines of the format:

<start addr> <size> <symbol>

[shivanirmishra]

Yes, the .map file looks correct.

7fff882e0000 344 generic_trampoline_jit
7fff882e0388 344 generic_trampoline_jump
7fff882e0710 340 generic_trampoline_rgctx_lazy_fetch
7fff882e0a98 344 generic_trampoline_aot
7fff882e0e20 344 generic_trampoline_aot_plt
7fff882e11a8 344 generic_trampoline_delegate
7fff882e1530 344 generic_trampoline_vcall
7fff882e18b8 a8 restore_context
7fff882e1968 1f0 call_filter
7fff882e1b60 d8 throw_exception
7fff882e1c40 d8 rethrow_exception
7fff882e1d20 d8 rethrow_preserve_exception
7fff882e1e00 108 throw_corlib_exception
7fff88101ef0 90 System.OutOfMemoryException:.ctor (string)

It's creating multiple sets of .map and .dump files in the /tmp/ folder for one particular workload.

Perf tool is able to read .map files to do symbol resolution and show the data. And due to this, we are able to see the functions in the profile but not the annotations inside that which requires a .dump file.

[brianrob]

Ok, so that's good. In that case, the issue is that there is something about the jitdump that isn't working. If you update MONO_ENV_OPTIONS to just --jitdump does that change anything? The perfmap file shouldn't be necessary if the jitdump file is present.

[anjiltech]

Hi @brianrob ,

When we try perf record dotnet run without using the --jitmap with MONO_ENV_OPTIONS, (post inject) the symbols are not resolved, and while trying to annotate it complains as (though we use --jitdump with MONO_ENV_OPTIONS)

"bjdump: '/tmp/perf-220107.map': No such file"

Error:──────────────────────────────┐ ▒
│Couldn't annotate 00007fff977b6b80: │ ▒
│Internal error: Invalid -1 error code│ ▒
│ │ ▒
│ │ ▒
│Press any key...

[brianrob]

Ok, so it sounds like there is something in the integration between the mono runtime and the perf. @lambdageek, @fanyang-mono, would you be able to look into this?

[lambdageek]

Is this a dupe of dotnet/runtime#82520

Could you try with .NET 8 Preview 4, that should have the fix.

[shivanirmishra]

@lambdageek
We tried with .NET8 Preview 4, but getting the below issue after using perf inject:

Also when we are doing Strace on the same, we can see the MMAP calls there:

So, what can cause this wrong jitdump version 2, expected 1 issue in perf inject?