pytorch.profiler cannot profile aten:mm on GPU

[scse-l]

I use pytorch.profiler to profile a program of matmul on GPU, it seems profiler does not record aten.mm correctly. There is stats in GPU kernel View,

but no GPU kernel stats in Trace view.

Sample code:

import torch
a = torch.rand([1, 1024, 2048], device='cuda')
b = torch.rand([2048, 2048], device='cuda')
with torch.profiler.profile(
    activities=[
        torch.profiler.ProfilerActivity.CPU,
        torch.profiler.ProfilerActivity.CUDA,
    ],
    on_trace_ready=torch.profiler.tensorboard_trace_handler("./mm-profile")
):
    torch.matmul(a, b)

[scse-l]

Overall view

[scse-l]

Run torch.matmul for 100 times, there will be stats in Trace view.
It seems that there is a lot of warm-up work for aten:mm's first run. What does cudaFreee do?

[aaronenyeshi]

Hi, you can try turning on with_stack=True. You may get a little more detail on why its in cudaFree.

[scse-l]

Hi, you can try turning on with_stack=True. You may get a little more detail on why its in cudaFree.

Sure, I'll try it. I appreciate your guidance.