Files exported by `profiler.export_stacks()` are not compatible with flamegraph

[samuela]

🐛 Describe the bug

I'm attempting to profile a bit of PyTorch code, and the stack traces as exported by profiler.export_stacks() are not accepted by flamegraph.pl. In particular I'm getting errors of the form

[nix-shell:~/dev/cifar10-fast]$ flamegraph.pl --title "pytorch single epoch" --countname "us." profiling/stacks_5.txt
ERROR: No stack counts found
<?xml version="1.0" standalone="no"?>
<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN" "http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd">
<svg version="1.1" width="1200" height="60" onload="init(evt)" viewBox="0 0 1200 60" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink">
<!-- Flame graph stack visualization. See https://github.com/brendangregg/FlameGraph for latest version, and http://www.brendangregg.com/flamegraphs.html for examples. -->
<!-- NOTES:  -->
<text  x="600.00" y="24" >ERROR: No valid input provided to flamegraph.pl.</text>
</svg>

You can see the operative bit of profiling code here:

https://github.com/samuela/cifar10-fast/blob/a660ec033899d10820a1e57b480f949d6033716e/dawn.py#L86-L109

For convenience I have also checked in the stack trace output files here.

To reproduce,

• Checkout https://github.com/samuela/cifar10-fast at samuela/cifar10-fast@a660ec0.
• Run nix-shell.
• Run nixGLNvidia-510.47.03 python dawn.py.

Versions

[nix-shell:~/dev/cifar10-fast]$ python collect_env.py 
PyTorch version: 1.10.0+cu113
Is debug build: False
CUDA used to build PyTorch: 11.3
ROCM used to build PyTorch: N/A

OS: Ubuntu 20.04.4 LTS (x86_64)
GCC version: (GCC) 10.3.0
Clang version: Could not collect
CMake version: Could not collect
Libc version: glibc-2.33

Python version: 3.9.10 (main, Jan 13 2022, 23:32:03)  [GCC 10.3.0] (64-bit runtime)
Python platform: Linux-5.11.0-1028-aws-x86_64-with-glibc2.33
Is CUDA available: False
CUDA runtime version: Could not collect
GPU models and configuration: GPU 0: Tesla V100-SXM2-16GB
Nvidia driver version: 510.47.03
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] Could not collect
[conda] Could not collect

cc @ilia-cher @robieta @chaekit @gdankel @bitfort @ngimel @orionr @nbcsm @guotuofeng @guyang3532 @gaoteng-git

[samuela]

For context, see jax-ml/jax#9669.

[samuela]

I'm attempting to follow the examples provided in the official docs here.

[aryam1]

Just wanted to check if you managed to solve this, coming across the same issue myself. Is there a workaround to convert the PyTorch output to a format suitable for FlameGraph?

[samuela]

no solution afaik

[aaronenyeshi]

@ilia-cher, the original author of the export_stacks() API. I haven't seen anyone use export_stacks() in the past 2-3 years. Wondering if there are any users? Users are welcome to open PRs to fix this issue. Otherwise, would deprecating this API make sense.
cc @malfet, @guotuofeng, @albanD

[albanD]

I would have pinged you asking the same haha
So if this is not working anymore, the profiler team doesn't plan on fixing it and expects users to use other tools, then yes deprecating and removal sounds good.

[aaronenyeshi]

@samuela, we've deprecated the document note that uses framegraphs for export_stacks. We are no longer supporting this feature, if the feature is still required, please feel free to open a PR to fix the issue. Thank you.
Ref: #123102