NVIDIA Nsight Systems and CUDA.@profile error

[guyvdbroeck]

Running the Nsight Systems example from the docs, an error is thrown and no .qdrep file is written to disk:

$ nsys launch julia
Warning: LBR backtrace method is not supported on this platform. DWARF backtrace method will be used.
               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.7.0 (2021-11-30)
 _/ |\__'_|_|_|\__'_|  |  Official https://julialang.org/ release
|__/                   |

julia> using CUDA

julia> a = CUDA.rand(1024,1024,512);

julia> sin.(a);

julia> CUDA.@profile sin.(a);
[ Info: Running under Nsight Systems, CUDA.@profile will automatically start the profiler

WARNING: CUDA tracing is required for cudaProfilerStart/Stop API support. Turning it on by default.
Capture range started in the application.
Generating '/tmp/nsys-report-a882.qdstrm'
Capture range ended in the application.
[1/1] [========================100%] report1.nsys-rep
Generated:
    /tmp/report1.nsys-rep
The request is not applicable in this state: StopRequest.
ERROR: failed process: Process(`/usr/local/bin/nsys stop`, ProcessExited(1)) [1]

Stacktrace:
 [1] pipeline_error
   @ ./process.jl:531 [inlined]
 [2] run(::Cmd; wait::Bool)
   @ Base ./process.jl:446
 [3] run
   @ ./process.jl:444 [inlined]
 [4] stop()
   @ CUDA.Profile ~/.julia/packages/CUDA/YpW0k/lib/cudadrv/profile.jl:90
 [5] top-level scope
   @ ~/.julia/packages/CUDA/YpW0k/lib/cudadrv/profile.jl:18
 [6] top-level scope
   @ ~/.julia/packages/CUDA/YpW0k/src/initialization.jl:52

This is using

$ /usr/local/bin/nsys --version
NVIDIA Nsight Systems version 2021.5.1.118-f89f9cd

[maleadt]

Try running with --trace=cuda; this looks like a bug in Nsight Systems.

[maleadt]

e65df15 and 20452a3

[guyvdbroeck]

Try running with --trace=cuda; this looks like a bug in Nsight Systems.

Doesn't seem to fix it for me?

$ nsys launch --trace=cuda julia
Warning: LBR backtrace method is not supported on this platform. DWARF backtrace method will be used.
               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.7.0 (2021-11-30)
 _/ |\__'_|_|_|\__'_|  |  Official https://julialang.org/ release
|__/                   |

julia> using CUDA

julia> a = CUDA.rand(1024,1024,512);

julia> sin.(a);

julia> CUDA.@profile sin.(a);
[ Info: Running under Nsight Systems, CUDA.@profile will automatically start the profiler

WARNING: CUDA tracing is required for cudaProfilerStart/Stop API support. Turning it on by default.
Capture range started in the application.
Generating '/tmp/nsys-report-af18.qdstrm'
Capture range ended in the application.
[1/1] [========================100%] report2.nsys-rep
Generated:
    /home/guy/report2.nsys-rep
The request is not applicable in this state: StopRequest.
ERROR: failed process: Process(`/usr/local/bin/nsys stop`, ProcessExited(1)) [1]

Stacktrace:
 [1] pipeline_error
   @ ./process.jl:531 [inlined]
 [2] run(::Cmd; wait::Bool)
   @ Base ./process.jl:446
 [3] run
   @ ./process.jl:444 [inlined]
 [4] stop()
   @ CUDA.Profile ~/.julia/packages/CUDA/YpW0k/lib/cudadrv/profile.jl:90
 [5] top-level scope
   @ ~/.julia/packages/CUDA/YpW0k/lib/cudadrv/profile.jl:18
 [6] top-level scope
   @ ~/.julia/packages/CUDA/YpW0k/src/initialization.jl:52

[maleadt]

Ah sorry, I thought this was the issue reported on Slack where no kernel launches were detected. This is different; nsys stop seems to fail.

[maleadt]

09ec093