GPU profiling no longer works in recent CodeXL releases

[csbnw]

I am not able to (remotely) profile my OpenCL applications using recent CodeXL releases. The last release that worked for me was 1.7.

My setup is as follows:

• Client running Fedora 26 (previously tested on Ubuntu 16.* and Fedora 25)
• Servers running CentOS 7.3 (previously tested on Ubuntu 14.04 and Centos 7.1/7.2)
• Installed same CodeXL versions on all machines
• Setup ssh-tunnel for connection between client and server: ssh -t -X -L 27015:server:27015 server /path/to/codexl/CodeXLRemoteAgent
• Run CodeXL locally on client (using provided CodeXL script)

Testing the connection works, I get the following popup:
Connection was established successfuly.

But starting the GPU Profile doesn't:
Communication failure: Please verify that both client and remote machines are connected to the network.

The CodeXLRemoteAgent reports the following:

********************************************************
CodeXL Remote Agent settings were extracted successfully
--------------------------------------------------------
Agent version is: 2.4.58.0
Read  timeout is: Infinite Timeout
Write timeout is: Infinite Timeout
Agent IP:Port is: 10.141.0.2:27015
********************************************************

CodeXL Remote Agent is listening on <10.141.0.2:27015>

Press any key to terminate CodeXL Remote Agent. . .


*** Connection: Received a connection from <10.141.0.2:63453> ***

*** Connection: Received a connection from <10.141.0.2:63965> ***

*** Disconnection: <10.141.0.2:63965>ended a session and disconnected ***

*** Connection: Received a connection from <10.141.0.2:64477> ***

*** Disconnection: <10.141.0.2:64477>ended a session and disconnected ***

*** Connection: Received a connection from <10.141.0.2:64989> ***

Which seems to suggest that the connection is fine.

Furthermore, when running CodeXL directly on the server, using x-forwarding, the GPU Profile button is gray and profiling is therefore not available either.

I have been able to reproduce this issue using various GPUs (both consumer and workstation/server models), architectures (Tahiti, Hawaii, Fiji, Vega) and drivers (14.301, 15.302, catalyst-15.9, radeon-crimson-15.12, amdgpu-pro-17.*), but I haven't found a combination that works. I also tried compiling CodeXL from source, but this did not resolve the issues either.

Any suggestions?

[chesik-amd]

Hi Bram,

Are you able to run the command line profiler on the server? In recent CodeXL versions, the command line profiler executable is named "rcprof".

To collect an application trace, use "rcprof --apitrace AppToProfile".
To collect performance counters, use "rcprof --perfcounter AppToProfile"

Thanks,
Chris

[csbnw]

Hi Chris,

Yes, I am able to run rcprof on the command line. It produces apitrace.atp and Session1.csv in my home directory, respectively.

These files seem to contain all the profiling data that I am after but are much harder to interpret without CodeXL's visualizations.

update:
I noticed that I used CodeXL 2.4.58 before. I tried again (both local and remote) using 2.4.60, but nothing changes.

[ekondis]

The atp file can be easily imported to CodeXL running on a different system.

[csbnw]

I followed these instructions to import the files into CodeXL. It mostly works (including the timeline view), but occupancy information is missing.

CodeXL does not seem to automatically including the .occupancy file during import.

I tried creating it manually by following the instructions from rcprof:

An example to generate an occupancy display page:
rcprof -P "/path/to/session.occupancy" --occupancyindex 2 -o "path/to/output.html"

This does not work for me: (I tried all Thread ID values in the .occupancy file)

Error generating occupancy display file.

This is not a very helpful error message, how to fix this?

Even provided that this workaround using the command line profiler worked, it would be much more convenient if CodeXL would just work (especially remotely), like before in version 1.7.

[chesik-amd]

I've tested several imports of both .atp and .csv files using CodeXL build 2.4.60 and in all cases I see that it picks up the .occupancy file correctly. When importing the .csv, the view shows has a "Kernel Occupancy" column, which indicates the .occupancy file was imported with the .csv file? Do you not see the "Kernel Occupancy" column after importing a .csv file?

Also, the --occupancyindex swtich takes a number which corresponds to the 0-based index of the occupancy line in the .occupancy file. So if you profile and application that dispatches 3 kernels, you can generate an occupancy display file for the second kernel using the following command line:

rcprof -P session.csv --occupancyindex 1 -o sessionoccupancy.html

[csbnw]

I tried again, both methods (generating the .html file and the import into CodeXL) now work fine. I also used CodeXL build 2.4.60, and a Fury X (driver 1800.11).

Your command, however, should be:

rcprof -P session.occupancy --occupancyindex 1 -o sessionoccupancy.html

This is an acceptable workaround, but I would still prefer to have a working remote-profiling setup like before (using CodeXL 1.7).

I am still unable to profile the Vega Frontier (driver 2442.7) using this method, rcprof hangs.

A benchmark indicates that OpenCL works and the GPU is running at full speed:

./clpeak                                                                                                       
Platform: AMD Accelerated Parallel Processing
Device: gfx900
Driver version  : 1.1 (HSA,LC)
Compute units   : 64
Clock frequency : 1600 MHz

Single-precision compute (GFLOPS)
     float:      11.30 ms,   12161.71 GFLOPS
    float2:      11.34 ms,   12118.63 GFLOPS
    float4:      11.14 ms,   12332.36 GFLOPS
    float8:      10.93 ms,   12578.65 GFLOPS
   float16:      11.04 ms,   12447.90 GFLOPS

And with profiling enabled:

rcprof -p -O ./clpeak
Radeon Compute Profiler V5.1.6396 is enabled

strace:

open("./clpeak", O_RDONLY) = 3
lseek(3, 0, SEEK_END)                   = 433656
lseek(3, 0, SEEK_CUR)                   = 433656
lseek(3, 0, SEEK_SET)                   = 0
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\2\0>\0\1\0\0\0P/@\0\0\0\0\0"..., 8191) = 8191
close(3)                                = 0
clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x2aaaaaaeac50) = 36171
wait4(36171, Radeon Compute Profiler V5.1.6396 is enabled

[tangjie77wd]

@bramveenboer @chesik-amd @ekondis
I create projects with OpenCL on linux . My machine : CentOS 7.3+eclipse + AMD Rx560 + amdgpu-pro-17.30-465504 + CodeXL_Linux_x86_64_2.4.60 .

After I installed all tools above , go into the 'CodeXL_Linux_x86_64_2.4.60' folder and open a terminal ,run the command : ./CodeXL ,then CodeXL gui will appear .
Click the 'Help' menu and then click 'Load the Teapot Sample' .
Click the 'Profile' menu and then clike 'GPU:Performance Counters' .
Press 'F5' on your keyboard and wait for a while.
Close the 'AMD Teapot Example' gui and wait for a while , you will see many performance indicators in the CodeXL gui.

But I have a question these days : the Teapot sample is provided by CodeXL not myself (It exists in CodeXL as a 'CodeXL Teapot.cxl' file ,so we can load it directly in CodeXL. ) , how can i use CodeXL to test the performance of my own OpenCL projects (.cpp file and .cl file) written in the eclipse ( no .cxl file )? The file 'CodeXL_Quick_Start_Guide.pdf' does not describe any about this. Any help will be much appriciate !

[csbnw]

@tangjie77wd
Your question is not really related to my issue. I am explicitly referring to features that worked in earlier releases of CodeXL but not in recent ones. You are asking how to perform OpenCL profiling using CodeXL in the first place.

Anyway, @chesik-amd suggested to use command-line profiling:

Are you able to run the command line profiler on the server? In recent CodeXL versions, the command line profiler executable is named "rcprof".
To collect an application trace, use "rcprof --apitrace AppToProfile".
To collect performance counters, use "rcprof --perfcounter AppToProfile"

You can than import the profiling files into CodeXL as described here under "Viewing Profiler Results".

Btw, I am currently evaluating Fedora 26 + Vega Frontier + ROCM drivers/kernel + CodeXL 2.5.25.

I have only tested the API trace so far, but it mostly seems to work. It however also still seems really fragile: I installed the system on Monday, found some upgrades in the repository on Tuedsay and today some packages seem to have been rolled-back again. The Tuesday packages seemed to be broken anyway, missing dependencies etc. But using some tricks (manually install RPMs, symlinks, skip digest, ...), I got it working again.

Given that the ROCm packages are available only for Ubuntu 16.04 and Fedora (and the glibc on CentOS 7.3 is too old), I would suggest to try one of these operating systems.

[tangjie77wd]

@bramveenboer Thank you anyway!

[chesik-amd]

The issue with the remote profiler not working should be fixed in the next version of CodeXL

[chesik-amd]

This should no longer be an issue in CodeXL 2.6