bazel coverage doesn't produce coverage data for cuda config for XLA.

[hmonishN]

Description of the bug:

bazel coverage doesn't seem to produce coverage data for cuda configuration for XLA tests.

gives output:
WARNING: There was no coverage found
Also, the coverage report is an empty file.

Which category does this issue belong to?

No response

What's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

Steps for cuda config :
(Can either use jax latest container or make sure cuda tk 12.3 is installed with cudnn 8.7.6 or higher is insatlled)
git clone <xla_repo_link>
cd xla
export TF_NEED_CUDA=1
export TF_CUDA_COMPUTE_CAPABILITIES="compute_80"
bazel coverage --config cuda --instrument_test_targets --experimental_cc_coverage --combined_report=lcov //xla/service/gpu:command_buffer_scheduling_test

Which operating system are you running Bazel on?

Ubuntu 22.04.4 LTS

What is the output of bazel info release?

release 6.1.2

If bazel info release returns development version or (@non-git), tell us how you built Bazel.

No response

What's the output of git remote get-url origin; git rev-parse HEAD ?

No response

If this is a regression, please try to identify the Bazel commit where the bug was introduced with bazelisk --bisect.

No response

Have you found anything relevant by searching the web?

No response

Any other information, logs, or outputs that you want to share?

No response

[hmonishN]

CC @dariuskia

[sgowroji]

Hi @hmonishN, Could you please provide sample code to reproduce this issue with complete steps?

[hmonishN]

Here are the steps to reproduce the issue on Ubuntu 22.04.4 LTS nvidia gpu (A100 or newer version) machine:
sudo docker run --gpus all --security-opt seccomp=unconfined ghcr.io/nvidia/jax:latest
cd /opt/xla
export TF_NEED_CUDA=1
export TF_CUDA_COMPUTE_CAPABILITIES="compute_80"
/opt/jax/build/bazel-6.1.2-linux-x86_64 coverage --config cuda --instrument_test_targets --experimental_cc_coverage --combined_report=lcov //xla/service/gpu:command_buffer_scheduling_test

[hmonishN]

The JAX container is an opensource container which comes with xla and cuda libs and toolkit installed in it with bazel 6.12 linux version. The definition of the xla test is defined here: https://github.com/openxla/xla/blob/main/xla/service/gpu/BUILD#L3261

[hmonishN]

Hi @sgowroji, any update on this? Is there a workaround that we can implement to make it work?

[comius]

cc @c-mita

[c-mita]

This doesn't strike me as a Bazel issue; instead it looks like an issue with the XLA/CUDA configuration or toolchains being used. Perhaps you should report an issue there.

If there's an issue with Bazel preventing them from supporting coverage then I could take a look, but I'd need more details.