[CI] Set PATH during build to include location of sccache wrapped nvcc

[clee2000]

Sccache wasn't working for nvcc on jammy, so manually set the path to include where nvcc is

I had problems with always making nvcc a wrapper in some inductor tests where I got

sccache: encountered fatal error
sccache: error: PCH not supported by nvcc
sccache: caused by: PCH not supported by nvcc

and I also got an error (only on clang) when trying to set CMAKE_CUDA_COMPILER_LAUNCHER to /opt/cache/bin/sccache or sccache

ccache: error: failed to execute compile
    sccache: caused by: Compiler not supported: "nvcc warning : Support for offline compilation for architectures prior to \'<compute/sm/lto>_75\' will be removed in a future release (Use -Wno-deprecated-gpu-targets to suppress warning).\nnvcc fatal   : Failed to preprocess host compiler properties.\n"

Non jammy cuda jobs' docker images used a different dockerfile, which set CMAKE_CUDA_COMPILER_LAUNCHER

pytorch/.ci/docker/ubuntu-cuda/Dockerfile

Line 110 in e895e96

ENV CMAKE_CUDA_COMPILER_LAUNCHER=/opt/cache/bin/sccache

Alt solution:
Given that I only get the error on clang, I could set CMAKE_CUDA_COMPILER_LAUNCHER=sccache only when not using clang

Setting CUDA_NVCC_EXECUTABLE doesn't fail but also doesn't result in cache hits/misses

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/155464

• 📄 Preview Python docs built from this PR
• 📄 Preview C++ docs built from this PR
• ❓ Need help or want to give feedback on the CI? Visit the bot commands wiki or our office hours

Note: Links to docs will display an error until the docs builds have been completed.

⏳ No Failures, 3 Pending

As of commit 9619911 with merge base 73220d5 ():
💚 Looks good so far! There are no failures yet. 💚

UNSTABLE - The following job is marked as unstable, possibly due to flakiness on trunk:

• ⏳ pull / linux-jammy-py3-clang12-executorch / test (executorch, 1, 1, linux.2xlarge, unstable) (gh)

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[malfet]

This is some dark magic, please create an issue and let's try to get to the bottom of it....

I.e. unwrapped nvcc should always be in /usr/local/cuda/bin/nvcc folder, if it's not there, than we are doing something really wrong

[clee2000]

This is some dark magic, please create an issue and let's try to get to the bottom of it....

I.e. unwrapped nvcc should always be in /usr/local/cuda/bin/nvcc folder, if it's not there, than we are doing something really wrong

The unwrapped version is in /usr/local/cuda/bin/nvcc, and the wrapped version is it /opt/cache/lib/nvcc, which is usually not in path, so we end up not using it

[huydhn]

Good catch! It's true that nvcc is at /usr/local/cuda/bin/nvcc and not the sccache-wrapper one. The surprising thing, although, is that cache hit rate looks decent https://github.com/pytorch/pytorch/actions/runs/15562870942/job/43819461677#step:16:472, maybe nvcc stats is not even show there?

[clee2000]

@pytorchbot merge -f "im impatient"

[pytorchmergebot]

Merge started

Your change will be merged immediately since you used the force (-f) flag, bypassing any CI checks (ETA: 1-5 minutes). Please use -f as last resort and instead consider -i/--ignore-current to continue the merge ignoring current failures. This will allow currently pending tests to finish and report signal before the merge.

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here