Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

XLA Check Failed: options.is_autotuning_compilation #20024

Closed
sjrothfuss opened this issue Feb 29, 2024 · 1 comment
Closed

XLA Check Failed: options.is_autotuning_compilation #20024

sjrothfuss opened this issue Feb 29, 2024 · 1 comment
Labels
bug Something isn't working

Comments

@sjrothfuss
Copy link

sjrothfuss commented Feb 29, 2024

Description

Issue installing JAX on CentOS 7. I have tried both the cuda11 and cuda12 pip installations of JAX. Looks similar to but distinct from #17730.

Error when using JAX after either installation:

2024-02-27 16:32:33.075782: F external/xla/xla/service/gpu/nvptx_compiler.cc:614] Check failed: options.is_autotuning_compilation 
Aborted (core dumped)

Any help would be much appreciated.

System info (python version, jaxlib version, accelerator, etc.)

Nvidia-smi:
NVIDIA-SMI 530.30.02 Driver Version: 530.30.02 CUDA Version: 12.1

pip list after installing cuda12:
image

pip list after installing cuda11:
image

@sjrothfuss
Copy link
Author

We solved this issue by upgrading CUDA on our machine.

NVIDIA-SMI 550.54.14 Driver Version: 550.54.14 CUDA Version: 12.4

If someone else has the same issue and can't upgrade, I recommend specifying cuDNN and CUDA Toolkit versions before installing, such as by running the following.

conda update -y -n base -c defaults conda
conda create -n JAXENV -y python=3.8
conda activate JAXENV
conda install -y -c nvidia cudnn=8.0.4
conda install -y -c conda-forge cudatoolkit-dev=11.1.1
chmod 755 /miniconda3/envs/JAXENV/pkgs/cuda-toolkit
pip install --upgrade pip
pip install --upgrade jax==0.3.25 jaxlib==0.3.25+cuda11.cudnn805 -f https://storage.googleapis.com/jax-releases/jax_cuda_releases.html

(This code installs CUDA 11 and the compatible JAX because conda-forge does not have toolkit version ≥ 12 at the time of writing.)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

1 participant