Fix driver version to install CUDA on ubuntu 18.04 #1599

hackerliang · 2020-06-12T19:39:53Z

The issue seems to appear while running line 102 of this doc to install CUDA on Ubuntu 18.04. When I run the following script as it's written in the documentation, I ran to an error.

sudo apt-get install --no-install-recommends cuda-10-1 libcudnn7=7.6.4.38-1+cuda10.1 libcudnn7-dev=7.6.4.38-1+cuda10.1

The error is as follows:

cuda-10-1 : Depends: cuda-runtime-10-1 (>= 10.1.243) but it is not going to be installed
            Depends: cuda-demo-suite-10-1 (>= 10.1.243) but it is not going to be installed
E: Unable to correct problems, you have held broken packages.

I finally figured out why this happened:

When you install nvidia-driver-430, apt actually installed nvidia-driver-440
After that, when you install development and runtime libraries,
sudo apt-get install --no-install-recommends cuda-10-1 libcudnn7=7.6.4.38-1+cuda10.1 libcudnn7-dev=7.6.4.38-1+cuda10.1
This will get the error that I elaborated above, cuda-10-1 dependencies cannot be installed.

That's because, NVIDIA has updated their package to cuda-drivers=450.36.06-1, which is compatible with CUDA 11, but not CUDA 10.
When you install cuda-10-1, you actually installing cuda-drivers=450.36.06-1, but this will make the whole system upgraded to nvidia-driver-450 with CUDA 11.

In conclude:
Update command from nvidia-driver-430 to nvidia-driver-440, because when you install 430, you actually installing 440.
Limit cuda-drivers version to 440.64.00-1, which means cuda-drivers=440.64.00-1

The issue seems to appear while running line 102 of this doc to install CUDA on Ubuntu 18.04. When I run the following script as it's written in the documentation, I ran to an error. <code class="devsite-terminal">sudo apt-get install --no-install-recommends \ cuda-10-1 \ libcudnn7=7.6.4.38-1+cuda10.1 \ libcudnn7-dev=7.6.4.38-1+cuda10.1 </code> The error is as follows: ``` cuda-10-1 : Depends: cuda-runtime-10-1 (>= 10.1.243) but it is not going to be installed Depends: cuda-demo-suite-10-1 (>= 10.1.243) but it is not going to be installed E: Unable to correct problems, you have held broken packages. ``` I finally figured out why this happened: When you install nvidia-driver-430, apt actually installed nvidia-driver-440 After that, when you install development and runtime libraries, <code class="devsite-terminal">sudo apt-get install --no-install-recommends \ cuda-10-1 \ libcudnn7=7.6.4.38-1+cuda10.1 \ libcudnn7-dev=7.6.4.38-1+cuda10.1 </code> This will get the error that I elaborated above, cuda-10-1 dependencies cannot be installed. That's because, NVIDIA has updated their package to cuda-drivers=450.36.06-1, which is compatible with CUDA 11, but not CUDA 10. When you install cuda-10-1, you actually installing cuda-drivers=450.36.06-1, but this will make the whole system upgraded to nvidia-driver-450 with CUDA 11. In conclude: Update command from nvidia-driver-430 to nvidia-driver-440, because when you install 430, you actually installing 440. Limit cuda-drivers version to 440.64.00-1, which means cuda-drivers=440.64.00-1

googlebot · 2020-06-12T19:39:58Z

Thanks for your pull request. It looks like this may be your first contribution to a Google open source project (if not, look below for help). Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

📝 Please visit https://cla.developers.google.com/ to sign.

Once you've signed (or fixed any issues), please reply here with @googlebot I signed it! and we'll verify it.

What to do if you already signed the CLA

Individual signers

It's possible we don't have your GitHub username or you're using a different email address on your commit. Check your existing CLA data and verify that your email is set on your git commits.

Corporate signers

Your company has a Point of Contact who decides which employees are authorized to participate. Ask your POC to be added to the group of authorized contributors. If you don't know who your Point of Contact is, direct the Google project maintainer to go/cla#troubleshoot (Public version).
The email used to register you as an authorized contributor must be the email used for the Git commit. Check your existing CLA data and verify that your email is set on your git commits.
The email used to register you as an authorized contributor must also be attached to your GitHub account.

ℹ️ Googlers: Go here for more info.

hackerliang · 2020-06-12T19:43:18Z

@googlebot I signed it!

googlebot · 2020-06-12T19:43:21Z

CLAs look good, thanks!

ℹ️ Googlers: Go here for more info.

hackerliang · 2020-06-12T19:49:25Z

Install nvidia-driver-430, actually installing nvidia-driver-440

Dependency error

When I limit cuda-drivers version, error fix

lamberta · 2020-06-12T19:58:23Z

Nice work figuring this out—appreciate it!

gunan · 2020-06-12T20:02:59Z

site/en/install/gpu.md

 # Reboot. Check that GPUs are visible using the command: nvidia-smi

 # Install development and runtime libraries (~4GB)
 <code class="devsite-terminal">sudo apt-get install --no-install-recommends \
    cuda-10-1 \
    libcudnn7=7.6.4.38-1+cuda10.1  \
-    libcudnn7-dev=7.6.4.38-1+cuda10.1
+    libcudnn7-dev=7.6.4.38-1+cuda10.1 \
+    cuda-drivers=440.64.00-1


The hardcoded versions look brittle. @nluehr is there a way where we can avoid hardcoding the versions?

The following commands are feasible, but there may be a better solution?

sudo apt-get install --no-install-recommends \ cuda-10-1 \ libcudnn7=7.6.4.38-1+cuda10.1 \ libcudnn7-dev=7.6.4.38-1+cuda10.1 \ cuda-drivers=440\*

This reverts commit 34b9d3b.

This reverts commit bb61445.

skq-5233 · 2020-06-18T02:13:38Z

Hello! When I run the sh.compile.sh command, it compiles successfully, but when I run train.py, I get an error: undefined symbol:_ZTIN10tensorflow8opkernelE. I checked a lot of information on the Internet and did not solve it. Hope you help me see what is the problem. Thanks again! ! !
ubuntu :16.04 ; nvidia-driver-440;cuda:10.0; cudnn:7.5.0; tensorflow-gpu:1.13.1; gcc/g++:5.4.0;
@
@hackerliang

hackerliang · 2020-06-18T15:48:32Z

Hello! When I run the sh.compile.sh command, it compiles successfully, but when I run train.py, I get an error: undefined symbol:_ZTIN10tensorflow8opkernelE. I checked a lot of information on the Internet and did not solve it. Hope you help me see what is the problem. Thanks again! ! !
ubuntu :16.04 ; nvidia-driver-440;cuda:10.0; cudnn:7.5.0; tensorflow-gpu:1.13.1; gcc/g++:5.4.0;
@
@hackerliang

Maybe you just install the compiled version rather than compiling by yourself

conda install tensorflow==1.13.1 tensorflow-gpu==1.13.1

skq-5233 · 2020-06-19T01:07:14Z

Thank you for your reply. The compilation problem has been solved, but a new problem has occurred, and it has no effect according to the online method. ValueError: Variable degridding/conv1/weights already exists, disallowed. Did you mean to set reuse=True or reuse=tf.AUTO_REUSE in VarScope? Originally defined at:

…

------------------ 原始邮件 ------------------ 发件人: "hackerliang"<notifications@github.com>; 发送时间: 2020年6月18日(星期四) 晚上11:48 收件人: "tensorflow/docs"<docs@noreply.github.com>; 抄送: "Dandelion's Fled"<794919561@qq.com>;"Comment"<comment@noreply.github.com>; 主题: Re: [tensorflow/docs] Fix driver version to install CUDA on ubuntu 18.04 (#1599) Hello! When I run the sh.compile.sh command, it compiles successfully, but when I run train.py, I get an error: undefined symbol:_ZTIN10tensorflow8opkernelE. I checked a lot of information on the Internet and did not solve it. Hope you help me see what is the problem. Thanks again! ! ! ubuntu :16.04 ; nvidia-driver-440;cuda:10.0; cudnn:7.5.0; tensorflow-gpu:1.13.1; gcc/g++:5.4.0; @ @hackerliang Maybe you just install the compiled version rather than compiling by yourself conda install tensorflow==1.13.1 tensorflow-gpu==1.13.1 — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or unsubscribe.

skq-5233 · 2020-06-19T09:05:13Z

Hello, I changed the version of tf==1.10.0, the compilation process is ok, but the error is still reported when running train.py

…

------------------ 原始邮件 ------------------ 发件人: "hackerliang"<notifications@github.com>; 发送时间: 2020年6月18日(星期四) 晚上11:48 收件人: "tensorflow/docs"<docs@noreply.github.com>; 抄送: "Dandelion's Fled"<794919561@qq.com>;"Comment"<comment@noreply.github.com>; 主题: Re: [tensorflow/docs] Fix driver version to install CUDA on ubuntu 18.04 (#1599) Hello! When I run the sh.compile.sh command, it compiles successfully, but when I run train.py, I get an error: undefined symbol:_ZTIN10tensorflow8opkernelE. I checked a lot of information on the Internet and did not solve it. Hope you help me see what is the problem. Thanks again! ! ! ubuntu :16.04 ; nvidia-driver-440;cuda:10.0; cudnn:7.5.0; tensorflow-gpu:1.13.1; gcc/g++:5.4.0; @ @hackerliang Maybe you just install the compiled version rather than compiling by yourself conda install tensorflow==1.13.1 tensorflow-gpu==1.13.1 — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or unsubscribe.

nluehr · 2020-07-08T19:31:19Z

Updating the drivers to the newly released r450 as in PR 1616 should fix this issue without needing to explicitly install cuda-drivers or pin its version.

hackerliang · 2020-07-08T19:58:38Z

@nluehr
The problem is that we don't know if Tensorflow can works with CUDA 11 well. NVIDIA Driver 450 is bound with CUDA 11 and this PR is to prevent the whole system from upgrading to NVIDIA Driver 450 with CUDA 11.
If you have tested that all functionality of TensorFlow can work normally with the CUDA 11, then it can be upgraded to CUDA 11.

yashk2810 · 2020-07-08T20:00:14Z

TF stable won't work with Cuda 11

nluehr · 2020-07-08T20:09:35Z

Nvidia drivers are not bound to specific CUDA toolkit versions, but are backward compatible with older toolkits. So the r450 driver supports CUDA 10.1 and will work with stable TF releases.

hackerliang · 2020-07-08T20:12:28Z

Nvidia drivers are not bound to specific CUDA toolkit versions, but are backward compatible with older toolkits. So the r450 driver supports CUDA 10.1 and will work with stable TF releases.

sudo apt install nvidia-driver-450

This command is actually installing NV Driver 450 with CUDA 11.

hackerliang · 2020-07-08T20:12:44Z

TF stable won't work with Cuda 11

So what we need to do is fix this thing ASAP in the following two ways:

Prevent the APT package manager from installing NV Driver 450 with CUDA 11(this PR).
Upgrade TensorFlow stable to make it works with CUDA 11.

The installation guide for the official website is outdated.

gunan · 2020-07-08T20:15:01Z

Unfortunately, upgrading tf stable to cuda 11 will not happen until 2.4, which will be cut in september.
Cuda 11 support did not make it into 2.3 release cut.

nluehr · 2020-07-08T20:33:44Z

@hackerliang, sudo apt install nvidia-driver-450 does not install the CUDA 11 toolkit, just the driver. The 10.1 toolkit is installed by sudo apt-get install --no-install-recommends cuda-10-1.

hackerliang · 2020-07-08T21:18:45Z

@nluehr
You can try to install nvidia-driver-450 then install CUDA 10.
You will find that there is a dependency error and you can not install CUDA 10 but only CUDA 11.

yashk2810 · 2020-07-08T21:22:37Z

There is a way to install driver 450 and cuda 10.1.

Use the .run files that nvidia provides which give you an option to only install the driver or the toolkit. So you can install 450 driver from the cuda 11 toolkit(https://developer.nvidia.com/cuda-downloads?target_os=Linux&target_arch=x86_64&target_distro=Ubuntu&target_version=1804&target_type=runfilelocal) and then install cuda 10.1 (but not the driver) from cuda 10.1 toolkit.

This works since I did this yesterday. But I don't think this can be added to the install instructions since its not straightforward.

nluehr · 2020-07-08T21:57:30Z

@hackerliang That issue should be fixed in the newly released 450.51.05 driver.
@yashk2810 The 450.51.05 driver is available in the same nvidia developer debian repo already used in the current instructions. So the install instructions should work by just replacing the nvidia-driver-430 with nvidia-driver-450 like here.

yashk2810 · 2020-07-08T21:59:04Z

When I tried that install command for 450, it gave me an error.

Can you paste your results from your try?

nluehr · 2020-07-09T15:37:43Z

Following the instructions (and installing the 450 driver) on a clean system I see no errors. If I first install the 430 driver, I see the following error when attempting to install the 10.1 toolkit (same as reported at the top of this issue thread).

apt-get install --no-install-recommends cuda-10-1 \
                                        libcudnn7=7.6.4.38-1+cuda10.1 \
                                        libcudnn7-dev=7.6.4.38-1+cuda10.1

Some packages could not be installed. This may mean that you have
requested an impossible situation or if you are using the unstable
distribution that some required packages have not yet been created
or been moved out of Incoming.
The following information may help to resolve the situation:

The following packages have unmet dependencies:
 cuda-10-1 : Depends: cuda-runtime-10-1 (>= 10.1.243) but it is not going to be installed
             Depends: cuda-demo-suite-10-1 (>= 10.1.243) but it is not going to be installed
E: Unable to correct problems, you have held broken packages.

Now if I attempt to install the 450 driver over the top of the installed 430 driver I get a similar error.

apt-get install --no-install-recommends nvidia-driver-450

Some packages could not be installed. This may mean that you have
requested an impossible situation or if you are using the unstable
distribution that some required packages have not yet been created
or been moved out of Incoming.
The following information may help to resolve the situation:

The following packages have unmet dependencies:
 nvidia-driver-450 : Depends: nvidia-dkms-450 (= 450.51.05-0ubuntu1) but it is not going to be installed
                     Depends: nvidia-kernel-source-450 (= 450.51.05-0ubuntu1) but it is not going to be installed
                     Depends: libnvidia-extra-450 (= 450.51.05-0ubuntu1) but it is not going to be installed
                     Depends: nvidia-compute-utils-450 (= 450.51.05-0ubuntu1) but it is not going to be installed
                     Depends: nvidia-utils-450 (= 450.51.05-0ubuntu1) but it is not going to be installed
                     Depends: libnvidia-ifr1-450 (= 450.51.05-0ubuntu1) but it is not going to be installed
                     Depends: libnvidia-fbc1-450 (= 450.51.05-0ubuntu1) but it is not going to be installed
E: Unable to correct problems, you have held broken packages.

This error is the result of the broken 430 driver package. Removing it first with apt-get remove nvidia-driver-430 and then installing the 450 driver succeeds. And with the 450 driver, the cuda 10.1 install also succeeds.

What error are you seeing @hackerliang?

hackerliang · 2020-07-12T11:38:51Z

@nluehr
This thing happened a month ago, and I don't remember the details anymore and I don't have a spare machine now, maybe someone else can help with this test. However, I found a table from NVIDIA's official website. I don't really know what N/A means.

https://docs.nvidia.com/deploy/cuda-compatibility/index.html#cuda-application-compatibility

nluehr · 2020-07-13T17:08:48Z

@hackerliang the table linked refers to the 'CUDA compatibility platform' which is a Tesla-only feature that provides limited FORWARD compatibility (using older driver with newer toolkits).

Using older toolkits with newer drivers does not depend on that feature and is documented in section 2.2 of the same doc.

yashk2810 · 2020-07-17T17:14:38Z

The driver was update to 450. Is this PR still valid?

hackerliang requested review from lamberta, MarkDaoust and yashk2810 as code owners June 12, 2020 19:39

googlebot added the cla: no CLA has not been signed label Jun 12, 2020

googlebot added cla: yes CLA has been signed and removed cla: no CLA has not been signed labels Jun 12, 2020

lamberta requested review from sanjoy and gunan June 12, 2020 19:57

gunan reviewed Jun 12, 2020

View reviewed changes

hackerliang and others added 3 commits June 13, 2020 04:06

Fix hardcoded problem

34b9d3b

Revert "Fix hardcoded problem"

bb61445

This reverts commit 34b9d3b.

Revert "Revert "Fix hardcoded problem""

4644f7c

This reverts commit bb61445.

hackerliang requested a review from gunan July 8, 2020 19:58

hackerliang closed this Jul 18, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix driver version to install CUDA on ubuntu 18.04 #1599

Fix driver version to install CUDA on ubuntu 18.04 #1599

hackerliang commented Jun 12, 2020

googlebot commented Jun 12, 2020

hackerliang commented Jun 12, 2020

googlebot commented Jun 12, 2020

hackerliang commented Jun 12, 2020

lamberta commented Jun 12, 2020

gunan Jun 12, 2020

hackerliang Jun 12, 2020

skq-5233 commented Jun 18, 2020

hackerliang commented Jun 18, 2020

skq-5233 commented Jun 19, 2020 via email

skq-5233 commented Jun 19, 2020 via email

nluehr commented Jul 8, 2020

hackerliang commented Jul 8, 2020

yashk2810 commented Jul 8, 2020

nluehr commented Jul 8, 2020

hackerliang commented Jul 8, 2020

hackerliang commented Jul 8, 2020

gunan commented Jul 8, 2020

nluehr commented Jul 8, 2020

hackerliang commented Jul 8, 2020

yashk2810 commented Jul 8, 2020

nluehr commented Jul 8, 2020

yashk2810 commented Jul 8, 2020 •

edited

nluehr commented Jul 9, 2020

hackerliang commented Jul 12, 2020

nluehr commented Jul 13, 2020

yashk2810 commented Jul 17, 2020

Fix driver version to install CUDA on ubuntu 18.04 #1599

Fix driver version to install CUDA on ubuntu 18.04 #1599

Conversation

hackerliang commented Jun 12, 2020

googlebot commented Jun 12, 2020

What to do if you already signed the CLA

Individual signers

Corporate signers

hackerliang commented Jun 12, 2020

googlebot commented Jun 12, 2020

hackerliang commented Jun 12, 2020

lamberta commented Jun 12, 2020

gunan Jun 12, 2020

Choose a reason for hiding this comment

hackerliang Jun 12, 2020

Choose a reason for hiding this comment

skq-5233 commented Jun 18, 2020

hackerliang commented Jun 18, 2020

skq-5233 commented Jun 19, 2020 via email

skq-5233 commented Jun 19, 2020 via email

nluehr commented Jul 8, 2020

hackerliang commented Jul 8, 2020

yashk2810 commented Jul 8, 2020

nluehr commented Jul 8, 2020

hackerliang commented Jul 8, 2020

hackerliang commented Jul 8, 2020

gunan commented Jul 8, 2020

nluehr commented Jul 8, 2020

hackerliang commented Jul 8, 2020

yashk2810 commented Jul 8, 2020

nluehr commented Jul 8, 2020

yashk2810 commented Jul 8, 2020 • edited

nluehr commented Jul 9, 2020

hackerliang commented Jul 12, 2020

nluehr commented Jul 13, 2020

yashk2810 commented Jul 17, 2020

yashk2810 commented Jul 8, 2020 •

edited