CUDA 9RC + cuDNN7 #12474
Comments
|
Thanks Toby. Looking forward to the performance numbers on P100. |
|
Quick Question (sorry if it's not the right place to ask, and if I'm missing something really obvious on NVIDIA's Website): Do CUDA 9 & cuDNN 7 provide better performance on non-Volta (Pascal) GPUs? Are these noticeable when training Neural Nets? |
|
In my testing cuDNN 7 (with CUDA 8) it does seem to have some improved
Kernels. cuDNN 6 did not seem to have much of an impact I saw a speed up
in Single GPU performance for VGG16 and ResNet when testing on P100s. I
have not tested on K80s. We are still working through any minor issues
with TensorFlow. The upgrades are not always plug and play. This is a
fine place to ask, I just did not want answer compile questions for
platforms I am not using for this very unofficial patch. :-)
…On Thu, Aug 24, 2017 at 12:43 AM, Tanmay Bakshi ***@***.***> wrote:
Quick Question (sorry if it's not the right place to ask, and if I'm
missing something really obvious on NVIDIA's Website): Do CUDA 9 & cuDNN 7
provide better performance on non-Volta (Pascal) GPUs? Are these noticeable
when training Neural Nets?
—
You are receiving this because you were assigned.
Reply to this email directly, view it on GitHub
<#12474 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AWZesiHeQSoFmVaTYQEy_lHu3vbmUAXDks5sbSmPgaJpZM4O-GMz>
.
|
|
I'm seeing a bit of a different result. With CUDA 8 & cuDNN 6, training a simple but deep Keras CNN model took me 225 seconds, but with CUDA 9 & cuDNN 7, it took 241 seconds (on a P100). |
|
My test was CUDA 8 + cuDNN 7 and CUDA 8 + cuDNN 6. For CUDA 9RC + cuDNN 7, testing not apples to apples as I was using a slightly different versions of TF than I did for CUDA 8 + cuDNN 7 but I think likely perform the same, compared to CUDA 8 + cuDNN 7 Resnet-50 was about the same and I saw a regression with VGG16 at 8xP100 SMX running replicated nccl. With 1 GPU I saw some differences but not large enough to be certain. There are things to sort out |
|
@terrylewis60 It looks like you are subscribed to the issue. If you unsubscribe you will not get notifications. I would also greatly appreciate it if you removed the profanity and childish name calling. You are free to express yourself, although you are likely outside the community guidelines. It just seems rather silly to complain about something you subscribed to. Thank you. |
|
Any updates on this? While the patch works with the specific tf-commit (also with Python3), the tf version says version is 1.2.1-rc1, so rather old (that version was incompatible with current bazel 0.5.3, so bazel needs to be downgraded in order to build the tf-patch version.) The cudnn 7 changes so far are not included in HEAD. Any possibility for a branch that supports the most recent released upstream software/drivers? |
|
The commit point is Aug 1st, which is partially post 1.3. The patch works
with the 1.3 branch/tag as well. The PRs are in progress from NVIDIA.
The eigen PR for CUDA 9RC is in review by the external eigen team.
…On Aug 26, 2017 1:45 AM, "Dominik Schlösser" ***@***.***> wrote:
Any updates on this? While the patch works with the specific tf-commit
(also with Python3), the tf version says *version* is 1.2.1-rc1, so
rather old (that version was incompatible with current bazel 0.5.3, so
bazel needs to be downgraded in order to build the tf-patch version.) The
cudnn 7 changes so far are not included in HEAD. Any possibility for a
branch that supports the most recent released upstream software/drivers?
—
You are receiving this because you were assigned.
Reply to this email directly, view it on GitHub
<#12474 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AWZeslZ2gAsAXT2MmYkdgvLNHyoZoeWYks5sb9sjgaJpZM4O-GMz>
.
|
|
Bazel 5.3 had issues which as silly as it sounds were beyond our control
and immediate knowledge. Our build servers do not run latest bazel. I
also like to stay a step behind. I think I was the first to hit the issue
having accidentally upgraded 1 hour after the release. I wasted half a
day, which meant making up that time into the night. I understand the
frustration.
On Aug 26, 2017 6:22 AM, "Toby Boyd" <tobyboyd@google.com> wrote:
The commit point is Aug 1st, which is partially post 1.3. The patch works
with the 1.3 branch/tag as well. The PRs are in progress from NVIDIA.
The eigen PR for CUDA 9RC is in review by the external eigen team.
…On Aug 26, 2017 1:45 AM, "Dominik Schlösser" ***@***.***> wrote:
Any updates on this? While the patch works with the specific tf-commit
(also with Python3), the tf version says *version* is 1.2.1-rc1, so
rather old (that version was incompatible with current bazel 0.5.3, so
bazel needs to be downgraded in order to build the tf-patch version.) The
cudnn 7 changes so far are not included in HEAD. Any possibility for a
branch that supports the most recent released upstream software/drivers?
—
You are receiving this because you were assigned.
Reply to this email directly, view it on GitHub
<#12474 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AWZeslZ2gAsAXT2MmYkdgvLNHyoZoeWYks5sb9sjgaJpZM4O-GMz>
.
|
|
The patch almost works with r1.3: it fails on tensorflow/workspace.bzl: |
|
I tested the patch on the 1.3 TAG v1.3.0-rc2 because I was doing
performance tests with the various versions. With the env from the
original post. The 1.3 final is not that different, you can look at the
log but I am fairly sure it was a few small cherry picks. I am a bit
surprised it did not apply. You can always apply to the TAG and then
merge. If the merging does not get too messy you could merge all the way
to head. Then make a branch, share it, and bask in some tasty karma. :-)
…On Sat, Aug 26, 2017 at 7:11 AM, Dominik Schlösser ***@***.*** > wrote:
The patch almost works with r1.3: it fails on tensorflow/workspace.bzl:
error: patch failed: tensorflow/workspace.bzl:664
error: tensorflow/workspace.bzl: patch does not apply
[cmake anyone? ;-) ]
—
You are receiving this because you were assigned.
Reply to this email directly, view it on GitHub
<#12474 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AWZeskl8RRNrxl1I77IU4JokByOjQm1lks5scCd7gaJpZM4O-GMz>
.
|
|
For now I patched the patch so that it works with release r1.3: (note: the attached patch-file needs to be renamed, the extension This patch will only work with r1.3. (workspace.bzl changes almost on a daily basis, which breaks the patch). |
|
Big thank you.
…On Aug 26, 2017 9:20 AM, "Dominik Schlösser" ***@***.***> wrote:
For now I patched the patch so that it works with release r1.3:
0002-TF-1.3-CUDA-9.0-and-cuDNN-7.0-support.patch.txt
<https://github.com/tensorflow/tensorflow/files/1253794/0002-TF-1.3-CUDA-9.0-and-cuDNN-7.0-support.patch.txt>
(note: the attached patch-file needs to be renamed, the extension .txt
needs to be removed.)
This patch will only work with r1.3. (workspace.bzl changes almost on a
daily basis, which breaks the patch).
—
You are receiving this because you were assigned.
Reply to this email directly, view it on GitHub
<#12474 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AWZeslTuD_Ccy3os6rQHIYeqRHrzqtr8ks5scEXNgaJpZM4O-GMz>
.
|
|
Could somebody please specify the required bazel version for this patch to work on r1.3? |
|
I've built with tf r1.3, 0002-patch and bazel 0.5.3 (latest bazel release). The original 0001-patch (first post) did not build with bazel 0.5.3, but required a downgrade to 0.5.2. |
|
Thank you @domschl ! |
|
Hi @tfboyd, how does the following command work? Because there is no "external/eigen_archive" folder in tensorflow repo. There is only "third_party/eigen3/Eigen", so this command and your next patch command does not work. |
|
You need to try to build tf once. That pulls the external dependencies like eigen3. |
|
The real PRs are very close and I think some people have used those to get
things working as well. Just an FYI. When they are all approved I will
update and close this issue. Dominik is correct you have to build (then
fail) then do the Eigen step then build again.
…On Wed, Aug 30, 2017 at 12:16 PM, Dominik Schlösser < ***@***.***> wrote:
You need to try to build tf once. That pulls the external dependencies
like eigen3.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#12474 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AWZespskw2egUJi4PHAJkekDNbyPACEbks5sdbT0gaJpZM4O-GMz>
.
|
|
Hi @tfboyd, I tried to use your patches, but after the first build step, there is no "external/eigen_archive" folder at all. It means the command "cd -P bazel-out/../../../external/eigen_archive" failed. So when this folder is generated? |
|
@hfutxrg it must be broken now. It will be in the official build in a few weeks or less. Step #6 that mentions EIGEN is what creates that folder. But if it did not work for you that is ok, this was for fun and it worked for a few people. |
|
Hi @tfboyd, how did you benchmark the performance of P100 and V100? I used the benchmarks mentioned in https://www.tensorflow.org/performance/benchmarks. But it seems the performance results are incorrect. With real imagenet dataset, I got 905 images/sec with 1 P100 and 1234 images/sec with 4 P100. But that benchmark website has 219 images/sec with 1 P100 and 852 images/sec. So there are huge difference here. My command to run the benchmark is: Is there anything wrong with my command? Also that benchmark website has similar performance with synthetic data, but if I remove "--data_dir", the performance is 12866 images/sec which is incorrect. Any comments and suggestions are appreciated! Thanks, |
|
You are running the trivial model not resnet50. |
|
Thanks @tfboyd. That's a stupid mistake. After I changed to the correct option, the benchmark works correctly now. |
|
fp16 is not ready just yet. We still need batch_norm updated to support it and another push from the script. Give me a few more weeks and I should have everything you need. I (and we) very much want to share the code with you as soon as it functions even if it has some issues. It will also be hard to use this patch as it is getting very stale so if all goes well in a few weeks, I hope less, we can have everything (Cuda9RC, cuDNN 7 and fp16 with something to play with) official. Zero worries on the mistake there are a bunch of flags and it is so easy to make a mistaken. |
|
Hi @tfboyd , in the benchmark code, one way for variable update is called "independent". This should never not be used in real training, right? Because in real training, all GPUs have to communicate each other to update the gradient. The page https://www.tensorflow.org/performance/performance_models also didn't mention "independent" variable update. |
|
@hfutxrg Correct independent means graidents are not shared between the towers. Only useful for testing hardware and no a "legit" way to test actual performance. FYI, it looks like cuDNN 7 is mostly merged. I have not tested it myself yet but just FYI. Trying to keep the thread updated as I have time. and CUDA 9 There will be followup PRs and I hope to publish FP16 examples very soon. There is a review still in progress for batch norm. |
|
Will this be 1.4.0? |
|
1.4 binaries will most likely be CUDA 8 + cuDNN 6. I suspect (which is not an estimate) that 1.5 will be built with CUDA 9 + cuDNN 7. I am testing the build now and working through some issues with CUDA 9 and cuDNN 7. Maybe I am doing something wrong, I will find out tomorrow. I really want to have it CUDA 8 and cuDNN 7 building in 1.4 source but if we miss the release train we miss it. There is no date for 1.4 but I suspect it will RC before the end of October. It is being gated on finishing some features but again it most likely will not have prebuilt binaries for CUDA 9 and cuDNN 7. FYI. I will close this issue when it builds from source with no "tricks". :-) |
|
@tfboyd Thanks for your great help for integrating cuda rc 9 + cudnn 7. So, if I have a V100 machine, I should be able to compile tensorflow from the source codes (1.3 branch) based on your patches above and enjoy the speed up from V100, right? Btw, will your patch work with the official cuda 9 (release few days ago) and cudnn 7? Or you will update the patch or you already did that? Thanks again. |
|
There are always errors for me to command eigen-...-diff file. I can't command the code cd -P bazel-out/../../../external/eigen_archive correctly. I tried when terminal was in tensorflow file and tensorflow1.3 file. But it doesn't work. Could you please tell me what I had tried to use this way: command cd -P bazel-out/(+the exact position of )external/eigen_archive and then I execute the patch, the errors just happened at each @@ -236,7 +236,7@@(for example, others are just the same form) and the program says can't find file to patch at input line xx. Do you know why these happen? |
|
I suggest taking TensorFlow from head and not using this patch. I will update the original comment. CUDA 9 with no other changes speeds up V100s over CUDA 8 in my testing of ResNet50 FP32. I hope to share the teams FP16 code and some results but we are still working through a variety of items. Technial the tf_cnn_benchmarks.py code currently published will do FP16 but I want to make sure it is running great with great convergence (high accuracy that matches or almost matches the FP32, e.e. ~75%) and scaling. |
|
I marked this closed but as always (or mostly) I am happy to answer questions and share status when I have time. |
|
Hello @tfboyd I want to offer more information about the first error . When I execute ( cd -P tensorflow1.3/lib/python2.7/.../external/eigen_archive ) I can enter the path. However, if I add bazel-out after -P, the terminal reports: No such file or dictionary. So I want to know the function of bazel-out. Is it a path or a command? I can’t find bazel-out in the bazel package. Thanks! |
|
Hi, I am a newbie. I want some help regarding tensorflow installation. |
|
Hi, my gcc version is 5.4.0 (Ubuntu 16.04), so I compile and install gcc 5.3.0 into /opt/gcc-5.3.0 ( gcc dependentments have already write into /etc/ld.so.conf.d/gcc-5.3.0.conf, and $HOME/.bashrc except gcc), and I have soft connection all /opt/gcc-5.3.0/bin to /usr/bin except the file whose name begins with 'x86_64-unknown-linux-' because I do not know the name in /usr/bin. So that, I fail My question is, do I have to compile it with 5.3.0? Anybody could help me? |
|
@Arnab0989 did you find some solution? Because I have exactly the same situation and I didn't find solution except that to install the old nvidia driver + cudnn6 + cuda8 |
|
Looking through the benchmark code and it mentions NCCL not supporting FP16, however Caffe has NCCL with FP16 support. Will this be added to Tensorflow for 1.5? - I'm currently getting worse results with FP16 on internal results. Current Hypotheses are it is either NCCL or TF.Layers. However, I've implemented a custom getter which I thought would circumvent the tf.layers issues. |
|
Can you provide more specifics? I assume V100 and where you are seeing FP16 worse than FP32 as far as throughput. And single or multi-gpu as well as which CUDA and cuDNN. That will help me to understand if and where the problem might be as well as get the right assistance and provide useful guidance. |
|
@Arnab0989 and @plgkm6 I suggest opening another issue to get windows help. This was just tracking getting CUDA 9 and cuDNN 7 support in the source. I know that is a little nuanced but you will get more attention for the windows problem outside this thread with a separate issue. I believe the team is still working though build issues with windows but I am not positive. |
|
@tfboyd Thanks for the quick reply, it's an internal benchmark which mirrors the TF benchmark but with medical images. GPUS: 8 P100s General computation time, for example I can double my batch size to 8 (max that fits in memory) but I only get 85 img/s vs 100 img/s with FP32 and batch size of 4. Model is in |
|
I would not advise FP16 on P100s for training unless it happens to work better for your situation. It is ok for testing that it works. I believe the issues is the V100 has the ability to do FP16 but do the accumulation in FP32. P100 cannot do that so you have to cast and do other "stuff". Apologies for being vague I am sharing what I have heard but I do not have direct experience. I am pretty sure I am generally correct but my details are likely off in some way. I will open an issue in the benchmark repo and link it here later today. Maybe I can get more data from the perf team. |
|
/CC @chsigg, who is planning on implementing FP16 nccl support. Not sure if this will be in by TF1.5 though. @joeyearsley As @tfboyd mentioned, P100s do not get a huge gain from fp16, and there is currently no nccl support for fp16. I don't think |
and others
Things have moved forward. I strongly suggest building from head with CUDA 9 and cuDNN 7. All of the necessary codes should be in the TF 1.4 tag but given we are still working on new features for FP16, I would build from head if that is of interest. I do not like to share anything I have not personally tested as I know how frustrating trying to get things to compile can be.
Everything below this line is OUT DATED as of 19-OCT
This is an unofficial and very not supported patch to make it possible to compile TensorFlow with CUDA9RC and cuDNN 7 or CUDA8 + cuDNN 7.
During testing on V100 (Volta) and ResNet-50 FP32 using CUDA 9RC + cuDNN 7 was significantly faster than CUDA 8 + cuDNN 6, which was not a surprise. I am about to test on P100s. I am sharing this patch informally so those that are interested can play with cuDNN 7 as well as CUDA 9RC before we have the official release. As we have more interesting code, e.g FP16 models, I will share it in this issue. I expect NVIDIA will start to submit official cuDNN 7 patches very soon.
Note: This patch may work on more recent versions of TensorFlow but it will likely bit rot so keep that in mind. Apply the cuDNN 7 patch and then fast-forwarding the branch might be the best approach. My git skills are not strong so do what you think is best.
Clone the tensorflow repo
https://github.com/tensorflow/tensorflow.gitCheckout the revision that the TensorFlow patch can apply to:
git checkout db596594b5653b43fcb558a4753b39904bb62cbd~Apply the TensorFlow patch:
git apply ~/Downloads/0001-CUDA-9.0-and-cuDNN-7.0-support.patchRun ./configure. When it asks for the CUDA version, put 9.0 (or 8). When it asks for the cudnn version, put 7.0.1 (Entering '7' worked fine for me). Make sure you put the paths to the right Cuda and cudnn versions and have your ldconfig or LD_LIBARY_PATH set to point to the CUDA 9 folder.
./configureAttempt to build TensorFlow, so that Eigen is downloaded. This build will fail if building for CUDA9RC but will succeed for CUDA8
bazel build --config=opt --config=cuda //tensorflow/tools/pip_package:build_pip_packageApply the Eigen patch:
cd - bazel build --config=opt --config=cuda //tensorflow/tools/pip_package:build_pip_packageI have run this process myself on Ubuntu 14.04 with Python 2.7.
Thank for NVIDIA for the early patch and @reedwm who created most of these instructions.
If you are using Python 2.7 and gcc 4.8+ here is a .whl where I followed the instructions above and one with CUDA 8 and cuDNN 7 which I have yet to test. Stress again, this was created by me and not the TensorFlow build team. My/our goal is to engage with anyone that wants to try this out and try to have a little fun. :-)
The text was updated successfully, but these errors were encountered: