Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

import tensorflow is very slow #37729

Open
hankcs opened this issue Mar 19, 2020 · 16 comments
Open

import tensorflow is very slow #37729

hankcs opened this issue Mar 19, 2020 · 16 comments
Assignees

Comments

@hankcs
Copy link

@hankcs hankcs commented Mar 19, 2020

Please make sure that this is an issue related to performance of TensorFlow.
As per our
GitHub Policy,
we only address code/doc bugs, performance issues, feature requests and
build/installation issues on GitHub. tag:performance_template

System information

  • Have I written custom code (as opposed to using a stock
    example script provided in TensorFlow): No
  • OS Platform and Distribution (e.g.,
    Linux Ubuntu 16.04): Ubuntu 18.04.3 LTS
  • Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if
    the issue happens on mobile device:
  • TensorFlow installed from (source or
    binary): binary
  • TensorFlow version (use command below): v2.1.0-rc2-17-ge5bf8de 2.1.0
  • Python version: Python 3.6.6
  • Bazel version (if compiling from source):
  • GCC/Compiler version (if compiling from
    source):
  • CUDA/cuDNN version:
    CUDA Version: 10.1
    cudnn-10.1
  • GPU model and memory:
    TITAN RTX
    24190MiB

You can collect some of this information using our environment capture
script
You can also obtain the TensorFlow version with: 1. TF 1.0: python -c "import tensorflow as tf; print(tf.GIT_VERSION, tf.VERSION)" 2. TF 2.0: python -c "import tensorflow as tf; print(tf.version.GIT_VERSION, tf.version.VERSION)"

Describe the current behavior
I'm thinking about using tensorflow 2.x for research. However, every time I import tf, it takes about 5 seconds. import tensorflow is 10 times slower thanimport torch.

Describe the expected behavior
I was expecting the two to be roughly the same level for a fresh import. I know these two libraries have their own design choices. But I'm just wondering is there any chance to speed up the import statement. As a researcher, I need to constantly debug and 5s is just not good.

Standalone code to reproduce the issue
Provide a reproducible test case that is the bare minimum necessary to generate
the problem. If possible, please share a link to Colab/Jupyter/any notebook.

from timeit import default_timer as timer
import os

print('import pytorch')
start = timer()
import torch

end = timer()
print('Elapsed time: ' + str(end - start))

print('import tensorflow')
start = timer()
import tensorflow

end = timer()
print('Elapsed time: ' + str(end - start))

Other info / logs Include any logs or source code that would be helpful to
diagnose the problem. If including tracebacks, please include the full
traceback. Large logs and files should be attached.

import pytorch
Elapsed time: 0.46338812075555325
import tensorflow
Elapsed time: 4.396180961281061
@hankcs
Copy link
Author

@hankcs hankcs commented Mar 19, 2020

The log to trace import python3 -v -c 'import tensorflow' 2>&1 | ts '%H:%M:%.S' | grep 'import ' > log.txt is attached below.

log.txt

@Saduf2019 Saduf2019 added the TF 2.1 label Mar 20, 2020
@Saduf2019
Copy link

@Saduf2019 Saduf2019 commented Mar 20, 2020

@hankcs
i have executed your code in nightly and this issue does not exist in nightly, please find the gist here

@hankcs
Copy link
Author

@hankcs hankcs commented Mar 20, 2020

@hankcs
i have executed your code in nightly and this issue does not exist in nightly, please find the gist here

Thank you for your reply. I'm still getting similar performance with tf-nightly. In colab pytorch got 0.2s but tf-nightly got 2s. I ran your gist too, the results are similar.

See https://colab.research.google.com/drive/1-NUqqa7W_6_rYP3r09uSm6pnYHQXtLoh

@jvishnuvardhan
Copy link
Contributor

@jvishnuvardhan jvishnuvardhan commented Mar 20, 2020

@hankcs I tried several time with TF2.1 and tf-nightly and I could not reproduce the issue. Some time TF is faster and sometime Pytorch is faster but the difference is not much. Please check the gist here.

Output is as follows

import pytorch
Elapsed time: 2.196739539999996
import tensorflow
Elapsed time: 1.7435289440000048

I have a question. Are you importing it in for loop?. You can import once and run your code multiple times to debug. Thanks!

Please close the issue if this was resolved for you. Thanks!

@hankcs
Copy link
Author

@hankcs hankcs commented Mar 20, 2020

I have a question. Are you importing it in for loop?. You can import once and run your code multiple times to debug. Thanks!

No, I'm importing only once. Redundant import takes no time.

I found some clues. If I terminate the colab session then execute your gist, pytorch takes 2 seconds and tf takes 1-4 seconds which shows they are on the same level.
hankcs com 2020-03-20 at 5 00 25 PM
However, if I restart the runtime, pytorch takes 0.2 second but tf still takes 2 seconds.
hankcs com 2020-03-20 at 5 01 01 PM

These shows that pytorch can make use of cache in memory while tf somehow can not. If you restart the session (reboot the system), there is no cache then they perform the same.

@jvishnuvardhan
Copy link
Contributor

@jvishnuvardhan jvishnuvardhan commented Mar 20, 2020

@hankcs I ran the colab you shared and I got the similar results as I showed. Check the following output.

import pytorch
Elapsed time: 2.00347342200007
import tensorflow
Elapsed time: 1.8452622999998312
@jvishnuvardhan
Copy link
Contributor

@jvishnuvardhan jvishnuvardhan commented Mar 20, 2020

@hankcs Can you please check the same in your local? Thanks!

@hankcs
Copy link
Author

@hankcs hankcs commented Mar 20, 2020

Sure, my local machine running 2.1.0 gives similar result to what I observe in colab.

import pytorch
Elapsed time: 0.45859437994658947
import tensorflow
2.1.0
Elapsed time: 4.311788005754352

I'll report tf-nightly later as I have some jobs running.

@mihaimaruseac
Copy link
Collaborator

@mihaimaruseac mihaimaruseac commented Mar 20, 2020

(t) mihaimaruseac@ankh:/tmp/t$ pip install -q tensorflow torch
(t) mihaimaruseac@ankh:/tmp/t$ time python -c "import tensorflow"
2020-03-20 14:47:02.352581: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libnvinfer.so.6'; dlerror: libnvinfer.so.6: cannot open shared object file: No such file or directory
2020-03-20 14:47:02.352679: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libnvinfer_plugin.so.6'; dlerror: libnvinfer_plugin.so.6: cannot open shared object file: No such file or directory
2020-03-20 14:47:02.352694: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:30] Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.

real    0m8.390s
user    0m2.386s
sys     0m9.376s
(t) mihaimaruseac@ankh:/tmp/t$ time python -c "import tensorflow"
2020-03-20 14:47:18.194391: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libnvinfer.so.6'; dlerror: libnvinfer.so.6: cannot open shared object file: No such file or directory
2020-03-20 14:47:18.194475: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libnvinfer_plugin.so.6'; dlerror: libnvinfer_plugin.so.6: cannot open shared object file: No such file or directory
2020-03-20 14:47:18.194490: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:30] Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.

real    0m1.616s
user    0m3.429s
sys     0m3.624s
(t) mihaimaruseac@ankh:/tmp/t$ time python -c "import pytorch"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'pytorch'

real    0m0.066s
user    0m0.058s
sys     0m0.014s
(t) mihaimaruseac@ankh:/tmp/t$ time python -c "import torch"

real    0m7.498s
user    0m2.299s
sys     0m10.536s
(t) mihaimaruseac@ankh:/tmp/t$ time python -c "import torch"

real    0m0.547s
user    0m2.314s
sys     0m3.578s

There is indeed a slowdown in tf import but it looks like first import time is the same in both pytorch and TF.

We have an internal bug assigned for speeding up TF import

@jvishnuvardhan jvishnuvardhan removed their assignment Mar 20, 2020
@hankcs
Copy link
Author

@hankcs hankcs commented Mar 20, 2020

Thank you. I'm grateful to know that tf teams are making efforts to address this issue. I'll promote tf in my research projects.

@epicmatthew21
Copy link

@epicmatthew21 epicmatthew21 commented Apr 19, 2020

getting the same issue, any update?

@kozmaz87
Copy link

@kozmaz87 kozmaz87 commented Jul 13, 2020

Also this seems to be a thing with caching by your operating system. On Windows the first time I load it by the time it imports may be 4 seconds+, which is unacceptable on an SSD on a Ryzen 9 computer. Then I close the interpreter and do it again, it imports in less than a second. This is why you fail to reproduce it in my humble opinion. Make sure you freshly restart your computer and run the test thereafter, also analyse the output of python -X importtime

@mihaimaruseac
Copy link
Collaborator

@mihaimaruseac mihaimaruseac commented Jul 13, 2020

In my case the benchmarks were immediately after installing the pips, so they were not loaded in a cache.

For Windows, there is already a known issue regarding slow down to disk access when there are multiple files around microsoft/WSL#4197

@zirui
Copy link

@zirui zirui commented Nov 27, 2020

any update on this issue?
I have similar issue one my local mac machine:

import pytorch
Elapsed time: 0.302831451
import tensorflow
Elapsed time: 3.103030106

System information

  • Have I written custom code (as opposed to using a stock
    example script provided in TensorFlow): No
  • OS Platform and Distribution (e.g.,Linux Ubuntu 16.04): macOS Big Sur version 11.01
  • TensorFlow installed from (source or binary): binary
  • TensorFlow version (use command below): 2.3.1
  • Python version: Python 3.7.1
@YashRunwal
Copy link

@YashRunwal YashRunwal commented Dec 21, 2020

Hello guys,

Any updates?

I am using a conda virtual environment.

import tensorflow as tf takes an absurd amount of time.

Note: I am using tensorflow cpu.

What can be done to reduce this import time?

@MJimitater
Copy link

@MJimitater MJimitater commented Jan 8, 2021

same issue, any updates?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
9 participants