Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Installation issue with Tensorflow-cpu, no module named '_pwyrap_tensorflow_internal' #22512

Closed
Overdrivr opened this Issue Sep 25, 2018 · 27 comments

Comments

Projects
None yet
@Overdrivr
Copy link

Overdrivr commented Sep 25, 2018

System information

  • Have I written custom code (as opposed to using a stock example script provided in TensorFlow): Yes
  • OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Windows 10 server in desktop mode
  • Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device: NA
  • TensorFlow installed from (source or binary): binary
  • TensorFlow version (use command below): tensorflow-gpu 1.10.0
  • Python version: Anaconda x64 3.6
  • Bazel version (if compiling from source): NA
  • GCC/Compiler version (if compiling from source): NA
  • CUDA/cuDNN version: 9.0/7.0
  • GPU model and memory: Q4000
  • Exact command to reproduce: N/A

Describe the problem

  1. Create a python application that imports tensorflow-gpu
  2. Package the application with pyinstaller
  3. Start the generated executable, that crashes while trying to import tensorflow.

The error trace is this:

Traceback (most recent call last):
  File "site-packages\tensorflow\python\pywrap_tensorflow.py", line 58, in <module>
  File "c:\users\paperspace\.virtualenvs\***-ecouzwts\lib\site-packages\PyInstaller\loader\pyimod03_importers.py", line 627, in exec_module
    exec(bytecode, module.__dict__)
  File "site-packages\tensorflow\python\pywrap_tensorflow_internal.py", line 18, in <module>
  File "site-packages\tensorflow\python\pywrap_tensorflow_internal.py", line 17, in swig_import_helper
  File "importlib\__init__.py", line 126, in import_module
ModuleNotFoundError: No module named 'tensorflow.python._pywrap_tensorflow_internal'


Failed to load the native TensorFlow runtime.

See https://www.tensorflow.org/install/install_sources#common_installation_problems

for some common reasons and solutions.  Include the entire stack trace
above this error message when asking for help.

On the other hand, doing the same process with tensorflow(-cpu) works fine.

I ruled out CUDA/cuDNN installation issue, because if I run my python application using python, it works fine with the GPU version.

I have tried checking at the .exe with dependency_walker, but appart from false-positive missing API-MS-* and EXT-MS-* DLLs, I don't see anything obvious.

So I'm stuck here.
What discrepancy could create an import issue with tensorflow-gpu and not with tensorflow-cpu ?

@tensorflowbutler

This comment has been minimized.

Copy link
Member

tensorflowbutler commented Sep 26, 2018

Thank you for your post. We noticed you have not filled out the following field in the issue template. Could you update them if they are relevant in your case, or leave them as N/A? Thanks.
Exact command to reproduce

@Overdrivr

This comment has been minimized.

Copy link
Author

Overdrivr commented Sep 26, 2018

I've updated the form accordingly

@Overdrivr

This comment has been minimized.

Copy link
Author

Overdrivr commented Sep 26, 2018

@ymodak I have made some progress. It turns out, I was building tensorflow from an Anaconda shell. I removed Anaconda and installed standard python 3.6.6 64 bits.

By doing so, the initial problem went away.
Executable runs fine on the build machine.
However, if I transfer the standalone application to another machine, it fails with the following error:

Traceback (most recent call last):
  File "site-packages\tensorflow\python\pywrap_tensorflow.py", line 58, in <module>
  File "c:\users\paperspace\.virtualenvs\nexture-studio-dist-ecouzwts\lib\site-packages\PyInstaller\loader\pyimod03_importers.py", line 627, in exec_module
  File "site-packages\tensorflow\python\pywrap_tensorflow_internal.py", line 18, in <module>
  File "site-packages\tensorflow\python\pywrap_tensorflow_internal.py", line 17, in swig_import_helper
  File "importlib\__init__.py", line 126, in import_module
ImportError: DLL load failed: A dynamic link library (DLL) initialization routine failed.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "nxt\nxt.py", line 2, in <module>
  File "c:\users\paperspace\.virtualenvs\nexture-studio-dist-ecouzwts\lib\site-packages\PyInstaller\loader\pyimod03_importers.py", line 627, in exec_module
  File "site-packages\tensorflow\__init__.py", line 22, in <module>
  File "c:\users\paperspace\.virtualenvs\nexture-studio-dist-ecouzwts\lib\site-packages\PyInstaller\loader\pyimod03_importers.py", line 627, in exec_module
  File "site-packages\tensorflow\python\__init__.py", line 49, in <module>
  File "c:\users\paperspace\.virtualenvs\nexture-studio-dist-ecouzwts\lib\site-packages\PyInstaller\loader\pyimod03_importers.py", line 627, in exec_module
  File "site-packages\tensorflow\python\pywrap_tensorflow.py", line 74, in <module>
ImportError: Traceback (most recent call last):
  File "site-packages\tensorflow\python\pywrap_tensorflow.py", line 58, in <module>
  File "c:\users\paperspace\.virtualenvs\nexture-studio-dist-ecouzwts\lib\site-packages\PyInstaller\loader\pyimod03_importers.py", line 627, in exec_module
  File "site-packages\tensorflow\python\pywrap_tensorflow_internal.py", line 18, in <module>
  File "site-packages\tensorflow\python\pywrap_tensorflow_internal.py", line 17, in swig_import_helper
  File "importlib\__init__.py", line 126, in import_module
ImportError: DLL load failed: A dynamic link library (DLL) initialization routine failed.


Failed to load the native TensorFlow runtime.

See https://www.tensorflow.org/install/install_sources#common_installation_problems

for some common reasons and solutions.  Include the entire stack trace
above this error message when asking for help.

Here is the code to reproduce:

import tensorflow as tf

def main():	
	hello = tf.constant('hello TF')
	sess = tf.Session()
	print(sess.run(hello))

if __name__ == '__main__':
    main()

Steps to reproduce:

  1. Run the program with python 3.6 and tensorflow-gpu 1.10.0. Works ok.
  2. Package into a standalone with PyInstaller
  3. Run the executable. Works ok.
  4. Move the folder containing the executable and all dependencies to another machine.
  5. Run the executable. Fails with error above.
@Overdrivr

This comment has been minimized.

Copy link
Author

Overdrivr commented Sep 26, 2018

I finally managed to catch the exception happening at import time, application is raising EXCEPTION_ILLEGAL_INSTRUCTION.

exception_illegal_instruction

Could this be related to AXV instructions not supported ? The processor of the VM is an Intel Xeon E5-2623 v4 that apparently supports AVX2.

@ymodak ymodak assigned wt-huang and unassigned ymodak Sep 26, 2018

@wt-huang

This comment has been minimized.

Copy link

wt-huang commented Sep 26, 2018

@Overdrivr This is not a bug. You can run some sanity checks on the second machine to make sure the environment is set up correctly and dependencies are installed properly.

@Overdrivr

This comment has been minimized.

Copy link
Author

Overdrivr commented Sep 27, 2018

You're correct, there is a configuration issue on the second machine.

I have installed python 3.6.6 on it, created a new virtual env, only containing tensorflow, and can reproduce the issue.

What puzzles me is that I have installed tensorflow-cpu on it, and I still get _pywrap_tensorflow_internal not found (And it's not the first time I've installed TF on a machine, I've done it for CPU and GPU version a good dozen of times).

I have ran the following sanity check (updated for tensorflow 1.10.0, Cuda 9.0, cuDNN 7.0.X):

import ctypes
import imp
import sys
from ctypes.util import find_library

def main():
    check()

def check():
    try:
        import tensorflow as tf
        print("TensorFlow successfully installed.")
        if tf.test.is_built_with_cuda():
            print("The installed version of TensorFlow includes GPU support.")
        else:
            print("The installed version of TensorFlow does not include GPU support.")
            #sys.exit(0)
    except ImportError as e:
        print("ERROR: Failed to import the TensorFlow module.")
        print('Reason: {}'.format(e))

    candidate_explanation = False

    python_version = sys.version_info.major, sys.version_info.minor
    print("- Python version is %d.%d." % python_version)
    if not (python_version == (3, 5) or python_version == (3, 6)):
        candidate_explanation = True
        print("- The official distribution of TensorFlow for Windows requires "
          "Python version 3.5 or 3.6.")

    try:
        _, pathname, _ = imp.find_module("tensorflow")
        print("- TensorFlow is installed at: %s" % pathname)
    except ImportError:
        candidate_explanation = False
        print("""- No module named TensorFlow is installed in this Python environment. You may
install it using the command `pip install tensorflow`.""")

    try:
        msvcp140 = ctypes.WinDLL("msvcp140.dll")
        msvcp140_path = find_library("msvcp140.dll")
        print('- msvcp140.dll Found at {}'.format(msvcp140_path))
    except OSError:
        candidate_explanation = True
        print("""
- Could not load 'msvcp140.dll'. You may install this DLL by downloading Microsoft Visual
  C++ 2015 Redistributable Update 3 from this URL:
  https://www.microsoft.com/en-us/download/details.aspx?id=53587""")

    try:
        cudart64_90 = ctypes.WinDLL("cudart64_90.dll")
        cudart64_90_path = find_library("cudart64_90.dll")
        print('- Cuda 9.0 found at {}'.format(cudart64_90_path))
        # TODO: Look for version.txt file in CUDA path
    except OSError:
        candidate_explanation = True
        print("""
- Could not load 'cudart64_90.dll'. Download and install CUDA 9.0 from
  this URL: https://developer.nvidia.com/cuda-toolkit""")

    try:
        nvcuda = ctypes.WinDLL("nvcuda.dll")
        nvcuda_path = find_library("nvcuda.dll")
        print('- nvcuda.dll found at {}'.format(nvcuda_path))
    except OSError:
        candidate_explanation = True
        print("""
- Could not load 'nvcuda.dll'. The GPU version of TensorFlow requires that
  this DLL be installed in a directory that is named in your %PATH%
  environment variable. Typically it is installed in 'C:\Windows\System32'.
  If it is not present, ensure that you have a CUDA-capable GPU with the
  correct driver installed.""")

    cudnn7_found = False
    try:
        cudnn7 = ctypes.WinDLL("cudnn64_7.dll")
        cudnn7_path = find_library("cudnn64_7.dll")
        print("- cuDNN Found at {}".format(cudnn7_path))
        cudnn7_found = True
    except OSError:
        candidate_explanation = True
        print("""
- Could not load 'cudnn64_7.dll'. The GPU version of TensorFlow
  requires that this DLL be installed in a directory that is named in
  your %PATH% environment variable. Note that installing cuDNN is a
  separate step from installing CUDA, and it is often found in a
  different directory from the CUDA DLLs. You may install the
  necessary DLL by downloading cuDNN 7.0 for Cuda 9.0 from this URL:
  https://developer.nvidia.com/cudnn""")

    if not candidate_explanation:
        print("""
- All required DLLs appear to be present. Please open an issue on the
  TensorFlow GitHub page: https://github.com/tensorflow/tensorflow/issues""")

    sys.exit(-1)


if __name__ == '__main__':
    main()

Output:

ERROR: Failed to import the TensorFlow module.
Reason: Traceback (most recent call last):
  File "C:\Users\Paperspace\.virtualenvs\test-tf-vXVQRlro\lib\site-packages\tensorflow\python\pywrap_tensorflow_internal.py", line 14, in swig_import_helper
    return importlib.import_module(mname)
  File "C:\Users\Paperspace\.virtualenvs\test-tf-vXVQRlro\lib\importlib\__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 994, in _gcd_import
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 658, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 571, in module_from_spec
  File "<frozen importlib._bootstrap_external>", line 922, in create_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
ImportError: DLL load failed: A dynamic link library (DLL) initialization routine failed.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\Paperspace\.virtualenvs\test-tf-vXVQRlro\lib\site-packages\tensorflow\python\pywrap_tensorflow.py", line 58, in <module>
    from tensorflow.python.pywrap_tensorflow_internal import *
  File "C:\Users\Paperspace\.virtualenvs\test-tf-vXVQRlro\lib\site-packages\tensorflow\python\pywrap_tensorflow_internal.py", line 17, in <module>
    _pywrap_tensorflow_internal = swig_import_helper()
  File "C:\Users\Paperspace\.virtualenvs\test-tf-vXVQRlro\lib\site-packages\tensorflow\python\pywrap_tensorflow_internal.py", line 16, in swig_import_helper
    return importlib.import_module('_pywrap_tensorflow_internal')
  File "C:\Users\Paperspace\.virtualenvs\test-tf-vXVQRlro\lib\importlib\__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
ModuleNotFoundError: No module named '_pywrap_tensorflow_internal'


Failed to load the native TensorFlow runtime.

See https://www.tensorflow.org/install/install_sources#common_installation_problems

for some common reasons and solutions.  Include the entire stack trace
above this error message when asking for help.
- Python version is 3.6.
- TensorFlow is installed at: C:\Users\Paperspace\.virtualenvs\test-tf-vXVQRlro\lib\site-packages\tensorflow
- msvcp140.dll Found at C:\Windows\system32\msvcp140.dll
- Cuda 9.0 found at C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v9.0\bin\cudart64_90.dll
- nvcuda.dll found at C:\Windows\system32\nvcuda.dll
- cuDNN Found at C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v9.0\bin\cudnn64_7.dll

- All required DLLs appear to be present. Please open an issue on the
  TensorFlow GitHub page: https://github.com/tensorflow/tensorflow/issues

Basically, the sanity check does not highlight any error, yet there seems to be an issue loading some DLL. This is very weird, because I have installed TF-cpu and GPU on another machine with exactly the same specs two hours before that one (They are both freshly created machines on a cloud provider with the same base Windows 10 image).

@Overdrivr Overdrivr changed the title Packaging issues with tensorflow-gpu Installation issue with Tensorflow-cpu, no module named '_pwyrap_tensorflow_internal' Sep 27, 2018

@Overdrivr

This comment has been minimized.

Copy link
Author

Overdrivr commented Sep 27, 2018

I have tried TF 1.5 and it solves the issue. TF can be imported just fine. So, clearly, it has be an issue with AVX.

But it is stated here (and everywhere else) that the Xeon E5-2623 v4 supports AVX2, so I guess it should support AVX too, right ?

To be sure, I wrote a small script that fetches CPUID and checks if the hardware supports AVX.
Install cpuid to run it.

from cpuid import *

def _is_set(id, reg_idx, bit):
    regs = cpuid(id)

    if (1 << bit) & regs[reg_idx]:
        return "Yes"
    else:
        return "--"

print("Vendor ID         : %s" % cpu_vendor())
print("CPU name          : %s" % cpu_name())
print("Microarchitecture : %s%s" % cpu_microarchitecture())
print("Vector instructions supported:")
print("SSE       : %s" % _is_set(1, 3, 25))
print("SSE2      : %s" % _is_set(1, 3, 26))
print("SSE3      : %s" % _is_set(1, 2, 0))
print("SSSE3     : %s" % _is_set(1, 2, 9))
print("SSE4.1    : %s" % _is_set(1, 2, 19))
print("SSE4.2    : %s" % _is_set(1, 2, 20))
print("SSE4a     : %s" % _is_set(0x80000001, 2, 6))
print("AVX       : %s" % _is_set(1, 2, 28))
print("AVX2      : %s" % _is_set(7, 1, 5))
print("BMI1      : %s" % _is_set(7, 1, 3))
print("BMI2      : %s" % _is_set(7, 1, 8))

And this is what I get:

Vendor ID         : GenuineIntel
CPU name          : Intel(R) Xeon(R) CPU E5-2623 v4 @ 2.60GHz
Microarchitecture : broadwellnoavx
Vector instructions supported:
SSE       : Yes
SSE2      : Yes
SSE3      : Yes
SSSE3     : Yes
SSE4.1    : Yes
SSE4.2    : Yes
SSE4a     : --
AVX       : --
AVX2      : --
BMI1      : Yes
BMI2      : Yes

Just the name of the micro-architecture is pretty self-explanatory.
Not only does he not support AVX2, he does not even support AVX1.

I would like to give a big THANKS to intel for advertising their processor to support AVX when they clearly don't (and for wasting our time).

@Overdrivr

This comment has been minimized.

Copy link
Author

Overdrivr commented Sep 27, 2018

While we're at it, I'm getting the same problem with tensorflow-cpu... on another freshly created machine.. that supports AVX. Fun ride.

Emotionnal rollercoaster

I'm leaving this here for documentation, it might help others.

In my production virtualenv with tensorflow-cpu + a bunch of other packages, I get the following error (the output comes from a tensorflow-install-checker tool that I will release in public domain):

INFO - Vendor ID         : GenuineIntel
INFO - CPU name          : Intel(R) Xeon(R) CPU E5-2623 v4 @ 2.60GHz
INFO - Microarchitecture : broadwell
OK  - Your processor supports AVX instructions
ERR - Failed to import the TensorFlow module.
Traceback (most recent call last):
  File "c:\users\paperspace\.virtualenvs\nexture-studio-dist-jrq2aiwp\lib\site-packages\tensorflow\python\pywrap_tensorflow_internal.py", line 14, in swig_import_helper
    return importlib.import_module(mname)
  File "c:\users\paperspace\.virtualenvs\nexture-studio-dist-jrq2aiwp\lib\importlib\__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 994, in _gcd_import
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 658, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 571, in module_from_spec
  File "<frozen importlib._bootstrap_external>", line 922, in create_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
ImportError: DLL load failed: The specified module could not be found.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "c:\users\paperspace\.virtualenvs\nexture-studio-dist-jrq2aiwp\lib\site-packages\tensorflow\python\pywrap_tensorflow.py", line 58, in <module>
    from tensorflow.python.pywrap_tensorflow_internal import *
  File "c:\users\paperspace\.virtualenvs\nexture-studio-dist-jrq2aiwp\lib\site-packages\tensorflow\python\pywrap_tensorflow_internal.py", line 17, in <module>
    _pywrap_tensorflow_internal = swig_import_helper()
  File "c:\users\paperspace\.virtualenvs\nexture-studio-dist-jrq2aiwp\lib\site-packages\tensorflow\python\pywrap_tensorflow_internal.py", line 16, in swig_import_helper
    return importlib.import_module('_pywrap_tensorflow_internal')
  File "c:\users\paperspace\.virtualenvs\nexture-studio-dist-jrq2aiwp\lib\importlib\__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
ModuleNotFoundError: No module named '_pywrap_tensorflow_internal'


Failed to load the native TensorFlow runtime.

See https://www.tensorflow.org/install/install_sources#common_installation_problems

for some common reasons and solutions.  Include the entire stack trace
above this error message when asking for help.
OK  - TensorFlow found at: c:\users\paperspace\.virtualenvs\nexture-studio-dist-jrq2aiwp\lib\site-packages\tensorflow
OK  - msvcp140.dll found at: c:\users\paperspace\.virtualenvs\nexture-studio-dist-jrq2aiwp\lib\site-packages\scipy\extra-dll\msvcp140.dll
OK  - Cuda 9.0 found at C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v9.0\bin\cudart64_90.dll
OK  - nvcuda.dll found at C:\Windows\system32\nvcuda.dll
OK  - cuDNN Found at C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v9.0\bin\cudnn64_7.dll

In another virtualenv with only tensorflow-cpu, I get this one:

Traceback (most recent call last):
  File "C:\Users\Paperspace\.virtualenvs\test_tf--gQQvE-L\lib\site-packages\tensorflow\python\platform\self_check.py", line 47, in preload_check
    ctypes.WinDLL(build_info.msvcp_dll_name)
  File "C:\Program Files\Python36\Lib\ctypes\__init__.py", line 348, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: [WinError 126] The specified module could not be found

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Users\Paperspace\.virtualenvs\test_tf--gQQvE-L\lib\site-packages\tensorflow\__init__.py", line 22, in <module>
    from tensorflow.python import pywrap_tensorflow  # pylint: disable=unused-import
  File "C:\Users\Paperspace\.virtualenvs\test_tf--gQQvE-L\lib\site-packages\tensorflow\python\__init__.py", line 49, in <module>
    from tensorflow.python import pywrap_tensorflow
  File "C:\Users\Paperspace\.virtualenvs\test_tf--gQQvE-L\lib\site-packages\tensorflow\python\pywrap_tensorflow.py", line 30, in <module>
    self_check.preload_check()
  File "C:\Users\Paperspace\.virtualenvs\test_tf--gQQvE-L\lib\site-packages\tensorflow\python\platform\self_check.py", line 55, in preload_check
    % build_info.msvcp_dll_name)
ImportError: Could not find 'msvcp140.dll'. TensorFlow requires that this DLL be installed in a directory that is named in your %PATH% environment variable. You may install this DLL by downloading Visual C++ 2015 Redistributable Update 3 from this URL: https://www.microsoft.com/en-us/download/details.aspx?id=53587

Hum weird right ? Let's summarize:

  • In the first env, msvcp140.dll is found, but TF fails to load
  • In the second env, msvcp140.dll is missing

If you look closely, the msvcp140.dll found in the first env does not come from system32 folder.
It comes from scipy (c:\users\paperspace\.virtualenvs\******-jrq2aiwp\lib\site-packages\scipy\extra-dll\msvcp140.dll).
Scipy is (to solve other issues) releasing msvcp140.dll in their extra-dll folder. This folder ends up in the PATH, and the dll in question picked up by the library finder.

Now, why that's a problem:

  • For machines that already have this DLL in system32 folder, it's fine because system path takes precedence over user path.
  • For machines without this DLL, instead of having TF crash with DLL missing, it crashes with some cryptic _pywrap_tensorflow_internal... error, most likely because the DLL found in scipy is different and not compatible that the latest installed by Microsoft Visual C++ 2015 Redistributable Update 3 RC. The workaround is then to simply install Microsoft Visual C++ 2015 Redistributable 64 bits. This way, the DLL is installed in system32 folder and takes precedences over the one installed by scipy.

After doing so on my VM, importing TF in both virtualenvs now runs properly. 🥂

INFO - Vendor ID         : GenuineIntel
INFO - CPU name          : Intel(R) Xeon(R) CPU E5-2623 v4 @ 2.60GHz
INFO - Microarchitecture : broadwell
OK  - Your processor supports AVX instructions
OK  - TensorFlow is functional.
ERR - TensorFlow does not include GPU support.
OK  - TensorFlow found at: c:\users\paperspace\.virtualenvs\nexture-studio-dist-jrq2aiwp\lib\site-packages\tensorflow
OK  - msvcp140.dll found at: C:\Windows\system32\msvcp140.dll
OK  - Cuda 9.0 found at C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v9.0\bin\cudart64_90.dll
OK  - nvcuda.dll found at C:\Windows\system32\nvcuda.dll
OK  - cuDNN Found at C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v9.0\bin\cudnn64_7.dll

Maybe it might be worth mentioning it in the install docs.

@gunan

This comment has been minimized.

Copy link
Member

gunan commented Sep 27, 2018

@lamberta Can we add this requirement to our windows installation docs?

For machines without this DLL, instead of having TF crash with DLL missing, it crashes with some cryptic _pywrap_tensorflow_internal... error, most likely because the DLL found in scipy is different and not compatible that the latest installed by Microsoft Visual C++ 2015 Redistributable Update 3 RC. The workaround is then to simply install Microsoft Visual C++ 2015 Redistributable 64 bits. This way, the DLL is installed in system32 folder and takes precedences over the one installed by scipy.

@lamberta

This comment has been minimized.

Copy link
Member

lamberta commented Sep 27, 2018

@gunan I can highlight the need to install Microsoft Visual C++ 2015 Redistributable and to make sure the DLL is installed in the system32 folder.

Can also add the error message to this page: https://www.tensorflow.org/install/errors (and link to this issue).

But is this something tensorflow can detect and provide a more useful error message? "Can not find msvcp140.dll in system32 directory" (or something)

@gunan

This comment has been minimized.

Copy link
Member

gunan commented Sep 27, 2018

Certainly. Both are things we need to do.

@tensorflowbutler

This comment has been minimized.

Copy link
Member

tensorflowbutler commented Oct 12, 2018

Nagging Assignee @wt-huang: It has been 14 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

@wt-huang

This comment has been minimized.

Copy link

wt-huang commented Oct 12, 2018

Closing this issue as the PR is merged.

@wt-huang wt-huang closed this Oct 12, 2018

@tamamjerbi

This comment has been minimized.

Copy link

tamamjerbi commented Dec 6, 2018

PLEASE HELP!!!
I've tried everything, can't understand why on earth is this hapenning!!
my system instructions set:-----------------------------------------
Instructions sets | MMX, SSE, SSE2, SSE3, SSSE3, SSE4.1, SSE4.2, EM64T, VT-x, AES, AVX, AVX2, FMA3

Win10-64bit
Core i5 8250U
and using tensorflow cpu 1.12

what I've tried so far
downgrading Tensorflow to 1.5 and 1.9
Installing C++ build tools
I've already latest C++ redist 2017
.................................................................................
don't know what to do else!!!!

Error reproduce on importing tensorflow

Python 3.6.0 (v3.6.0:41df79263a11, Dec 23 2016, 08:06:12) [MSC v.1900 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.

import tensorflow

Traceback (most recent call last):
File "", line 1, in
File "C:\Users\ASUS\AppData\Local\Programs\Python\Python36\lib\site-packages\tensorflow_init_.py", line 24, in
from tensorflow.python import pywrap_tensorflow # pylint: disable=unused-import
File "C:\Users\ASUS\AppData\Local\Programs\Python\Python36\lib\site-packages\tensorflow\python_init_.py", line 59, in
from tensorflow.core.framework.graph_pb2 import *
File "C:\Users\ASUS\AppData\Local\Programs\Python\Python36\lib\site-packages\tensorflow\core\framework\graph_pb2.py", line 6, in
from google.protobuf import descriptor as _descriptor
File "C:\Users\ASUS\AppData\Local\Programs\Python\Python36\lib\site-packages\google\protobuf\descriptor.py", line 47, in
from google.protobuf.pyext import _message
ImportError: DLL load failed: The specified procedure could not be found.

please help!!!

@Overdrivr

This comment has been minimized.

Copy link
Author

Overdrivr commented Dec 6, 2018

It's because it's not TF that fails to load, it's protobuf: protocolbuffers/protobuf#5046 (comment)

Apparently, downgrading to protobuf 3.6.0 (most likely you have 3.6.1) does the trick

@praneetmehta

This comment has been minimized.

Copy link

praneetmehta commented Jan 23, 2019

It's because it's not TF that fails to load, it's protobuf: protocolbuffers/protobuf#5046 (comment)

Apparently, downgrading to protobuf 3.6.0 (most likely you have 3.6.1) does the trick

The problem exists for every version of tensorflow 1.5 - 1.12 (cpu). But downgrading protobuf from 3.6.1 to 3.6.0 does the trick. @Overdrivr thanks for the solution

@kumarrajesh996

This comment has been minimized.

Copy link

kumarrajesh996 commented Feb 24, 2019

import keras
Using TensorFlow backend.
Traceback (most recent call last):
File "C:\Users\Rajesh Kumar Khatik\AppData\Local\Programs\Python\Python36\lib\site-packages\tensorflow\python\pywrap_tensorflow.py", line 58, in
from tensorflow.python.pywrap_tensorflow_internal import *
File "C:\Users\Rajesh Kumar Khatik\AppData\Local\Programs\Python\Python36\lib\site-packages\tensorflow\python\pywrap_tensorflow_internal.py", line 28, in
_pywrap_tensorflow_internal = swig_import_helper()
File "C:\Users\Rajesh Kumar Khatik\AppData\Local\Programs\Python\Python36\lib\site-packages\tensorflow\python\pywrap_tensorflow_internal.py", line 24, in swig_import_helper
_mod = imp.load_module('_pywrap_tensorflow_internal', fp, pathname, description)
File "C:\Users\Rajesh Kumar Khatik\AppData\Local\Programs\Python\Python36\Lib\imp.py", line 243, in load_module
return load_dynamic(name, filename, file)
File "C:\Users\Rajesh Kumar Khatik\AppData\Local\Programs\Python\Python36\Lib\imp.py", line 343, in load_dynamic
return _load(spec)
ImportError: DLL load failed: The specified module could not be found.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "", line 1, in
File "C:\Users\Rajesh Kumar Khatik\AppData\Local\Programs\Python\Python36\lib\site-packages\keras_init_.py", line 3, in
from . import utils
File "C:\Users\Rajesh Kumar Khatik\AppData\Local\Programs\Python\Python36\lib\site-packages\keras\utils_init_.py", line 6, in
from . import conv_utils
File "C:\Users\Rajesh Kumar Khatik\AppData\Local\Programs\Python\Python36\lib\site-packages\keras\utils\conv_utils.py", line 9, in
from .. import backend as K
File "C:\Users\Rajesh Kumar Khatik\AppData\Local\Programs\Python\Python36\lib\site-packages\keras\backend_init_.py", line 89, in
from .tensorflow_backend import *
File "C:\Users\Rajesh Kumar Khatik\AppData\Local\Programs\Python\Python36\lib\site-packages\keras\backend\tensorflow_backend.py", line 5, in
import tensorflow as tf
File "C:\Users\Rajesh Kumar Khatik\AppData\Local\Programs\Python\Python36\lib\site-packages\tensorflow_init_.py", line 24, in
from tensorflow.python import pywrap_tensorflow # pylint: disable=unused-import
File "C:\Users\Rajesh Kumar Khatik\AppData\Local\Programs\Python\Python36\lib\site-packages\tensorflow\python_init_.py", line 49, in
from tensorflow.python import pywrap_tensorflow
File "C:\Users\Rajesh Kumar Khatik\AppData\Local\Programs\Python\Python36\lib\site-packages\tensorflow\python\pywrap_tensorflow.py", line 74, in
raise ImportError(msg)
ImportError: Traceback (most recent call last):
File "C:\Users\Rajesh Kumar Khatik\AppData\Local\Programs\Python\Python36\lib\site-packages\tensorflow\python\pywrap_tensorflow.py", line 58, in
from tensorflow.python.pywrap_tensorflow_internal import *
File "C:\Users\Rajesh Kumar Khatik\AppData\Local\Programs\Python\Python36\lib\site-packages\tensorflow\python\pywrap_tensorflow_internal.py", line 28, in
_pywrap_tensorflow_internal = swig_import_helper()
File "C:\Users\Rajesh Kumar Khatik\AppData\Local\Programs\Python\Python36\lib\site-packages\tensorflow\python\pywrap_tensorflow_internal.py", line 24, in swig_import_helper
_mod = imp.load_module('_pywrap_tensorflow_internal', fp, pathname, description)
File "C:\Users\Rajesh Kumar Khatik\AppData\Local\Programs\Python\Python36\Lib\imp.py", line 243, in load_module
return load_dynamic(name, filename, file)
File "C:\Users\Rajesh Kumar Khatik\AppData\Local\Programs\Python\Python36\Lib\imp.py", line 343, in load_dynamic
return _load(spec)
ImportError: DLL load failed: The specified module could not be found.

Failed to load the native TensorFlow runtime.

See https://www.tensorflow.org/install/errors

for some common reasons and solutions. Include the entire stack trace
above this error message when asking for help.

Please help me with this error

@anisayari

This comment has been minimized.

Copy link

anisayari commented Mar 6, 2019

Downgrading protobuf to 3.6.0 bring me this error, don't know who should support this fix, protobuf or tensorflow... :
tensorflow-gpu 1.13.1 has requirement protobuf>=3.6.1, but you'll have protobuf 3.6.0 which is incompatible.

@gargisri68

This comment has been minimized.

Copy link

gargisri68 commented Mar 7, 2019

I am following https://towardsdatascience.com/python-environment-setup-for-deep-learning-on-windows-10-c373786e36d1 blog and I get the following error on writing import tensorflow:
image
Please help.

@yuwancumian666

This comment has been minimized.

Copy link

yuwancumian666 commented Mar 12, 2019

@gargisri68 same issue. Have you solved the problem?

@gargisri68

This comment has been minimized.

Copy link

gargisri68 commented Mar 12, 2019

@yuwancumian666

This comment has been minimized.

Copy link

yuwancumian666 commented Mar 12, 2019

I encountered this problem when I installed tensorflow-gpu 2.0.0a0. But tensorflow-cpu 2.0.0a0 works normally. After I reinstalled tensorflow-gpu1.12.0, it's fine. Even the tensorflow-gpu 1.13 will have the problem. Unknown reason. 😒

@gargisri68

This comment has been minimized.

Copy link

gargisri68 commented Mar 12, 2019

@dip-scd

This comment has been minimized.

Copy link

dip-scd commented Mar 12, 2019

@gargisri68, @yuwancumian666 solution seems to be installing CUDA 10.0, not CUDA 10.1 (actually, I left both of them installed). Windows 10 Pro in my case.

I've checked _pywrap_tensorflow_internal.pyd with Dependency Walker and there were some missing DLLs from CUDA 10 in the list (such as cudart64_100.dll). So I've installed CUDA 10.
Also, I've added "C:\Program Files\NVIDIA Corporation\NvStreamSrv" to my PATH in environment variables because one of the missing DLLs were found there. Not sure if it was necessary.

Now tensorflow is successfully imported and my model is using GPU, according to the performance monitor.

So, if you have CUDA 10.1 but not CUDA 10.0 installed, I propose to give this solution a try. I'm using tensorflow-gpu==2.0.0-a0 but I had the same issue with DLL in 1.13. So I would expect that fix will work for 1.13 as well.

@AzureByte

This comment has been minimized.

Copy link

AzureByte commented Mar 14, 2019

I had a fresh install of tensorflow-gpu and all the other latest NVIDIA requirements. Only had to downgrade CUDA to 10.0 from 10.1 for it to work.

@cl0ckt0wer

This comment has been minimized.

Copy link

cl0ckt0wer commented Mar 16, 2019

Here's a list of all of the CUDA packages to download:

https://developer.nvidia.com/cuda-toolkit-archive

@MrPiavushka

This comment has been minimized.

Copy link

MrPiavushka commented Mar 20, 2019

i had the same issue using (conda install tensorflow-gpu) solve the problem.
cuda 10.0
cudnn 7.41.5 for cuda 10.0
python 7.1

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.