New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Theano 0.9.0 - Could not initialize pygpu, support disabled #6417

Closed
pedro-dreyer opened this Issue Sep 15, 2017 · 21 comments

Comments

Projects
None yet
7 participants
@pedro-dreyer

pedro-dreyer commented Sep 15, 2017

Just installed Theano using conda and tried to check if it was using the GPU.

Here is the ouput of the test program in the tutorial :

 $ THEANO_FLAGS=device=cuda0 python gpu_tutorial1.py
ERROR (theano.gpuarray): Could not initialize pygpu, support disabled
Traceback (most recent call last):
  File "/home/deep/anaconda3/lib/python3.6/site-packages/theano/gpuarray/__init__.py", line 164, in <module>
    use(config.device)
  File "/home/deep/anaconda3/lib/python3.6/site-packages/theano/gpuarray/__init__.py", line 151, in use
    init_dev(device)
  File "/home/deep/anaconda3/lib/python3.6/site-packages/theano/gpuarray/__init__.py", line 60, in init_dev
    sched=config.gpuarray.sched)
  File "pygpu/gpuarray.pyx", line 634, in pygpu.gpuarray.init
  File "pygpu/gpuarray.pyx", line 584, in pygpu.gpuarray.pygpu_init
  File "pygpu/gpuarray.pyx", line 1057, in pygpu.gpuarray.GpuContext.__cinit__
pygpu.gpuarray.GpuArrayException: b'cuDevicePrimaryCtxRetain: CUDA_ERROR_OUT_OF_MEMORY: out of memory'
[Elemwise{exp,no_inplace}(<TensorType(float32, vector)>)]
Looping 1000 times took 3.258797 seconds
Result is [ 1.23178029  1.61879337  1.52278066 ...,  2.20771813  2.29967761
  1.62323284]
Used the cpu

Running Ubuntu 16.04/ Cuda 8.0/ python 3.6/ Titan XP

@nouiz

This comment has been minimized.

Show comment
Hide comment
@nouiz

nouiz Sep 15, 2017

Member
Member

nouiz commented Sep 15, 2017

@pedro-dreyer

This comment has been minimized.

Show comment
Hide comment
@pedro-dreyer

pedro-dreyer Sep 15, 2017

A new error appears after I update to the newest beta release:

$ THEANO_FLAGS=device=cuda0 python gpu_tutorial1.py
ERROR (theano.gpuarray): pygpu was configured but could not be imported or is too old (version 0.6 or higher required)
NoneType: None
[Elemwise{exp,no_inplace}(<TensorType(float32, vector)>)]
Looping 1000 times took 3.225526 seconds
Result is [ 1.23178029  1.61879337  1.52278066 ...,  2.20771813  2.29967761
  1.62323284]
Used the cpu

PS: I think there is an typo in the command to install the beta release. shouldn't it be libgpuarray?

pedro-dreyer commented Sep 15, 2017

A new error appears after I update to the newest beta release:

$ THEANO_FLAGS=device=cuda0 python gpu_tutorial1.py
ERROR (theano.gpuarray): pygpu was configured but could not be imported or is too old (version 0.6 or higher required)
NoneType: None
[Elemwise{exp,no_inplace}(<TensorType(float32, vector)>)]
Looping 1000 times took 3.225526 seconds
Result is [ 1.23178029  1.61879337  1.52278066 ...,  2.20771813  2.29967761
  1.62323284]
Used the cpu

PS: I think there is an typo in the command to install the beta release. shouldn't it be libgpuarray?

@nouiz

This comment has been minimized.

Show comment
Hide comment
@nouiz

nouiz Sep 15, 2017

Member
Member

nouiz commented Sep 15, 2017

@pedro-dreyer

This comment has been minimized.

Show comment
Hide comment
@pedro-dreyer

pedro-dreyer Sep 15, 2017

0.7.2

Note that I needed to change the command slightly to

python -c "import pygpu; print(pygpu.__version__)"

Since I am using python 3.6

pedro-dreyer commented Sep 15, 2017

0.7.2

Note that I needed to change the command slightly to

python -c "import pygpu; print(pygpu.__version__)"

Since I am using python 3.6

@notoraptor

This comment has been minimized.

Show comment
Hide comment
@notoraptor

notoraptor Sep 15, 2017

Collaborator

Can you check your theano version too ?

python -c "import theano; print(theano.__version__)"
Collaborator

notoraptor commented Sep 15, 2017

Can you check your theano version too ?

python -c "import theano; print(theano.__version__)"
@pedro-dreyer

This comment has been minimized.

Show comment
Hide comment
@pedro-dreyer

pedro-dreyer Sep 15, 2017

0.9.0.dev-c697eeab84e5b8a74908da654b66ec9eca4f1291

pedro-dreyer commented Sep 15, 2017

0.9.0.dev-c697eeab84e5b8a74908da654b66ec9eca4f1291

@notoraptor

This comment has been minimized.

Show comment
Hide comment
@notoraptor

notoraptor Sep 15, 2017

Collaborator

Ok. So, theano is still not updated to the last version (should be 0.10...).

Can you retry conda update theano -c mila-udem and post the output ?

Collaborator

notoraptor commented Sep 15, 2017

Ok. So, theano is still not updated to the last version (should be 0.10...).

Can you retry conda update theano -c mila-udem and post the output ?

@nouiz

This comment has been minimized.

Show comment
Hide comment
@nouiz

nouiz Sep 15, 2017

Member
Member

nouiz commented Sep 15, 2017

@pedro-dreyer

This comment has been minimized.

Show comment
Hide comment
@pedro-dreyer

pedro-dreyer Sep 15, 2017

Can you retry conda update theano -c mila-udem and post the output ?

$ conda update theano -c mila-udem
Fetching package metadata ...........
Solving package specifications: .

# All requested packages already installed.
# packages in environment at /home/deep/anaconda3:
#
theano                    0.9.0                    py36_0  

Before doing the conda update, make sure to uninstall Theano first.

$ conda uninstall theano
Fetching package metadata .........
Solving package specifications: .

Package plan for package removal in environment /home/deep/anaconda3:

The following packages will be REMOVED:

    theano: 0.9.0-py36_0

Proceed ([y]/n)? y

Then conda install theano -c mila-udem

$ conda install theano -c mila-udem
Fetching package metadata ...........
Solving package specifications: .

Package plan for installation in environment /home/deep/anaconda3:

The following NEW packages will be INSTALLED:

    theano: 0.9.0-py36_0

Proceed ([y]/n)? y

Even after uninstalling and reinstalling the version is still 0.9.0

pedro-dreyer commented Sep 15, 2017

Can you retry conda update theano -c mila-udem and post the output ?

$ conda update theano -c mila-udem
Fetching package metadata ...........
Solving package specifications: .

# All requested packages already installed.
# packages in environment at /home/deep/anaconda3:
#
theano                    0.9.0                    py36_0  

Before doing the conda update, make sure to uninstall Theano first.

$ conda uninstall theano
Fetching package metadata .........
Solving package specifications: .

Package plan for package removal in environment /home/deep/anaconda3:

The following packages will be REMOVED:

    theano: 0.9.0-py36_0

Proceed ([y]/n)? y

Then conda install theano -c mila-udem

$ conda install theano -c mila-udem
Fetching package metadata ...........
Solving package specifications: .

Package plan for installation in environment /home/deep/anaconda3:

The following NEW packages will be INSTALLED:

    theano: 0.9.0-py36_0

Proceed ([y]/n)? y

Even after uninstalling and reinstalling the version is still 0.9.0

@nouiz

This comment has been minimized.

Show comment
Hide comment
@nouiz

nouiz Sep 15, 2017

Member
Member

nouiz commented Sep 15, 2017

@pedro-dreyer

This comment has been minimized.

Show comment
Hide comment
@pedro-dreyer

pedro-dreyer Sep 15, 2017

I did

conda update -c mila-udem/label/pre theano libgpuarray pygpu

which updated the packages to the following versions:

libgpuarray               0.7.2.dev1           h3972f8e_0    mila-udem/label/pre
pygpu                     0.7.2.dev1       py36he6660cf_0    mila-udem/label/pre
theano                    0.10.0beta2        pyha8067b5_0    mila-udem/label/pre

A new error happens now:

$ THEANO_FLAGS=device=cuda0 python gpu_tutorial1.py
ERROR (theano.gpuarray): Could not initialize pygpu, support disabled
Traceback (most recent call last):
  File "/home/deep/anaconda3/lib/python3.6/site-packages/theano/gpuarray/__init__.py", line 204, in <module>
    use(config.device)
  File "/home/deep/anaconda3/lib/python3.6/site-packages/theano/gpuarray/__init__.py", line 191, in use
    init_dev(device, preallocate=preallocate)
  File "/home/deep/anaconda3/lib/python3.6/site-packages/theano/gpuarray/__init__.py", line 60, in init_dev
    pygpu_version = pygpu_parse_version(pygpu.__version__)
  File "/home/deep/anaconda3/lib/python3.6/site-packages/theano/gpuarray/__init__.py", line 51, in pygpu_parse_version
    patch = int(pieces[2].split('+', 1)[0])
ValueError: invalid literal for int() with base 10: '2.dev1'
[Elemwise{exp,no_inplace}(<TensorType(float32, vector)>)]
Looping 1000 times took 3.038952 seconds
Result is [ 1.23178029  1.61879337  1.52278066 ...,  2.20771813  2.29967761
  1.62323284]
Used the cpu

pedro-dreyer commented Sep 15, 2017

I did

conda update -c mila-udem/label/pre theano libgpuarray pygpu

which updated the packages to the following versions:

libgpuarray               0.7.2.dev1           h3972f8e_0    mila-udem/label/pre
pygpu                     0.7.2.dev1       py36he6660cf_0    mila-udem/label/pre
theano                    0.10.0beta2        pyha8067b5_0    mila-udem/label/pre

A new error happens now:

$ THEANO_FLAGS=device=cuda0 python gpu_tutorial1.py
ERROR (theano.gpuarray): Could not initialize pygpu, support disabled
Traceback (most recent call last):
  File "/home/deep/anaconda3/lib/python3.6/site-packages/theano/gpuarray/__init__.py", line 204, in <module>
    use(config.device)
  File "/home/deep/anaconda3/lib/python3.6/site-packages/theano/gpuarray/__init__.py", line 191, in use
    init_dev(device, preallocate=preallocate)
  File "/home/deep/anaconda3/lib/python3.6/site-packages/theano/gpuarray/__init__.py", line 60, in init_dev
    pygpu_version = pygpu_parse_version(pygpu.__version__)
  File "/home/deep/anaconda3/lib/python3.6/site-packages/theano/gpuarray/__init__.py", line 51, in pygpu_parse_version
    patch = int(pieces[2].split('+', 1)[0])
ValueError: invalid literal for int() with base 10: '2.dev1'
[Elemwise{exp,no_inplace}(<TensorType(float32, vector)>)]
Looping 1000 times took 3.038952 seconds
Result is [ 1.23178029  1.61879337  1.52278066 ...,  2.20771813  2.29967761
  1.62323284]
Used the cpu
@nouiz

This comment has been minimized.

Show comment
Hide comment
@nouiz

nouiz Sep 18, 2017

Member

Thanks for the report. This is now fixed in this PR on Theano:

#6424

It was our first libgpuarray dev version. So it seam Theano didn't parsed that well!

Member

nouiz commented Sep 18, 2017

Thanks for the report. This is now fixed in this PR on Theano:

#6424

It was our first libgpuarray dev version. So it seam Theano didn't parsed that well!

@lamblin

This comment has been minimized.

Show comment
Hide comment
@lamblin

lamblin Sep 22, 2017

Member

Closed by #6424.

Member

lamblin commented Sep 22, 2017

Closed by #6424.

@lamblin lamblin closed this Sep 22, 2017

@XiaoqiaoChen

This comment has been minimized.

Show comment
Hide comment
@XiaoqiaoChen

XiaoqiaoChen Oct 4, 2017

@nouiz Why do I still have the totally same problems?

XiaoqiaoChen commented Oct 4, 2017

@nouiz Why do I still have the totally same problems?

@nouiz

This comment has been minimized.

Show comment
Hide comment
@nouiz

nouiz Oct 5, 2017

Member

Which error? There was a few in this issue. Can you give your full error? small detail could give different reason. Do you use the github version of Theano? When did you update it? If not, which version of Theano do you use?

Member

nouiz commented Oct 5, 2017

Which error? There was a few in this issue. Can you give your full error? small detail could give different reason. Do you use the github version of Theano? When did you update it? If not, which version of Theano do you use?

@msiraj83

This comment has been minimized.

Show comment
Hide comment
@msiraj83

msiraj83 Nov 7, 2017

@nouiz ...thanks .... it work for me and the remove the error....

conda update theano pygpu libgpuarray -c mila-udem

msiraj83 commented Nov 7, 2017

@nouiz ...thanks .... it work for me and the remove the error....

conda update theano pygpu libgpuarray -c mila-udem

@bricksdont

This comment has been minimized.

Show comment
Hide comment
@bricksdont

bricksdont Nov 14, 2017

@nouiz I'm also getting

ERROR (theano.gpuarray): Could not initialize pygpu, support disabled

Does the newest release candidate require a specific version of cuDNN?

bricksdont commented Nov 14, 2017

@nouiz I'm also getting

ERROR (theano.gpuarray): Could not initialize pygpu, support disabled

Does the newest release candidate require a specific version of cuDNN?

@nouiz

This comment has been minimized.

Show comment
Hide comment
@nouiz

nouiz Nov 14, 2017

Member

This error isn't related to cudnn. try just "import pygpu" it could give you a better error. Theano 1.0rc1 need pygpu version 0.7*

Member

nouiz commented Nov 14, 2017

This error isn't related to cudnn. try just "import pygpu" it could give you a better error. Theano 1.0rc1 need pygpu version 0.7*

@bricksdont

This comment has been minimized.

Show comment
Hide comment
@bricksdont

bricksdont Nov 14, 2017

UPDATE: I found a solution, the problem was that cuDNN is installed in a non-standard location. This works:

% cat ~/.theanorc
[dnn]
include_path = /usr/include
library_path = /usr/lib/x86_64-linux-gnu

$ python -c "import pygpu"

does not give me an error at all, but:

$ python -c "import theano"
WARNING (theano.tensor.blas): Using NumPy C-API based implementation for BLAS functions.
ERROR (theano.gpuarray): Could not initialize pygpu, support disabled
Traceback (most recent call last):
  File "/mnt/storage/clfiles/projects/clresources/resources/applications/toolkit/anaconda/v2curr/lib/python2.7/site-packages/theano/gpuarray/__init__.py", line 220, in <module>
    use(config.device)
  File "/mnt/storage/clfiles/projects/clresources/resources/applications/toolkit/anaconda/v2curr/lib/python2.7/site-packages/theano/gpuarray/__init__.py", line 207, in use
    init_dev(device, preallocate=preallocate)
  File "/mnt/storage/clfiles/projects/clresources/resources/applications/toolkit/anaconda/v2curr/lib/python2.7/site-packages/theano/gpuarray/__init__.py", line 110, in init_dev
    context.cudnn_handle = dnn._make_handle(context)
  File "/mnt/storage/clfiles/projects/clresources/resources/applications/toolkit/anaconda/v2curr/lib/python2.7/site-packages/theano/gpuarray/dnn.py", line 118, in _make_handle
    cudnn = _dnn_lib()
  File "/mnt/storage/clfiles/projects/clresources/resources/applications/toolkit/anaconda/v2curr/lib/python2.7/site-packages/theano/gpuarray/dnn.py", line 105, in _dnn_lib
    raise RuntimeError('Could not load cudnn library')
RuntimeError: Could not load cudnn library

conda installed packages:

$ conda list | grep "gpu\|theano"
libgpuarray               0.7.4                hdd06f6c_0    mila-udem
pygpu                     0.7.4            py27h5d96ec3_0    mila-udem
theano                    1.0.0rc1         py27h672a420_0    mila-udem

cuDNN:

$ cat /usr/include/cudnn.h | grep CUDNN_MAJOR -A 2
#define CUDNN_MAJOR      5
#define CUDNN_MINOR      1
#define CUDNN_PATCHLEVEL 10

bricksdont commented Nov 14, 2017

UPDATE: I found a solution, the problem was that cuDNN is installed in a non-standard location. This works:

% cat ~/.theanorc
[dnn]
include_path = /usr/include
library_path = /usr/lib/x86_64-linux-gnu

$ python -c "import pygpu"

does not give me an error at all, but:

$ python -c "import theano"
WARNING (theano.tensor.blas): Using NumPy C-API based implementation for BLAS functions.
ERROR (theano.gpuarray): Could not initialize pygpu, support disabled
Traceback (most recent call last):
  File "/mnt/storage/clfiles/projects/clresources/resources/applications/toolkit/anaconda/v2curr/lib/python2.7/site-packages/theano/gpuarray/__init__.py", line 220, in <module>
    use(config.device)
  File "/mnt/storage/clfiles/projects/clresources/resources/applications/toolkit/anaconda/v2curr/lib/python2.7/site-packages/theano/gpuarray/__init__.py", line 207, in use
    init_dev(device, preallocate=preallocate)
  File "/mnt/storage/clfiles/projects/clresources/resources/applications/toolkit/anaconda/v2curr/lib/python2.7/site-packages/theano/gpuarray/__init__.py", line 110, in init_dev
    context.cudnn_handle = dnn._make_handle(context)
  File "/mnt/storage/clfiles/projects/clresources/resources/applications/toolkit/anaconda/v2curr/lib/python2.7/site-packages/theano/gpuarray/dnn.py", line 118, in _make_handle
    cudnn = _dnn_lib()
  File "/mnt/storage/clfiles/projects/clresources/resources/applications/toolkit/anaconda/v2curr/lib/python2.7/site-packages/theano/gpuarray/dnn.py", line 105, in _dnn_lib
    raise RuntimeError('Could not load cudnn library')
RuntimeError: Could not load cudnn library

conda installed packages:

$ conda list | grep "gpu\|theano"
libgpuarray               0.7.4                hdd06f6c_0    mila-udem
pygpu                     0.7.4            py27h5d96ec3_0    mila-udem
theano                    1.0.0rc1         py27h672a420_0    mila-udem

cuDNN:

$ cat /usr/include/cudnn.h | grep CUDNN_MAJOR -A 2
#define CUDNN_MAJOR      5
#define CUDNN_MINOR      1
#define CUDNN_PATCHLEVEL 10
@nouiz

This comment has been minimized.

Show comment
Hide comment
@nouiz

nouiz Nov 14, 2017

Member
Member

nouiz commented Nov 14, 2017

@bricksdont

This comment has been minimized.

Show comment
Hide comment
@bricksdont

bricksdont Nov 14, 2017

Yes, that's the culprit. See my update above - thanks for your help!

bricksdont commented Nov 14, 2017

Yes, that's the culprit. See my update above - thanks for your help!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment