ERROR (theano.gof.opt): Optimization failure due to: constant_folding (python 3.6 issue ?)

[jc-Rosichini]

Hi I just installed a new windows 10 configuration with exactly the same theano/lasagne code than runs properly on another similar machine. Both using the latest versions of theano/lasagne and gpuarray.

On the new one at compile time, I'm getting a bunch or errors:
ERROR (theano.gof.opt): Optimization failure due to: constant_folding
....
TypeError: ("a bytes-like object is required, not 'str'", '[GpuFromHost(TensorConstant{(1,) of 1e-06})]')

(see below and attached dump)

which doesn't occur on the other configuration.

As far as I can see the only difference is that the new one uses Anaconda Python 3.6 versus Anaconda Python 3.5.

Here is part of the dump:

Python 3.6.0 |Anaconda 4.3.1 (64-bit)| (default, Dec 23 2016, 11:57:41) [MSC v.1900 64 bit (AMD64)] on win32
Using cuDNN version 5105 on context None
Mapped name None to device cuda0: GeForce GTX 1080 (0000:01:00.0)
theano: 0.9.0.dev-b3d19ce8e8d7cdf87c3f84599be81647803aa1f6
numpy: 1.11.3
pygpu: 0.6.2
lasagne: 0.2.dev1

Compilation starts at 2017-03-25 19:57:39
optimizer: rmsprop, initial learning rate: 0.001
ERROR (theano.gof.opt): Optimization failure due to: constant_folding
ERROR (theano.gof.opt): node: GpuFromHost(TensorConstant{(1,) of 1e-06})
ERROR (theano.gof.opt): TRACEBACK:
ERROR (theano.gof.opt): Traceback (most recent call last):
File "C:\Users\MachineLearning\AppData\Local\Continuum\Anaconda3\lib\site-packages\theano\gof\opt.py", line 1996, in process_node
replacements = lopt.transform(node)
File "C:\Users\MachineLearning\AppData\Local\Continuum\Anaconda3\lib\site-packages\theano\tensor\opt.py", line 6497, in constant_folding
no_recycling=[], impl=impl)
File "C:\Users\MachineLearning\AppData\Local\Continuum\Anaconda3\lib\site-packages\theano\gof\op.py", line 935, in make_thunk
no_recycling)
File "C:\Users\MachineLearning\AppData\Local\Continuum\Anaconda3\lib\site-packages\theano\gof\op.py", line 839, in make_c_thunk
output_storage=node_output_storage)
File "C:\Users\MachineLearning\AppData\Local\Continuum\Anaconda3\lib\site-packages\theano\gof\cc.py", line 1190, in make_thunk
keep_lock=keep_lock)
File "C:\Users\MachineLearning\AppData\Local\Continuum\Anaconda3\lib\site-packages\theano\gof\cc.py", line 1131, in compile
keep_lock=keep_lock)
File "C:\Users\MachineLearning\AppData\Local\Continuum\Anaconda3\lib\site-packages\theano\gof\cc.py", line 1586, in cthunk_factory
key=key, lnk=self, keep_lock=keep_lock)
File "C:\Users\MachineLearning\AppData\Local\Continuum\Anaconda3\lib\site-packages\theano\gof\cmodule.py", line 1159, in module_from_key
module = lnk.compile_cmodule(location)
File "C:\Users\MachineLearning\AppData\Local\Continuum\Anaconda3\lib\site-packages\theano\gof\cc.py", line 1489, in compile_cmodule
preargs=preargs)
File "C:\Users\MachineLearning\AppData\Local\Continuum\Anaconda3\lib\site-packages\theano\gof\cmodule.py", line 2309, in compile_str
tf.write('===============================\n')
File "C:\Users\MachineLearning\AppData\Local\Continuum\Anaconda3\lib\tempfile.py", line 483, in func_wrapper
return func(*args, **kwargs)
TypeError: ("a bytes-like object is required, not 'str'", '[GpuFromHost(TensorConstant{(1,) of 1e-06})]')

ERROR (theano.gof.opt): Optimization failure due to: constant_folding
ERROR (theano.gof.opt): node: GpuFromHost(TensorConstant{(1,) of 0.9})
ERROR (theano.gof.opt): TRACEBACK:
ERROR (theano.gof.opt): Traceback (most recent call last):
File "C:\Users\MachineLearning\AppData\Local\Continuum\Anaconda3\lib\site-packages\theano\gof\opt.py", line 1996, in process_node
replacements = lopt.transform(node)

[jc-Rosichini]

dump file
theano_pb.txt

[jc-Rosichini]

The compile is ok if done in CPU mode

Compilation starts at 2017-03-25 20:29:11
optimizer: rmsprop, initial learning rate: 0.001
WARNING (theano.gof.compilelock): Overriding existing lock by dead process '11720' (I am process '11424')
train compilation took 171.393 seconds
Compilation starts at 2017-03-25 20:32:04

[jc-Rosichini]

The compile is ok too if using device=gpu

The issue is limited to new backend device=cuda0

[nouiz]

This error hide another one. Can you try this PR that remove the second error and will allow you to see the first error?

#5773

[nouiz]

Another fix was merged. Update Theano and you should see the original error.

[jc-Rosichini]

Just updated Theano from github

Here is the current status:

Compilation starts at 2017-03-27 21:54:22
optimizer: rmsprop, initial learning rate: 0.001

You can find the C code in this temporary file: C:\Users\MACHIN~1\AppData\Local\Temp\theano_compilation_error_uqtmxc29
ERROR (theano.gof.opt): Optimization failure due to: constant_folding
ERROR (theano.gof.opt): node: GpuFromHost(TensorConstant{(1,) of 1e-06})
ERROR (theano.gof.opt): TRACEBACK:
ERROR (theano.gof.opt): Traceback (most recent call last):
File "C:\Users\MachineLearning\AppData\Local\Continuum\Anaconda3\lib\site-packages\theano\gof\opt.py", line 1996, in process_node
replacements = lopt.transform(node)
File "C:\Users\MachineLearning\AppData\Local\Continuum\Anaconda3\lib\site-packages\theano\tensor\opt.py", line 6497, in constant_folding
no_recycling=[], impl=impl)
File "C:\Users\MachineLearning\AppData\Local\Continuum\Anaconda3\lib\site-packages\theano\gof\op.py", line 935, in make_thunk
no_recycling)
File "C:\Users\MachineLearning\AppData\Local\Continuum\Anaconda3\lib\site-packages\theano\gof\op.py", line 839, in make_c_thunk
output_storage=node_output_storage)
File "C:\Users\MachineLearning\AppData\Local\Continuum\Anaconda3\lib\site-packages\theano\gof\cc.py", line 1190, in make_thunk
keep_lock=keep_lock)
File "C:\Users\MachineLearning\AppData\Local\Continuum\Anaconda3\lib\site-packages\theano\gof\cc.py", line 1131, in compile
keep_lock=keep_lock)
File "C:\Users\MachineLearning\AppData\Local\Continuum\Anaconda3\lib\site-packages\theano\gof\cc.py", line 1586, in cthunk_factory
key=key, lnk=self, keep_lock=keep_lock)
File "C:\Users\MachineLearning\AppData\Local\Continuum\Anaconda3\lib\site-packages\theano\gof\cmodule.py", line 1159, in module_from_key
module = lnk.compile_cmodule(location)
File "C:\Users\MachineLearning\AppData\Local\Continuum\Anaconda3\lib\site-packages\theano\gof\cc.py", line 1489, in compile_cmodule
preargs=preargs)
File "C:\Users\MachineLearning\AppData\Local\Continuum\Anaconda3\lib\site-packages\theano\gof\cmodule.py", line 2339, in compile_str
(status, compile_stderr.replace('\n', '. ')))
Exception: ('Compilation failed (return status=1): D:\theano_compiledir\compiledir_Windows-10-10.0.14393-SP0-Intel64_Family_6_Model_94_Stepping_3_GenuineIntel-3.6.0-64\tmpkbn5jd8o\mod.cpp:4:28: fatal error: gpuarray/array.h: No such file or directory\r. compilation terminated.\r. ', '[GpuFromHost(TensorConstant{(1,) of 1e-06})]')

You can find the C code in this temporary file: C:\Users\MACHIN~1\AppData\Local\Temp\theano_compilation_error_bv5sk_4z
ERROR (theano.gof.opt): Optimization failure due to: constant_folding
ERROR (theano.gof.opt): node: GpuFromHost(TensorConstant{(1,) of 0.9})
ERROR (theano.gof.opt): TRACEBACK:
ERROR (theano.gof.opt): Traceback (most recent call last):
File "C:\Users\MachineLearning\AppData\Local\Continuum\Anaconda3\lib\site-packages\theano\gof\opt.py", line 1996, in process_node
replacements = lopt.transform(node)
File "C:\Users\MachineLearning\AppData\Local\Continuum\Anaconda3\lib\site-packages\theano\tensor\opt.py", line 6497, in constant_folding
no_recycling=[], impl=impl)
File "C:\Users\MachineLearning\AppData\Local\Continuum\Anaconda3\lib\site-packages\theano\gof\op.py", line 935, in make_thunk
no_recycling)
File "C:\Users\MachineLearning\AppData\Local\Continuum\Anaconda3\lib\site-packages\theano\gof\op.py", line 839, in make_c_thunk
output_storage=node_output_storage)
File "C:\Users\MachineLearning\AppData\Local\Continuum\Anaconda3\lib\site-packages\theano\gof\cc.py", line 1190, in make_thunk
keep_lock=keep_lock)
File "C:\Users\MachineLearning\AppData\Local\Continuum\Anaconda3\lib\site-packages\theano\gof\cc.py", line 1131, in compile
keep_lock=keep_lock)
File "C:\Users\MachineLearning\AppData\Local\Continuum\Anaconda3\lib\site-packages\theano\gof\cc.py", line 1586, in cthunk_factory
key=key, lnk=self, keep_lock=keep_lock)
File "C:\Users\MachineLearning\AppData\Local\Continuum\Anaconda3\lib\site-packages\theano\gof\cmodule.py", line 1159, in module_from_key
module = lnk.compile_cmodule(location)
File "C:\Users\MachineLearning\AppData\Local\Continuum\Anaconda3\lib\site-packages\theano\gof\cc.py", line 1489, in compile_cmodule
preargs=preargs)
File "C:\Users\MachineLearning\AppData\Local\Continuum\Anaconda3\lib\site-packages\theano\gof\cmodule.py", line 2339, in compile_str
(status, compile_stderr.replace('\n', '. ')))
Exception: ('Compilation failed (return status=1): D:\theano_compiledir\compiledir_Windows-10-10.0.14393-SP0-Intel64_Family_6_Model_94_Stepping_3_GenuineIntel-3.6.0-64\tmpe_c4xe85\mod.cpp:4:28: fatal error: gpuarray/array.h: No such file or directory\r. compilation terminated.\r. ', '[GpuFromHost(TensorConstant{(1,) of 0.9})]')

[jc-Rosichini]

This seems related to the way pygpu is installed on Windows.
Some directories are not clearly accessible.

(1) Whenever libgpuarray is compiled from its git source code (using in this cas Visual Studio 2015 and Cmake), it gets installed as follows:

Build started: Project: INSTALL, Configuration: Release x64 ------
1> -- Install configuration: "Release"
1> -- Installing: C:/Program Files/libgpuarray/include/gpuarray/array.h
1> -- Installing: C:/Program Files/libgpuarray/include/gpuarray/blas.h
1> -- Installing: C:/Program Files/libgpuarray/include/gpuarray/collectives.h
1> -- Installing: C:/Program Files/libgpuarray/include/gpuarray/buffer.h
1> -- Installing: C:/Program Files/libgpuarray/include/gpuarray/buffer_blas.h
1> -- Installing: C:/Program Files/libgpuarray/include/gpuarray/buffer_collectives.h
1> -- Installing: C:/Program Files/libgpuarray/include/gpuarray/abi_version.h
1> -- Installing: C:/Program Files/libgpuarray/include/gpuarray/config.h
1> -- Installing: C:/Program Files/libgpuarray/include/gpuarray/elemwise.h
1> -- Installing: C:/Program Files/libgpuarray/include/gpuarray/error.h
1> -- Installing: C:/Program Files/libgpuarray/include/gpuarray/extension.h
1> -- Installing: C:/Program Files/libgpuarray/include/gpuarray/ext_cuda.h
1> -- Installing: C:/Program Files/libgpuarray/include/gpuarray/kernel.h
1> -- Installing: C:/Program Files/libgpuarray/include/gpuarray/types.h
1> -- Installing: C:/Program Files/libgpuarray/include/gpuarray/util.h
1> -- Installing: C:/Program Files/libgpuarray/include/gpuarray/wincompat/stdint.h
1> -- Installing: C:/Program Files/libgpuarray/lib/gpuarray.lib
1> -- Installing: C:/Program Files/libgpuarray/bin/gpuarray.dll
1> -- Installing: C:/Program Files/libgpuarray/lib/gpuarray-static.lib

(2) Running test via python -c "import pygpu;pygpu.test()" is fine provided the environment variable
GPUARRAY_TEST_DEVICE is set to cuda0

python -c "import pygpu;pygpu.test()"
pygpu is installed in C:\Users\MachineLearning\AppData\Local\Continuum\Anaconda3\lib\site-packages\pygpu-0.6.2-py3.6-win-amd64.egg\pygpu
NumPy version 1.11.3
NumPy relaxed strides checking option: False
NumPy is installed in C:\Users\MachineLearning\AppData\Local\Continuum\Anaconda3\lib\site-packages\numpy
Python version 3.6.0 |Anaconda 4.3.1 (64-bit)| (default, Dec 23 2016, 11:57:41) [MSC v.1900 64 bit (AMD64)]
nose version 1.3.7
*** Testing for GeForce GTX 1080
...... Ran 7000 tests in 204.289s!!

(3) I did manage to instruct the Theano compiler to fetch the header gpu/array.h by creating the environment variable:
CPLUS_INCLUDE_PATH to C:/Program Files/libgpuarray/include
this fix the previous reported error

(4) then creating the environment variable LIBRARY_PATH pointing to C:/Program Files/libgpuarray/lib/ fixes the remaining error

BOTTOM LINE: For the benefit of users that are generating libgpuarray from thje gpuarray github directory on Windows, I suggest to clarify the procedure (ie directories and environment variables)