ImportError when importing Theano on Cygwin: No module named cutils_ext #2253

Closed
Wainberg opened this Issue Nov 13, 2014 · 27 comments

Projects

None yet

7 participants

@Wainberg

I just installed the trunk version of Theano (commit c416c5e) on Cygwin and receive the error "ImportError: No module named cutils_ext" when importing Theano. However, cutils_ext is correctly created and can be imported after the fact:

Python 2.7.8 (default, Jul 25 2014, 14:04:36)
[GCC 4.8.3] on cygwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import theano
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "theano/__init__.py", line 72, in <module>
    from theano.scan_module import scan, map, reduce, foldl, foldr, clone
  File "theano/scan_module/__init__.py", line 41, in <module>
    from theano.scan_module import scan_opt
  File "theano/scan_module/scan_opt.py", line 65, in <module>
    from theano import tensor
  File "theano/tensor/__init__.py", line 7, in <module>
    from theano.tensor.subtensor import *
  File "theano/tensor/subtensor.py", line 26, in <module>
    import theano.gof.cutils  # needed to import cutils_ext
  File "theano/gof/cutils.py", line 295, in <module>
    compile_cutils()
  File "theano/gof/cutils.py", line 260, in compile_cutils
    preargs=args)
  File "theano/gof/cmodule.py", line 2014, in compile_str
    return dlimport(lib_filename)
  File "theano/gof/cmodule.py", line 289, in dlimport
    rval = __import__(module_name, {}, {}, [module_name])
ImportError: No module named cutils_ext
>>> import cutils_ext
>>> cutils_ext.__file__
'/home/User/.theano/compiledir_CYGWIN_NT-6.1-1.7.32-0.274-5-3-x86_64-64bit--2.7.8-64/cutils_ext/__init__.py'
>>> import os; os.listdir('/home/User/.theano/compiledir_CYGWIN_NT-6.1-1.7.32-0.274-5-3-x86_64-64bit--2.7.8-64/cutils_ext')
['cutils_ext.so', 'mod.cpp', '__init__.py', '__init__.pyc']

This may be related to this StackOverflow post and this earlier bug report.

@lamblin
Member
lamblin commented Nov 13, 2014

What happens if you try the following? Is there an error message?

>>> from cutils_ext.cutils_ext import *
@Wainberg

Yes, I get ImportError: No module named cutils_ext.

@lamblin
Member
lamblin commented Nov 17, 2014

It is possible that a wrong cutils_ext was imported, or that cutils_ext.so was not able to be loaded.
Can you post the result of dir(cutils_ext), after you have imported it?

@Wainberg

It looks like the library is not being loaded properly.

>>> dir(cutils_ext)
['__builtins__', '__doc__', '__file__', '__name__', '__package__', '__path__']
@lamblin
Member
lamblin commented Nov 25, 2014

OK, so I just tried it, and I have the same result when I call dir on the parent cutils_ext, so the problem is probably not from there.

@lamblin
Member
lamblin commented Nov 25, 2014

Oh wait, I just realized that in the cutils_ext/ directory, you have 'cutils_ext.so', but Python on Windows would expect it to be cutils_ext.pyd! (and maybe for there to be a cutils_ext.dll as well, I don't recall).
So I guess the logic to determine the appropriate file extension given the platform has to be adjusted for Cygwin. Can you have a look?

@Wainberg

Good catch! Manually renaming cutils_ext.so to cutils_ext.dll allowed Theano to import.

@nouiz
Member
nouiz commented Nov 28, 2014

Can you help us find the fix? Can you delete the folder where
cutils_ext.dll is, put a break point in theano/gof/cmodule.py in the method
compile_str. Can you check in that method why we generate a .so file?

Normally, in that cmodule file, it is the method get_lib_extension() that
generate the extention. But it is a trivial method:

def get_lib_extension():
"""Return the platform-dependent extension for compiled modules."""
if sys.platform == 'win32':
return 'pyd'
else:
return 'so'

So I do not understand why a .so could have happened in your case.

Fred

On Tue, Nov 25, 2014 at 10:43 PM, Wainberg notifications@github.com wrote:

Good catch! Manually renaming cutils_ext.so to cutils_ext.dll allowed
Theano to import.


Reply to this email directly or view it on GitHub
#2253 (comment).

@Wainberg

That's easy: sys.platform == 'cygwin' on cygwin! So the fix is presumably (haven't tested):

def get_lib_extension():
    """Return the platform-dependent extension for compiled modules."""
    if sys.platform == 'win32':
        return 'pyd'
    elif sys.platform == 'cygwin':
        return 'dll'
    else:
        return 'so'
@nouiz
Member
nouiz commented Jan 17, 2015

I made a guess fix to theano in this PR:

#2413

Can you test it? To test it correctly, you need to delete Theano compiledir.

thanks

On Sat, Nov 29, 2014 at 12:06 PM, Wainberg notifications@github.com wrote:

That's easy: sys.platform == 'cygwin' on cygwin!


Reply to this email directly or view it on GitHub
#2253 (comment).

@Wainberg

@nouiz It's still broken with the latest version of Theano. I think that fix is incorrect because the extension should be dll for cygwin, not pyd. See my suggested fix above.

@abergeron
Member

No, python uses .pyd for compiled modules on windows. So the correct fix is to use that.

@Wainberg

@abergeron Manually renaming the file to .dll worked for me, and .pyd didn't.

@nouiz
Member
nouiz commented Jan 20, 2015

Is it possible that the cygwin python doesn't work the same way as windows
python?

On Mon, Jan 19, 2015 at 1:57 PM, Wainberg notifications@github.com wrote:

@abergeron https://github.com/abergeron Manually renaming the file to
.dll worked for me, and .pyd didn't.


Reply to this email directly or view it on GitHub
#2253 (comment).

@abergeron
Member

Apparently, yes. It's somewhat crippled in that only the fallback name (.dll) works for compiled modules.

@ih4cku
ih4cku commented May 31, 2015

@nouiz Theano 0.7.0 still has the issue on Cygwin. Renaming pyd to dll also works for me. @Wainberg 's code is just fine.

@nouiz
Member
nouiz commented Jun 2, 2015

#2990

@ih4cku, can you delete your theano cache and try this PR? Just to be sure I get it right. There is different case (like cuda_ndarray that is a python lib and a shared lib other python lib link again, a python lib that do not link to other dll, a python lib that link to cuda_ndarray, and finaly, like cuda_convnet, a dll lib, where python lib link again). I would like to be sure they are all fine with .dll.

thanks

@rilut
Contributor
rilut commented Feb 14, 2016

Affects me as well, I'm on Windows 10 x64. It always fails whether I'm on Anaconda Python2 or Python3, installing from pip (0.7.0) or github (0.8.0-dev).

Btw, when I search cutils* in my Anaconda2 folder on Windows Explorer, I only found:

  • cutils.pyc on C:\Anaconda2\Lib\site-packages\theano\gof
  • cutils.py on C:\Anaconda2\Lib\site-packages\theano\gof
  • cutils.pyd on C:\Anaconda2\Lib\site-packages\sqlalchemy

No other pyd/so/dll files

Update:

There is cutils_ext.pyd in C:\Users\mrlut\AppData\Local\Theano\compiledir_Windows-10-10.0.10586-Intel64_Family_6_Model_42_Stepping_7_GenuineIntel-2.7.11-64\cutils_ext. But I keep failed importing Theano

@nouiz nouiz added the Windows label Feb 17, 2016
@nouiz nouiz modified the milestone: 0.8, 0.9 Feb 17, 2016
@darkaeon10
darkaeon10 commented Nov 11, 2016 edited

I'm having the same problem as @rilut.

There is a 'cutils_ext.pyd' in 'C:\Users\user\AppData\Local\Theano\compiledir_Windows-10-10.0.14393-SP0-Intel64_Family_6_Model_69_Stepping_1_GenuineIntel-3.5.2-64\cutils_ext'.

Renaming that file to 'cutils_ext.dll' as suggested by @Wainberg doesn't work for me. I also tried editing the get_lib_extension() function in cmodule.py according to @Wainberg 's suggested fix, but still no luck. I'm on Windows 10 x64 by the way.

Any updates on this?

@nouiz
Member
nouiz commented Nov 11, 2016

The update would be to not use cygwin and use this PR with anaconda:

#5207

We didn't tested that with the GPU.

On Fri, Nov 11, 2016 at 10:24 AM, Alron notifications@github.com wrote:

I'm having the same problem as @rilut https://github.com/rilut.

There is a 'cutils_ext.pyd' in 'C:\Users\user\AppData\Local
Theano\compiledir_Windows-10-10.0.14393-SP0-Intel64_Family_
6_Model_69_Stepping_1_GenuineIntel-3.5.2-64\cutils_ext'.

Renaming that file to 'cutils_ext.dll' as suggested by @Wainberg
https://github.com/Wainberg doesn't work for me. I also tried editing
the get_lib_extension() function in cmodule.py accordingt o @Wainberg
https://github.com/Wainberg 's suggested fix, but still no luck. I'm on
Windows 10 x64 by the way.

Any updates on this?


You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#2253 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/AALC-1ziBFpEMZDDj_wMB8fEgx33pMdmks5q9IiTgaJpZM4C6bxm
.

@rilut
Contributor
rilut commented Nov 11, 2016 edited

Hi, so it was 9 months after my comment and I couldn't surely remember if:

  • I can solve that problem or not; or
  • whether it solved itself or by restarting; or
  • it's related to compilers (mingw or TDM-GCC); or
  • it's the same problem with my current laptop. But I think it's the same cutils_ext problem

So, on my current laptop, I've failed to install Theano too.
Python said a dll/pyd missing, even though it exists when I opened it in Windows Explorer. When I copy-pasted and cd-ed the directory in the command prompt, it errors, because of... a space. Thus I realized, the culprit was a space in the user path.

So, Windows 10 sometimes use your full name as the name of your user folder. Instead the one word rilut, it used my full name.

I solved it by creating a new user account: without spaces and without logging in to my Microsoft account (so it doesn't use your full name, but you can sign in to your Microsoft account later).

@darkaeon10
darkaeon10 commented Nov 11, 2016 edited

@rilut ah well, i don't think there were any spaces in my path (C:\Users\user\AppData\Local\Theano\compiledir_Windows-10-10.0.14393-SP0-Intel64_Family_6_Model_69_Stepping_1_GenuineIntel-3.5.2-64\cutils_ext). It still doesn't work. :(

@rilut
Contributor
rilut commented Nov 11, 2016

Ah, so it's a different problem with my current laptop then

@darkaeon10

Yeah it seems so. Thanks for the help though @rilut !

@darkaeon10
darkaeon10 commented Nov 11, 2016 edited

@nouiz , is it correct to simply replace the theano folder in the conda environment with the PR you referred me to?

Update:

I tried this and it still had the 'The specified module could not be found.'
I really don't get it. The file is right there at "C:\Users\user\AppData\Local\Theano\compiledir_Windows-10-10.0.14393-SP0-Intel64_Family_6_Model_69_Stepping_1_GenuineIntel-3.5.2-64\cutils_ext\cutils_ext.pyd".

I'm not using Cygwin, so the "pyd" file is correct right?

I don't understand why the module cannot be found. Is this a matter of Theano not being able to find and load this pyd file properly?

@darkaeon10

Just an update, I was able to fix my problem by setting-up theano using this tutorial: http://ankivil.com/installing-keras-theano-and-dependencies-on-windows-10/

I'm not too sure how this is different from installing through the conda command, but this time I created a .theanorc file with this line in the global setting: "base_compiledir=C:\theano_compiledir".

@nouiz
Member
nouiz commented Nov 30, 2016

We go to support mainly only Conda on Windows. Otherwise it is too much time consuming. We will accept PR to give better support for Cygwin, but we won't work on that.

@nouiz nouiz closed this Nov 30, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment