Distutils fails to build extension in path with spaces

[klouo]

BPO	16907
Nosy	@loewis, @tarekziade, @merwok, @zware, @serhiy-storchaka
Superseder	bpo-4508: distutils compiler not handling spaces in path to output/src files

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2016-01-18.21:59:58.078>
created_at = <Date 2013-01-09.12:22:42.650>
labels = ['type-bug', 'library', 'OS-windows']
title = 'Distutils fails to build extension in path with spaces'
updated_at = <Date 2016-01-18.21:59:58.076>
user = 'https://bugs.python.org/klouo'

bugs.python.org fields:

activity = <Date 2016-01-18.21:59:58.076>
actor = 'zach.ware'
assignee = 'none'
closed = True
closed_date = <Date 2016-01-18.21:59:58.078>
closer = 'zach.ware'
components = ['Distutils', 'Windows']
creation = <Date 2013-01-09.12:22:42.650>
creator = 'klo.uo'
dependencies = []
files = []
hgrepos = []
issue_num = 16907
keywords = []
message_count = 10.0
messages = ['179434', '179439', '179440', '179444', '179473', '179482', '179484', '179485', '224333', '258553']
nosy_count = 8.0
nosy_names = ['loewis', 'tarek', 'eric.araujo', 'BreamoreBoy', 'Ramchandra Apte', 'zach.ware', 'serhiy.storchaka', 'klo.uo']
pr_nums = []
priority = 'normal'
resolution = 'duplicate'
stage = 'resolved'
status = 'closed'
superseder = '4508'
type = 'behavior'
url = 'https://bugs.python.org/issue16907'
versions = ['Python 2.7', 'Python 3.4', 'Python 3.5']

[klouo]

I noticed this issue, while trying to compile Cython extension, when source file is in path with spaces. Extension wouldn't compile because LIBDIR is passed to MinGW g++ (though not to gcc) or MSVC compilers unquoted. I tracked the problem to "distutils/build_ext.py" in get_ext_fullpath() function.

I patched it, this way:

========================================

--- build_ext.bak
+++ build_ext.py
@@ -647,6 +647,11 @@
         package = '.'.join(modpath[0:-1])
         build_py = self.get_finalized_command('build_py')
         package_dir = os.path.abspath(build_py.get_package_dir(package))
+        try:
+            from win32api import GetShortPathName
+            package_dir = GetShortPathName(package_dir)
+        except:
+            pass
         # returning
         #   package_dir/filename
         return os.path.join(package_dir, filename)

========================================

which is just one way to do it.

[RamchandraApte]

Can you fix the bare except?

[klouo]

Can you fix the bare except?

========================================
...
except ImportError:
pass
========================================

[serhiy-storchaka]

get_ext_fullpath() has no relations to this bug. The bug is probably in spawn() functions, perhaps in _nt_quote_args() or something like. It looks as Windows specific bug (Posix passes command arguments as a list, not joining it into string).

[loewis]

klo.uo: can you kindly provide a working (or, rather, failing) example? A trivial "hello-world" kind of package could do, along with a report what path you unpacked it in, and what error you get.

Your patch is not applicable to Python, since it requires the win32 extensions, which we cannot assume to be present.

[merwok]

I think this is a duplicate report; could you please search for an existing open report for the same bug?

[klouo]

klo.uo: can you kindly provide a working (or, rather, failing)
example? A trivial "hello-world" kind of package could do, along
with a report what path you unpacked it in, and what error you get.

As mentioned in opening thread, this doesn't happen with Cython's
"hello world" example, as gcc somehow doesn't trigger this problem.
Maybe because it's handled by Cython's own distutils copies, maybe it's
handled by Numpy's distutils version or it could be by Python's
distutils.

I noticed this issue while using Numpy with Cython. Here is simple example:

C:\Documents and Settings\klo\My Documents\code\python\misc\test.pyx:
========================================
cimport numpy

def sum(x):
    cdef numpy.ndarray[int, ndim=1] arr = x
    cdef int i, s = 0
    for i in range(arr.shape[0]):
        s += arr[i]
    return s

========================================

C:\Documents and Settings\klo\My Documents\code\python\misc\setup.py:
========================================

from distutils.core import setup
from distutils.extension import Extension
from Cython.Distutils import build_ext
from numpy.distutils.misc_util import get_numpy_include_dirs

setup(
cmdclass = {'build_ext': build_ext},
ext_modules = [Extension("test", ["test.pyx"], include_dirs=get_numpy_include_dirs())]
)
========================================

command line: python setup.py build_ext --inplace

MinGW result:
========================================
...
g++ -shared build\temp.win32-2.7\Release\test.o -LC:\Python27\libs -LC:\Python27\PCbuild -lpython27 -lmsvcr90 -o C:\Documents and Settings\klo\My Documents\code\python\misc\test.pyd
Found executable C:\MinGW\bin\g++.exe
g++.exe: error: and: No such file or directory
g++.exe: error: Settings\klo\My: No such file or directory
g++.exe: error: Documents\code\python\misc\test.pyd: No such file or directory
error: Command "g++ -shared build\temp.win32-2.7\Release\test.o -LC:\Python27\libs -LC:\Python27\PCbuild -lpython27 -lmsvcr90 -o C:\Documents and Settings\klo\My Documents\code\python\misc\test.pyd" failed with exit status 1
========================================

Similar result if I use MSVC compiler:

MSVC result:
========================================
building 'test' extension
c:\Program Files\Microsoft Visual Studio 10.0\VC\BIN\cl.exe /c /nologo /Ox /MD /W3 /GS- /DNDEBUG -IC:\Python27\lib\site-packages\numpy\core\include -IC:\Python27\include -IC:\Python27\PC /Tctest.c /Fobuild\temp.win32-2.7\Release\test.obj
Found executable c:\Program Files\Microsoft Visual Studio 10.0\VC\BIN\cl.exe
c:\Program Files\Microsoft Visual Studio 10.0\VC\BIN\link.exe /DLL /nologo /INCREMENTAL:NO /LIBPATH:C:\Python27\libs /LIBPATH:C:\Python27\PCbuild /EXPORT:inittest build\temp.win32-2.7\Release\test.obj /OUT:C:\Documents and Settings\klo\My Documents\code\python\misc\test.pyd /IMPLIB:build\temp.win32-2.7\Release\test.lib /MANIFESTFILE:build\temp.win32-2.7\Release\test.pyd.manifest /MANIFEST
Found executable c:\Program Files\Microsoft Visual Studio 10.0\VC\BIN\link.exe
LINK : fatal error LNK1181: cannot open input file 'and.obj'
error: Command "c:\Program Files\Microsoft Visual Studio 10.0\VC\BIN\link.exe /DLL /nologo /INCREMENTAL:NO /LIBPATH:C:\Python27\libs /LIBPATH:C:\Python27\PCbuild /EXPORT:inittest build\temp.win32-2.7\Release\test.obj /OUT:C:\Documents and Settings\klo\My Documents\code\python\misc\test.pyd /IMPLIB:build\temp.win32-2.7\Release\test.lib /MANIFESTFILE:build\temp.win32-2.7\Release\test.pyd.manifest /MANIFEST" failed with exit status 1181
========================================

So issue is with unquoted path for output file, and not for LIBDIR as I wrote
previously (perhaps I misread the logs, as I was trying to fix similar issue
for Theano, at the same time)

Your patch is not applicable to Python, since it requires the win32
extensions, which we cannot assume to be present.

Yes, I provided patch that worked for me temporarily, as I'm not
familiar with distutils, and that's as far as I went.
But source issue is probably elsewhere, as Serhiy suggested.

[klouo]

I found two similar issues:

1. distutils compiler not handling spaces in path to output/src files: http://bugs.python.org/issue4508

with patches for unix compiler and cygwin compiler

2. Distutils does not put quotes around paths that contain spaces when compiling with MSVC: http://bugs.python.org/issue13765

with some patch for MSVC

Problem seems to be deeper than provided patches, as evident also from my report

[BreamoreBoy]

Of the two issues mentioned in msg179485 bpo-4508 is still open but bpo-13765 has been closed "not a bug".

[zware]

This isn't obviously a distutils bug, but sounds strikingly similar to bpo-4508. Closing as a duplicate of bpo-4508.