distutils build_ext.get_outputs returns wrong result

[kxroberto]

BPO	5977
Nosy	@devdanzin, @benjaminp, @tarekziade, @merwok, @zooba
Files	ext_filename.patch build_ext._compiler.diff: Revert build_ext.compiler -> build_ext._compiler issue5977-w32.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/merwok'
closed_at = <Date 2021-02-03.18:10:47.393>
created_at = <Date 2009-05-09.13:29:48.721>
labels = ['type-bug', 'library']
title = 'distutils build_ext.get_outputs returns wrong result'
updated_at = <Date 2021-02-03.18:10:47.392>
user = 'https://bugs.python.org/kxroberto'

bugs.python.org fields:

activity = <Date 2021-02-03.18:10:47.392>
actor = 'steve.dower'
assignee = 'eric.araujo'
closed = True
closed_date = <Date 2021-02-03.18:10:47.393>
closer = 'steve.dower'
components = ['Distutils']
creation = <Date 2009-05-09.13:29:48.721>
creator = 'kxroberto'
dependencies = []
files = ['13941', '13973', '13983']
hgrepos = []
issue_num = 5977
keywords = ['patch']
message_count = 16.0
messages = ['87494', '87495', '87506', '87524', '87525', '87643', '87654', '87662', '87663', '87669', '87670', '87759', '87761', '143918', '213606', '386281']
nosy_count = 8.0
nosy_names = ['kxroberto', 'ajaksu2', 'benjamin.peterson', 'tarek', 'eric.araujo', 'rpetrov', 'alexis', 'steve.dower']
pr_nums = []
priority = 'normal'
resolution = 'out of date'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue5977'
versions = ['Python 2.7', 'Python 3.3', 'Python 3.4']

[kxroberto]

with --inplace etc. build_ext.get_outputs returns wrong extension paths;
tries to computes them out of the air *again* and ..

Tools which need to further know the produced module file, e.g.
pyximport (Cython) do crash then ...

patch see attachment: stores and uses the real path in use.
writes a new attribute ext_filename to Extension object - should it be
underscored _ext_filename or ... ?

[tarekziade]

can you provide a test case that demonstrates the problem ?

[kxroberto]

>> test_build_ext: The test must be run in a python build dir

don't have a build setup currently. maybe in future.

One can yet see in build_ext.py:

after "if self.inplace:" (line485)
there are 2 different computations for ext_filename

while computation in get_outputs() is not aware of self.inplace (line447

[tarekziade]

I was talking about your use case (pyximport (Cython) do crash then ?)

If I can have a real example, with the output and the expected one,
I will build a test with it in Distutils before I fix it

[kxroberto]

its with any .pyx (Cython) module , when after pyximport.install() and
injection of --inplace option a .pyx module is imported.

the relevant lines in pyxbuild.py are

        dist.run_commands()
        return dist.get_command_obj("build_ext").get_outputs()[0]

which use the buggy "get_outputs" and shall return the full path of
built module to pyximport.py :

    so_path = pyxbuild.pyx_to_dll(pyxfilename, extension_mod,
                                  build_in_temp=build_in_temp,
                                  pyxbuild_dir=pyxbuild_dir,
                                  setup_args=sargs )
    assert os.path.exists(so_path), "Cannot find: %s" % so_path

=> crash with "Cannot find..." before pyximport.load_module goes to
import it.

A stripped down test case should perhaps 'build_ext' any arbitrary
extension module with --inplace ( a option of base command 'build' ) and
something like ...

dist.get_command_obj("build_ext").inplace=1
dist.run_commands()
so_path = dist.get_command_obj("build_ext").get_outputs()[0]
assert os.path.isfile(so_path) and os.path.dirname(so_path) in ('','.')

... will produce a invalid so_path: not pointing to actual locally
in-place built xy.pyd/.so, but to a non-existing or old file in the
build folders

[tarekziade]

done in r72585 and propagated in 2.6, 3.0, 3.1 branches

Thank you !

[devdanzin]

Tarek,
It looks like all buildbots are red because setup.py doesn't work with
this change:

running build
running build_ext
Traceback (most recent call last):
  File "./setup.py", line 1896, in <module>
    main()
  File "./setup.py", line 1891, in main
    'Lib/smtpd.py']
  File
"/home2/buildbot/slave/trunk.loewis-sun/build/Lib/distutils/core.py",
line 149, in setup
    dist.run_commands()
  File
"/home2/buildbot/slave/trunk.loewis-sun/build/Lib/distutils/dist.py",
line 974, in run_commands
    self.run_command(cmd)
  File
"/home2/buildbot/slave/trunk.loewis-sun/build/Lib/distutils/dist.py",
line 994, in run_command
    cmd_obj.run()
  File
"/home2/buildbot/slave/trunk.loewis-sun/build/Lib/distutils/command/build.py",
line 132, in run
    self.run_command(cmd_name)
  File
"/home2/buildbot/slave/trunk.loewis-sun/build/Lib/distutils/cmd.py",
line 326, in run_command
    self.distribution.run_command(command)
  File
"/home2/buildbot/slave/trunk.loewis-sun/build/Lib/distutils/dist.py",
line 994, in run_command
    cmd_obj.run()
  File
"/home2/buildbot/slave/trunk.loewis-sun/build/Lib/distutils/command/build_ext.py",
line 348, in run
    self.build_extensions()
  File "./setup.py", line 103, in build_extensions
    missing = self.detect_modules()
  File "./setup.py", line 302, in detect_modules
    add_dir_to_list(self.compiler.library_dirs, '/usr/local/lib')
AttributeError: 'NoneType' object has no attribute 'library_dirs'
[31295 refs]
*** Error code 1
The following command caused the error:
case $MAKEFLAGS in \
*s*)  CC='gcc' LDSHARED='gcc -shared' OPT='-g -Wall -Wstrict-prototypes'
./python -E ./setup.py -q build;; \
*)  CC='gcc' LDSHARED='gcc -shared' OPT='-g -Wall -Wstrict-prototypes'
./python -E ./setup.py build;; \
esac
make: Fatal error: Command failed for target `sharedmods'
program finished with exit code 1

[tarekziade]

Sorry about that, I was travelling and didn't see that the buildbots
turned red. (I ran the test but didn't build python again)

I have changed that because build_ext.compiler is a string option, e.g.
the compiler type. Turning it into a compiler instance when the command
is run twice it breaks because it is used as a string argument to
create a compiler.

Maybe a solution would be to rename the option name into "compiler_type"
and keep build_ext.compiler as a compiler instance.

[devdanzin]

Tarek,
Sorry about breaking the compiler option, I was aware that passing None
there would have that risk.

I think anything that breaks people's setup.py shouldn't go into
release26-maint without a really important motivation.

Is there a way to keep self.compiler as an attribute and make
new_compiler work correctly with it?

[tarekziade]

new_compiler takes "compiler" as a compiler name. It would make much
sense to make it accept a compiler instance.

now for people setup.py, I wasn't aware that people are using
build_ext.compiler as a compiler, besides Python itself.

IMHO we should rename the compiler option to "compiler_type" then
provide an API build_ext.get_compiler() then fix Python setup.py file.

Then this should not go into 2.6 but in 2.7, and we should put back the
option in new_compiler like it was (it's wrong but at least it works if
you run the command just once, which is the most common case)