Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

distutils doesn't parallelize extension module compilation #49559

Closed
pitrou opened this issue Feb 19, 2009 · 30 comments
Closed

distutils doesn't parallelize extension module compilation #49559

pitrou opened this issue Feb 19, 2009 · 30 comments
Labels
build The build process and cross-build type-feature A feature request or enhancement

Comments

@pitrou
Copy link
Member

pitrou commented Feb 19, 2009

BPO 5309
Nosy @pitrou, @scoder, @vstinner, @giampaolo, @tiran, @tarekziade, @merwok, @berkerpeksag
Files
  • build_ext_parallel.patch
  • build_ext_parallel2.patch
  • build_ext_parallel3.patch
  • build_ext_parallel4.patch
  • build_ext_parallel5.patch
  • build_ext_parallel6.patch
  • issue5309-doc.diff
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
    closed_at = <Date 2014-09-30.13:04:24.612>
    created_at = <Date 2009-02-19.01:39:51.797>
    labels = ['type-feature', 'build']
    title = "distutils doesn't parallelize extension module compilation"
    updated_at = <Date 2016-09-17.19:05:32.783>
    user = 'https://github.com/pitrou'

    bugs.python.org fields:

    activity = <Date 2016-09-17.19:05:32.783>
    actor = 'giampaolo.rodola'
    assignee = 'none'
    closed = True
    closed_date = <Date 2014-09-30.13:04:24.612>
    closer = 'pitrou'
    components = ['Build']
    creation = <Date 2009-02-19.01:39:51.797>
    creator = 'pitrou'
    dependencies = []
    files = ['32378', '32380', '32382', '36545', '36564', '36671', '37824']
    hgrepos = []
    issue_num = 5309
    keywords = ['patch', 'gsoc']
    message_count = 30.0
    messages = ['82447', '116603', '178993', '179010', '201392', '201396', '201400', '226412', '226415', '226424', '226521', '226523', '226546', '226646', '227188', '227651', '227652', '227877', '227883', '227885', '227887', '227888', '227902', '227906', '227907', '234147', '234514', '234803', '240505', '276819']
    nosy_count = 13.0
    nosy_names = ['pitrou', 'scoder', 'vstinner', 'giampaolo.rodola', 'christian.heimes', 'tarek', 'eric.araujo', 'Arfrever', 'python-dev', 'jtaylor', 'berker.peksag', 'bfroehle', 'Sjlver']
    pr_nums = []
    priority = 'low'
    resolution = 'fixed'
    stage = 'resolved'
    status = 'closed'
    superseder = None
    type = 'enhancement'
    url = 'https://bugs.python.org/issue5309'
    versions = ['Python 3.5']

    @pitrou
    Copy link
    Member Author

    pitrou commented Feb 19, 2009

    When run with "make -jN", the Python compilation process is able to
    parallelize across N concurrent processes for the interpreter core, but
    extension modules are still compiled sequentially.

    @pitrou pitrou added the stdlib Python modules in the Lib dir label Feb 19, 2009
    @pitrou pitrou added the type-feature A feature request or enhancement label Feb 19, 2009
    @BreamoreBoy BreamoreBoy mannequin removed the stdlib Python modules in the Lib dir label Sep 16, 2010
    @merwok
    Copy link
    Member

    merwok commented Sep 16, 2010

    Do you want to work on a patch?

    @merwok merwok changed the title setup.py doesn't parallelize extension module compilation packaging doesn't parallelize extension module compilation Apr 8, 2011
    @pitrou
    Copy link
    Member Author

    pitrou commented Jan 3, 2013

    The original request is really about setup.py, not packaging. I don't care about packaging, and it's not in the stdlib.

    @pitrou pitrou added the build The build process and cross-build label Jan 3, 2013
    @pitrou pitrou changed the title packaging doesn't parallelize extension module compilation setup.py doesn't parallelize extension module compilation Jan 3, 2013
    @tiran
    Copy link
    Member

    tiran commented Jan 4, 2013

    +1

    I think it's sufficient to parallelize the compilation step (.c -> .o) and ignore the linker step. Linking is usually so fast that it doesn't matter.

    Idea:

    • separate compile step from link step
    • run all compile calls for all extensions in parallel until all objects of all extensions are generated
    • do linking and error reporting in serial mode

    @Arfrever Arfrever mannequin changed the title setup.py doesn't parallelize extension module compilation distutils doesn't parallelize extension module compilation Oct 25, 2013
    @pitrou
    Copy link
    Member Author

    pitrou commented Oct 26, 2013

    Here is a working patch.

    @pitrou
    Copy link
    Member Author

    pitrou commented Oct 26, 2013

    Here is an updated patch which only enables parallel building in setup.py.

    @tiran
    Copy link
    Member

    tiran commented Oct 26, 2013

    Antoine's patch doesn't work for a fresh copy of Python

    • add import error report to build_ext parallel (distutils doesn't use logging and doesn't have a working log.exception() function)
    • make time a builtin module
    • add hack for concurrent.futures.process
    • add --parallel=i / -ji (jobs) argument to build_ext

    @Sjlver
    Copy link
    Mannequin

    Sjlver mannequin commented Sep 5, 2014

    Is there a reason this has not landed? The patch works perfectly for me, except for one issue:

    @@ -268,6 +275,9 @@
             if self.undef:
                 self.undef = self.undef.split(',')
     
    +        if self.parallel:
    +            self.parallel = int(self.parallel)
    +
             if self.swig_opts is None:
                 self.swig_opts = []
             else:
    

    If self.parallel is True, this will set self.parallel to 1, causing it to use one worker instead of n (where n is the number of processors).

    An updated patch is attached.

    @scoder
    Copy link
    Contributor

    scoder commented Sep 5, 2014

    Yes, please. The sequential build in distutils is very annoying.

    (too late for Py3.4, though, I guess...)

    @Sjlver
    Copy link
    Mannequin

    Sjlver mannequin commented Sep 5, 2014

    With this patch, and on Ubuntu 14.04, occasionally modules fail to build with the following error:

    *** WARNING: renaming "_testbuffer" since importing it failed: dlopen: cannot load any more object with static TLS

    I'm not 100% sure if this is really due to the patch... but I've never seen it before. I'll try investigate a bit more.

    @pitrou
    Copy link
    Member Author

    pitrou commented Sep 6, 2014

    Updated patch:

    • add the -j option to the "build" command as well (even though it's only used by build_ext)
    • in setup.py, only force parallel compilation if make was called with -j

    @Arfrever
    Copy link
    Mannequin

    Arfrever mannequin commented Sep 6, 2014

    • Is support for infinite jobs planned?
      (GNU make supports it for -j without number argument.)

    • self.parallel = int(self.parallel) would raise ValueError when a non-number is passed. I suggest to print user-friendly error message.

    @merwok
    Copy link
    Member

    merwok commented Sep 7, 2014

    self.parallel = int(self.parallel) would raise ValueError when a
    non-number is passed. I suggest to print user-friendly error message.

    The distutils idiom would be to catch the TypeError/ValueError and raise DistutilsOptionError. Higher layers convert that to a message.

    @Sjlver
    Copy link
    Mannequin

    Sjlver mannequin commented Sep 9, 2014

    I've checked the dlopen issue. This was due to a problem with UndefinedBehaviorSanitizer, and was solved by upgrading to Clang 3.5. It has little to do with this patch.

    Using this patch and http://bugs.python.org/issue22359 , I now get reliable parallel builds.

    @pitrou
    Copy link
    Member Author

    pitrou commented Sep 20, 2014

    Updated patch raising DistutilsOptionError. I couldn't find any docs so I didn't update them :-)

    @python-dev
    Copy link
    Mannequin

    python-dev mannequin commented Sep 26, 2014

    New changeset bbe57429eba0 by Antoine Pitrou in branch 'default':
    Issue bpo-5309: distutils' build and build_ext commands now accept a -j
    https://hg.python.org/cpython/rev/bbe57429eba0

    @pitrou
    Copy link
    Member Author

    pitrou commented Sep 26, 2014

    This is now pushed.

    @pitrou pitrou closed this as completed Sep 26, 2014
    @vstinner
    Copy link
    Member

    vstinner commented Sep 30, 2014

    It looks like compilation of Python 3.5 fails on FreeBSD 6.4 because of the changeset bbe57429eba0a9ec21fb0f1178f409f1bba44c22:

    http://buildbot.python.org/all/builders/x86%20FreeBSD%206.4%203.x/builds/5061

    Compile log:
    ---

    building '_ctypes' extension
    creating build/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/libffi
    creating build/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/libffi/src
    creating build/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/libffi/src/x86
    gcc -pthread -fPIC -fno-strict-aliasing -Wsign-compare -g -O0 -Wall -Wstrict-prototypes -Ibuild/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/libffi/include -Ibuild/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/libffi -I/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/libffi/src -I./Include -I. -IInclude -I/usr/local/include -I/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Include -I/usr/home/db3l/buildarea/3.x.bolen-freebsd/build -c /usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/_ctypes.c -o build/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/_ctypes.o -Wall -fexceptions
    gcc -pthread -fPIC -fno-strict-aliasing -Wsign-compare -g -O0 -Wall -Wstrict-prototypes -Ibuild/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/libffi/include -Ibuild/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/libffi -I/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/libffi/src -I./Include -I. -IInclude -I/usr/local/include -I/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Include -I/usr/home/db3l/buildarea/3.x.bolen-freebsd/build -c /usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/callbacks.c -o build/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/callbacks.o -Wall -fexceptions
    gcc -pthread -fPIC -fno-strict-aliasing -Wsign-compare -g -O0 -Wall -Wstrict-prototypes -Ibuild/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/libffi/include -Ibuild/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/libffi -I/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/libffi/src -I./Include -I. -IInclude -I/usr/local/include -I/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Include -I/usr/home/db3l/buildarea/3.x.bolen-freebsd/build -c /usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/callproc.c -o build/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/callproc.o -Wall -fexceptions
    /usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/callproc.c: In function `py_dl_open':
    /usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/callproc.c:1336: warning: initialization discards qualifiers from pointer target type
    gcc -pthread -fPIC -fno-strict-aliasing -Wsign-compare -g -O0 -Wall -Wstrict-prototypes -Ibuild/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/libffi/include -Ibuild/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/libffi -I/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/libffi/src -I./Include -I. -IInclude -I/usr/local/include -I/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Include -I/usr/home/db3l/buildarea/3.x.bolen-freebsd/build -c /usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/stgdict.c -o build/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/stgdict.o -Wall -fexceptions
    gcc -pthread -fPIC -fno-strict-aliasing -Wsign-compare -g -O0 -Wall -Wstrict-prototypes -Ibuild/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/libffi/include -Ibuild/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/libffi -I/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/libffi/src -I./Include -I. -IInclude -I/usr/local/include -I/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Include -I/usr/home/db3l/buildarea/3.x.bolen-freebsd/build -c /usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/cfield.c -o build/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/cfield.o -Wall -fexceptions
    gcc -pthread -fPIC -fno-strict-aliasing -Wsign-compare -g -O0 -Wall -Wstrict-prototypes -Ibuild/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/libffi/include -Ibuild/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/libffi -I/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/libffi/src -I./Include -I. -IInclude -I/usr/local/include -I/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Include -I/usr/home/db3l/buildarea/3.x.bolen-freebsd/build -c /usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/libffi/src/prep_cif.c -o build/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/libffi/src/prep_cif.o -Wall -fexceptions
    gcc -pthread -fPIC -fno-strict-aliasing -Wsign-compare -g -O0 -Wall -Wstrict-prototypes -Ibuild/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/libffi/include -Ibuild/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/libffi -I/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/libffi/src -I./Include -I. -IInclude -I/usr/local/include -I/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Include -I/usr/home/db3l/buildarea/3.x.bolen-freebsd/build -c /usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/libffi/src/closures.c -o build/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/libffi/src/closures.o -Wall -fexceptions
    gcc -pthread -fPIC -fno-strict-aliasing -Wsign-compare -g -O0 -Wall -Wstrict-prototypes -Ibuild/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/libffi/include -Ibuild/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/libffi -I/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/libffi/src -I./Include -I. -IInclude -I/usr/local/include -I/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Include -I/usr/home/db3l/buildarea/3.x.bolen-freebsd/build -c /usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/libffi/src/x86/ffi.c -o build/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/libffi/src/x86/ffi.o -Wall -fexceptions
    gcc -pthread -fPIC -fno-strict-aliasing -Wsign-compare -g -O0 -Wall -Wstrict-prototypes -Ibuild/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/libffi/include -Ibuild/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/libffi -I/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/libffi/src -I./Include -I. -IInclude -I/usr/local/include -I/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Include -I/usr/home/db3l/buildarea/3.x.bolen-freebsd/build -c /usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/libffi/src/x86/freebsd.S -o build/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/libffi/src/x86/freebsd.o -Wall -fexceptions
    gcc -pthread -shared build/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/_ctypes.o build/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/callbacks.o build/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/callproc.o build/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/stgdict.o build/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/cfield.o build/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/libffi/src/prep_cif.o build/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/libffi/src/closures.o build/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/libffi/src/x86/ffi.o build/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/libffi/src/x86/freebsd.o -L/usr/local/lib -o build/lib.freebsd-6.4-RELEASE-i386-3.5-pydebug/_ctypes.so
    *** WARNING: renaming "_ssl" since importing it failed: build/lib.freebsd-6.4-RELEASE-i386-3.5-pydebug/_ssl.so: Undefined symbol "X509_check_ca"
    *** WARNING: renaming "_ctypes" since importing it failed: build/lib.freebsd-6.4-RELEASE-i386-3.5-pydebug/_ctypes.so: Undefined symbol "ffi_call_win32"
    error: [Errno 2] No such file or directory: 'build/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/_ctypes.o'
    *** Error code 1
    ---

    Before:
    ---
    building '_ctypes' extension
    gcc -pthread -fPIC -fno-strict-aliasing -Wsign-compare -g -O0 -Wall -Wstrict-prototypes -Ibuild/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/libffi/include -Ibuild/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/libffi -I/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/libffi/src -I./Include -I. -IInclude -I/usr/local/include -I/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Include -I/usr/home/db3l/buildarea/3.x.bolen-freebsd/build -c /usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/_ctypes.c -o build/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/_ctypes.o -Wall -fexceptions
    gcc -pthread -fPIC -fno-strict-aliasing -Wsign-compare -g -O0 -Wall -Wstrict-prototypes -Ibuild/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/libffi/include -Ibuild/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/libffi -I/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/libffi/src -I./Include -I. -IInclude -I/usr/local/include -I/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Include -I/usr/home/db3l/buildarea/3.x.bolen-freebsd/build -c /usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/callbacks.c -o build/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/callbacks.o -Wall -fexceptions
    gcc -pthread -fPIC -fno-strict-aliasing -Wsign-compare -g -O0 -Wall -Wstrict-prototypes -Ibuild/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/libffi/include -Ibuild/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/libffi -I/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/libffi/src -I./Include -I. -IInclude -I/usr/local/include -I/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Include -I/usr/home/db3l/buildarea/3.x.bolen-freebsd/build -c /usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/callproc.c -o build/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/callproc.o -Wall -fexceptions
    /usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/callproc.c: In function `py_dl_open':
    /usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/callproc.c:1336: warning: initialization discards qualifiers from pointer target type
    gcc -pthread -fPIC -fno-strict-aliasing -Wsign-compare -g -O0 -Wall -Wstrict-prototypes -Ibuild/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/libffi/include -Ibuild/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/libffi -I/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/libffi/src -I./Include -I. -IInclude -I/usr/local/include -I/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Include -I/usr/home/db3l/buildarea/3.x.bolen-freebsd/build -c /usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/stgdict.c -o build/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/stgdict.o -Wall -fexceptions
    gcc -pthread -fPIC -fno-strict-aliasing -Wsign-compare -g -O0 -Wall -Wstrict-prototypes -Ibuild/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/libffi/include -Ibuild/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/libffi -I/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/libffi/src -I./Include -I. -IInclude -I/usr/local/include -I/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Include -I/usr/home/db3l/buildarea/3.x.bolen-freebsd/build -c /usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/cfield.c -o build/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/cfield.o -Wall -fexceptions
    gcc -pthread -fPIC -fno-strict-aliasing -Wsign-compare -g -O0 -Wall -Wstrict-prototypes -Ibuild/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/libffi/include -Ibuild/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/libffi -I/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/libffi/src -I./Include -I. -IInclude -I/usr/local/include -I/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Include -I/usr/home/db3l/buildarea/3.x.bolen-freebsd/build -c /usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/libffi/src/prep_cif.c -o build/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/libffi/src/prep_cif.o -Wall -fexceptions
    gcc -pthread -fPIC -fno-strict-aliasing -Wsign-compare -g -O0 -Wall -Wstrict-prototypes -Ibuild/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/libffi/include -Ibuild/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/libffi -I/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/libffi/src -I./Include -I. -IInclude -I/usr/local/include -I/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Include -I/usr/home/db3l/buildarea/3.x.bolen-freebsd/build -c /usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/libffi/src/closures.c -o build/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/libffi/src/closures.o -Wall -fexceptions
    gcc -pthread -fPIC -fno-strict-aliasing -Wsign-compare -g -O0 -Wall -Wstrict-prototypes -Ibuild/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/libffi/include -Ibuild/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/libffi -I/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/libffi/src -I./Include -I. -IInclude -I/usr/local/include -I/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Include -I/usr/home/db3l/buildarea/3.x.bolen-freebsd/build -c /usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/libffi/src/x86/ffi.c -o build/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/libffi/src/x86/ffi.o -Wall -fexceptions
    gcc -pthread -fPIC -fno-strict-aliasing -Wsign-compare -g -O0 -Wall -Wstrict-prototypes -Ibuild/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/libffi/include -Ibuild/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/libffi -I/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/libffi/src -I./Include -I. -IInclude -I/usr/local/include -I/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Include -I/usr/home/db3l/buildarea/3.x.bolen-freebsd/build -c /usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/libffi/src/x86/freebsd.S -o build/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/libffi/src/x86/freebsd.o -Wall -fexceptions
    gcc -pthread -shared build/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/_ctypes.o build/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/callbacks.o build/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/callproc.o build/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/stgdict.o build/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/cfield.o build/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/libffi/src/prep_cif.o build/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/libffi/src/closures.o build/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/libffi/src/x86/ffi.o build/temp.freebsd-6.4-RELEASE-i386-3.5-pydebug/usr/home/db3l/buildarea/3.x.bolen-freebsd/build/Modules/_ctypes/libffi/src/x86/freebsd.o -L/usr/local/lib -o build/lib.freebsd-6.4-RELEASE-i386-3.5-pydebug/_ctypes.so
    *** WARNING: renaming "_ctypes" since importing it failed: build/lib.freebsd-6.4-RELEASE-i386-3.5-pydebug/_ctypes.so: Undefined symbol "ffi_call_win32"
    ---

    @vstinner vstinner reopened this Sep 30, 2014
    @vstinner
    Copy link
    Member

    vstinner commented Sep 30, 2014

    There is a similar error on OpenIndiana buildbot, Python cannot be compiled since this build:

    http://buildbot.python.org/all/builders/AMD64%20OpenIndiana%203.x/builds/8664/steps/compile/logs/stdio

    gcc -fPIC -Wsign-compare -g -O0 -Wall -Wstrict-prototypes -I/usr/local/include/ncursesw -m64 -Werror=declaration-after-statement -Ibuild/temp.solaris-2.11-i86pc.64bit-3.5-pydebug/libffi/include -Ibuild/temp.solaris-2.11-i86pc.64bit-3.5-pydebug/libffi -I/export/home/buildbot/64bits/3.x.cea-indiana-amd64/build/Modules/_ctypes/libffi/src -I./Include -I. -IInclude -I/usr/local/include -I/export/home/buildbot/64bits/3.x.cea-indiana-amd64/build/Include -I/export/home/buildbot/64bits/3.x.cea-indiana-amd64/build -c /export/home/buildbot/64bits/3.x.cea-indiana-amd64/build/Modules/_ctypes/libffi/src/x86/sysv.S -o build/temp.solaris-2.11-i86pc.64bit-3.5-pydebug/export/home/buildbot/64bits/3.x.cea-indiana-amd64/build/Modules/_ctypes/libffi/src/x86/sysv.o -Wall -fexceptions
    gcc -fPIC -Wsign-compare -g -O0 -Wall -Wstrict-prototypes -I/usr/local/include/ncursesw -m64 -Werror=declaration-after-statement -Ibuild/temp.solaris-2.11-i86pc.64bit-3.5-pydebug/libffi/include -Ibuild/temp.solaris-2.11-i86pc.64bit-3.5-pydebug/libffi -I/export/home/buildbot/64bits/3.x.cea-indiana-amd64/build/Modules/_ctypes/libffi/src -I./Include -I. -IInclude -I/usr/local/include -I/export/home/buildbot/64bits/3.x.cea-indiana-amd64/build/Include -I/export/home/buildbot/64bits/3.x.cea-indiana-amd64/build -c /export/home/buildbot/64bits/3.x.cea-indiana-amd64/build/Modules/_ctypes/libffi/src/x86/win32.S -o build/temp.solaris-2.11-i86pc.64bit-3.5-pydebug/export/home/buildbot/64bits/3.x.cea-indiana-amd64/build/Modules/_ctypes/libffi/src/x86/win32.o -Wall -fexceptions
    /var/tmp//ccqBa4n2.s: Assembler messages:
    /var/tmp//ccqBa4n2.s:8: Error: suffix or operands invalid for push' /var/tmp//ccqBa4n2.s:17: Error: suffix or operands invalid for push'
    /var/tmp//ccqBa4n2.s:18: Error: suffix or operands invalid for push' /var/tmp//ccqBa4n2.s:79: Error: suffix or operands invalid for jmp'
    /var/tmp//ccqBa4n2.s:138: Error: suffix or operands invalid for pop' /var/tmp//ccqBa4n2.s:147: Error: suffix or operands invalid for push'
    /var/tmp//ccqBa4n2.s:155: Error: suffix or operands invalid for push' /var/tmp//ccqBa4n2.s:156: Error: suffix or operands invalid for push'
    /var/tmp//ccqBa4n2.s:163: Error: suffix or operands invalid for push' /var/tmp//ccqBa4n2.s:176: Error: suffix or operands invalid for pop'
    /var/tmp//ccqBa4n2.s:210: Error: suffix or operands invalid for jmp' /var/tmp//ccqBa4n2.s:251: Error: suffix or operands invalid for pop'
    /var/tmp//ccqBa4n2.s:261: Error: suffix or operands invalid for pop' /var/tmp//ccqBa4n2.s:268: Error: suffix or operands invalid for push'
    /var/tmp//ccqBa4n2.s:272: Error: suffix or operands invalid for push' /var/tmp//ccqBa4n2.s:315: Error: suffix or operands invalid for jmp'
    /var/tmp//ccqBa4n2.s:358: Error: suffix or operands invalid for pop' /var/tmp//ccqBa4n2.s:359: Error: suffix or operands invalid for pop'
    /var/tmp//ccqBa4n2.s:370: Error: suffix or operands invalid for push' /var/tmp//ccqBa4n2.s:383: Error: suffix or operands invalid for pop'
    /var/tmp//ccqBa4n2.s:416: Error: suffix or operands invalid for jmp' /var/tmp//ccqBa4n2.s:459: Error: suffix or operands invalid for pop'
    /var/tmp//ccqBa4n2.s:460: Error: suffix or operands invalid for pop' /var/tmp//ccqBa4n2.s:461: Error: suffix or operands invalid for pop'
    /var/tmp//ccqBa4n2.s:471: Error: suffix or operands invalid for `jmp'
    *** WARNING: renaming "_curses" since importing it failed: ld.so.1: python: fatal: relocation error: file build/lib.solaris-2.11-i86pc.64bit-3.5-pydebug/_curses.so: symbol newscr: referenced symbol not found
    *** WARNING: renaming "_curses_panel" since importing it failed: No module named '_curses'
    error: [Errno 2] No such file or directory: 'build/temp.solaris-2.11-i86pc.64bit-3.5-pydebug/export/home/buildbot/64bits/3.x.cea-indiana-amd64/build/Modules/_decimal/_decimal.o'
    make: *** [sharedmods] Error 1

    @pitrou
    Copy link
    Member Author

    pitrou commented Sep 30, 2014

    Thanks for noticing this.
    Looking at an old build, the ctypes and curses compile errors under OpenIndiana are not new:
    http://buildbot.python.org/all/builders/AMD64%20OpenIndiana%203.x/builds/8658/steps/compile/logs/stdio

    Simply, it seems that a ctypes compile error is now interpreted as a failure of the compilation build step. I'll try to fix that.

    (the problem looks similar on the FreeBSD 6.4 buildbot)

    @vstinner
    Copy link
    Member

    vstinner commented Sep 30, 2014

    the ctypes and curses compile errors under OpenIndiana are not new

    FYI the compilation error of curses on OpenIndiana is *old*, at least 3 years old :-(
    http://bugs.python.org/issue13552

    I just opened the issue bpo-22521 for the compilation error of ctypes on FreeBSD.

    @Sjlver
    Copy link
    Mannequin

    Sjlver mannequin commented Sep 30, 2014

    I don't understand where this error comes from...

    The compilation commands are exactly the same in both the "before" and "after" logs. The order of commands is also the same.

    The only difference is this message:

    *** WARNING: renaming "_ssl" since importing it failed: build/lib.freebsd-6.4-RELEASE-i386-3.5-pydebug/_ssl.so: Undefined symbol "X509_check_ca"
    

    This message probably occurs because _ssl.so is now built in parallel with _ctypes.

    In the logs, one can see that the command to build _ctypes.o is executed; because of that, I wonder why the file is missing. Maybe we could check for wrong dependency tracking in the _ctypes makefile (similar to http://bugs.python.org/issue22359)

    @python-dev
    Copy link
    Mannequin

    python-dev mannequin commented Sep 30, 2014

    New changeset 94af1af93670 by Antoine Pitrou in branch 'default':
    Remove pointless "vile hack" that can cause the build step to fail when some extension modules can't be imported.
    https://hg.python.org/cpython/rev/94af1af93670

    @pitrou
    Copy link
    Member Author

    pitrou commented Sep 30, 2014

    The change I just pushed should fix the failures, waiting for the buildbots to compile now.

    @pitrou
    Copy link
    Member Author

    pitrou commented Sep 30, 2014

    Looks ok on OpenIndiana, closing now.

    @pitrou pitrou closed this as completed Sep 30, 2014
    @jtaylor
    Copy link
    Mannequin

    jtaylor mannequin commented Jan 16, 2015

    very nice, thanks for adding this.

    coincidentally numpy added the same to numpy.distutils independently just a week later, though numpy also accepts an environment variable to set the number of jobs.
    This is useful for e.g. pip installations where one does not control the command line. Also an environment variable allows parallel jobs in environments where it is not guaranteed that the feature is available. E.g. you could just put it into your .bashrc and when building with 3.5 it will just work and 2.7 will not fail.

    Is the naming --parallel/j already fixed? I'll change the numpy options to the same name then.

    Please also add it to the release notes so the feature can be discovered easier.

    @berkerpeksag
    Copy link
    Member

    berkerpeksag commented Jan 22, 2015

    Here's a doc patch.

    @python-dev
    Copy link
    Mannequin

    python-dev mannequin commented Jan 27, 2015

    New changeset 107669985805 by Berker Peksag in branch 'default':
    Add whatsnew entry for issue bpo-5309.
    https://hg.python.org/cpython/rev/107669985805

    @python-dev
    Copy link
    Mannequin

    python-dev mannequin commented Apr 11, 2015

    New changeset fc28c67fbbd3 by doko in branch 'default':

    @giampaolo
    Copy link
    Contributor

    giampaolo commented Sep 17, 2016

    I have 2 complaints about this:

    1 - doc is missing: the only way to be aware of this is either by reading the 3.6 what's new doc or by checking the cmdline helper

    2 - -j "N" parameter could be optional: if not specified os.cpu_count() can be used.

    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Labels
    build The build process and cross-build type-feature A feature request or enhancement
    Projects
    None yet
    Development

    No branches or pull requests

    7 participants