C++ compilation support for distutils

[ahonkela]

BPO	1222585
Nosy	@vstinner, @tiran, @benjaminp, @tarekziade, @merwok, @dhduvall, @skrah, @jdemeyer, @grimreaper, @ryandesign
Files	python_distutils_c++_support.patch: Patch against CVS HEAD to add C++ compilation support to distutils. py26_cxx_compiler.patch: Patch against release26-maint py25_cxx_compiler.patch: Patch against release25-maint distutils2-C++.patch: Patch for Distutils2 python-2.7-distutils-C++.patch python-3.3-distutils-C++.patch python-3.4-distutils-C++.patch python-3.x-distutils-C++.patch python-3.8-distutils-C++.patch python-3.x-distutils-C++.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2020-10-22.21:08:31.465>
created_at = <Date 2005-06-17.12:29:06.000>
labels = ['3.7', 'type-feature', 'library']
title = 'C++ compilation support for distutils'
updated_at = <Date 2020-10-22.21:08:31.465>
user = 'https://bugs.python.org/ahonkela'

bugs.python.org fields:

activity = <Date 2020-10-22.21:08:31.465>
actor = 'eric.araujo'
assignee = 'none'
closed = True
closed_date = <Date 2020-10-22.21:08:31.465>
closer = 'eric.araujo'
components = ['Distutils']
creation = <Date 2005-06-17.12:29:06.000>
creator = 'ahonkela'
dependencies = []
files = ['6696', '15389', '15390', '16742', '32218', '32219', '32220', '47046', '47669', '48016']
hgrepos = []
issue_num = 1222585
keywords = ['patch', 'needs review']
message_count = 35.0
messages = ['48477', '59307', '93211', '95629', '95631', '95633', '95640', '95641', '100804', '100805', '100806', '100807', '101588', '101602', '101626', '101627', '102216', '102231', '115762', '144117', '144191', '158200', '158223', '178660', '213211', '242706', '299396', '299403', '314859', '337380', '371688', '371689', '371757', '371840', '379348']
nosy_count = 21.0
nosy_names = ['mdboom', 'vstinner', 'christian.heimes', 'ahonkela', 'benjamin.peterson', 'tarek', 'eric.araujo', 'rpetrov', 'Arfrever', 'dhduvall', 'skrah', 'drkirkby', 'jdemeyer', 'eitan.adler', 'barcc', 'Michael.Crusoe', 'ehvatum', 'Christian H', 'Josef Kemetm\xc3\xbcller', 'bje', 'ryandesign']
pr_nums = []
priority = 'normal'
resolution = 'wont fix'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue1222585'
versions = ['Python 3.6', 'Python 3.7']

[ahonkela]

Add more proper C++ compilation support to distutils by
propagating information on used language to compiler as
well as linker. The information is used by
unixccompiler, cygwinccompiler and emxccompiler to call
a C++ compiler instead of a C compiler. Options for the
call are formed by as a union of options to both C and
C++ compilers.

The old version worked by accident on some platforms
where you can compile C++ programs by calling the C
compiler (e.g. gcc), but not on others (e.g. Tru64
Unix, probably also others not using gcc).

The more compilcated handling of options than with
linker is necessary to properly support manual
overrides such as "CXX=g++ -foo".

The attached patch is against CVS HEAD, but applies
with minimal changes at least to Python 2.3.

[tiran]

Isn't it already fixed? Need to check it at the upcoming bug day.

[drkirkby]

Just to add, I'd much appreciate if this bug could be squashed. I see
there is a patch, but it is against a 3-year old version, and is not a
small patch. Hence I am reluctant to try to apply it myself.

It is causing a problem in the Sage project, as I tried to compile Sage
with Sun's compiler on Solaris, rather than gcc.

[mdboom]

This is a rather serious bug for matplotlib -- since the C++ compiler is
not correctly selected, it can't be built with many Unix compilers, such
as Sun Studio.

[tarekziade]

Has someone tried to apply it with a recent Python ?

I can work on it but I'll need some help from someone matplotlib or Sage
I guess..

[mdboom]

I'm testing this against release25_maint and release26_maint as we
speak. I'll post new patches when that's done.

[mdboom]

Attached is a patch against the release26-maint branch. This works for
me on Sun Workshop 6.

Note that on Sun, the C compiler is 'cc' and the C++ compiler is 'CC',
but distutils uses 'cc' and 'c++'. However, the CXX environment
variable now works (after applying the patch), so this is less of an
issue, if it can be considered a bug at all.

[mdboom]

Here's another patch against release25_maint for the benefit of those
still using Python 2.5. It's different only in some whitespace changes.

[Arfrever]

I'm attaching my own patch, which has the following advantages:

• Determination of target language is performed per source file instead of per list of source files passed to CCompiler.compile().
• Proper determination of linker.
• Proper support for CXXFLAGS and other variables.

[tarekziade]

Moved to distutils2 (distutils is feature frozen now)

[mdboom]

Does the reassignment only apply to the more recent more ambitious patch? The original patch (and my forward porting of it) is purely a bugfix, not a feature freeze. This is a showstopper bug for matplotlib on Solaris, for example, and I would hate to see its application further delayed.

[tarekziade]

If we consider that Distutils didn't provide c++ support, and that it partially worked by accident (through gcc), I would call it a new feature. Especially since it requires a new option.

We are freezing the API so we can't add options to methods/functions.

That's why it could be done on distutils2 instead, which will be released when 2.7 final is released (but not in the stdlib), and backward compatible with 2.4, 2.5, 2.6.

If we could do the proper job for C++ support in distutils2 that would be better.

Notice that distutils2 is a fork of how distutils was in trunk last month, so it should be fairly simple to depend on it in your project, to benefit from new features etc. The goal is to push it back in the stdlib in 3.3.

If you want to start a dev. branch of your project using distutils2, I am willing to help there.

[Arfrever]

To have proper support for C++ linking in distutils2 at least in when used with Python 2.7, Makefile should set a variable, which will specify default C++ linker. I think that LDCXXSHARED can be used as a name for this new variable. I'm attaching the patch for configure.in and Makefile.pre.in.

[rpetrov]

I could not found reasons python build to support C++ compiler as all source is C!
Another case is distutils where is possible external module to use C++ code.

[Arfrever]

Python can be compiled using C++ compiler.
See http://docs.python.org/extending/extending.html#writing-extensions-in-c

[Arfrever]

Also you would have to pass --with-cxx-main=<compiler> option to configure.

[benjaminp]

I think python-LDCXXSHARED.patch is fine.

[tarekziade]

LDCXXSHARED added in r79652 and r79657.

Now we can work on distutils2 side

[merwok]

Arfrever’s patch looks good. There is no tests for detect_language now; should the patch be blocked by that?

(Note: Remember to use “hg import --user 'Arfrever Frehtes Taifersar Arahesis <preferred email>'” to give proper credit)

[djc]

Could this get some attention, please?

[merwok]

Well, there’s no rush, Python 3.3b1 is not next week.

I’m willing to apply the patch to packaging and distutils2 (in a week or two when I’ve finished a massive cleanup and caught up with packaging), but I’d like tests for detect_language, and if possible tests for compiling C++ extensions/libs too.

[djc]

Ping, again. I'm sorry, I didn't write any of these patches and would not be a great fit for writing tests.

[merwok]

I should be able to port the patch and add tests for detect_language, but I know very little about C++ and may not be able to write a full test that really compiles and checks a C++ program. We’re having a sprint on the 21, I’ll see if I can work with another participant to do this. A sample short C++ source file would help (just use some Python/C function and print something).

[Arfrever]

I attach updated patches for distutils in case somebody wants to use them. (I privately update them once per week.)

[merwok]

Now that distutils2 development is stopped and that distutils is no longer under a feature freeze, this is a reasonable thing to add to 3.5. Thanks for your efforts Arfrever, and sorry you had to port and maintain this to so many different versions.

[ehvatum]

The 2012-12-31 of this patch (currently the latest) has the surprising effect of causing distutils CFLAGS to be dropped. See for reference: https://bugs.gentoo.org/show_bug.cgi?id=548776

The issue I am experiencing is resolved by changing the section of the patch that reads:

+- (cc, cxx, opt, cflags, ccshared, ldshared, shlib_suffix, ar, ar_flags) = \
+- get_config_vars('CC', 'CXX', 'OPT', 'CFLAGS',
+- 'CCSHARED', 'LDSHARED', 'SHLIB_SUFFIX', 'AR', 'ARFLAGS')
++ (cc, cxx, ccshared, ldshared, ldcxxshared, shlib_suffix, ar, ar_flags) = \
++ get_config_vars('CC', 'CXX', 'CCSHARED', 'LDSHARED', 'LDCXXSHARED',
++ 'SHLIB_SUFFIX', 'AR', 'ARFLAGS')
++
++ cflags = ''
++ cxxflags = ''

to:

+- (cc, cxx, opt, cflags, ccshared, ldshared, shlib_suffix, ar, ar_flags) = \
+- get_config_vars('CC', 'CXX', 'OPT', 'CFLAGS',
+- 'CCSHARED', 'LDSHARED', 'SHLIB_SUFFIX', 'AR', 'ARFLAGS')
++ (cc, cxx, cflags, ccshared, ldshared, ldcxxshared, shlib_suffix, ar, ar_flags) = \
++ get_config_vars('CC', 'CXX', 'CFLAGS', 'CCSHARED', 'LDSHARED', 'LDCXXSHARED',
++ 'SHLIB_SUFFIX', 'AR', 'ARFLAGS')
++
++ cxxflags = cflags

This change causes the CFLAGS outputted by "python-config --cflags" to be used by distutils, as they are without the patch applied. To me, losing those CFLAGS, especially when building plain C python extensions, seems like a significant regression.

[JosefKemetmller]

python-3.4-distutils-C++.patch with changes of Erik Hvatum applied.

[JosefKemetmller]

python-3.4-distutils-C++.patch with Erik Hvatum's suggestions applied.

[bje]

Is there a simple workaround that one can put into setup.py in the meantime? I tried using compiler.compiler.remove('-Wstrict-prototypes') to no avail.

[jdemeyer]

I tried using compiler.compiler.remove('-Wstrict-prototypes') to no avail.

The -Wstrict-prototypes issue is a separate bug. It is fixed in Python >= 3.6 and there is an open backport PR for 2.7: #7476

[ryandesign]

What needs to happen to get this 15 year old bug fixed? It prevents C++ Python modules from being compiled in situations where the user needs to supply CXXFLAGS.

[tiran]

Please report the issue with setuptools. distutils is no longer under development. We recommend that all users use setuptools.

[vstinner]

What needs to happen to get this 15 year old bug fixed? It prevents C++ Python modules from being compiled in situations where the user needs to supply CXXFLAGS.

Someone has to propose a PR, someone else to review it, and a core developer has to take the responsibility to merge that PR. So far, there is no PR created.

[ryandesign]

Christian, thanks for the pointer. I think you're right, I probably am actually wanting this to be fixed in setuptools, not distutils, since setuptools is what people are using today. Since setuptools is an offshoot of distutils, I had assumed that the developers of setuptools would take ownership of any remaining distutils bugs that affected setuptools but I guess not. I looked through the setuptools issue tracker and this was the closest existing bug I could find: pypa/setuptools#1732

[merwok]

setuptools and other active build tools are the better target for this feature.