distutils.sysconfig.customize_compiler() overrides CFLAGS var with OPT var if CFLAGS env var is set

[vstinner]

BPO	36235
Nosy	@vstinner, @ned-deily, @koobs, @yan12125, @miss-islington
PRs	bpo-36235: Fix CFLAGS in distutils customize_compiler() #12236 [3.7] bpo-36235: Fix CFLAGS in distutils customize_compiler() (GH-12236) #12348 [2.7] bpo-36235: Fix CFLAGS in distutils customize_compiler() (GH-12236) #12349 bpo-36235: make test_customize_compiler more robust #12380 bpo-36235: Enhance distutils test_customize_compiler() #12403 [3.7] bpo-36235: Enhance distutils test_customize_compiler() (GH-12403) #12415 [2.7] bpo-36235: Enhance distutils test_customize_compiler() (GH-12403) #12417 [2.7] bpo-36235: Fix distutils test_customize_compiler() on macOS #12751 bpo-36235: Fix distutils test_customize_compiler() on macOS #12764 [3.7] bpo-36235: Fix distutils test_customize_compiler() on macOS (GH-12764) #12768

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2019-04-11.01:07:51.548>
created_at = <Date 2019-03-08.12:23:45.361>
labels = ['3.8', 'build', '3.7']
title = 'distutils.sysconfig.customize_compiler() overrides CFLAGS var with OPT var if CFLAGS env var is set'
updated_at = <Date 2019-04-11.01:07:51.547>
user = 'https://github.com/vstinner'

bugs.python.org fields:

activity = <Date 2019-04-11.01:07:51.547>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2019-04-11.01:07:51.548>
closer = 'vstinner'
components = ['Build']
creation = <Date 2019-03-08.12:23:45.361>
creator = 'vstinner'
dependencies = []
files = []
hgrepos = []
issue_num = 36235
keywords = ['patch']
message_count = 19.0
messages = ['337468', '337505', '337990', '337994', '337995', '338051', '338199', '338229', '338252', '338254', '338264', '338265', '338267', '338268', '339785', '339795', '339859', '339899', '339901']
nosy_count = 5.0
nosy_names = ['vstinner', 'ned.deily', 'koobs', 'yan12125', 'miss-islington']
pr_nums = ['12236', '12348', '12349', '12380', '12403', '12415', '12417', '12751', '12764', '12768']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue36235'
versions = ['Python 2.7', 'Python 3.7', 'Python 3.8']

[vstinner]

When a C extension is built by distutils, distutils.sysconfig.customize_compiler() is used to configure compiler flags. Extract of the code:

(cc, cxx, opt, cflags, ccshared, ldshared, shlib_suffix, ar, ar_flags) = \
    get_config_vars('CC', 'CXX', 'OPT', 'CFLAGS',
                    'CCSHARED', 'LDSHARED', 'SHLIB_SUFFIX', 'AR', 'ARFLAGS')

...
if 'CFLAGS' in os.environ:
    cflags = opt + ' ' + os.environ['CFLAGS']
    ldshared = ldshared + ' ' + os.environ['CFLAGS']
if 'CPPFLAGS' in os.environ:
    cpp = cpp + ' ' + os.environ['CPPFLAGS']
    cflags = cflags + ' ' + os.environ['CPPFLAGS']
    ldshared = ldshared + ' ' + os.environ['CPPFLAGS']
...

If the CFLAGS environment variable is set, the 'CFLAGS' configuration variable is overriden with the 'OPT' configuration variable: cflags = opt + ...

This bug has been fixed since 2013 in Fedora and RHEL by this patch:
https://src.fedoraproject.org/rpms/python2/blob/master/f/00168-distutils-cflags.patch

RHEL bug report:
https://bugzilla.redhat.com/show_bug.cgi?id=849994

I converted that patch to a pull request.

[ned-deily]

This appears to be a duplicate of bpo-969718.

[vstinner]

New changeset 86082c2 by Victor Stinner in branch 'master':
bpo-36235: Fix CFLAGS in distutils customize_compiler() (GH-12236)
86082c2

[vstinner]

New changeset 37f6971 by Victor Stinner in branch '2.7':
bpo-36235: Fix CFLAGS in distutils customize_compiler() (GH-12236) (GH-12349)
37f6971

[vstinner]

New changeset 6c0e0d1 by Victor Stinner in branch '3.7':
bpo-36235: Fix CFLAGS in distutils customize_compiler() (GH-12236) (GH-12348)
6c0e0d1

[yan12125]

After this patch, test_distutils failed if $CPPFLAGS is not empty when building CPython. For example:

$ export CPPFLAGS=-DFOO=BAR
$ ./configure
$ make
$ ./python -m test test_distutils   
Run tests sequentially
0:00:00 load avg: 0.45 [1/1] test_distutils
test test_distutils failed -- Traceback (most recent call last):
  File "/home/yen/Projects/cpython/Lib/distutils/tests/test_sysconfig.py", line 99, in test_customize_compiler
    self.assertEqual(comp.exes['compiler'], 'my_cc --sysconfig-cflags --mycflags')
AssertionError: 'my_cc --sysconfig-cflags --mycflags -DFOO=BAR' != 'my_cc --sysconfig-cflags --mycflags'
- my_cc --sysconfig-cflags --mycflags -DFOO=BAR
?                                    

---

+ my_cc --sysconfig-cflags --mycflags

test_distutils failed

== Tests result: FAILURE ==

1 test failed:
test_distutils

Total duration: 2 sec 192 ms
Tests result: FAILURE

Tested with commit d2fdd1f.

This is an issue as Arch Linux uses CPPFLAGS="-D_FORTIFY_SOURCE=2" for creating packages [1].

[1] https://git.archlinux.org/svntogit/packages.git/tree/trunk/makepkg.conf?h=packages/pacman#n39

[vstinner]

Chih-Hsuan Yen: After this patch, test_distutils failed if $CPPFLAGS is not empty when building CPython.

Aha, more sysconfig configuration variables and environment variables should be mocked. I wrote PR 12403 to mock all variables. Would you mind to test and review it?

[yan12125]

STINNER Victor: Thank you very much for that much better fix! I've tested it on Arch Linux. With that patch, test_distutils passes as usual. The pull request looks good, too. Just a question: I think it should be backported to 3.7 and 2.7 branches like #12236?

[vstinner]

New changeset 72c7b37 by Victor Stinner in branch 'master':
bpo-36235: Enhance distutils test_customize_compiler() (GH-12403)
72c7b37

[vstinner]

Just a question: I think it should be backported to 3.7 and 2.7 (...)

Right, I wrote PRs for that. I didn't use GitHub labels because the bot is currently broken.

[vstinner]

New changeset dd1cfef by Victor Stinner in branch '3.7':
bpo-36235: Enhance distutils test_customize_compiler() (GH-12403) (GH-12415)
dd1cfef

[vstinner]

New changeset 8c380e9 by Victor Stinner in branch '2.7':
bpo-36235: Enhance distutils test_customize_compiler() (GH-12403) (GH-12417)
8c380e9

[vstinner]

Thanks for the review, I merged my PR and the test enhancements.

[yan12125]

Oh, I didn't know the bot is not working for now. Thank you for making the test more robust!

[vstinner]

Oh, the test failed on Python 2.7 on macOS:

x86-64 High Sierra 2.7
https://buildbot.python.org/all/#/builders/140/builds/211

======================================================================
FAIL: test_customize_compiler (distutils.tests.test_sysconfig.SysconfigTestCase)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/Users/buildbot/buildarea/2.7.billenstein-sierra/build/Lib/distutils/tests/test_sysconfig.py", line 134, in test_customize_compiler
    'sc_cc -E')
AssertionError: '/usr/bin/clang -E' != 'sc_cc -E'

[vstinner]

New changeset 22de4ce by Victor Stinner in branch '2.7':
bpo-36235: Fix distutils test_customize_compiler() on macOS (GH-12751)
22de4ce

[vstinner]

Good, my change fixed x86-64 High Sierra 2.7 buildbot:
https://buildbot.python.org/all/#/builders/140/builds/216

I wrote PR 12764 to forward-port the fix to the master branch.

[vstinner]

New changeset a9bd892 by Victor Stinner in branch 'master':
bpo-36235: Fix distutils test_customize_compiler() on macOS (GH-12764)
a9bd892

[miss-islington]

New changeset d9b25a2 by Miss Islington (bot) in branch '3.7':
bpo-36235: Fix distutils test_customize_compiler() on macOS (GH-12764)
d9b25a2