rename the platform directory from plat-$(MACHDEP) to plat-$(PLATFORM_TRIPLET)

[doko42]

BPO	23968
Nosy	@warsaw, @doko42, @ncoghlan, @vstinner, @larryhastings, @ned-deily, @freakboy3742, @xdegaye, @zware, @koobs, @tpetazzoni, @rkuska, @wscullin, @yan12125
Files	platdir.diff: patch platdir2.diff issue23968-android-regression.patch: Fix a regression after changeset 5c77488830bc

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2016-08-29.18:04:43.978>
created_at = <Date 2015-04-15.19:04:07.449>
labels = ['expert-installation', 'release-blocker']
title = 'rename the platform directory from plat-$(MACHDEP) to plat-$(PLATFORM_TRIPLET)'
updated_at = <Date 2016-08-31.18:22:20.390>
user = 'https://github.com/doko42'

bugs.python.org fields:

activity = <Date 2016-08-31.18:22:20.390>
actor = 'yan12125'
assignee = 'none'
closed = True
closed_date = <Date 2016-08-29.18:04:43.978>
closer = 'doko'
components = ['Installation']
creation = <Date 2015-04-15.19:04:07.449>
creator = 'doko'
dependencies = []
files = ['39049', '43214', '44260']
hgrepos = []
issue_num = 23968
keywords = ['patch', 'needs review']
message_count = 32.0
messages = ['241143', '243874', '243896', '244362', '244411', '244605', '248423', '267342', '267352', '268528', '268530', '268536', '268829', '268833', '269237', '269240', '269360', '271636', '271690', '271691', '271692', '271693', '272790', '273179', '273182', '273185', '273875', '273876', '273903', '273904', '274042', '274056']
nosy_count = 17.0
nosy_names = ['barry', 'doko', 'ncoghlan', 'vstinner', 'larry', 'ned.deily', 'Arfrever', 'freakboy3742', 'xdegaye', 'python-dev', 'zach.ware', 'koobs', 'sYnfo', 'thomas-petazzoni', 'rkuska', 'wscullin', 'yan12125']
pr_nums = []
priority = 'release blocker'
resolution = 'fixed'
stage = 'needs patch'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue23968'
versions = ['Python 3.6']

[doko42]

Currently there is only one platform directory for all linux architectures, there are two reasons to have a separate directory for each architecture:

• the contents of the files in this directory differ
  (most prominent case are the RTLD_* constants on various
  architectures).

• The _sysconfigdata.py module is installed in the platdir.
  A cross build of the extensions in the standard library
  is done by running the python interpreter for the build
  machine with the platdir for the host machine. This
  currently works for cross builds targeting a different
  os, but not cross building from one linux architecture
  to another.

This would then remove Lib/plat-linux and introduce some Lib/plat-$(PLATFORM_TRIPLET) directories.

[doko42]

ping? I would like to get this into 3.5.

[ncoghlan]

+1 from me.

[sYnfo]

+1 from the Python team from Fedora, the patch looks good from downstream standpoint.

[warsaw]

I don't know what problems I might have run into previously, but it's working now and seems fine to me. +1

[Arfrever]

What Lib/plat-$(PLATFORM_TRIPLET) directories will be introduced?

This patch would make situation worse on rarely used architectures (these without Lib/plat-$(PLATFORM_TRIPLET) distributed in repository of CPython) if patch for bug bpo-12619 is not applied.

[wscullin]

This would likely improve life for folks with cross-compile toolchains.

[doko42]

attaching an updated patch. besides renaming of the platform directory, the following changes are made:

• it makes the name of the internal _sysconfigdata module dependent
  on the abiflags
• it moves the _sysconfigdata module into the platform directory.
  This address as well bpo-22724, it should address bpo-18083 as well,
  provided that some multiarch tuples for solaris are defined.
• it fixes regeneration of the Python modules from C headers for
  cross builds.
• it introduces a sys.implementation._multiarch attribute to construct
  the name of the platform directory without having the
  _sysconfigdata module available (which now is located in the platform
  dir)

One outstanding issue probably is the generation of the platform directory in the source tree, not in the build tree (but this is existing behaviour).

This should address bpo-12619 (this patch probably needs an update),

[ncoghlan]

This may be of interest for iOS and Android cross-compilation, so I've added Xavier de Gaye to the nosy list, while Russell Keith-Magee added himself.

[python-dev]

New changeset 78d2cb7f66b6 by doko in branch 'default':

• Issue bpo-23968: Rename the platform directory from plat-$(MACHDEP) to
  https://hg.python.org/cpython/rev/78d2cb7f66b6

[doko42]

now checked in, after 3.6.0a2, to watch behaviour on different platforms. Still needs doc updates.

[python-dev]

New changeset a062c60c9f7d by doko in branch 'default':

• Issue bpo-23968: Update distutils/sysconfig.py to look for the renamed
  https://hg.python.org/cpython/rev/a062c60c9f7d

[python-dev]

New changeset e2b09c00ee24 by Ned Deily in branch 'default':
Issue bpo-23968: Fix installs of the renamed config directory for OS X
https://hg.python.org/cpython/rev/e2b09c00ee24

[ned-deily]

There is a problem with the current implementation. When running ./configure using compilers other than a current gcc, for example, when using clang or older versions of gcc, the build variable MULTIARCH may not get set because those compilers do not support --print-multiarch:

configure.ac:744
MULTIARCH=$($CC --print-multiarch 2>/dev/null)

But, a bit later in ./configure, PLATFORM_TRIPLET gets set as a result of running conftest.c. The end result is that PLATFORM_TRIPET is non-empty but MULTIARCH is empty. Still later in configure.ac (around 4477:

if test x$PLATFORM_TRIPLET = x; then
LIBPL='$(prefix)'"/lib/python${VERSION}/config-${LDVERSION}"
else
LIBPL='$(prefix)'"/lib/python${VERSION}/config-${LDVERSION}-${PLATFORM_TRIPLET}"
fi

So, the value of LIBPL is extended to include PLATFORM_TRIPLET regardless of whether MULTIARCH has a value.

This causes problems at runtime because get_makefile_filename() in Lib/sysconfig.py now uses the value of sys.implemntation's _multiarch attribute (whose value is compiled in sysmodule.c from MULTIARCH) to form the path to the config directory (the equivalent of LIBPL). In these cases, sys.implementation will have no_multiarch attribute so sysconfig looks for the config directory in the pre-3.6 location (e.g. .../lib/python3.6/config-3.6m) even though it was installed in the new location (.../lib/python3.6/config-3.6m-darwin). Fortunately, both test_sysconfig and test_distutils catch this:

======================================================================
FAIL: test_srcdir (test.test_sysconfig.TestSysConfig)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/py/dev/3x/root/uxd/lib/python3.6/test/test_sysconfig.py", line 346, in test_srcdir
    self.assertTrue(os.path.isdir(srcdir), srcdir)
AssertionError: False is not true : /py/dev/3x/root/uxd/lib/python3.6/config-3.6dm

======================================================================
FAIL: test_get_makefile_filename (test.test_sysconfig.MakefileTests)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/py/dev/3x/root/uxd/lib/python3.6/test/test_sysconfig.py", line 414, in test_get_makefile_filename
    self.assertTrue(os.path.isfile(makefile), makefile)
AssertionError: False is not true : /py/dev/3x/root/uxd/lib/python3.6/config-3.6dm/Makefile

======================================================================
FAIL: test_srcdir (distutils.tests.test_sysconfig.SysconfigTestCase)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/py/dev/3x/root/uxd/lib/python3.6/distutils/tests/test_sysconfig.py", line 61, in test_srcdir
    self.assertTrue(os.path.isdir(srcdir), srcdir)
AssertionError: False is not true : /py/dev/3x/root/uxd/lib/python3.6/config-3.6dm

---

I'm not sure what the best solution is.

[xdegaye]

The same problem occurs when cross-compiling for Android and running test_sysconfig and test_distutils.

Some other points:

• The generation of the platform directory breaks the support of read only source trees. The files of this directory are not tracked by mercurial and printed by 'hg status'.

• The new files in the platform directory are regenerated from C headers of the build machine.

[xdegaye]

The same problem occurs when cross-compiling for Android and running test_sysconfig and test_distutils.

This is misleading. Actually, test_sysconfig and test_distutils do not fail when run from the source tree, but fail on an installed python, whatever the compilation mode: native or cross-compilation.

[xdegaye]

Regeneration of the platdir files needs to be aware of the value of sysroot [1]. For example on Android cross-builds, sysroot must be set to the path of the headers and libraries for the targeted Android API level corresponding to a version of the Android libc and a version of Android. So, automatically regenerating those files in this case requires to look into the CC, CFLAGS and CPPFLAGS environment variables for sysroot. OTOH RTLD_* constants are exposed in the posix module (bpo-13226) and the platdir files usefulness is limited now.

Cross building from one linux architecture to another is possible with gnu make VPATH support by building outside the source tree. Android is also a linux architecture and building outside the source tree makes sense, not only to work around the problem described in the second item of msg241143, but also to build for different Android API levels and identify the results of those builds.

It is not clear that the changes made in this issue fixes correctly bpo-22724, see msg269359.

I think the scope of this isssue should be restricted to multiarch build systems, i.e. when gcc has been configured with '--enable-multiarch' [2].

[1] Options for Directory Search: https://gcc.gnu.org/onlinedocs/gcc/Directory-Options.html
[2] Installing GCC: Configuration: https://gcc.gnu.org/install/configure.html

[xdegaye]

To reproduce the problem on any system where Python is built with configure:

$ mkdir -p tmp/build tmp/install
$ cd tmp
$ hg clone https://hg.python.org/cpython cpython
$ cd build
$ $(cd ../cpython && pwd)/configure --prefix=$(cd ../install && pwd) && make install
$ ../install/bin/python3 -m test -v test_sysconfig
[snip]

======================================================================
FAIL: test_srcdir (test.test_sysconfig.TestSysConfig)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/path_to/tmp/install/lib/python3.6/test/test_sysconfig.py", line 346, in test
_srcdir
    self.assertTrue(os.path.isdir(srcdir), srcdir)
AssertionError: False is not true : /path_to/tmp/install/lib/python3.6/config-3.6m

======================================================================
FAIL: test_get_makefile_filename (test.test_sysconfig.MakefileTests)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/path_to/tmp/install/lib/python3.6/test/test_sysconfig.py", line 414, in test
_get_makefile_filename
    self.assertTrue(os.path.isfile(makefile), makefile)
AssertionError: False is not true : /path_to/tmp/install/lib/python3.6/config-3.6m/Makefile

---

Ran 21 tests in 0.063s

FAILED (failures=2, skipped=2)
test test_sysconfig failed
test_sysconfig failed
1 test failed:
test_sysconfig
Total duration: 0:00:01

[xdegaye]

This problem occured for the first time on the "installed Python" buildbot (setup by Zachary) at
http://buildbot.python.org/all/builders/x86%20Gentoo%20Installed%20with%20X%203.x/builds/747
on the day changeset 78d2cb7f66b6 has been commited (see msg268528).

Those "installed Python" buildbot are very helpful, thanks Zachary.

[xdegaye]

The reason I would like this problem fixed is that the test suite is run on an installed Python when cross-compiling, obviously you cannot run the test suite with the cross-compiled Python on the source tree of the build system.

[ned-deily]

It's not just broken for cross-compiling. As I noted earlier, it's broken when using other than gcc, such as is the case with current OS X systems. This needs to be fixed before release.

[xdegaye]

It is also broken for gcc, see the buildbot 'compile' output.
It seems broken for all non windows builds whatever the compiler.

[python-dev]

New changeset ddc4bdae5e41 by Ned Deily in branch 'default':
Issue bpo-23968: Make OS X installer build script aware of renamed platform
https://hg.python.org/cpython/rev/ddc4bdae5e41

[vstinner]

Any update on this issue?

test_srcdir() of test_sysconfig still fails on "x86 Gentoo Installed with X 3.x" buildbot. I opened the issue bpo-27798 which was closed a duplicate of this issue.

First failure of the buildbot:

http://buildbot.python.org/all/builders/x86%20Gentoo%20Installed%20with%20X%203.x/builds/747

This build contains many changes. I don't know which one introduced the issue?

• change a062c60c9f7d60db2b63af88e2d676e617398d01: Update distutils/sysconfig.py to look for the renamed _sysconfigdata module too
• change 78d2cb7f66b602ec496800fbbe71e590c0f56b00: Rename the platform directory from plat-$(MACHDEP) to plat-$(PLATFORM_TRIPLET)

Ned Deily: "This needs to be fixed before release."

I agree. I use the release blocker priority as a reminder. Well, if the issue cannot be fixed, I suggest to revert changes related to this issue.

"It's not just broken for cross-compiling. As I noted earlier, it's broken when using other than gcc, such as is the case with current OS X systems."

test_sysconfig fails on "x86 Gentoo Installed with X 3.x" buildbot which uses GCC. Extract of the configure output:

checking for gcc... gcc

[doko42]

please could somebody test the attached patch (I'm also trying to setup a gentoo env)? The idea is to keep the multiarch and triplet macros in sync.

diff -r a7f3678f9509 configure.ac
--- a/configure.ac	Sat Aug 20 03:05:13 2016 +0200
+++ b/configure.ac	Sat Aug 20 03:54:39 2016 +0200
@@ -882,6 +882,8 @@
   if test x$PLATFORM_TRIPLET != x$MULTIARCH; then
     AC_MSG_ERROR([internal configure error for the platform triplet, please file a bug report])
   fi
+elif test x$PLATFORM_TRIPLET != x && test x$MULTIARCH = x; then
+  MULTIARCH=$PLATFORM_TRIPLET
 fi
 if test x$PLATFORM_TRIPLET = x; then
   PLATDIR=plat-$MACHDEP

[yan12125]

I've just tried doko's patch on Arch Linux. Before this patch there are three test failures in test_distutils and test_sysconfig, just like Gentoo. After the patch, all tests in test_distutils and test_sysconfig pass.

On my PC both gcc -print-multiarch and gcc --print-multiarch print an empty string.

By the way, could you attach this patch file to roundup, so that others can download it?

Reference: Arch Linux's build script of python-hg: https://aur.archlinux.org/cgit/aur.git/tree/PKGBUILD?h=python-hg

[python-dev]

New changeset 5c77488830bc by doko in branch 'default':

• Issue bpo-23968, keep platform_triplet and multiarch macros in sync
  https://hg.python.org/cpython/rev/5c77488830bc

[doko42]

now fixed.

[yan12125]

Changeset 5c77488830bc brings a regression - Android build fails with the following message:

export H2PY; H2PY="$PYTHON_FOR_BUILD /home/yen/python3-android/src/cpython/build-target/../Tools/scripts/h2py.py"; \
cd ../Lib/plat-aarch64-linux-gnu; ./regen
+ _PYTHON_PROJECT_BASE=/home/yen/python3-android/src/cpython/build-target _PYTHON_HOST_PLATFORM=linux-aarch64 PYTHONPATH=/home/yen/python3-android/src/cpython/build-target/build/lib.linux-aarch64-3.6:../Lib:../Lib/plat-aarch64-linux-gnu python3.6 /home/yen/python3-android/src/cpython/build-target/../Tools/scripts/h2py.py -i '(u_long)' /opt/android-ndk/platforms/android-21/arch-arm64/usr/include/sys/types.h /opt/android-ndk/platforms/android-21/arch-arm64/usr/include/netinet/in.h /opt/android-ndk/platforms/android-21/arch-arm64/usr/include/dlfcn.h
./regen: line 33: _PYTHON_PROJECT_BASE=/home/yen/python3-android/src/cpython/build-target: No such file or directory
make[1]: *** [Makefile:1324: ../Lib/plat-aarch64-linux-gnu] Error 127
make[1]: Leaving directory '/home/yen/python3-android/src/cpython/build-target'
make: *** [Makefile:36: python] Error 2

The previous changeset 1902e1d79e25 builds fine. Prepending PYTHON_FOR_BUILD in configure.ac with env makes the whole build work again.

From the message seems plat-* directories are re-generated with a different name. Is it expected?

[yan12125]

Oh didn't see the title of this issue. Here's the patch for Android.

[doko42]

I don't see how the suggested android fix is related, and what it is supposed to fix. It adds the env command before setting the environment variables, which should be a no-op.

However I think we should introduce different platform triplets for android. Please follow-up in http://bugs.python.org/issue27917

[yan12125]

Well changes in this issue are not the root cause of my problem - it just exposes a broken implementation done long time ago. I'll examine more and open a new bug.