Python fails to build _asyncio on module on AIX

[vstinner]

BPO	39936
Nosy	@vstinner, @aixtools, @isidentical
PRs	bpo-39936: _aix_support uses _bootsubprocess #18970
Files	setup-aix.patch make.log _aix_support.py

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2020-03-13.07:47:36.675>
created_at = <Date 2020-03-11.17:42:14.411>
labels = ['build', '3.9']
title = 'Python fails to build _asyncio on module on AIX'
updated_at = <Date 2020-03-13.07:47:36.674>
user = 'https://github.com/vstinner'

bugs.python.org fields:

activity = <Date 2020-03-13.07:47:36.674>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2020-03-13.07:47:36.675>
closer = 'vstinner'
components = ['Build']
creation = <Date 2020-03-11.17:42:14.411>
creator = 'vstinner'
dependencies = []
files = ['48970', '48971', '48972']
hgrepos = []
issue_num = 39936
keywords = ['patch']
message_count = 20.0
messages = ['363946', '363948', '363950', '363951', '363952', '363960', '363965', '363967', '363970', '363971', '363972', '363977', '363979', '363987', '364037', '364045', '364048', '364054', '364075', '364076']
nosy_count = 3.0
nosy_names = ['vstinner', 'Michael.Felt', 'BTaskaya']
pr_nums = ['18970']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue39936'
versions = ['Python 3.9']

[vstinner]

The commit 1ec63b6 of bpo-39763 broke the Python compilation on AIX:
https://bugs.python.org/issue39763#msg363749

--

The last successful build was before the commit 1ec63b6:
https://buildbot.python.org/all/#/builders/119/builds/383

_socket compilation:
(...) -o build/lib.aix-7100-9898-32-3.9-pydebug/_socket.so

pythoninfo:

sys.path: [
'/home/buildbot/buildarea/3.x.aixtools-aix-power6/build',
'/home/buildbot/buildarea/3.x.aixtools-aix-power6/build/target/lib/python39.zip',
'/home/buildbot/buildarea/3.x.aixtools-aix-power6/build/Lib',
'/home/buildbot/buildarea/3.x.aixtools-aix-power6/build/build/lib.aix-7100-9898-32-3.9-pydebug',
'/home/buildbot/.local/lib/python3.9/site-packages']

Both steps use "build/lib.aix-7100-9898-32-3.9-pydebug/" directory.

--

Recent failure:
https://buildbot.python.org/all/#/builders/119/builds/451

_socket compilation:

(...) -o build/lib.aix-7104-1806-32-3.9-pydebug/_socket.so

pythoninfo:

sys.path: [
'/home/buildbot/buildarea/3.x.aixtools-aix-power6/build',
'/home/buildbot/buildarea/3.x.aixtools-aix-power6/build/target/lib/python39.zip',
'/home/buildbot/buildarea/3.x.aixtools-aix-power6/build/Lib',
'/home/buildbot/buildarea/3.x.aixtools-aix-power6/build/build/lib.aix-7100-9898-32-3.9-pydebug',
'/home/buildbot/.local/lib/python3.9/site-packages']

So the compilation uses "build/lib.aix-7104-1806-32-3.9-pydebug/" directory, whereas pythoninfo uses "build/lib.aix-7100-9898-32-3.9-pydebug/" directory.

It can explain why setup.py fails to import _socket later: it was built in a different directory.

--

I see that _aix_support._aix_bosmp64() has two code paths depending on the subprocess module can be imported:

# subprocess is not necessarily available early in the build process
# if not available, the config_vars are also definitely not available
# supply substitutes to bootstrap the build
try:
import subprocess
_have_subprocess = True
_tmp_bd = get_config_var("AIX_BUILDDATE")
_bgt = get_config_var("BUILD_GNU_TYPE")
except ImportError: # pragma: no cover
_have_subprocess = False
_tmp_bd = None
_bgt = "powerpc-ibm-aix6.1.7.0"

def _aix_bosmp64():
    # type: () -> Tuple[str, int]
    """
    Return a Tuple[str, int] e.g., ['7.1.4.34', 1806]
    The fileset bos.mp64 is the AIX kernel. It's VRMF and builddate
    reflect the current ABI levels of the runtime environment.
    """
    if _have_subprocess:
        # We expect all AIX systems to have lslpp installed in this location
        out = subprocess.check_output(["/usr/bin/lslpp", "-Lqc", "bos.mp64"])
        out = out.decode("utf-8").strip().split(":")  # type: ignore
        # Use str() and int() to help mypy see types
        return str(out[2]), int(out[-1])
    else:
        from os import uname

    osname, host, release, version, machine = uname()
    return "{}.{}.0.0".format(version, release), _MISSING_BD

--

_aix_support._aix_bosmp64() is called by _aix_support.aix_platform() which is called by get_host_platform() of distutils.util.

Currently, setup.py does:

• Inject _bootsubprocess into sys.modules['subprocess'] so "import subprocess" works
• Build all C extensions
• Remove sys.modules['subprocess'], so the next "import subprocess" may or may not load Lib/subprocess.py which uses the newly built C extensions like _posixsubprocess and select
• Attempt to load C extensions: if an import fails, rename the file: it happens for _asyncio on AIX, that's the bug

[vstinner]

AIX_BUILDDATE variable is written in pyconfig.h by configure:

https://buildbot.python.org/all/#/builders/119/builds/383

configure: "checking for the system builddate... 1806"

https://buildbot.python.org/all/#/builders/119/builds/451

configure: "checking for the system builddate... 1806"

So AIX_BUILDDATE seems to be set in both case.

But on build 451, the "build/lib.aix-7100-9898-32-3.9-pydebug" directory name contains 9898: this number comes from _aix_support._MISSING_BD. It's used when _have_subprocess is false or when get_config_var("AIX_BUILDDATE") cannot be converted into an integer.

[aixtools]

While looking through the history of bot builds - I consistently see 420 tests being done on one bot - but test_socket does not always show up in the list.

I'll look at this as I can, but "free-time" is limited. Delay is not a lack of interest.

[vstinner]

Michael.Felt: Can you please try to build the master branch of Python with the attached setup-aix.patch applied?

Something like:

git checkout master
git apply setup-aix.patch
./configure CFLAGS="-O0"
make 2>&1|tee make.log

Then attach make.log.

[vstinner]

Another approach to patch _aix_support.py, replace:

try:
import subprocess
_have_subprocess = True
_tmp_bd =
_bgt = get_config_var("BUILD_GNU_TYPE")
except ImportError: # pragma: no cover
_have_subprocess = False
_tmp_bd = None
_bgt = "powerpc-ibm-aix6.1.7.0"

with:

try:
import subprocess
_have_subprocess = True
_tmp_bd = get_config_var("AIX_BUILDDATE")
_bgt = get_config_var("BUILD_GNU_TYPE")
except ImportError: # pragma: no cover
import _bootsubprocess as subprocess
_have_subprocess = True
_tmp_bd = None
_bgt = "powerpc-ibm-aix6.1.7.0"

Use _bootsubprocess when subprocess is not available. If this approach works, _have_subprocess=False code path can be removed.

[isidentical]

I tried your patch on AIX 7.2, looks like you need to shift sys import to the top

Traceback (most recent call last):
  File "/home/isidentical/cpython/./setup.py", line 16, in <module>
    sys.modules['subprocess'] = _bootsubprocess
NameError: name 'sys' is not defined
make: The error code from the last command is 1.

Stop.

[vstinner]

Yeah, sys must be imported first. Did you try to fix the patch? Does it work?

[aixtools]

I'll take a look at what you are suggesting.

The starting point (before the rm command) is the make command that I run again.

What I notice - read am thinking - is that _socket.so is being created by the "setup.py build" - so, if you can help me make that bit more verbose, I'll work on that too.

+++
Script command is started on Wed Mar 11 15:29:44 CDT 2020.
$
$
$
$ rm ./build/lib.aix-7200-1543-32-3.9/_socket.so ./build/temp.aix-7200-1543-32-3.9/home/aixtools/python/cpython-master/Modules/socketmodule.o
$
$
$ make V=1
CC='gcc -pthread' LDSHARED='Modules/ld_so_aix gcc -pthread -bI:Modules/python.exp ' OPT='-DNDEBUG -g -fwrapv -O3 -Wall' _TCLTK_INCLUDES='' _TCLTK_LIBS='' ./python -E ./setup.py build
running build
running build_ext
ldd: /lib/libreadline.a: File is an archive.
building '_asyncio' extension
gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter -Wno-missing-field-initializers -Werror=implicit-function-declaration -fvisibility=hidden -I./Include/internal -I./Include
-I. -I/home/aixtools/python/cpython-master/Include -I/home/aixtools/python/cpython-master -c /home/aixtools/python/cpython-master/Modules/_asynciomodule.c -o build/temp.aix-7200-1543-32-3.9/home/aixtools/python/cpython-master/Modules/_asynciomodule.o
Modules/ld_so_aix gcc -pthread -bI:Modules/python.exp build/temp.aix-7200-1543-32-3.9/home/aixtools/python/cpython-master/Modules/_asynciomodule.o -o build/lib.aix-7200-1543-32-3.9/_asyncio.so
building '_socket' extension
gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter -Wno-missing-field-initializers -Werror=implicit-function-declaration -fvisibility=hidden -I./Include/internal -I./Include
-I. -I/home/aixtools/python/cpython-master/Include -I/home/aixtools/python/cpython-master -c /home/aixtools/python/cpython-master/Modules/socketmodule.c -o build/temp.aix-7200-1543-32-3.9/home/aixtools/python/cpython-master/Modules/socketmodule.o
Modules/ld_so_aix gcc -pthread -bI:Modules/python.exp build/temp.aix-7200-1543-32-3.9/home/aixtools/python/cpython-master/Modules/socketmodule.o -o build/lib.aix-7200-1543-32-3.9/_socket.so
*** WARNING: renaming "_asyncio" since importing it failed: No module named '_socket'

The following modules found by detect_modules() in setup.py, have been
built by the Makefile instead, as configured by the Setup files:
_abc atexit pwd
time

Following modules built successfully but were removed because they could not be imported:
_asyncio

[aixtools]

So, with the patch - the process stops at:

aixtools@x064:[/home/aixtools/python-3.9]make
 CC='xlc_r' LDSHARED='Modules/ld_so_aix xlc_r -bI:Modules/python.exp    ' OPT='-DNDEBUG -O'  _TCLTK_INCLUDES='' _TCLTK_LIBS=''  ./python -E ./setup.py  build
Traceback (most recent call last):
  File "/home/aixtools/python-3.9/Lib/subprocess.py", line 72, in <module>
    import msvcrt
ModuleNotFoundError: No module named 'msvcrt'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/aixtools/python-3.9/./setup.py", line 4, in <module>
    import subprocess
  File "/home/aixtools/python-3.9/Lib/subprocess.py", line 77, in <module>
    import _posixsubprocess
ModuleNotFoundError: No module named '_posixsubprocess'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/aixtools/python-3.9/./setup.py", line 16, in <module>
    sys.modules['subprocess'] = _bootsubprocess
NameError: name 'sys' is not defined
make: 1254-004 The error code from the last command is 1.

Stop.

[aixtools]

So, this is what I have on screen. Will add the log in a moment.

[1] + Done make 2>&1|tee make.log &
aixtools@x064:[/home/aixtools/python-3.9]find . | grep socket
./Doc/howto/sockets.rst
./Doc/library/socket.rst
./Doc/library/socketserver.rst
./Lib/pycache/socket.cpython-39.opt-1.pyc
./Lib/pycache/socket.cpython-39.opt-2.pyc
./Lib/pycache/socket.cpython-39.opt-4.pyc
./Lib/pycache/socket.cpython-39.pyc
./Lib/pycache/socketserver.cpython-39.pyc
./Lib/socket.py
./Lib/socketserver.py
./Lib/test/pycache/mock_socket.cpython-39.pyc
./Lib/test/pycache/test_socket.cpython-39.pyc
./Lib/test/pycache/test_socketserver.cpython-39.pyc
./Lib/test/mock_socket.py
./Lib/test/test_socket.py
./Lib/test/test_socketserver.py
./Modules/socketmodule.c
./Modules/socketmodule.h
./PCbuild/_socket.vcxproj
./PCbuild/_socket.vcxproj.filters
./build/lib.aix-7104-1806-32-3.9/_socket.so
./build/temp.aix-7104-1806-32-3.9/home/aixtools/python-3.9/Modules/socketmodule.o
aixtools@x064:[/home/aixtools/python-3.9]rm ./build/lib.aix-7104-1806-32-3.9/_socket.so ./build/temp.aix-7104-1806-32-3.9/home/aixtools/python-3.9/Modules/socketmodule.o

aixtools@x064:[/home/aixtools/python-3.9]make V=1
CC='xlc_r' LDSHARED='Modules/ld_so_aix xlc_r -bI:Modules/python.exp ' OPT='-DNDEBUG -O' _TCLTK_INCLUDES='' _TCLTK_LIBS='' ./python -E ./setup.py build
running build
running build_ext
INFO: Can't locate Tcl/Tk libs and/or headers
building '_asyncio' extension
xlc_r -DNDEBUG -O -I./Include/internal -I./Include -I. -I/home/aixtools/python-3.9/Include -I/home/aixtools/python-3.9 -c /home/aixtools/python-3.9/Modules/_asynciomodule.c -o build/temp.aix-7104-1806-32-3.9/home/aixtools/python-3.9/Modules/_asynciomodule.o
Modules/ld_so_aix xlc_r -bI:Modules/python.exp build/temp.aix-7104-1806-32-3.9/home/aixtools/python-3.9/Modules/_asynciomodule.o -o build/lib.aix-7104-1806-32-3.9/_asyncio.so
building '_socket' extension
xlc_r -DNDEBUG -O -I./Include/internal -I./Include -I. -I/home/aixtools/python-3.9/Include -I/home/aixtools/python-3.9 -c /home/aixtools/python-3.9/Modules/socketmodule.c -o build/temp.aix-7104-1806-32-3.9/home/aixtools/python-3.9/Modules/socketmodule.o
1500-030: (I) INFORMATION: PyInit__socket: Additional optimization may be attained by recompiling and specifying MAXMEM option with a value greater than 8192.
Modules/ld_so_aix xlc_r -bI:Modules/python.exp build/temp.aix-7104-1806-32-3.9/home/aixtools/python-3.9/Modules/socketmodule.o -o build/lib.aix-7104-1806-32-3.9/_socket.so
*** WARNING: renaming "_asyncio" since importing it failed: No module named '_socket'

Python build finished successfully!
The necessary bits to build these optional modules were not found:
_gdbm _lzma _sqlite3
_tkinter readline
To find the necessary bits, look in setup.py in detect_modules() for the module's name.

The following modules found by detect_modules() in setup.py, have been
built by the Makefile instead, as configured by the Setup files:
_abc atexit pwd
time

Following modules built successfully but were removed because they could not be imported:
_asyncio

[aixtools]

OK. I removed the _aix_support.py from the formula.

After make see the new (rather) old build paths for "socket"

aixtools@x064:[/home/aixtools/python-3.9]find . | grep socket
./Doc/howto/sockets.rst
./Doc/library/socket.rst
./Doc/library/socketserver.rst
./Lib/pycache/socket.cpython-39.opt-1.pyc
./Lib/pycache/socket.cpython-39.opt-2.pyc
./Lib/pycache/socket.cpython-39.opt-4.pyc
./Lib/pycache/socket.cpython-39.pyc
./Lib/pycache/socketserver.cpython-39.pyc
./Lib/socket.py
./Lib/socketserver.py
./Lib/test/pycache/mock_socket.cpython-39.pyc
./Lib/test/pycache/test_socket.cpython-39.pyc
./Lib/test/pycache/test_socketserver.cpython-39.pyc
./Lib/test/mock_socket.py
./Lib/test/test_socket.py
./Lib/test/test_socketserver.py
./Modules/socketmodule.c
./Modules/socketmodule.h
./PCbuild/_socket.vcxproj
./PCbuild/_socket.vcxproj.filters
./build/lib.aix-7.1-3.9/_socket.so
./build/temp.aix-7.1-3.9/home/aixtools/python-3.9/Modules/socketmodule.o

I remove these two files again and run make:

aixtools@x064:[/home/aixtools/python-3.9]make
CC='xlc_r' LDSHARED='Modules/ld_so_aix xlc_r -bI:Modules/python.exp ' OPT='-DNDEBUG -O' _TCLTK_INCLUDES='' _TCLTK_LIBS='' ./python -E ./setup.py build
running build
running build_ext
INFO: Can't locate Tcl/Tk libs and/or headers
building '_asyncio' extension
xlc_r -DNDEBUG -O -I./Include/internal -I./Include -I. -I/home/aixtools/python-3.9/Include -I/home/aixtools/python-3.9 -c /home/aixtools/python-3.9/Modules/_asynciomodule.c -o build/temp.aix-7.1-3.9/home/aixtools/python-3.9/Modules/_asynciomodule.o
Modules/ld_so_aix xlc_r -bI:Modules/python.exp build/temp.aix-7.1-3.9/home/aixtools/python-3.9/Modules/_asynciomodule.o -o build/lib.aix-7.1-3.9/_asyncio.so
building '_socket' extension
xlc_r -DNDEBUG -O -I./Include/internal -I./Include -I. -I/home/aixtools/python-3.9/Include -I/home/aixtools/python-3.9 -c /home/aixtools/python-3.9/Modules/socketmodule.c -o build/temp.aix-7.1-3.9/home/aixtools/python-3.9/Modules/socketmodule.o
1500-030: (I) INFORMATION: PyInit__socket: Additional optimization may be attained by recompiling and specifying MAXMEM option with a value greater than 8192.
Modules/ld_so_aix xlc_r -bI:Modules/python.exp build/temp.aix-7.1-3.9/home/aixtools/python-3.9/Modules/socketmodule.o -o build/lib.aix-7.1-3.9/_socket.so
*** WARNING: renaming "_asyncio" since importing it failed: No module named '_socket'

Python build finished successfully!
The necessary bits to build these optional modules were not found:
_gdbm _lzma _sqlite3
_tkinter readline
To find the necessary bits, look in setup.py in detect_modules() for the module's name.

The following modules found by detect_modules() in setup.py, have been
built by the Makefile instead, as configured by the Setup files:
_abc atexit pwd
time

Following modules built successfully but were removed because they could not be imported:
_asyncio

The diff!

diff --git a/Lib/_aix_support.py b/Lib/_aix_support.py
index 2c5cd3297d..c7f4491633 100644
--- a/Lib/_aix_support.py
+++ b/Lib/_aix_support.py
@@ -12,7 +12,8 @@ try:
     _tmp_bd = get_config_var("AIX_BUILDDATE")
     _bgt = get_config_var("BUILD_GNU_TYPE")
 except ImportError:  # pragma: no cover
-    _have_subprocess = False
+    import _bootsubprocess as subprocess
+    _have_subprocess = True
     _tmp_bd = None
     _bgt = "powerpc-ibm-aix6.1.7.0"

diff --git a/Lib/distutils/util.py b/Lib/distutils/util.py
index 4b002ecef1..513af52ecd 100644
--- a/Lib/distutils/util.py
+++ b/Lib/distutils/util.py
@@ -79,8 +79,9 @@ def get_host_platform():
             machine += ".%s" % bitness[sys.maxsize]
         # fall through to standard osname-release-machine representation
     elif osname[:3] == "aix":
-        from _aix_support import aix_platform
-        return aix_platform()
+        return "%s-%s.%s" % (osname, version, release)
+        # from _aix_support import aix_platform
+        # return aix_platform()
     elif osname[:6] == "cygwin":
         osname = "cygwin"
         rel_re = re.compile (r'[\d.]+', re.ASCII)
diff --git a/Lib/sysconfig.py b/Lib/sysconfig.py
index 4003726dc9..b116e96007 100644
--- a/Lib/sysconfig.py
+++ b/Lib/sysconfig.py
@@ -666,8 +666,9 @@ def get_platform():
             machine += ".%s" % bitness[sys.maxsize]
         # fall through to standard osname-release-machine representation
     elif osname[:3] == "aix":
-        from _aix_support import aix_platform
-        return aix_platform()
+        return "%s-%s.%s" % (osname, version, release)
+        # from _aix_support import aix_platform
+        # return aix_platform()
     elif osname[:6] == "cygwin":
         osname = "cygwin"
         import re

IMHO - this problem has nothing to do with _aix_support.py - That issue was fixed earlier by the new _bootsubprocess.