"Out of tree" build fails on AIX #54865

pedz · 2010-12-09T03:35:03Z

BPO	10656
Nosy	@merwok, @haubi, @vadmium
PRs	[Do Not Merge] Convert `Misc/NEWS` so that it is managed by towncrier #552
Files	aix-fix-out-of-source.diff: Patch to apply on 2.7 branch issue10656-out-of-source-build-on-aix.patch: patch for 3.4

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2016-11-21.03:31:23.763>
created_at = <Date 2010-12-09.03:35:03.276>
labels = ['build', '3.7']
title = '"Out of tree" build fails on AIX'
updated_at = <Date 2018-01-02.15:40:35.893>
user = 'https://bugs.python.org/pedz'

bugs.python.org fields:

activity = <Date 2018-01-02.15:40:35.893>
actor = 'David.Edelsohn'
assignee = 'none'
closed = True
closed_date = <Date 2016-11-21.03:31:23.763>
closer = 'martin.panter'
components = ['Build']
creation = <Date 2010-12-09.03:35:03.276>
creator = 'pedz'
dependencies = []
files = ['21489', '35476']
hgrepos = ['252']
issue_num = 10656
keywords = ['patch']
message_count = 10.0
messages = ['123668', '128275', '128315', '132649', '219740', '279474', '279475', '279526', '281196', '281307']
nosy_count = 8.0
nosy_names = ['sable', 'eric.araujo', 'haubi', 'pedz', 'python-dev', 'Tristan.Carel', 'martin.panter', 'David.Edelsohn']
pr_nums = ['552']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'compile error'
url = 'https://bugs.python.org/issue10656'
versions = ['Python 2.7', 'Python 3.5', 'Python 3.6', 'Python 3.7']

The text was updated successfully, but these errors were encountered:

pedz · 2010-12-09T03:35:03Z

I do "out of tree" builds (I think that is the right term). I make an
empty directory, cd into it, and then do:

<path to src directory>/configure

followed by make. This works on most open source pages. It appears to
be very close to working with Python 2.7.1 but the problem is that it assumes ld_so_aix is in the source tree but it is in the build tree.

I did an "in tree" build using basically just ./configure && make and so far (I'm still testing) that has worked fine.

sable · 2011-02-10T10:15:20Z

There are some unit tests concerning distutils which are currently failing on AIX and may be an illustration of the same problem:

======================================================================
ERROR: test_build_ext (distutils.tests.test_build_ext.BuildExtTestCase)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/san_cis/home/cis/.buildbot/python-aix6/py3k-aix6-xlc/build/Lib/distutils/unixccompiler.py", line 254, in link
    self.spawn(linker + ld_args)
  File "/san_cis/home/cis/.buildbot/python-aix6/py3k-aix6-xlc/build/Lib/distutils/ccompiler.py", line 911, in spawn
    spawn(cmd, dry_run=self.dry_run)
  File "/san_cis/home/cis/.buildbot/python-aix6/py3k-aix6-xlc/build/Lib/distutils/spawn.py", line 34, in spawn
    _spawn_posix(cmd, search_path, dry_run=dry_run)
  File "/san_cis/home/cis/.buildbot/python-aix6/py3k-aix6-xlc/build/Lib/distutils/spawn.py", line 138, in _spawn_posix
    % (cmd[0], exit_status))
distutils.errors.DistutilsExecError: command './Modules/ld_so_aix' failed with exit status 1

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/san_cis/home/cis/.buildbot/python-aix6/py3k-aix6-xlc/build/Lib/distutils/tests/test_build_ext.py", line 85, in test_build_ext
    cmd.run()
  File "/san_cis/home/cis/.buildbot/python-aix6/py3k-aix6-xlc/build/Lib/distutils/command/build_ext.py", line 347, in run
    self.build_extensions()
  File "/san_cis/home/cis/.buildbot/python-aix6/py3k-aix6-xlc/build/Lib/distutils/command/build_ext.py", line 456, in build_extensions
    self.build_extension(ext)
  File "/san_cis/home/cis/.buildbot/python-aix6/py3k-aix6-xlc/build/Lib/distutils/command/build_ext.py", line 543, in build_extension
    target_lang=language)
  File "/san_cis/home/cis/.buildbot/python-aix6/py3k-aix6-xlc/build/Lib/distutils/ccompiler.py", line 719, in link_shared_object
    extra_preargs, extra_postargs, build_temp, target_lang)
  File "/san_cis/home/cis/.buildbot/python-aix6/py3k-aix6-xlc/build/Lib/distutils/unixccompiler.py", line 256, in link
    raise LinkError(msg)
distutils.errors.LinkError: command './Modules/ld_so_aix' failed with exit status 1

======================================================================
ERROR: test_get_outputs (distutils.tests.test_build_ext.BuildExtTestCase)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/san_cis/home/cis/.buildbot/python-aix6/py3k-aix6-xlc/build/Lib/distutils/unixccompiler.py", line 254, in link
    self.spawn(linker + ld_args)
  File "/san_cis/home/cis/.buildbot/python-aix6/py3k-aix6-xlc/build/Lib/distutils/ccompiler.py", line 911, in spawn
    spawn(cmd, dry_run=self.dry_run)
  File "/san_cis/home/cis/.buildbot/python-aix6/py3k-aix6-xlc/build/Lib/distutils/spawn.py", line 34, in spawn
    _spawn_posix(cmd, search_path, dry_run=dry_run)
  File "/san_cis/home/cis/.buildbot/python-aix6/py3k-aix6-xlc/build/Lib/distutils/spawn.py", line 138, in _spawn_posix
    % (cmd[0], exit_status))
distutils.errors.DistutilsExecError: command './Modules/ld_so_aix' failed with exit status 1

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/san_cis/home/cis/.buildbot/python-aix6/py3k-aix6-xlc/build/Lib/distutils/tests/test_build_ext.py", line 343, in test_get_outputs
    cmd.run()
  File "/san_cis/home/cis/.buildbot/python-aix6/py3k-aix6-xlc/build/Lib/distutils/command/build_ext.py", line 347, in run
    self.build_extensions()
  File "/san_cis/home/cis/.buildbot/python-aix6/py3k-aix6-xlc/build/Lib/distutils/command/build_ext.py", line 456, in build_extensions
    self.build_extension(ext)
  File "/san_cis/home/cis/.buildbot/python-aix6/py3k-aix6-xlc/build/Lib/distutils/command/build_ext.py", line 543, in build_extension
    target_lang=language)
  File "/san_cis/home/cis/.buildbot/python-aix6/py3k-aix6-xlc/build/Lib/distutils/ccompiler.py", line 719, in link_shared_object
    extra_preargs, extra_postargs, build_temp, target_lang)
  File "/san_cis/home/cis/.buildbot/python-aix6/py3k-aix6-xlc/build/Lib/distutils/unixccompiler.py", line 256, in link
    raise LinkError(msg)
distutils.errors.LinkError: command './Modules/ld_so_aix' failed with exit status 1

I will take a look.

sable · 2011-02-10T17:12:46Z

I tried with trunk (python 3.2) on AIX:
./build/configure --with-pydebug --without-computed-gotos
make
and that worked correctly.

Not sure with python 2.7.

TristanCarel · 2011-03-31T11:15:40Z

I have experienced the same problem under ppc64 aix 6.1.2.0 while compiling Python 2.7.1

$ cd Python-2.7.1
$ mkdir __build
$ cd __build
$ ../configure --with-gcc=xlc_r --with-cxx-main=xlC_r --with-threads --disable-ipv6
$ /usr/linux/bin/make
[...]
../Modules/ld_so_aix xlc_r -bI:Modules/python.exp -L.. -L/opt/freeware/lib -L/opt/freeware/lib build/temp.aix-6.1-2.7/data/tcarel/ngbuilder/python/2.7.1/Python-2.7.1/Modules/_struct.o -L/opt/freeware/lib -L/usr/local/lib -o build/lib.aix-6.1-2.7/_struct.so
unable to execute ../Modules/ld_so_aix: No such file or directory

There are references to $(srcdir)/Modules/ld_so_aix in both configure.in and Makefile.pre.in whereas ld_so_aix is created in build tree.

==================================================

diff -r f4e6e18cae4b Makefile.pre.in
--- a/Makefile.pre.in	Tue Mar 29 17:28:25 2011 -0700
+++ b/Makefile.pre.in	Thu Mar 31 13:05:45 2011 +0200
@@ -1028,7 +1028,7 @@
 		$(INSTALL_SCRIPT) $(srcdir)/Modules/makexp_aix	\
 				$(DESTDIR)$(LIBPL)/makexp_aix;		\
 		echo "$(LIBPL)/makexp_aix";			\
-		$(INSTALL_SCRIPT) $(srcdir)/Modules/ld_so_aix	\
+		$(INSTALL_SCRIPT) Modules/ld_so_aix	\
 				$(DESTDIR)$(LIBPL)/ld_so_aix;		\
 		echo "$(LIBPL)/ld_so_aix";			\
 		echo; echo "See Misc/AIX-NOTES for details.";	\
diff -r f4e6e18cae4b configure.in
--- a/configure.in	Tue Mar 29 17:28:25 2011 -0700
+++ b/configure.in	Thu Mar 31 13:05:45 2011 +0200
@@ -1778,7 +1778,7 @@
 then
 	case $ac_sys_system/$ac_sys_release in
 	AIX*)
-		BLDSHARED="\$(srcdir)/Modules/ld_so_aix \$(CC) -bI:\$(srcdir)/Modules/python.exp"
+		BLDSHARED="Modules/ld_so_aix \$(CC) -bI:Modules/python.exp -L\$(srcdir)"
 		LDSHARED="\$(BINLIBDEST)/config/ld_so_aix \$(CC) -bI:\$(BINLIBDEST)/config/python.exp"
 		;;
 	BeOS*)

==================================================

After fixing these 2 files, Modules/ld_so_aix is found and properly executed by compilation process, but then there is a new issue:

Modules/ld_so_aix xlc_r -bI:Modules/python.exp -L.. -L/opt/freeware/lib -L/opt/freeware/lib build/temp.aix-6.1-2.7/data/tcarel/Python-2.7.1/Modules/_struct.o -L/opt/freeware/lib -L/usr/local/lib -o build/lib.aix-6.1-2.7/_struct.so
Modules/ld_so_aix[183]: Modules/makexp_aix: not found.

This time, makexp_aix is expected to be is the same directory than ld_so_aix whereas makeexp_aix remains in the source tree.

Here is a fix:

==================================================

diff -r f4e6e18cae4b Modules/ld_so_aix.in
--- a/Modules/ld_so_aix.in	Tue Mar 29 17:28:25 2011 -0700
+++ b/Modules/ld_so_aix.in	Thu Mar 31 13:01:57 2011 +0200
@@ -69,7 +69,10 @@
   echo $usage; exit 2
 fi
 
-makexp=`dirname $0`/makexp_aix
+makexp=`dirname $0`/@srcdir@/makexp_aix
+if test ! -e "$makexp" ; then
+  makeexp=makexp_aix
+fi
 
 # Check for existence of compiler.
 CC=$1; shift

==================================================

The first value fix the issue while building Python. The second will use PATH to find makeexp_aix as explained in section `SHARED LIBRARY SUPPORT' in AIX-NOTES (for building shared extension modules).

You can find a patch for 2.7 branch in attachment.

Regards

Tristan Carel
Exalead

haubi · 2014-06-04T06:20:01Z

Basically the same as Tristan's patch, with a little improvement to not rely on PATH to find makexp_aix within ld_so_aix.

Thanks!

vadmium · 2016-10-26T01:05:00Z

The idea of the patches makes sense to me.

Embedding the name of whatever directory Python was built in into the installed version of Modules/ld_so_aix.in does not seem very clean to me. Is there another way of doing this? Perhaps a file copy or symlink?

vadmium · 2016-10-26T01:14:39Z

Actually, why do you remove $(srcdir) for Modules/python.exp? This code was added in r88426; see <https://bugs.python.org/issue941346#msg128586\>.

haubi · 2016-10-27T06:32:24Z

Modules/python.exp is generated at build-time, thus does not belong to $(srcdir).

vadmium · 2016-11-19T03:18:58Z

Okay I will assume that $(srcdir) reference was just due to a mistake or bad cargo-culting then.

Will commit this soon unless anyone has a good idea to avoid embedding @abs_srcdir@.

python-dev · 2016-11-20T23:24:12Z

New changeset 48526666321a by Martin Panter in branch '3.5':
Issue bpo-10656: Fix out-of-tree building on AIX
https://hg.python.org/cpython/rev/48526666321a

New changeset 76d1f8001e27 by Martin Panter in branch '3.6':
Issue bpo-10656: Merge AIX build fix from 3.5
https://hg.python.org/cpython/rev/76d1f8001e27

New changeset 180f046b597e by Martin Panter in branch 'default':
Issue bpo-10656: Merge AIX build fix from 3.6
https://hg.python.org/cpython/rev/180f046b597e

New changeset ca46883fc5cf by Martin Panter in branch '2.7':
Issue bpo-10656: Fix out-of-tree building on AIX
https://hg.python.org/cpython/rev/ca46883fc5cf

pedz mannequin added build The build process and cross-build labels Dec 9, 2010

haubi mannequin changed the title ~~"Out of tree" build fails on AIX 5.3~~ "Out of tree" build fails on AIX Jun 4, 2014

vadmium added the 3.7 only security fixes label Oct 26, 2016

vadmium closed this as completed Nov 21, 2016

ezio-melotti transferred this issue from another repository Apr 10, 2022

"Out of tree" build fails on AIX #54865

"Out of tree" build fails on AIX #54865

pedz mannequin commented Dec 9, 2010

pedz mannequin commented Dec 9, 2010

sable mannequin commented Feb 10, 2011

sable mannequin commented Feb 10, 2011

TristanCarel mannequin commented Mar 31, 2011

haubi mannequin commented Jun 4, 2014

vadmium commented Oct 26, 2016

vadmium commented Oct 26, 2016

haubi mannequin commented Oct 27, 2016

vadmium commented Nov 19, 2016

python-dev mannequin commented Nov 20, 2016

"Out of tree" build fails on AIX #54865

"Out of tree" build fails on AIX #54865

Comments

pedz mannequin commented Dec 9, 2010

pedz mannequin commented Dec 9, 2010

sable mannequin commented Feb 10, 2011

sable mannequin commented Feb 10, 2011

TristanCarel mannequin commented Mar 31, 2011

Regards

haubi mannequin commented Jun 4, 2014

vadmium commented Oct 26, 2016

vadmium commented Oct 26, 2016

haubi mannequin commented Oct 27, 2016

vadmium commented Nov 19, 2016

python-dev mannequin commented Nov 20, 2016