OS X pythonw.c compile error with 10.4 or earlier deployment target: no spawn.h

[ned-deily]

BPO	7658
Nosy	@loewis, @ronaldoussoren, @ned-deily
Files	issue-pythonw-10-4-py3k-trunk.txt pythonw-on-10.4.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/ronaldoussoren'
closed_at = <Date 2010-02-07.20:07:29.537>
created_at = <Date 2010-01-08.09:06:05.620>
labels = ['OS-mac', 'build']
title = 'OS X pythonw.c compile error with 10.4 or earlier deployment target: no spawn.h'
updated_at = <Date 2010-02-07.20:07:29.536>
user = 'https://github.com/ned-deily'

bugs.python.org fields:

activity = <Date 2010-02-07.20:07:29.536>
actor = 'ronaldoussoren'
assignee = 'ronaldoussoren'
closed = True
closed_date = <Date 2010-02-07.20:07:29.537>
closer = 'ronaldoussoren'
components = ['macOS']
creation = <Date 2010-01-08.09:06:05.620>
creator = 'ned.deily'
dependencies = []
files = ['15785', '15833']
hgrepos = []
issue_num = 7658
keywords = ['patch']
message_count = 20.0
messages = ['97397', '97399', '97400', '97401', '97402', '97403', '97404', '97405', '97409', '97437', '97448', '97616', '97626', '97687', '97689', '97700', '97701', '97955', '98127', '99013']
nosy_count = 4.0
nosy_names = ['loewis', 'ronaldoussoren', 'ned.deily', 'srid']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'compile error'
url = 'https://bugs.python.org/issue7658'
versions = ['Python 2.7', 'Python 3.2']

[ned-deily]

r77031 (trunk) and r77032 (py3k) for bpo-6834 enhanced the
the OS X python interpreter launcher, python/pythonw, to
allow user-selection of the interpreter execution architecture
for multiple architecture builds, i.e. 32-bit vs 64-bit, by
using the enhanced capability of the posix_spawn system call
instead of execv. However, posix_spawn does not exist prior
to OS X 10.5 and it is still important for the OS X installer
to support builds and execution on 10.4 and earlier systems.

The solution here is to modify Mac/Tools/pythonw.c to
conditionally compile calls to execv for builds with a
deployment target of 10.4 or earlier and use posix_spawn
for deployments of 10.5 or above.

Also, correct 32-bit arch names added in configure/configure.in:
for 10.4, they must be the more general "ppc", not "ppc7400".

APPLIES py3k, trunk

[loewis]

Would it be possible to make this a runtime choice? I.e. use posix_spawn when available, and exec otherwise?

[ned-deily]

Not cleanly, I think. The problem is there's no spawn.h in the 10.4u SDK so it can't be built on 10.4 without supplying a copy of a system header file.

[ned-deily]

Also, the new functionality is really only needed for 32-bit/64-bit multi-arch builds which aren't support with the 10.4u SDK anyway.

[loewis]

To elaborate, it should be possible to declare the posix_spawnp functions as __attribute__((weak)), and then test at run-time whether the function (pointers) are null. See http://tinyurl.com/y8myawg for an example.

[ronaldoussoren]

Ned: the new functionality is also needed for 2-way univeral binaries, it makes pythonw behave much more as if you execute the real interpreter instead of a stub executable.

That posix_spawn doesn't exist sucks, and I'm a bit annoyed with myself for not noticing that before the commit.

It is pretty easy to create executables on 10.5 that use posix_spawnv when available using weak linking (we already do that in posixmodule.c for other functionality).

That won't work for builds on 10.4, or linking to the 10.4u SDK, if the posix_spawn symbols aren't present there: weak linking still has to resolve the symbol at link-time and only ensures that the executable won't fail to start when the symbol isn't present at runtime.

I don't have the 10.4u SKD on my machine at the moment and cannot investigate further at the moment, but will do ASAP.

Another thing I'd like to do before 2.7 is released is to see if it is possible to perform the build using the default SDK and patch distutils to use the 10.4u SDK when building extensions for a universal build on a n OSX 10.4 system (because the default SDK on those systems cannot build universal binaries).

[ronaldoussoren]

BTW. The patch is incorrect in it current form:

• The change to LIPO_32BIT_FLAGS is unconditional, the current values are needed to build on modern system, I guess the proposed new value would be needed for building on 10.4?

• The patch changes pythonw to use execv unconditionally when building a binary that should work on 10.4, as noted in the discussion it is easy enough to build an executable that uses posix_spawnv when it is available and falls back to execv when it is not. A configure check can be used to detect if posix_spawnv is available at link-time.

[ned-deily]

As far as I can see, the only possible shortcoming of the patch is that it restores the current behavior with a 2-way fat ppc/i386 build with 10.4u (i.e. the way python.org installers are currently built); that is, you would still not be able to use "arch" to select PPC-emulation on an Intel machine when a 10.4 SDK build is installed on a 10.5 or later Intel system. That's no different than what we have today and I don't know of any demand for that feature. There are other fairly easy ways around it, if really necessary. But perhaps I'm overlooking something. On the other hand, I don't see anything wrong with going the weak linking route (thanks for the pointer, Martin), either, other than the additional effort and complexity.

[ned-deily]

"* The change to LIPO_32BIT_FLAGS is unconditional, the current values
are needed to build on modern system, I guess the proposed new
value would be needed for building on 10.4?"

Ah, yes, sorry. I built and tested with and on 10.6 with 10.5 SDK/gcc-4.0, on 10.5 with gcc-4.0, and on 10.4u/gcc-4.0 but I see I did overlook trying 3-way on 10.6 with 10.6 SDK/gcc-4.2, where ppc7400 is indeed needed. Arch ppc is definitely needed for building on 10.6 and 10.5 with the 10.4u SDK and gcc-4.0:

lipo: -extract ppc7400 specified but fat file: pythonw does not contain that architecture

If you don't get to it first, I'll fix and resubmit the patch later.

[loewis]

As far as I can see, the only possible shortcoming of the patch is
that it restores the current behavior with a 2-way fat ppc/i386 build
with 10.4u (i.e. the way python.org installers are currently built);
that is, you would still not be able to use "arch" to select
PPC-emulation on an Intel machine when a 10.4 SDK build is installed
on a 10.5 or later Intel system. That's no different than what we
have today and I don't know of any demand for that feature. There
are other fairly easy ways around it, if really necessary.

For users, what matters are really the binaries provided by python.org
(and those provided by Apple). So if you restore the previous behavior,
that's basically "for good" - that you still have the compile option to
support arch won't matter for users.

Now, ISTM that users do indeed request the ability to chose between a
32-bit interpreter and a 64-bit interpreter at launch time. I know that
they do on Windows (just because you can), and I would be puzzled if
there weren't OSX users that wanted it one way or the other (depending
on the application you intend to run).

[ned-deily]

Martin, just to be clear: the purpose of the new feature *is* to allow the choice between 32-bit/64-bit and that is important. My comment was that the downside of the submitted fix, as it stands, would be to not allow choosing archs only for pythons built with the 10.4u SDKs. And with the 10.4u SDK, those pythons are and can only be 32-bit so the only arch choice there would be between 32-bit Intel and 32-bit PPC and the only environment where that could mean anything is on a 10.5 or 10.6 Intel machine where, in theory, it should be possible to run 32-bit PPC pythons in emulation mode. For pythons built with 10.5 or 10.6 SDKs (in as yet to be supplied python.org installers), then the arch feature allows choice among the up-to 4 archs in the build: 32/64, Intel/PPC. The proposed fix doesn't alter that aspect of the new feature at all. But the patch needs to be redone anyway so, if the weak linking is added, there wouldn't even be that very small drawback.

OS X multi-architecture executables: complicated!

[srid]

Our nightly ActivePython builds broke because of this issue. Note that we still build on 10.4 w/ 10.4u SDK.

I should be able to help verify the new fix - once it is uploaded - in our build machine.

[ronaldoussoren]

I've attached a patch that should fix the build issues with the 10.4 SDK.

The patch touches configure.in, run autoconf and autoheader after applying the patch.

I haven't tested the patch yet beyond compilation on 10.6 system without the 10.4 SDK.

This patch does not attempt to fix the issues Ned has with LIPO_32BIT_FLAGS, I will look into that later today.

[ned-deily]

Thanks, Ronald. The patch looks good. I've got a patch in progress for the LIPO flags part: looks like the key is 'ppc' for DEPLOYMENT_TARGET <= 10.4 and 'ppc7400' for DEPLOYMENT_TARGET >= 10.5 for either gcc-4.0 or -4.2. I'll test the two together on the various platforms tomorrow.

[srid]

Ronald, with your patch I see this on 10.4 (Xcode 2.5):

[...]
lipo -extract ppc7400 -extract i386 -output /Users/apy/rrun/build/activepython-DEV/build/pyhg_trunk-macosx-hgtip27-rrun/image/Library/Frameworks/Python.framework/Versions/2.7/bin/python2.7-32 pythonw
lipo: -extract ppc7400 specified but fat file: pythonw does not contain that architecture
[...]

[ned-deily]

@sridhar: that's the part that's not fixed yet. We'll have a patch for that shortly.

[ronaldoussoren]

Ned and Sridhar: IMO we need a configure test to detect which argument should be used to extract ppc code (ppc or ppc7400).

[ronaldoussoren]

I've committed a fix for this in r77585, please test.

I can now compile python-trunk on OSX 10.6 while targetting the 10.4 SDK, and have compiled on OSX 10.4 as well.

I will forward port this to 3.2.

[srid]

RONALD: I've committed a fix for this in r77585, please test.

Verified on Tiger with 10.4u SDK.

[ronaldoussoren]

This is now fixed in the 3.2 branch as well.