sysconfig: $MACOSX_DEPLOYMENT_TARGET mismatch: now "10.3" but "10.5" during configure #53725
Comments
|
I cannot find correct repro steps for this, but: /Library/Frameworks/Python.framework/Versions/2.7/bin/python" -B -s -c "import sys;print('%d.%d' % tuple(sys.version_info)[:2])
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site.py", line 558, in <module>
main()
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site.py", line 540, in main
known_paths = addusersitepackages(known_paths)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site.py", line 264, in addusersitepackages
user_site = getusersitepackages()
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site.py", line 239, in getusersitepackages
user_base = getuserbase() # this will also set USER_BASE
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site.py", line 229, in getuserbase
USER_BASE = get_config_var('userbase')
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/sysconfig.py", line 518, in get_config_var
return get_config_vars().get(name)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/sysconfig.py", line 421, in get_config_vars
_init_posix(_CONFIG_VARS)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/sysconfig.py", line 300, in _init_posix
raise IOError(msg)
IOError: $MACOSX_DEPLOYMENT_TARGET mismatch: now "10.3" but "10.5" during configurePython was built on a Snow Leopard machine with MACOSX_DEPLOYMENT_TARGET=10.5 environment variable. But on the user's 10.6 machine, no such environment variable is necessarily set. Why is this check required? Shouldn't it be restricted to building modules using distutils, and not happen during an innocuous "import site" that happens in interpreter startup? |
|
I agree that this behavior wrong, this should only trigger an error when building packages. How did you build python? I guess something like this: $ export MACOSX_DEPLOYMENT_TARGET=10.5
$ configure --enable-framework
$ make installThis should result in getting the right deployment target into config/Makefile, which means there may be two bugs here:
BTW. I tend to configure like this: $ configure --enable-framework MACOSX_DEPLOYMENT_TARGET=10.5That way automatic reruns of configure pick up the right environment variables. |
|
We build ActivePython 2.7 on Mac as follows: $ export MACOSX_DEPLOYMENT_TARGET=10.5
$ ./configure --enable-framework --enable-universalsdk=/Developer/SDKs/MacOSX10.5.sdk/ --with-universal-archs=intel
$ make(the environment variable is also used by other builds) We explicitly specify the SDK path here because the build happens on a 10.6 machine (to ensure that tkinter is built for 64-bit arch). Also, we explicitly drop PPC support. |
|
Assigning to myself because I intend to work on this. More questions: does the error occur on the 10.6 machine you used to do the build or another machine? Is MACOSX_DEPLOYMENT_TARGET set in the environment when you run the command that gives the error message? |
|
Another machine.
BTW, I just figured that following command will reliably reproduce this $ MACOSX_DEPLOYMENT_TARGET=10.3 python2.7 -B -s -c "import
sys;print('%d.%d' % tuple(sys.version_info)[:2])"
Traceback (most recent call last):
File
"/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site.py",
line 558, in <module>
main()
File
"/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site.py",
line 540, in main
known_paths = addusersitepackages(known_paths)
File
"/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site.py",
line 264, in addusersitepackages
user_site = getusersitepackages()
File
"/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site.py",
line 239, in getusersitepackages
user_base = getuserbase() # this will also set USER_BASE
File
"/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site.py",
line 229, in getuserbase
USER_BASE = get_config_var('userbase')
File
"/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/sysconfig.py",
line 518, in get_config_var
return get_config_vars().get(name)
File
"/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/sysconfig.py",
line 421, in get_config_vars
_init_posix(_CONFIG_VARS)
File
"/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/sysconfig.py",
line 300, in _init_posix
raise IOError(msg)
IOError: $MACOSX_DEPLOYMENT_TARGET mismatch: now "10.3" but "10.5"
during configureThough, I do recall seeing this error without having that environment |
srid
mannequin
changed the title
|
Looks like reply-by-email stripped some parts of the message.
Another 10.6 machine. |
|
Even simply invoking the interpreter raises this exception! $ MACOSX_DEPLOYMENT_TARGET=10.3 python2.7
[...]
IOError: $MACOSX_DEPLOYMENT_TARGET mismatch: now "10.3" but "10.5" during configure
$ |
|
I can reproduce this with a script that builds and installs a couple of python versions, annoyingly enough I don't understand why that code fails. In particular, in my script the run of python that fails is started with this code: lg.debug("Run setup script with '%s'", python)
pprint.pprint(os.environ)
p = subprocess.Popen([
python, "setup.py", "install"],
cwd=distribute_dir, stdout=subprocess.PIPE,
stderr=subprocess.STDOUT)This prints the environment and then runs a setup.py script. What confuses me is that the printed environment does *not* contain MACOSX_DEPLOYMENT_TARGET while that is the only way to trigger this bug. |
|
This (untested) patch should fix the issue: Index: Lib/sysconfig.py --- Lib/sysconfig.py (revision 84147)
+++ Lib/sysconfig.py (working copy)
@@ -295,9 +295,8 @@
cur_target = cfg_target
os.putenv('MACOSX_DEPLOYMENT_TARGET', cfg_target)
elif map(int, cfg_target.split('.')) > map(int, cur_target.split('.')):
- msg = ('$MACOSX_DEPLOYMENT_TARGET mismatch: now "%s" but "%s" '
- 'during configure' % (cur_target, cfg_target))
- raise IOError(msg)
+ os.putenv('MACOSX_DEPLOYMENT_TARGET', cfg_target)
+ cfg_target = cur_target
# On AIX, there are wrong paths to the linker scripts in the Makefile
# -- these paths are relative to the Python source, but when installedThis removes the exception, and instead replaces the incompatible environment setting by the configured setting. It might be better to just have: Index: Lib/sysconfig.py --- Lib/sysconfig.py (revision 84147)
+++ Lib/sysconfig.py (working copy)
@@ -291,13 +291,8 @@
if sys.platform == 'darwin' and 'MACOSX_DEPLOYMENT_TARGET' in vars:
cfg_target = vars['MACOSX_DEPLOYMENT_TARGET']
cur_target = os.getenv('MACOSX_DEPLOYMENT_TARGET', '')
- if cur_target == '':
- cur_target = cfg_target
+ if cur_target != cfg_target:
os.putenv('MACOSX_DEPLOYMENT_TARGET', cfg_target)
- elif map(int, cfg_target.split('.')) > map(int, cur_target.split('.')):
- msg = ('$MACOSX_DEPLOYMENT_TARGET mismatch: now "%s" but "%s" '
- 'during configure' % (cur_target, cfg_target))
- raise IOError(msg)
# On AIX, there are wrong paths to the linker scripts in the Makefile
# -- these paths are relative to the Python source, but when installedThis entirely ignores the environment variable and always uses the value that was present during the configure run. |
|
I now understand why my script fails, and it is caused by this issue. The sysconfig.py code has another major issue: the use of os.putenv. This changes the environment, without changing os.environ. The use of os.putenv should be replaced by setting keys in os.environ to make it easier to discover that changes have been made. Even that is no good: setting the environment variable should only be done in distutils to ensure that the right build environment is used. It should not be set globally where it will affect code that it was never intended to affect. BTW. Sridhar: could this be the reason you cannot find the correct reproduction steps for this? Do you use a build script that is writting in python and run with a copy of python where sysconfig.get_config_var('MACOSX_DEPLOYMENT_TARGET') returns '10.3'? |
|
On 2010-08-17, at 9:01 PM, Ronald Oussoren wrote:
I agree that environment variable should only be affected during distutils build, and not globally. I now recall debugging this issue (with 2.7 alpha/beta, I guess) and arriving at the same confusion.
Yes, but I am not entirely sure if that copy of python returned 10.3, and I no longer have that copy ... as I've been upgrading 2.6 and 2.7 pretty often in our two Mac build machines. |
|
I just ran into this with Python 2.7, installed using port. I'm using python to process a text file in an Xcode 3.2 Script build step, and it's throwing the error because my Xcode target has a base sdk of 10.5. |
|
I've attached a patch for 3.2 that should fix the issue. The patch adds a couple of testcases (1 for sysconfig and 1 for distutils.command.build_ext), adjust a couple more and implements the following functional changes:
I'm not 100% sure about the (lack of) change to distutils.sysconfig._init_posix. It might be better to just always use the deployment target that was specified during the build of python itself. There is a use-case for overriding the deployment target though: locally build an extension that you use on your machine and that uses compiler/library features that require a newer deployment target than the one used to build python itself. |
|
Another way this issue can show up: when building Python itself. If Parser/Python.asdl needs to be built (as with a new source checkout), the makefile target executes a python script (Parser/asdl_c.py) via /usr/bin/env python (a bootstrap dependency). If that python happens to be python2.7 and the build MACOSX_DEPLOYMENT_TARGET target is not the same as that as that of the python2.7, the python2.7 fails with the $MACOSX_DEPLOYMENT_TARGET mismatch from sysconfig.py _init_posix and thus the build fails. The patch looks good to me except that all of the "assertEquals" (now deprecated) should be changed to "assertEqual". I think parts 1 and 2 should definitely be applied to py3k, 3.2, and 2.7. I'm less certain about the distutils changes (parts 3 and 4) but I agree they are probably the right thing to do. Then there's the issue of getting corresponding changes into distutils2 if needed. |
|
Tarek: could you comment on this patch, in particular: is it OK to commit this to 2.7, 3.2 and head? (I haven't checked yet if the patch still applies cleanly, will do that later today) |
merwok
changed the title
|
This issue is very annoying when you use python's with different deployment targets and should IMHO be fixed in the next release. |
|
Looks acceptable to me. A few details in the code could be improved: + @unittest.skipUnless(sys.platform == 'darwin', 'MacOSX test') + finally: + def _try_compile_deployment_target(self): + fp.close() + tgt = '%02d%01d0'%(tgt) + except CompileError: + self.assertEquals(get_platform(), 'macosx-10.4-fat') + + stderr=open('/dev/null'), |
|
I've attached a v2 of the patch which adresses Éric's comments. I haven't changed the CompileError handling though, that would change a test failure into a test error. IMHO test errors should only happend due to bugs in the test code. AddCleanup is neat trick, I keep forgetting the nice new features that were added to unittest recently. The assertEquals issue is one that keeps biting me, I know the correct form is assertEqual but every single time I write tests I keep using assertEquals for some reason. Maybe its time for some electroshock therapy... |
|
|
I trust this patch does the right thing -- can you apply it before the rc this weekend? |
|
I'll apply the patch late tonight (I won't be home until at least 22:30 CEST) |
|
Attached the backport to 2.7 for my v2 patch. |
|
New changeset 9874f213edb2 by Ronald Oussoren in branch '2.7': New changeset 5b108229a978 by Ronald Oussoren in branch '3.2': New changeset 978016199be8 by Ronald Oussoren in branch '2.7': New changeset 25040a6a68e9 by Ronald Oussoren in branch '3.2': New changeset 412d5f2c995f by Ronald Oussoren in branch 'default': |
|
I've applied the patches to 3.3, 3.2 and 2.7 |
|
Don’t you get ResourceWarnings from the popens and the unclosed file? |
|
I ported the patch to packaging. Please test. |
|
There are several issues here now. With the patches as now applied, when running the tests with standard OS X installer Pythons, I saw occasional failures of the new test_deployment_target test case ("Unexpected target"). After ensuring that the build_ext tests actually are executed consistently (see the patches for bpo-12141), I found that the test case failed consistently when test_distutils was run after test___all__, which is the default. There was also a telltale warning that test___all__ altered the execution environment. After investigating, I've come to the conclusion that the root cause of this test failure *and* of the original reported problem in this issue is the original implementation of forcing MACOSX_DEPLOYMENT_TARGET way back in r38123. The main problem with r38123 is that, while the issue needing fixing is limited to Distutils-spawned build steps, the solution unpredictably can modify the environment of *all* subprocesses (as Ronald noted earlier in msg114195):
0
>>> import distutils.command.build
>>> os.system('echo $MACOSX_DEPLOYMENT_TARGET')
10.6
0The modification occurs in distutils.sysconfig._init_posix when distutils.sysconfig is imported so the env change can be a side-effect of importing other distutils modules, like above. This behavior has been the case since r38123 but its effects were somewhat masked within the Python interpreter process by the use of os.putenv. Now that the patches for this issue replaced os.putenv with setting os.environ directly, the changed MACOSX_DEPLOYMENT_TARGET is now visible in the interpreter, too. And that's the cause of the new test___all__ "altered the execution environment" message. Running test___all__ causes distutils.sysconfig to be imported which causes the _init_posix one-time initialization to take place which will set MACOSX_DEPLOYMENT_TARGET (if it wasn't already set externally). Regrtest now sees the change after running test___all__ and restores the original environment thereby deleting MACOSX_DEPLOYMENT_TARGET. Because _init_posix is only run once, test_distutils fails afterwards because MACOSX_DEPLOYMENT_TARGET is no longer set and will never be set. So I am now convinced that, indeed, the right solution is to only set MACOSX_DEPLOYMENT_TARGET in the Distutils-spawned processes. Setting it globally is too fragile and has unintended side-effects (like in the reported problems earlier in this issue). Beyond that there are some issues with the new test cases. As it stands, the new test_deployment_target test case tries to run two subtests but the second doesn't actually work: distutils.build_ext doesn't bother recompiling it since the object file leftover from the first subtest appears to be up-to-date. Also the test cases do not fully test the intended behavior of the deployment target processing: allowing the deployment target to be overridden to be equal or greater than that of the interpreter build but not less. Another issue is that with packaging now landed in the default branch (for 3.3), similar code and tests are needed there. Éric ported the new test cases but distutils.sysconfig does not exist in packaging: it uses the standard sysconfig. So there is currently no deployment target checking or setting (what's done in distutils.sysconfig._init_posix) at all when using packaging. The attached patches for 3.3, 3.2, and 2.7 try to address all these issues:
|
|
Can the 3.2 part of this be resolved this weekend? |
|
I would like Ronald's take on it (also, I expect to be off-line for the weekend). Note, as it stands now, 3.2.1 (without any further patches) would have the same less than ideal behavior as 2.7.2. |
|
I agree with Ned, the changes to the environment should only be done in subprocesses started by distutils. The patch looks fine, but I haven't tested the patches yet. |
|
New changeset 29c670f0d7d5 by Ned Deily in branch '2.7': New changeset 01434498e55e by Ned Deily in branch '2.7': New changeset e9e1fb4e3752 by Ned Deily in branch '2.7': New changeset bb89023191a0 by Ned Deily in branch '3.2': New changeset b95eee310931 by Ned Deily in branch '3.2': New changeset 8e0cfba9c8c4 by Ned Deily in branch '3.2': New changeset 371a818687a5 by Ned Deily in branch 'default': New changeset 47b670d386cd by Ned Deily in branch 'default': New changeset c1ae16fce76d by Ned Deily in branch 'default': New changeset 1bd45742751b by Ned Deily in branch 'default': |
|
While I am a little concerned about applying these fixes, it is clear that the previous behavior was broken and the initial set of patches as applied did not improve matters. The only risk I can see is that there is a slight chance that there *might* be some 3rd-party package that unknowingly depended on the previous behavior of setting MACOSX_DEPLOYMENT_TARGET globally and which might now fail. There is no simple way to find such packages short of attempting to build them and test them. However, if there *should* be such packages, the simple fix for them is to export the desired MACOSX_DEPLOYMENT_TARGET value into the interpreter process (via a shell variable, for instance). So, I think it best to bite the bullet. I've applied the Distutils patches to 2.7 (for 2.7.3), to 3.2 (for 3.2.1), to default (for 3.3) and the packaging patches to default (for 3.3). |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
The text was updated successfully, but these errors were encountered: