urllib ctypes error on Mac OS X Server 10.5

[asoki]

BPO	5413
Nosy	@theller, @ronaldoussoren, @ned-deily
Files	urllib5413.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/ronaldoussoren'
closed_at = <Date 2009-09-20.11:00:47.961>
created_at = <Date 2009-03-03.23:55:43.392>
labels = ['OS-mac', 'ctypes', 'type-crash']
title = 'urllib ctypes error on Mac OS X Server 10.5'
updated_at = <Date 2009-09-20.11:00:47.960>
user = 'https://bugs.python.org/asoki'

bugs.python.org fields:

activity = <Date 2009-09-20.11:00:47.960>
actor = 'ronaldoussoren'
assignee = 'ronaldoussoren'
closed = True
closed_date = <Date 2009-09-20.11:00:47.961>
closer = 'ronaldoussoren'
components = ['macOS', 'ctypes']
creation = <Date 2009-03-03.23:55:43.392>
creator = 'asoki'
dependencies = []
files = ['13489']
hgrepos = []
issue_num = 5413
keywords = ['patch']
message_count = 9.0
messages = ['83111', '83112', '83117', '84654', '84990', '85004', '86165', '86763', '92888']
nosy_count = 4.0
nosy_names = ['theller', 'ronaldoussoren', 'ned.deily', 'asoki']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = None
status = 'closed'
superseder = None
type = 'crash'
url = 'https://bugs.python.org/issue5413'
versions = ['Python 2.6', 'Python 3.0', 'Python 3.1']

[asoki]

i trying to compile Python 2.6.1 from scratch on Mac OS X Server (ppc).
After configure/make/make install i test the compiled python with
the followig file (t.py) (example from
http://docs.python.org/library/urllib.html):

Contents of /tmp/t.py:
-------------

import urllib
params = urllib.urlencode({'spam': 1, 'eggs': 2, 'bacon': 0})
f = urllib.urlopen("http://www.musi-cal.com/cgi-bin/query", params)
print f.read()

---

and i get this error:

python /tmp/t.py 
Traceback (most recent call last):
  File "/tmp/t.py", line 3, in <module>
    f = urllib.urlopen("http://www.musi-cal.com/cgi-bin/query", params)
  File "/usr/local/test/python/lib/python2.6/urllib.py", line 82, in urlopen
    opener = FancyURLopener()
  File "/usr/local/test/python/lib/python2.6/urllib.py", line 611, in
__init__
    URLopener.__init__(self, *args, **kwargs)
  File "/usr/local/test/python/lib/python2.6/urllib.py", line 129, in
__init__
    proxies = getproxies()
  File "/usr/local/test/python/lib/python2.6/urllib.py", line 1555, in
getproxies
    return getproxies_environment() or getproxies_macosx_sysconf()
  File "/usr/local/test/python/lib/python2.6/urllib.py", line 1449, in
getproxies_macosx_sysconf
    _CFSetup(sc)
  File "/usr/local/test/python/lib/python2.6/urllib.py", line 1330, in
_CFSetup
    sc.CFStringCreateWithCString.argtypes = [ c_void_p, c_char_p, c_int32 ]
  File "/usr/local/test/python/lib/python2.6/ctypes/__init__.py", line
366, in __getattr__
    func = self.__getitem__(name)
  File "/usr/local/test/python/lib/python2.6/ctypes/__init__.py", line
371, in __getitem__
    func = self._FuncPtr((name_or_ordinal, self))
AttributeError: dlsym(RTLD_DEFAULT, CFStringCreateWithCString): symbol
not found
Exception AttributeError: "FancyURLopener instance has no attribute
'tempcache'" in <bound method FancyURLopener.__del__ of
<urllib.FancyURLopener instance at 0x3743a0>> ignored

than tested it whit this code (Thanks to Ned Deily)
See the mailinglist thread at:
http://mail.python.org/pipermail/python-list/2009-March/703067.html

Contents of /tmp/tt.py:
-----------------------

from ctypes import cdll
from ctypes.util import find_library
sc = cdll.LoadLibrary(find_library("SystemConfiguration"))
x = sc.CFStringCreateWithCString(0, "HTTPEnable", 0)

---

/usr/local/test/python/bin/python2.6 /tmp/tt.py
Traceback (most recent call last):
  File "/tmp/tt.py", line 4, in <module>
    x = sc.CFStringCreateWithCString(0, "HTTPEnable", 0)
  File "/usr/local/test/python/lib/python2.6/ctypes/__init__.py", line
366,
in __getattr__
    func = self.__getitem__(name)
  File "/usr/local/test/python/lib/python2.6/ctypes/__init__.py", line
371,
in __getitem__
    func = self._FuncPtr((name_or_ordinal, self))
AttributeError: dlsym(RTLD_DEFAULT, CFStringCreateWithCString): symbol
not
found

Both test fails on my PPC Mac and runs without error on my Intel MacBook

Some detailed info about the macs:

ppc:
Model Name: Xserve G5
Model Identifier: RackMac3,1
Processor Name: PowerPC G5 (3.1)
Processor Speed: 2.3 GHz
Number Of CPUs: 2
L2 Cache (per CPU): 512 KB
Boot ROM Version: 5.1.7f2
System Version: Mac OS X Server 10.5.6 (9G55)
Kernel Version: Darwin 9.6.0

intel:
Model Name: MacBook Pro
Model Identifier: MacBookPro3,1
Processor Name: Intel Core 2 Duo
Processor Speed: 2.4 GHz
Number Of Processors: 1
Total Number Of Cores: 2
L2 Cache: 4 MB
Bus Speed: 800 MHz
Boot ROM Version: MBP31.0070.B07
SSC Version: 1.16f8
System Version: Mac OS X 10.5.6 (9G55)
Kernel Version: Darwin 9.6.0

I recompiled Python-2.6.1 with MACOSX_DEPLOYMENT_TARGET=10.3, but no
change..
I updated developertools to xcode312_2621, no change..

Btw.. Python-2.5.4 compiled from scratch works fine.

I dig a bit deeper with otool and discover some differences:

Intel MacOS.so: Carbon, libSystem.B.dylib, CoreServices,
ApplicationServices

PPC MacOS.so: Carbon, libmx.A.dylib, libSystem.B.dylib

Intel _ctypes.so: libSystem.B.dylib
PPC _ctypes.so: libmx.A.dylib, libSystem.B.dylib

-----------
Intel with deploymant target 10.3, xcode312:
-----------
sh-3.2# otool
-L /usr/local/test/python/lib/python2.6/lib-dynload/MacOS.so
/usr/local/test/python/lib/python2.6/lib-dynload/MacOS.so:
/System/Library/Frameworks/Carbon.framework/Versions/A/Carbon
(compatibility version 2.0.0, current version 136.0.0)
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current
version
111.1.3)

/System/Library/Frameworks/CoreServices.framework/Versions/A/CoreServices
(compatibility version 1.0.0, current version 32.0.0)

/System/Library/Frameworks/ApplicationServices.framework/Versions/A/Applicat
ionServices (compatibility version 1.0.0, current version 34.0.0)

-------

sh-3.2# otool
-L /usr/local/test/python/lib/python2.6/lib-dynload/_ctypes.so
/usr/local/test/python/lib/python2.6/lib-dynload/_ctypes.so:
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current
version
111.1.3)

----------
PPC with deployment_target 10.3, xcode312_2621
----------

otool -L /usr/local/test/python/lib/python2.6/lib-dynload/MacOS.so
/usr/local/test/python/lib/python2.6/lib-dynload/MacOS.so:
/System/Library/Frameworks/Carbon.framework/Versions/A/Carbon
(compatibility version 2.0.0, current version 136.0.0)
/usr/lib/libmx.A.dylib (compatibility version 1.0.0, current version
47.1.0)
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current
version
111.1.3)

-------

otool -L /usr/local/test/python/lib/python2.6/lib-dynload/_ctypes.so
/usr/local/test/python/lib/python2.6/lib-dynload/_ctypes.so:
/usr/lib/libmx.A.dylib (compatibility version 1.0.0, current version
47.1.0)
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current
version
111.1.3)

How i compiled it:

export MACOSX_DEPLOYMENT_TARGET=10.5
configure --prefix=/usr/local/test/python
make
make install

How i test:

export
PYTHONPATH=/usr/local/test/python:/usr/local/test/python/lib/python2.6/site-packages
/usr/local/test/python/bin/ is in the PATH
python /tmp/t.py

[ned-deily]

For more info, see discussion starting here:
<http://mail.python.org/pipermail/python-list/2009-March/703067.html\>

I suspect a difference in the proxy configuration available between OS X
and OS X Server, although I don't have a OS X Server configuration at hand
to test it. (For 2.6, the OS X proxy configuration code was rewritten to
use ctypes instead of the deprecated mac Carbon wrappers.)

[ronaldoussoren]

I don't have time to investigate this right now, I do have access to a
10.5 server box though. With some luck I'll be able to create a patch
later today.

Looking at the code in urllib I found the most likely reason for the
problem: the Python script tries to resolve "CFStringCreateWithCString"
through the SystemConfiguration framework. It may well be that this
doesn't work on PPC systems and that this symbol should explicitly be
loaded from the CoreFoundation framework.

BTW. This probably also affects 3.x.

[ronaldoussoren]

I don't have a PPC machine handy at the moment, but I hope the attached
copy of urllib fixes the issue. Could you please test if that's indeed the
case?

[asoki]

no luck.
output for t.py:

Traceback (most recent call last):
  File "t.py", line 3, in <module>
    f = urllib.urlopen("http://www.musi-cal.com/cgi-bin/query", params)
  File "/var/root/pytest/python/lib/python2.6/urllib.py", line 82, in
urlopen
    opener = FancyURLopener()
  File "/var/root/pytest/python/lib/python2.6/urllib.py", line 611, in
__init__
    URLopener.__init__(self, *args, **kwargs)
  File "/var/root/pytest/python/lib/python2.6/urllib.py", line 129, in
__init__
    proxies = getproxies()
  File "/var/root/pytest/python/lib/python2.6/urllib.py", line 1558, in
getproxies
    return getproxies_environment() or getproxies_macosx_sysconf()
  File "/var/root/pytest/python/lib/python2.6/urllib.py", line 1452, in
getproxies_macosx_sysconf
    _CFSetup(sc)
TypeError: _CFSetup() takes exactly 2 arguments (1 given)
Exception AttributeError: "FancyURLopener instance has no attribute
'tempcache'" in <bound method FancyURLopener.__del__ of
<urllib.FancyURLopener instance at 0x372288>> ignored

output for tt.py:

Traceback (most recent call last):
  File "tt.py", line 4, in <module>
    x = sc.CFStringCreateWithCString(0, "HTTPEnable", 0)
  File "/var/root/pytest/python/lib/python2.6/ctypes/__init__.py", line
366, in __getattr__
    func = self.__getitem__(name)
  File "/var/root/pytest/python/lib/python2.6/ctypes/__init__.py", line
371, in __getitem__
    func = self._FuncPtr((name_or_ordinal, self))
AttributeError: dlsym(RTLD_DEFAULT, CFStringCreateWithCString): symbol
not found

notes:
t.py fails on both with the same error
tt.py runs on Intel/Mac OS X and fails on PPC/Mac OS X Server

[ronaldoussoren]

I'll install 10.5 server on my PPC test machine when I'm back from PyCon,
this seem to need some serious debugging.

[ronaldoussoren]

The attached patch is not entirely correct, I'm not yet uploading an
updated patch because the fix is obvious.

I cannot reproduce this with a the 2.6.2 installer (build on PPC) nor
with a recent build from subversion (python-trunk, build on X86). My
tests were done on a PPC G4 iMac running the following version of OSX:

ProductName: Mac OS X
ProductVersion: 10.4.11
BuildVersion: 8S165

Xcode is:

$ xcodebuild -version
Component versions: DevToolsCore-660.0; DevToolsSupport-651.0

I'm currently installing 10.5 server in a VM, and will try to test that
as well (using Rosetta). I don't have access to a PPC machine running
10.5 server, the 10.4 system I mention above cannot be easily
reinstalled.

BTW. I intend to commit the fixed version of my patch regardless of
whether this fixes the issue because it is slightly more correct to
fetch CoreFoundation symbols directly from the CoreFoundation framework
instead of relying on SystemConfiguration's "connection" to that
framework.

[theller]

Seems ronald takes care of this issue.

[ronaldoussoren]

I've just committed a fix for another issue and that completely removes
the ctypes-based code that causes this issue (replacing it by a C
extension)

Therefore this issue should no longer occur and hence can be closed.