urllib.urlopen crashes in a thread on Snow Leopard

[jweber]

BPO	6851
Nosy	@warsaw, @ronaldoussoren
Files	crash_log.txt: Details from crash reporter dialog urllib-crash.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/ronaldoussoren'
closed_at = <Date 2009-09-20.10:58:24.418>
created_at = <Date 2009-09-07.00:00:57.021>
labels = ['OS-mac', 'type-crash', 'release-blocker']
title = 'urllib.urlopen crashes in a thread on Snow Leopard'
updated_at = <Date 2009-09-20.10:58:24.417>
user = 'https://bugs.python.org/jweber'

bugs.python.org fields:

activity = <Date 2009-09-20.10:58:24.417>
actor = 'ronaldoussoren'
assignee = 'ronaldoussoren'
closed = True
closed_date = <Date 2009-09-20.10:58:24.418>
closer = 'ronaldoussoren'
components = ['macOS']
creation = <Date 2009-09-07.00:00:57.021>
creator = 'jweber'
dependencies = []
files = ['14849', '14904']
hgrepos = []
issue_num = 6851
keywords = ['patch', '26backport']
message_count = 6.0
messages = ['92340', '92651', '92652', '92680', '92721', '92887']
nosy_count = 4.0
nosy_names = ['barry', 'ronaldoussoren', 'jweber', 'emlyn']
pr_nums = []
priority = 'release blocker'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'crash'
url = 'https://bugs.python.org/issue6851'
versions = ['Python 2.6', 'Python 2.5']

[jweber]

The following program works fine under Mac OS 10.5.x. But in 10.6, it
crashes Python, and displays Apple's crash reporter dialog. I've tried
it on Python 2.6 and 2.5, both the 64-bit and 32-bit versions.

The crash seems to happen any time urllib.urlopen is done in a thread.
The same thing happens if I do it in a BaseHTTPServer.HTTPServer, using
SocketServer.ThreadingMixIn.

#!/usr/bin/python
import threading, urllib

class MyThread (threading.Thread):
	def run(self):
		c = urllib.urlopen("http://www.google.com")

MyThread().start()

[ronaldoussoren]

It seems that CoreFoundation doesn't like being loaded on a secondairy
thread:

#0 0x00007fff8301bb90 in __CFInitialize ()
#1 0x00007fff5fc0d5ce in
__dyld__ZN16ImageLoaderMachO11doImageInitERKN11ImageLoader11LinkContextE
()
#2 0x00007fff5fc0d607 in
__dyld__ZN16ImageLoaderMachO16doInitializationERKN11ImageLoader11LinkCon
textE ()
#3 0x00007fff5fc0bcec in
__dyld__ZN11ImageLoader23recursiveInitializationERKNS_11LinkContextEj ()
#4 0x00007fff5fc0bc9d in
__dyld__ZN11ImageLoader23recursiveInitializationERKNS_11LinkContextEj ()
#5 0x00007fff5fc0bda6 in
__dyld__ZN11ImageLoader15runInitializersERKNS_11LinkContextE ()
#6 0x00007fff5fc08fbb in __dyld_dlopen ()
#7 0x00007fff84902d40 in dlopen ()
#8 0x000000010070a0d2 in py_dl_open ()
#9 0x00000001000bb4ed in PyEval_EvalFrameEx ()
(More stack frames follow...)

[ronaldoussoren]

This probably means that the ctypes code in urllib.py needs to be ported
to C, although we won't know if that helps until said C code is written
:-(

Doing that would be a good idea anyway, while trying to create a
workaround I noticed that the ctypes code is invalid anyway because
there are not enough ctypes annotations when running in 64-bit mode
(where sizeof(int) != sizeof(long))

In 32-bit mode you can avoid the crash by calling urllib.urlopen or
urllib.proxy_bypass once on the main thread, but you have to use a
qualified name ("localhost.localdomain", not "localhost").

[warsaw]

Agreed this should be a release blocker for 2.6.3

[ronaldoussoren]

The attached patch seems to fix the issue, but needs further testing.

Warning: the patch is not entirely clean, the patch contains an
unrelated change to setup.py.

The patch replaces some code that uses ctypes to read configuration data
using the SystemConfiguration framework by a regular extension.

With this patch the crasher goes away, I still have to do more testing
and proper checking of the C code.

The patch is for the trunk, I expect it will cleanly port to 2.6. It
will also need porting to 3.x later on.

[ronaldoussoren]

The attached patch was not 100% correct in the error handling in the
set_proxies functions. That is fixed in the actually commit.

Committed as r74962 (trunk), r74963 (2.6).