Exceed Recursion Limit in Thread

[jaredlang]

BPO	9670
Nosy	@ronaldoussoren, @pitrou, @ned-deily, @bitdancer, @ambv
Files	issue9670.patch issue9670-v2.txt test_issue9670.py issue9670-v3.patch issue9670-v4.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/ronaldoussoren'
closed_at = <Date 2012-06-30.01:28:56.951>
created_at = <Date 2010-08-24.13:23:18.915>
labels = ['OS-mac', 'type-crash']
title = 'Exceed Recursion Limit in Thread'
updated_at = <Date 2012-06-30.01:28:56.934>
user = 'https://bugs.python.org/jaredlang'

bugs.python.org fields:

activity = <Date 2012-06-30.01:28:56.934>
actor = 'ned.deily'
assignee = 'ronaldoussoren'
closed = True
closed_date = <Date 2012-06-30.01:28:56.951>
closer = 'ned.deily'
components = ['macOS']
creation = <Date 2010-08-24.13:23:18.915>
creator = 'jaredlang'
dependencies = []
files = ['19298', '21126', '21127', '21527', '21907']
hgrepos = []
issue_num = 9670
keywords = ['patch', 'needs review']
message_count = 26.0
messages = ['114787', '114788', '114789', '114793', '119210', '119214', '119217', '119219', '119222', '130873', '130874', '132916', '133217', '133415', '133417', '133418', '133430', '133431', '135271', '135292', '137108', '137109', '164338', '164342', '164344', '164363']
nosy_count = 8.0
nosy_names = ['ronaldoussoren', 'pitrou', 'ned.deily', 'r.david.murray', 'santagada', 'lukasz.langa', 'jaredlang', 'python-dev']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'crash'
url = 'https://bugs.python.org/issue9670'
versions = ['Python 3.1', 'Python 2.7', 'Python 3.2']

[jaredlang]

Recursion within a thread on OSX can result in a crash by exceeding the systems recursion limit. Recursion behaves as expected if not in thread, meaning it throws a RunTimeError with the message "maximum recursion depth exceeded."

The crash is able to be avoided if the recursion limit is set to a lower number, ie. 800, via sys.setrecursionlimit.

Example program which crashes on OSX:

>>> def f():
...     return f()
... 
>>> import threading
>>> thread = threading.Thread(target=f)
>>> thread.start()
Bus error

Alternatively, the following works as expected:

>>> import sys
>>> def f():
...     return f()
... 
>>> import threading
>>> thread = threading.Thread(target=f)
>>> sys.setrecursionlimit(800)
>>> thread.start()

>>> Exception in thread Thread-1:
Traceback (most recent call last):
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/threading.py", line 532, in __bootstrap_inner
    self.run()
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/threading.py", line 484, in run
    self.__target(*self.__args, **self.__kwargs)
  File "<stdin>", line 2, in f
  File "<stdin>", line 2, in f
  File "<stdin>", line 2, in f
  File "<stdin>", line 2, in f
  File "<stdin>", line 2, in f
  File "<stdin>", line 2, in f
...

RuntimeError: maximum recursion depth exceeded

[bitdancer]

This is not the first recursion limit related problem we've seen with FreeBSD derived systems. See for example bpo-1201456. It would be nice to have an actual fix for this class of problem, but I have no clue what that would look like. There may also be a specific fix for this issue. At least, we can hope :)

Python 2.6 is in security-fix-only mode, so I've updated the versions. I can confirm that this reproduces on 2.7 and py3k trunk on OS/X 10.4 using a non-framework build. I haven't tested 3.1 but I am sure it must behave the same.

[ronaldoussoren]

This is the same issue as bpo-6763.

The root cause of this issue is that the default stack size for threads other than the main thread is small.

One way to fix this issue is to increase the default stacksize for new threads.

[bitdancer]

As I mentioned on the other ticket, I think bumping the default stack size is probably the most useful solution. This is is a clearer description of the stack problem, since it doesn't get involved in MIMETypes internals, so I'm not closing it as a duplicate.

[ronaldoussoren]

The attached patch explicitly sets the minimal stack size to about 704K, which is the minimal size where 'def f(): f()' doesn't cause a buserror when run in a thread.

[santagada]

Why not make it bigger so it doesn't crash for much bigger functions?

[ronaldoussoren]

On 20 Oct, 2010, at 17:41, Leonardo Santagada wrote:

Leonardo Santagada <santagada@gmail.com> added the comment:

Why not make it bigger so it doesn't crash for much bigger functions?

I don't mind making it bigger, but a larger size does come with a cost: the larger the initial stack size the less threads you can have without trashing the system (or running out of address space on a 32-bit build).

I'm fine with increasing it to 1M, which is a nice round number build not larger.

BTW. I haven't looked up the default stack size for the main thread on OSX.

Ronald

----------
nosy: +santagada

---

Python tracker <report@bugs.python.org>
<http://bugs.python.org/issue9670\>

---

[santagada]

The default stack size in osx is 8MB. I think it is a safer bet, and no one is supposed to be using more than like 40-50 threads anyway (specially in 32 bit only systems limited to 8 cores).

Is there a user case for hundreds/thousands of threads in python?

[bitdancer]

It would be nice if we could expand this fix to include FreeBSD, which as I understand it has the same problem.

[ronaldoussoren]

I've slightly updated the patch:

1. Increase default stack size to 1MByte

2. Change the preprocessor guard to also trigger on FreeBSD

Without this patch I get crashes for 32-bit and 64-bit Intel architectures on OSX 10.6, with this patch those crashes go away.

I don't have access to a freebsd box so cannot test there.

[ronaldoussoren]

The attached unittest file triggers the issue until the patch is applied.

I'm not sure if I'd be a good idea to add this test as is to the testsuite, given the hard crash when it fails.

[ned-deily]

Here's an updated combined patch. I've revised Ronald's test to run via subprocess so it should be safe to add to the standard tests. It works as expected on OS X. If there are no objections, I will commit it and monitor the buildbots.