segfault on printing deeply nested structures (2.7 only)

[jonaskoelker]

BPO	1069092
Nosy	@birkenfeld, @terryjreedy, @vstinner, @benjaminp, @bitdancer

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2013-11-05.06:10:45.109>
created_at = <Date 2004-11-18.23:31:34.000>
labels = ['interpreter-core', 'type-crash']
title = 'segfault on printing deeply nested structures (2.7 only)'
updated_at = <Date 2013-11-05.06:10:45.107>
user = 'https://bugs.python.org/jonaskoelker'

bugs.python.org fields:

activity = <Date 2013-11-05.06:10:45.107>
actor = 'benjamin.peterson'
assignee = 'none'
closed = True
closed_date = <Date 2013-11-05.06:10:45.109>
closer = 'benjamin.peterson'
components = ['Interpreter Core']
creation = <Date 2004-11-18.23:31:34.000>
creator = 'jonaskoelker'
dependencies = []
files = []
hgrepos = []
issue_num = 1069092
keywords = []
message_count = 11.0
messages = ['60597', '60598', '60599', '74020', '75364', '101503', '101520', '101525', '114391', '202192', '202196']
nosy_count = 7.0
nosy_names = ['georg.brandl', 'terry.reedy', 'vstinner', 'jonaskoelker', 'benjamin.peterson', 'r.david.murray', 'BreamoreBoy']
pr_nums = []
priority = 'normal'
resolution = 'wont fix'
stage = 'needs patch'
status = 'closed'
superseder = None
type = 'crash'
url = 'https://bugs.python.org/issue1069092'
versions = ['Python 2.6', 'Python 2.7']

[jonaskoelker]

## bug1.py
loop = None,
while True: loop = loop, None

bash$ python -i bug1.py
(I press C-c)
>>> print loop
((((((((((((((((( <skipping a large amount of parens
for brevity> (((((((((segfault
bash$ 

bash$ python -i bug1.py
(I press C-c)
>>> str(loop)
segfault
bash$ 

bash$ python -i bug1.py
(I press C-c)
>>> repr(loop)
segfault
bash$

--- ---
the results are the same for bug2.py, bug3.py and
bug.py4 which have the same contents, except that loop
is a list, None is replaced by Ellipsis or both. The
behavior (i.e. segfaulting) is the same.
also, when pressing C-c, python's memory usage remains
constant. gc.garbage is empty; gc.collect() returns 0
and doesn't free memory.

my python and uname:

bash$ python
Python 2.3.4 (#2, Sep 24 2004, 08:39:09)
[GCC 3.3.4 (Debian 1:3.3.4-12)] on linux2

bash$ uname --all
Linux koelker 2.4.18-bf2.4 #1 Son Apr 14 09:53:28 CEST
2002 i686 GNU/Linux

I have also reproduced it on
Python 2.3.3 (#1, May 7 2004, 10:31:40)
[GCC 3.3.3 20040412 (Red Hat Linux 3.3.3-7)] on linux2

bash$ uname --all
Linux horse09 2.6.8-1.521smp #1 SMP Man Aug 16 09:25:06
EDT 2004 i686 i686 i386 GNU/Linux

and on
Python 2.4a3 (#1, Oct 3 2004, 12:05:12)
[GCC 3.3.2] on linux2
(on the same machine)

it didn't reproduce on
Python 1.5.2 (#1, Mar 9 2000, 17:40:34) [GCC 2.95.2
19991024 (release)] on hp-uxB

bash$ uname -a
HP-UX aragorn B.11.11 U 9000/782 2006786478
unlimited-user license

but it did on
Python 1.5.2 (#1, Jul 5 2001, 03:02:19) [GCC 2.96
20000731] (Red Hat Linux 7.1 2 on linux-i386

bash$ uname
Linux legolas 2.4.20-20.7smp #1 SMP Mon Aug 18 14:46:14
EDT 2003 i686 unknown

[terryjreedy]

Logged In: YES
user_id=593130

On <<<Python 2.2.1 (#34, Apr  9 2002, 19:34:33) [MSC 32 
bit (Intel)] on win32>>> I got (interactive mode, about 5 
sec before ^C)  appropriate output:

((((((((((((((((((((((((((((((((((((((((((((((((((((((
Traceback (most recent call last):
  File "<stdin>", line 1, in ?
MemoryError: stack overflow

The nested depth was
>>> i=0
>>> while loop: loop,i = loop[0], i+1
...
>>> i
50695

Note: a test case would need a finite loop such as:
for i in range(100000)
instead of while True (until ^C pressed).

[birkenfeld]

Logged In: YES
user_id=1188172

I can reproduce this here on Linux with 100000x None. Will
try to investigate.

[vstinner]

This issue is a stack overflow: your code do recursive calls to
internal_print().

Backtrace from gdb:
--------------------------------------------
#0 0xb7e92064 in _IO_new_file_overflow ()
from /lib/tls/i686/cmov/libc.so.6
#1 0xb7e94bc3 in __overflow () from /lib/tls/i686/cmov/libc.so.6
#2 0xb7e8e03b in fputc () from /lib/tls/i686/cmov/libc.so.6
#3 0x080a2f2d in tupleprint (op=0xb7b5cc2c, fp=0xb7f744e0, flags=0)
at Objects/tupleobject.c:193
#4 0x0808b9f7 in internal_print (op=0xb7b5cc2c, fp=0xb7f744e0,
flags=0, nesting=0) at Objects/object.c:309
#5 0x0808ba60 in PyObject_Print (op=0xb7b5cc2c, fp=0xb7f744e0,
flags=0) at Objects/object.c:324
(...)
bpo-4628 0x0808ba60 in PyObject_Print (op=0xb7b68e4c, fp=0xb7f744e0,
flags=0) at Objects/object.c:324
bpo-4629 0x080a2f9e in tupleprint (op=0xb7b68e6c, fp=0xb7f744e0, flags=0)
at Objects/tupleobject.c:201
bpo-4630 0x0808b9f7 in internal_print (op=0xb7b68e6c, fp=0xb7f744e0,
flags=0, nesting=0) at Objects/object.c:309
bpo-4631 0x0808ba60 in PyObject_Print (op=0xb7b68e6c, fp=0xb7f744e0,
flags=0) at Objects/object.c:324
(...)
bpo-4638 0x080a2f9e in tupleprint (op=0xb7b68ecc, fp=0xb7f744e0, flags=0)
at Objects/tupleobject.c:201
bpo-4639 0x0808b9f7 in internal_print (op=0xb7b68ecc, fp=0xb7f744e0,
flags=0, nesting=0) at Objects/object.c:309
bpo-4640 0x0808ba60 in PyObject_Print (op=0xb7b68ecc, fp=0xb7f744e0,
flags=0) at Objects/object.c:324
bpo-4641 0x080a2f9e in tupleprint (op=0xb7b68eec, fp=0xb7f744e0, flags=0)
at Objects/tupleobject.c:201
bpo-4642 0x0808b9f7 in internal_print (op=0xb7b68eec, fp=0xb7f744e0,
flags=0, nesting=0) at Objects/object.c:309
(...)
--------------------------------------------

Informations using my debugger (python-ptrace):
------------------------------------------------------------
PID: 1201
Signal: SIGSEGV
STACK OVERFLOW! Stack pointer is in 0xbf10d000-0xbf90d000 => [stack]
(rw-p)

• instruction: CALL 0x80d3305
• mapping: 0xbf10cff0 is not mapped in memory
• register <stack ptr>=0xbf10cff0
  ------------------------------------------------------------

Some solutions:

• avoid creation of nested sequence
• create generic protection against stack overflow (use
  setjmp/longjmp)
• avoid recursive call to internal_print()

[vstinner]

I propose a generic solution in issue bpo-3999 to catch stack overflow
(and be able to continue after such "exception").

[vstinner]

This issue is 6 years old. Python has some protections against stack overflow, but there is no perfect solution. Since there is no patch, I consider that I can close this issue (as wont fix).

[terryjreedy]

General policy is that ordinary code (not using, for instance, ctypes) should not crash or segfault the interpreter. I believe there is a 'crashers' subdirectory somewhere in the tree for examples that do so that people so inclined can work on them.

The OP reported a crash on 2.3/2.4a on Linux, but not 1.5. I could not reproduce it on 2.2 on Windows. Instead, I (properly) got an exception. Trying again with 3.1, I get a similar exception: RuntimeError: maximum recursion depth exceeded while getting the repr of a tuple. List instead of tuple does similar.

If 
>>> t=None,
>>> for i in range(50000): t = t,None

>> print(t)

still crashes on 2.6/2.7, at least with Linux, then there is still a bug to be fixed and the issue should be left open. If it now raises an exception as above, then this should be closed as fixed.

I am pretty sure this issue has nothing to do with None and Ellipsis but only with the structure (not necessarily a sequence) being deeply, deeply nested. So I think the title should be: "Segfault on printing deeply nested structures."

I think the deeper issue is the use of recursion on the C stack to print. If the print routine instead usee iteration with an auxiliary Python stack (list), then there should be no stack overflow to worry about. [When in increase the recursion limit to 100000 and try to print the 50000 nested tuples, I get "MemoryError: stack overflow" instead of the RuntimeError above. So there might be more than one fix needed.]

[bitdancer]

On linux: on Py3 (trunk and 3.1) I get the recursion depth exceeded message, but on py2 trunk I get the segfault if I use 100000 for the range. So somebody fixed this crasher in py3, somehow.