Error on test_issue_1395_5 with Python 2.7 and VS2010

[sable]

BPO	13461
Nosy	@loewis, @amauryfa, @pitrou, @briancurtin, @serhiy-storchaka
Files	issue13461-27.patch issue13461-3x.patch: similar patch for 3.x to not use numerical literals

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/serhiy-storchaka'
closed_at = <Date 2013-08-20.18:24:04.174>
created_at = <Date 2011-11-23.14:02:38.396>
labels = ['easy', 'expert-IO', 'type-crash']
title = 'Error on test_issue_1395_5 with Python 2.7 and VS2010'
updated_at = <Date 2013-08-20.18:24:04.172>
user = 'https://bugs.python.org/sable'

bugs.python.org fields:

activity = <Date 2013-08-20.18:24:04.172>
actor = 'serhiy.storchaka'
assignee = 'serhiy.storchaka'
closed = True
closed_date = <Date 2013-08-20.18:24:04.174>
closer = 'serhiy.storchaka'
components = ['IO']
creation = <Date 2011-11-23.14:02:38.396>
creator = 'sable'
dependencies = []
files = ['30238', '30239']
hgrepos = []
issue_num = 13461
keywords = ['patch', 'easy']
message_count = 10.0
messages = ['148184', '148186', '148257', '148327', '148328', '182102', '182103', '189061', '195704', '195708']
nosy_count = 9.0
nosy_names = ['loewis', 'amaury.forgeotdarc', 'pitrou', 'sable', 'brian.curtin', 'python-dev', 'serhiy.storchaka', 'prlw1', 'Yogesh.Chaudhari']
pr_nums = []
priority = 'high'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'crash'
url = 'https://bugs.python.org/issue13461'
versions = ['Python 2.7', 'Python 3.3', 'Python 3.4']

[sable]

I am trying to get Python working when compiled with Visual Studio 2010 (cf bpo-13210).

When running the tests with the python 2.7 branch compiled with VS2010, the "test_issue_1395_5" in test_io.py will cause Python to eat the whole memory within a few seconds and make the server completely unresponsive.

[sable]

The problem is in CTextIOWrapperTest.test_issue1395_5
Here is the backtrace:

msvcr100d.dll!memset()  Line 145	Asm

msvcr100d.dll!_heap_alloc_dbg_impl(unsigned __int64 nSize, int nBlockUse, const char * szFileName, int nLine, int * errno_tmp) Line 498 C++
msvcr100d.dll!_nh_malloc_dbg_impl(unsigned __int64 nSize, int nhFlag, int nBlockUse, const char * szFileName, int nLine, int * errno_tmp) Line 239 + 0x22 bytes C++
msvcr100d.dll!_nh_malloc_dbg(unsigned __int64 nSize, int nhFlag, int nBlockUse, const char * szFileName, int nLine) Line 302 + 0x2a bytes C++
msvcr100d.dll!malloc(unsigned __int64 nSize) Line 56 + 0x21 bytes C++
python27_d.dll!PyObject_Malloc(unsigned __int64 nbytes) Line 944 C
python27_d.dll!_PyObject_DebugMallocApi(char id, unsigned __int64 nbytes) Line 1445 + 0xa bytes C
python27_d.dll!_PyObject_DebugMalloc(unsigned __int64 nbytes) Line 1413 C
python27_d.dll!PyString_FromStringAndSize(const char * str, __int64 size) Line 88 + 0x11 bytes C
python27_d.dll!do_mkvalue(const char * * p_format, char * * p_va, int flags) Line 427 + 0xf bytes C
python27_d.dll!va_build_value(const char * format, char * va, int flags) Line 537 + 0x14 bytes C
python27_d.dll!_Py_VaBuildValue_SizeT(const char * format, char * va) Line 511 C
python27_d.dll!_PyObject_CallMethod_SizeT(_object * o, char * name, char * format, ...) Line 2671 + 0xf bytes C
python27_d.dll!textiowrapper_tell(textio * self, _object * args) Line 2222 + 0x2c bytes C

So the problem happens when calling in textio.c:
{{{
PyObject *decoded = PyObject_CallMethod(
            self->decoder, "decode", "s#", input, 1);
}}}

self->decoder is of type "_io.IncrementalNewlineDecoder" and input is "BBB".

This will result in PyString_FromStringAndSize being called with size = 4294967297, which will cause the server to fall.

[pitrou]

What if you replace:

PyObject *decoded = PyObject_CallMethod(
            self->decoder, "decode", "s#", input, 1);

with:

PyObject *decoded = PyObject_CallMethod(
            self->decoder, "decode", "s#", input, (Py_ssize_t) 1);

[sable]

Thanks Antoine! It solved the issue.

I will check soon with Python trunk to see if the same thing applies.

[amauryfa]

I've identified a few other cases where a '#' format is passed a numeric literal:

Python/codecs.c:514: return Py_BuildValue("(u#n)", &end, 0, end);
Modules/_io/textio.c:2323: DECODER_DECODE(input, 1, n);

[serhiy-storchaka]

Can we fix this easy issue before 2.7.4 release?

[pitrou]

Ageed, it's probably easy enough.

[YogeshChaudhari]

This patch should work for 2.7 branch

[python-dev]

New changeset 826233404be8 by Serhiy Storchaka in branch '3.3':
Issue bpo-13461: Fix a crash in the TextIOWrapper.tell method on 64-bit platforms.
http://hg.python.org/cpython/rev/826233404be8

New changeset 6c9d49b8e3ec by Serhiy Storchaka in branch 'default':
Issue bpo-13461: Fix a crash in the TextIOWrapper.tell method on 64-bit platforms.
http://hg.python.org/cpython/rev/6c9d49b8e3ec

New changeset 5e679ef2a55c by Serhiy Storchaka in branch '2.7':
Issue bpo-13461: Fix a crash in the TextIOWrapper.tell method and in the "replace"
http://hg.python.org/cpython/rev/5e679ef2a55c

[serhiy-storchaka]

Thank you for the report Sébastien. Thank you for the patch Yogesh.