assertEqual memory issues with large text inputs

[voidspace]

BPO	11763
Nosy	@rhettinger, @pitrou, @ezio-melotti, @voidspace, @serhiy-storchaka
Files	issue11763.diff: Patch to add a _diffThreshold of 2**16 (2.7) issue11763-2.diff issue11763_safe_repr.diff issue11763-3.diff

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/ezio-melotti'
closed_at = <Date 2014-08-03.09:36:44.389>
created_at = <Date 2011-04-04.20:16:03.858>
labels = ['type-bug', 'library']
title = 'assertEqual memory issues with large text inputs'
updated_at = <Date 2014-08-03.09:36:44.387>
user = 'https://github.com/voidspace'

bugs.python.org fields:

activity = <Date 2014-08-03.09:36:44.387>
actor = 'ezio.melotti'
assignee = 'ezio.melotti'
closed = True
closed_date = <Date 2014-08-03.09:36:44.389>
closer = 'ezio.melotti'
components = ['Library (Lib)']
creation = <Date 2011-04-04.20:16:03.858>
creator = 'michael.foord'
dependencies = []
files = ['21535', '21536', '22462', '32104']
hgrepos = []
issue_num = 11763
keywords = ['patch']
message_count = 12.0
messages = ['132965', '132986', '132987', '134529', '134530', '139061', '139080', '139081', '181551', '182423', '199830', '224622']
nosy_count = 7.0
nosy_names = ['rhettinger', 'pitrou', 'ezio.melotti', 'michael.foord', 'python-dev', 'sara.magliacane', 'serhiy.storchaka']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue11763'
versions = ['Python 2.7', 'Python 3.2', 'Python 3.3', 'Python 3.4']

[voidspace]

>>> s = "x" * (2**29)
>>> case.assertEqual(s + "a", s + "b")
Traceback (most recent call last):
 File "<stdin>", line 1, in <module>
 File "/home/antoine/cpython/default/Lib/unittest/case.py", line 643,
in assertEqual assertion_func(first, second, msg=msg)
 File "/home/antoine/cpython/default/Lib/unittest/case.py", line 984,
in assertMultiLineEqual secondlines = [second + '\n']
MemoryError

assertEqual delegates to assertMultilineEqual for comparing text which uses difflib for comparisons. This has performance issues (as well as memory issues) for very large inputs, so should fallback to a simple comparison (or simpler diff generation technique) for very large inputs.

[ezio-melotti]

Attached patch adds a _diffThreshold attribute of 2**16 and uses _baseAssertEqual whenever one of the two string is longer than 2**16 chars.

[pitrou]

Rather than hardwiring self.addCleanup(lambda: setattr(self, '_diffThreshold', 2**16)), you should retrieve the previous value.

[python-dev]

New changeset 8dbf661c0a63 by Ezio Melotti in branch '2.7':
bpo-11763: don't use difflib in TestCase.assertMultiLineEqual if the strings are too long.
http://hg.python.org/cpython/rev/8dbf661c0a63

New changeset 04e64f77c6c7 by Ezio Melotti in branch '3.1':
bpo-11763: don't use difflib in TestCase.assertMultiLineEqual if the strings are too long.
http://hg.python.org/cpython/rev/04e64f77c6c7

New changeset b316019638df by Ezio Melotti in branch '3.2':
bpo-11763: merge with 3.1.
http://hg.python.org/cpython/rev/b316019638df

New changeset df154c872b0c by Ezio Melotti in branch 'default':
bpo-11763: merge with 3.2.
http://hg.python.org/cpython/rev/df154c872b0c

[ezio-melotti]

I committed a slightly modified version of the patch, so the issue should be fixed now.

There are two related problems though:

1. difflib is used in other places, so those should probably be checked too;
2. _baseAssertEqual should check if the len of the msg (or the sum of the lengths of the two args) is greater than maxDiff and use safe_repr(arg, short=True) to avoid long diffs.

[saramagliacane]

Here is a small patch addressing the second point of the last message, but the tests are still missing.

[voidspace]

The basic idea of the patch is good, but instead of introducing _MAX_LENGTH, maxDiff should be reused.

[voidspace]

Sorry, ignore that. I see that the patch already passes maxDiff to truncate_str.

[serhiy-storchaka]

I left comments on Rietveld. Tests needed.

[ezio-melotti]

There are some tests in the bpo-11763-2.diff patch.