MemoryError in test_strptime

[serhiy-storchaka]

BPO	25029
Nosy	@abalkin, @vstinner, @larryhastings, @serhiy-storchaka, @eryksun, @zooba
Files	25029_1.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/zooba'
closed_at = <Date 2015-09-10.20:01:59.642>
created_at = <Date 2015-09-08.10:30:13.592>
labels = ['extension-modules', 'type-crash', 'release-blocker']
title = 'MemoryError in test_strptime'
updated_at = <Date 2015-09-10.20:01:59.640>
user = 'https://github.com/serhiy-storchaka'

bugs.python.org fields:

activity = <Date 2015-09-10.20:01:59.640>
actor = 'steve.dower'
assignee = 'steve.dower'
closed = True
closed_date = <Date 2015-09-10.20:01:59.642>
closer = 'steve.dower'
components = ['Extension Modules']
creation = <Date 2015-09-08.10:30:13.592>
creator = 'serhiy.storchaka'
dependencies = []
files = ['40407']
hgrepos = []
issue_num = 25029
keywords = ['patch']
message_count = 20.0
messages = ['250177', '250190', '250192', '250197', '250199', '250201', '250205', '250206', '250207', '250212', '250215', '250219', '250225', '250266', '250271', '250276', '250280', '250312', '250404', '250416']
nosy_count = 8.0
nosy_names = ['belopolsky', 'vstinner', 'larry', 'python-dev', 'serhiy.storchaka', 'eryksun', 'steve.dower', 'David.Edelsohn']
pr_nums = []
priority = 'release blocker'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'crash'
url = 'https://bugs.python.org/issue25029'
versions = ['Python 3.5', 'Python 3.6']

[serhiy-storchaka]

All builds on edelsohn-fedora-ppc64 are red starting from build 55 for 3.x and build 41 for 3.5. 3.4 is green.
http://buildbot.python.org/all/builders/PPC64%20Fedora%203.x/builds/55/
http://buildbot.python.org/all/builders/PPC64%20Fedora%203.5/builds/41/

======================================================================
ERROR: test_TimeRE_recreation (test.test_strptime.CacheTests)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/home/shager/cpython-buildarea/3.5.edelsohn-fedora-ppc64/build/Lib/test/test_strptime.py", line 565, in test_TimeRE_recreation
    _strptime._strptime_time('10', '%d')
  File "/home/shager/cpython-buildarea/3.5.edelsohn-fedora-ppc64/build/Lib/_strptime.py", line 494, in _strptime_time
    tt = _strptime(data_string, format)[0]
  File "/home/shager/cpython-buildarea/3.5.edelsohn-fedora-ppc64/build/Lib/_strptime.py", line 312, in _strptime
    _TimeRE_cache = TimeRE()
  File "/home/shager/cpython-buildarea/3.5.edelsohn-fedora-ppc64/build/Lib/_strptime.py", line 190, in __init__
    self.locale_time = LocaleTime()
  File "/home/shager/cpython-buildarea/3.5.edelsohn-fedora-ppc64/build/Lib/_strptime.py", line 75, in __init__
    self.__calc_am_pm()
  File "/home/shager/cpython-buildarea/3.5.edelsohn-fedora-ppc64/build/Lib/_strptime.py", line 114, in __calc_am_pm
    am_pm.append(time.strftime("%p", time_tuple).lower())
MemoryError

---

[larryhastings]

Any clue here? Is this unaligned access?

[DavidEdelsohn]

PPC64 is not a strict alignment system. The system is running a non-recent release of Fedora, so it could be a bad interaction with libc.

[vstinner]

I ran test_strptime & test_os with tracemalloc: test_os memory peak is higher than test_strptime memory peak (I ran the two tests independently). So the bug cannot be reproduce on my Fedora 22 (x86_64).

[vstinner]

Oh wait, the VmPeak is *much* high when running test_strptime:

• test_os VmPeak: 380 992 kB
• test_strptime VmPeak: 8 608 972 kB (8 GB!?)

1 GB should be enough for everybody:

$ bash -c 'ulimit -v 1000000; ./python -m test -v test_strptime'
..

======================================================================
ERROR: test_TimeRE_recreation (test.test_strptime.CacheTests)
----------------------------------------------------------------------

Traceback (most recent call last):
  ...
  File "/home/haypo/prog/python/default/Lib/_strptime.py", line 75, in __init__
    self.__calc_am_pm()
  File "/home/haypo/prog/python/default/Lib/_strptime.py", line 114, in __calc_am_pm
    am_pm.append(time.strftime("%p", time_tuple).lower())
MemoryError

[vstinner]

It's a regression introduced by c31dad22c80d (Issue bpo-24917).

[abalkin]

I would blame this change: <https://hg.python.org/cpython/rev/c31dad22c80d/#l3.41\>. The rest should not have any effect on Linux.

[vstinner]

Script to reproduce the issue:
---

import time
import locale
import pprint

time_tuple = time.struct_time((1999,3,17,1,44,55,2,76,0))

p1 = time.strftime("%p", time_tuple)
print("current LC_TIME", repr(p1))

locale.setlocale(locale.LC_TIME, ('de_DE', 'UTF8'))
p2 = time.strftime("%p", time_tuple)
print("de_DE (UTF8)", repr(p2))

---

Output:
---

$ python3.4 bug.py
current LC_TIME 'AM'
de_DE (UTF8) ''

---

The problem is that strftime()/wcsftime() *can* return 0, it's not an error. Whereas c31dad22c80d considers that if buflen is 0 but fmtlen is smaller than 5, we must retry with a larger buffer.

[vstinner]

Again, this issue remembers me the idea of rewriting strftime() in Python. We already have _strptime.py.

[zooba]

Sorry Larry.

I'll fix it.

[eryksun]

Steve, it seems to me that for MSVC the EINVAL test should come first:
   
    _Py_BEGIN_SUPPRESS_IPH
    olderr = errno;
    errno = 0;
    buflen = format_time(outbuf, i, fmt, &buf);
    err = errno;
    errno = olderr;
    _Py_END_SUPPRESS_IPH
    
    if (buflen == 0 && err == EINVAL) {
        PyErr_SetString(PyExc_ValueError, "Invalid format string");
        break;
    }

Then the old test could be restored, i.e. (buflen > 0 || i >= 256 * fmtlen).

[zooba]

@eryksun - that's exactly what I've just done :)

Unfortunately, I don't have a _locale module, so I can't do Victor's test. Attached my patch in case Victor is around to test it.

[vstinner]

My test runs the following command on Linux (Fedora 22):

make && bash -c 'ulimit -v 1000000; ./python -u -m test -v test_strptime'

• current default branch (rev 3c0c153d6b02): MemoryError
• with 25029_1.patch: the test pass

[zooba]

Thanks. I'll submit a PR for 3.5.0 later tonight - can't seem to clone Larry's repo successfully at work for some reason.

[zooba]

PR at https://bitbucket.org/larry/cpython350/pull-requests/21/issue-25029-memoryerror-in-test_strptime/diff

[larryhastings]

Does this need a Misc/NEWS entry?

[zooba]

Probably should, since the fix that caused it was in for rc3. There's no section for 3.5.0 final yet though (that's my excuse, anyway :) )

[python-dev]

New changeset bd7aed300a1b by Steve Dower in branch '3.5':
Issue bpo-25029: MemoryError in test_strptime
https://hg.python.org/cpython/rev/bd7aed300a1b

New changeset 706b9eaba734 by Steve Dower in branch '3.5':
Merge fix for bpo-25029
https://hg.python.org/cpython/rev/706b9eaba734

New changeset cceaccb6ec5a by Steve Dower in branch '3.5':
Adds Mics/NEWS entry for issue bpo-25029.
https://hg.python.org/cpython/rev/cceaccb6ec5a

[vstinner]

The test doesn't fail anymore on buildbots. I ran manually test_strptime with a memory limit of 1 GB on Python 3.5 and 3.6: the test pass.

Can we close the issue?

[zooba]

Done.