test_array fails on FreeBSD7 amd64

[robrien]

BPO	3862
Nosy	@loewis, @birkenfeld, @terryjreedy, @mdickinson
Files	64bit_2.patch: Possible fix?

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2013-03-28.10:00:00.143>
created_at = <Date 2008-09-14.01:30:18.257>
labels = ['tests', 'type-crash']
title = 'test_array fails on FreeBSD7 amd64'
updated_at = <Date 2013-03-28.10:00:00.107>
user = 'https://bugs.python.org/robrien'

bugs.python.org fields:

activity = <Date 2013-03-28.10:00:00.107>
actor = 'georg.brandl'
assignee = 'none'
closed = True
closed_date = <Date 2013-03-28.10:00:00.143>
closer = 'georg.brandl'
components = ['Tests']
creation = <Date 2008-09-14.01:30:18.257>
creator = 'robrien'
dependencies = []
files = ['11503']
hgrepos = []
issue_num = 3862
keywords = ['patch']
message_count = 16.0
messages = ['73204', '73209', '73225', '73256', '73257', '73299', '73335', '73367', '73368', '73379', '74076', '74078', '74096', '74113', '107979', '185413']
nosy_count = 6.0
nosy_names = ['loewis', 'georg.brandl', 'terry.reedy', 'aimacintyre', 'mark.dickinson', 'robrien']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'closed'
superseder = None
type = 'crash'
url = 'https://bugs.python.org/issue3862'
versions = ['Python 2.6', 'Python 2.5', 'Python 3.0']

[robrien]

or in FreeBSD?

2.6rc1 and 3.0b3 both fail test_array on FreeBSD7 amd64

test_array passes in 2.5.2 on the same machine but fails test_list the
same as test_array...

*** Signal 9

[aimacintyre]

2.6rc1 test_array passes on FreeBSD 6.3 i386. I don't have a 7.0 box
(either i386 or amd64) accessible at the moment to cross check.

Can you run the test on its own in verbose mode to get a bit more of an
idea where its blowing up? - ie:

user@box$ ./python -E -tt Lib/test/regrtest.py -v test_array

from the build directory.

[robrien]

2.6rc2 and Python-3.0b3 test_array detail

test_alloc_overflow (test.test_array.DoubleTest) ... Killed

Fills swap space and dumps core.

2.5.2

test_list
test_addmul (test.test_list.ListTest) ... ok
test_append (test.test_list.ListTest) ... ok
Killed

The FreeBSD ports patches fix this in 2.5.2. Specifically patching
seq_tests.py to limit test_bigrepeat() to if sys.maxint <= 2147483647.
no other tests fail; so I don't know immediately what else is patched.
Although there are about 25 patches for the 2.5 port.

[aimacintyre]

I've temporarily got a 7.0 amd64 system running and can confirm what you
see. I checked out the 2.5.2 port patches you mentioned, but all the
ones that seemed related are already in the 2.6rc1 sources.

On a hunch, I used ulimit -v is used to set a process virtual memory
size limit, and the test completed satisfactorily (I set it to 1048756,
ie 1GB).

I'm as yet none the wiser as to what to do with this info...

[aimacintyre]

On FreeBSD 7.0 i386, test_array passes without ulimit -v needing to be
set (ie its still the default "unlimited".

[mdickinson]

Does the attached patch help fix this at all? I encountered possibly the
same problem on a 64-bit Mac build of Python, and this patch fixed the
problem for me.

[aimacintyre]

Mark, your patch will probably get the test to pass, but the underlying
reason the test is failing appears to be unexpected behaviour of the
platform malloc().

FreeBSD 7.0 introduced a new malloc() implementation that relies on
mmap() and this is behaving differently to the malloc() implementation
in FreeBSD 6.3 which relied on sbrk().

I have posted a query about the new malloc()'s behaviour to a FreeBSD
forum and will report what I find out.

[mdickinson]

Thanks, Alan. I realised my answer was a shallow one after reading (too
late!) the thread you started on python-dev.

However, I have a suspicion that that particular array test
(test_alloc_overflow) was merely meant to test the code in
newarrayobject, at around line 427 of arraymodule.c, which looks like:

 	/* Check for overflow */
	if (nbytes / descr->itemsize != (size_t)size) {
		return PyErr_NoMemory();
	}

and that the test dated from an era when it was fairly safe to assume
that a size_t was at most 32 bits. I'd guess that test_alloc_overflow
was never intended to be a test of OS malloc failure behaviour.

So the array test is wrong, and I think this patch should be applied
anyway. I admit this doesn't help with the much more interesting
question of what's going on with malloc on FreeBSD.

[mdickinson]

s/Alan/Andrew/. Need more coffee. Apologies.

[loewis]

The patch looks fine to me, except that the failure message is now out
of date.

[aimacintyre]

On FreeBSD 7.0 amd64, applying Mark's patch does get the test to pass.

As noted on python-dev, the response I got from the author of the new
malloc() package in FreeBSD 7.x indicates that as of 7.1 the allocator
defaults to using sbrk() rather than mmap() (strategy selection is
tunable with an environment variable) so the problematic behaviour
should be masked once 7.1 is released.

Martin: I'm missing something regarding your comment about the failure
message - the exception text seems literally correct (though not
particularly useful), though I haven't spent a lot of time trying to
understand how test_array works. What would you suggest?

[mdickinson]

The 2**16 in the error message is wrong. Maybe the error message should
be something like:

"Attempt to create array of size larger than maxsize failed to raise
MemoryError."

A bit verbose, but more accurate.

There's another error in that patch, too. The line:

b * maxsize//3 + 1

should be

b * (maxsize//3 + 1)

(I was mistakenly reading the '*' as '*=', as in the first test.)

[loewis]

The 2**16 in the error message is wrong.

As he says.

[aimacintyre]

Yes, the bleeding obvious became so after some sleep.

Committed to trunk as r66708 with the extra fix and a more abbreviated
failure message, which I hope is still semantically the same as Mark's
suggested text.

Will forward port as appropriate to py3k.

[terryjreedy]

Was the fix forward ported or is this an issue for 3.1/2?

[birkenfeld]

Closing this old pending issue due to lack of activity.