test_queue fails occasionally

[mwhudson]

BPO	1010777
Nosy	@mwhudson, @tim-one

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/mwhudson'
closed_at = <Date 2004-08-20.12:41:13.000>
created_at = <Date 2004-08-17.14:54:16.000>
labels = ['interpreter-core']
title = 'test_queue fails occasionally'
updated_at = <Date 2004-08-20.12:41:13.000>
user = 'https://github.com/mwhudson'

bugs.python.org fields:

activity = <Date 2004-08-20.12:41:13.000>
actor = 'mwh'
assignee = 'mwh'
closed = True
closed_date = None
closer = None
components = ['Interpreter Core']
creation = <Date 2004-08-17.14:54:16.000>
creator = 'mwh'
dependencies = []
files = []
hgrepos = []
issue_num = 1010777
keywords = []
message_count = 5.0
messages = ['22099', '22100', '22101', '22102', '22103']
nosy_count = 2.0
nosy_names = ['mwh', 'tim.peters']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = None
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue1010777'
versions = ['Python 2.4']

[mwhudson]

I guess this is some kind of race condition. I see
test_queue fail about 5% of the time (judged by running
./python ../Lib/test/regrtest.py test_queue test_queue
test_queue test_queue ... with a couple of hundred
test_queue's, but I've seen it fail as part of a more
normal test run too).

It's always this failure:

test test_queue failed -- blocking function '<bound
method FailingQueue.put of
<test.test_queue.FailingQueue instance at 0x4041b184>>'
appeared not to block

and then, at least sometimes, you get this traceback:

Exception in thread Thread-3408:
Traceback (most recent call last):
  File
"/home/mwh/src/python/dist/src/Lib/threading.py", line
442, in __bootstrap
    self.run()
  File
"/home/mwh/src/python/dist/src/Lib/test/test_queue.py",
line 23, in run
    self.fn(*self.args)
  File "/home/mwh/src/python/dist/src/Lib/Queue.py",
line 129, in get
    item = self._get()
  File
"/home/mwh/src/python/dist/src/Lib/test/test_queue.py",
line 58, in _get
    return Queue.Queue._get(self)
AttributeError: 'NoneType' object has no attribute 'Queue'

somewhat later, usually while a subsequent test is running.

This is a debug build from CVS as of an hour or so back
on Redhat Linux 9 (so, in particular, with the NPTL
threading libray, not LinuxThreads).

[tim-one]

Logged In: YES
user_id=31435

FYI, Guido and I first noticed that test_queue sometimes fails
a few years ago. It's apparently much more likely to fail on a
non-Windows box, perhaps because Windows is happy to
switch threads frequently. I ran it for half an hour today on
a WinXP box, and it didn't fail at all. IIRC, it's not easy to fix -

• there are threads, and some of the coordination is done via
  time.sleep() calls.

I'll leave this assigned to me for a few more days, but it's
unlikely I'll find time to "do something" about it.

[tim-one]

Logged In: YES
user_id=31435

OK, it looks like your particular primary failure mode is
inevitable at times. If I boost the sleep in _TriggerThread.run
(), it fails every time, in the way you describe, on my box.

The difficulty appears to be one of test design.
_doBlockingTest believes that no block_func passed to it can
possibly return while the queue remains blocked, but the
failing test is passing it a block_func that returns via raising
FailingQueueException regardless of whether the queue is
blocked. So what we see is just the result of a thread race
in the test.

But as Freud said, insight does not imply a cure <wink>.

[tim-one]

Logged In: YES
user_id=31435

Michael, please give current CVS a try. I believe I truly fixed
the test race at the heart of your specific primary problem,
and greatly decreased the likelihood of two other broad
classes of sporadic race-related failures in the test driver.

[mwhudson]

Logged In: YES
user_id=6656

That seems to have done the trick! At least, I've run
test_queue a couple of hundred times with no failure, and
I'd have expected ~10 or so before I updated.

After reading the checkin comments: I suspect you guessed as
much, but system load was not a factor here.

Closing.