Queue doesn't recognize it is full after shrinking maxsize

[jaraco]

BPO	10110
Nosy	@rhettinger, @jaraco, @merwok
Files	shrinking_queue_not_full.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/rhettinger'
closed_at = <Date 2010-10-31.17:58:21.170>
created_at = <Date 2010-10-15.01:25:04.157>
labels = []
title = "Queue doesn't recognize it is full after shrinking maxsize"
updated_at = <Date 2010-10-31.17:58:21.140>
user = 'https://github.com/jaraco'

bugs.python.org fields:

activity = <Date 2010-10-31.17:58:21.140>
actor = 'rhettinger'
assignee = 'rhettinger'
closed = True
closed_date = <Date 2010-10-31.17:58:21.170>
closer = 'rhettinger'
components = []
creation = <Date 2010-10-15.01:25:04.157>
creator = 'jaraco'
dependencies = []
files = ['19240']
hgrepos = []
issue_num = 10110
keywords = ['patch']
message_count = 7.0
messages = ['118736', '118828', '118834', '118835', '118839', '118840', '120073']
nosy_count = 3.0
nosy_names = ['rhettinger', 'jaraco', 'eric.araujo']
pr_nums = []
priority = 'low'
resolution = 'fixed'
stage = 'patch review'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue10110'
versions = ['Python 3.2']

[jaraco]

The Queue object has a maxsize parameter and attribute, but due to the test for a full queue, shrinking the maxsize could result in the Queue not recognizing that it is full.

The attached patch (against the Python 3 trunk) demonstrates this limitation with a unit test and fixes the failing test case.

[jaraco]

I should point out that in Python 2.5, it was possible for a subclass to override the _full method to account for this situation, but with Python 2.6 and later, the calculation in _full was hand-inlined... so it's not readily possible for a subclass to correct for this behavior. It might be desirable to refactor that calculation into a _full method, though I suspect this would have performance implications, which is why I stuck with just adjusting the comparison in its various uses.

[rhettinger]

I've briefly looked at the patch and it seems reasonable. Will look it in more detail by 3.2 goes out.

I'm curious about your use case for wanting to change the maxsize of an existing Queue that is already in use. That seems like a odd design.

[jaraco]

I'm not sure what our use case is. I discovered this when I was looking at our project's util library, and we have a Queue subclass that overrides _full to handle the scenario where the queue shrinks. I'm guessing it's being used to dynamically adjust the queue size for performance reasons. Consider where you have a queue of items, but depending on the run time environment, that queue might need to be more or less limiting. Maybe the queue is larger during the day when requests are heavier and shrinks at night (so it's not accepting more items than it can handle when day breaks).

Surely that's a contrived example, and I suspect our actual usage is more concrete. I'll see if I can learn where we're using that functionality.

If shrinking maxsize is not allowed, that should be enforced in the interface (such as with a read-only/increase-only property) and documented. If you would rather proceed that way, I'll draft the patch for that instead.

[rhettinger]

That won't be necessary. The change from == to <= is innocuous.

There's no need to lock-up maxsize in a read-only property. We're consenting adults. Besides, it would probably break someone-else's odd use case. I don't want to expand the API, nor do I want to cripple anyone's ability to do weird stuff with it.

FWIW, the full() and empty() methods are usually not a good idea. It's better to catch a Full exception. Otherwise, the information can be out of date by the time you try to use it.

I'm going to mark this as a 3.2 only change. There were no guarantees about the behavior when maxsize is changed, nor should we make such guarantees.

[jaraco]

Thanks!

[rhettinger]

Applied in r86049.