Add closing and iteration to threading.Queue

[hemflit]

BPO	35034
Nosy	@tim-one, @rhettinger, @pitrou, @hemflit
PRs	bpo-35034: Add closing and iteration to threading.Queue #10018 bpo-35034: Add closing and iteration to threading.Queue #10018 bpo-35034: Add closing and iteration to threading.Queue #10018 bpo-35034: Add closing and iteration to threading.Queue #10018

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/rhettinger'
closed_at = <Date 2018-11-04.22:19:48.542>
created_at = <Date 2018-10-20.23:29:15.497>
labels = ['3.8', 'type-feature', 'library']
title = 'Add closing and iteration to threading.Queue'
updated_at = <Date 2018-11-09.12:23:18.813>
user = 'https://github.com/hemflit'

bugs.python.org fields:

activity = <Date 2018-11-09.12:23:18.813>
actor = 'hemflit'
assignee = 'rhettinger'
closed = True
closed_date = <Date 2018-11-04.22:19:48.542>
closer = 'rhettinger'
components = ['Library (Lib)']
creation = <Date 2018-10-20.23:29:15.497>
creator = 'hemflit'
dependencies = []
files = []
hgrepos = []
issue_num = 35034
keywords = ['patch', 'patch', 'patch', 'patch']
message_count = 5.0
messages = ['328181', '328186', '328952', '329269', '329510']
nosy_count = 4.0
nosy_names = ['tim.peters', 'rhettinger', 'pitrou', 'hemflit']
pr_nums = ['10018', '10018', '10018', '10018']
priority = 'normal'
resolution = 'rejected'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue35034'
versions = ['Python 3.8']

[hemflit]

Code using threading.Queue often needs to coordinate a "work is finished as far as far as I care" state between the producing and consuming side.

When going from the producer to the consumer ("No more items after this, so don't bother waiting"), this is usually implemented with sentinel objects, which is at best needlessly verbose and at worst tricky to get right (as with multiple consumers, or communicating a non-trivial sentinel object).
When going the other direction ("I'm not interested in consuming any more, so you can stop putting them on the queue"), or when a third component needs to notify both sides ("You two start wrapping up, but don't drop any in-flight items") there isn't even a clear usual solution.

Adding a close() method to the Queue (with accompanying exception treatment etc.) would solve all of this in a very clean way. It would not change anything for code that doesn't want to use it. It would simplify a lot of everyday uses of Queue. Many simple producers could reduce their coordination code to a with closing(queue) idiom. A further iter() method would enable many simple consumers to safely cut all their coordination boilerplate down to just for item in queue.

I've got a sample implementation ready for Queue and its children, and I'm about to submit it as a PR.

I'd be happy to contribute an equivalent (within limits of its API promises) implementation for SimpleQueue, but I don't have it written, and I'm unsure if I should make a separate issue for that.

[rhettinger]

Note we already have the task_done() / join() API to handle the common case of notification for when work is done. At one point, Guido opined that he didn't want the queue module to go any further into the world of task management (because it is beyond the scope of what the module is trying to do).

Also, because this is a significant expansion of an otherwise simple API and because it would establish new idioms, it would need to be discussed on python-ideas before going forward. Also, to evaluate the proposal, it would be help to have examples of existing code that would be improved with the new API. Likewise, it would be helpful to survey other languages to see if they have anything like this and see whether their experiences were possible or negative.

If the python-ideas discussion is fruitful, feel free to re-open this feature request. The existence of the PR will help make the discussion more concrete and give people a chance to try it out.

[hemflit]

Hi Raymond!

I've posted to python-ideas: https://mail.python.org/pipermail/python-ideas/2018-October/054238.html

The amount of attention it got was modest, so I couldn't exactly say the community has thoroughly vetted and enthusiastically endorsed this proposal :)

But all responses were in the narrow range of neutral to MILDLY positive.
There weren't any objections at all to either the idea or the code. Nor did anybody question whether a feature of this kind should be added at all.

I don't think I could realistically expect much more support than that for a change this minor. (Well, it would have been better if at least one person'd had an (uncontested) strongly positive response.)

Some problems with other kinds of implementations were brought up, but this one isn't vulnerable to them. I believe I've also addressed all your comments from this BPO issue in the opening post there.

An older discussion was linked, which had also shown tacit consensus that this type of feature in general would be welcome.

I suppose it's time to decide if that level of response is enough.

[rhettinger]

I've put some more thought into this and have discussed it with another core developer. While the idea was inspired, I think we should decline the API expansion.

When I last talked with Guido about this module, he opined that he didn't want to module to get further into the business of moderating between the producer and consumer, that it should focus on its core task of being a one-way thread-safe message queue.

I agree that users do craft various ways of communicating via sentinel objects; however, there is no one pattern that dominates so much that it warrants deforming the API to accommodate. FWIW, I use email as the basis for my mental model of how Queue is used. With email, we don't don't need a special part of the API to communicate that we're not going to send messages any more; instead, we either stop sending messages or send a farewell message.

Of the two ideas, making queues iterable is more plausible. However, it doesn't fit will common ways of using a queue. For non-blocking queues it makes zero sense. For daemon threads that block until sent a message, the usual pattern is while True loop that blocks on a get(). In such a case, a for-loop might be more compact but doesn't reflect how we think about the problem (implying that the loop may terminate and it obscures that the get() is what is actually blocking).

Sorry for taking a while to evaluate this. I looked at the effort you put in to the patch and wanted to really think it through. This idea is inspired, but in the end I don't think it is a good fit for how people typically use the Queue module (or its cousin in the Multiprocessing module).

Do consider posting a queue variant to the Python Package Index. Though not for all users, there may well be some uptake for a few users.

[hemflit]

Hi Raymond!

Thanks for the attention you've given this and for the words of encouragement. I'll try to make something like this for PyPI.

I do feel I should critique your metaphor of Queues as email.
(Please don't take this as a request to re-evaluate anything about this proposal or re-hash the "farewell messages are an inferior solution" from python-ideas. It's really only about that "basis for mental model" in general, which I think is doing you a disservice.)

Email is principally bidirectional; with Python's Queues, only one side has the freedom to send any kind of coordination instructions via the same transport that's used for workload messages.

Email is principally one-to-one. The *-to-many Queue case is specifically the one where propagation of farewell messages to all recipients is error-prone, with recipients suddenly needing to worry about the existence of other recipients.

(And if you feel that's not valid because you view multi-recipient as part of the core idea of email, consider that such email is basically "broadcast", with every message reaching every recipient - even further removed from the idea of Queues than single-recipient email is.)

The email metaphor tends to imply FIFO consumption, while Queues support other orderings. These other orderings further complicate the code for dealing with farewell messages.

With email, participants generally have handles on each other, and treat the transport as very abstract. With Queues it's exactly the opposite: they have a handle on the transport mechanism and abstract away the other participants; the transport explicitly takes over some burden of coordination and dispatch; and thanks to that it's very easy for participants to enter or leave an ongoing "conversation" without having to introduce themselves to the others.

(I'm realizing now that email is a pretty good metaphor for Erlang's IPC model though.)