Add timeout parameter to Queue.join()

[kdlucas]

BPO	9634
Nosy	@rhettinger, @ncoghlan, @pitrou, @anacrolix

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/rhettinger'
closed_at = <Date 2019-08-24.18:33:06.482>
created_at = <Date 2010-08-18.17:46:20.704>
labels = ['type-feature', 'library']
title = 'Add timeout parameter to Queue.join()'
updated_at = <Date 2019-08-24.18:33:06.476>
user = 'https://bugs.python.org/kdlucas'

bugs.python.org fields:

activity = <Date 2019-08-24.18:33:06.476>
actor = 'rhettinger'
assignee = 'rhettinger'
closed = True
closed_date = <Date 2019-08-24.18:33:06.482>
closer = 'rhettinger'
components = ['Library (Lib)']
creation = <Date 2010-08-18.17:46:20.704>
creator = 'kdlucas'
dependencies = []
files = []
hgrepos = []
issue_num = 9634
keywords = []
message_count = 17.0
messages = ['114257', '114263', '114264', '114271', '114276', '114283', '114284', '114286', '115346', '157454', '157455', '157458', '157461', '157510', '157518', '233464', '350392']
nosy_count = 8.0
nosy_names = ['rhettinger', 'ncoghlan', 'pitrou', 'anacrolix', 'kdlucas', 'isoschiz', 'Yuri.Bochkarev', 'Swen.Wenzel']
pr_nums = []
priority = 'low'
resolution = 'out of date'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue9634'
versions = ['Python 3.3']

[kdlucas]

I've seen quite a few people requesting to add a timeout value to the Queue.join() method, as it seems like a nice feature to have when waiting for queue's to finish.

Please add a feature so that Queue.join() will issue a self.all_tasks_done.release() when the timeout value is reached.

[rhettinger]

Can you provide links to the other requests?

This seems to be at odds with the whole premise
of what Queue.join() is trying to do (wait until
all worker threads have marked their tasks as done).

[rhettinger]

FWIW, if you want to do other tasks in a main thread and need to check on the status task completion in worker threads, you can already loop on:

 while q.unfinished_tasks:
      do_other_fun_stuff()
 q.join()                  # this shouldn't block anymore

[kdlucas]

Here are a few that I saw:

http://stackoverflow.com/questions/1564501/add-timeout-argument-to-pythons-queue-join

http://www.eggheadcafe.com/software/aspnet/36145181/max-time-threads.aspx

http://efreedom.com/Question/1-1564501/Add-timeout-argument-to-python-s-Queue-join

[pitrou]

We already have an optional timeout on threading.Thread.join(), which I suppose means that people need/want that feature.

[rhettinger]

One of the provided links is duplicate (repost from StackOverflow). None of the links offer use cases which can be used to inform the design (i.e. is this the correct approach, should a timeout raise a RuntimeError like Condition.wait() does, should a timeout set a variable to indicate the timeout thread.join() does, should it just return with no indication of the reason for returning, should we just document the unfinished_tasks variable as read-only so that users can check it periodically, what are the common calling patterns, how does this fit in with the original design intent of letting worker threads finish all pending tasks before moving on to code that assumes that the work is done).

Without motivating use cases, it is hard to create a correct design or to know whether this should be done at all. Unlike, thread.join() which has a long history in many contexts, the Queue.join/task_done API is newer; accordingly, the API should be grown somewhat cautiously. I don't think we've established using queue.join() with a time-out is a best practice or preferred pattern to solving a given problem. Instead, all we have is "it seems like a nice feature" which falls short of our usual standards.

[kdlucas]

This is easy enough to implement by subclassing the Queue class and overriding join(), so it's not a big deal. Just seems like it would be a nice thing to have.

[rhettinger]

I will leave this open for a while.
It may be that it is a good idea
and that some good use cases will
emerge to inform a correct design.

In the meantime, it would be great
if you could post your subclass to
the ASPN Cookbook to collect feedback.

[rhettinger]

Kelly, thanks for posting the idea.
I'm going to close this one for now
but it can be reopened if compelling
use cases arise.

[ncoghlan]

I just created bpo-14487 to request a documented API (Queue.pending()) that provides a formal mechanism for asking whether or not any jobs are still pending (analagous to the existing empty() and full() query methods).

Specifically, what I have is a client process that is executed periodically, gathers up a set of tasks and uses a thread pool to submit them in parallel to a synchronous API.

If all tasks complete with no new tasks being scheduled, then the client should terminate. However, if a new task arrives while any existing task is still in progress, then the client should submit it "immediately" (where, due to the time scales involved, "immediately" actually means "within the next few minutes" so I have plenty of scope to let the client block for a while instead of implementing a busy loop).

So, a timeout on join() would actually fit my use case better than the pending() API I proposed in the other issue. The processing loop would then look something like:

while have_tasks_to_process():
   submit_tasks_to_queue()
   try:
       task_queue.join(timeout)
   except Pending:
       pass

The advantage of having the timeout is that it would avoid the clumsy workarounds needed to avoid the busy loop created by the use of a query based approach. (Of course, I'm going to have to use the workaround anyway, since my client runs on Python 2.6, but still, I believe it meets the "concrete use case" criterion).

From a design aesthetic point of view, a pending() query API and a timeout on join() that throws a Pending exception would match the existing empty()/get()/Empty and full()/put()/Full API triples.