Is it possible to get results from a list as they complete?

[pfmoore]

If I have a list of result values, for example results = [fetch_url_task(u) for u in urls], is there a way to ask for the results as they become available? Basically, something like concurrent.futures.as_completed().

I can simply poll the list repeatedly, but that seems inefficient. I might be able to build something by adding signal handlers for SIGNAL_COMPLETE and SIGNAL_ERROR, but it's not immediately obvious to me how I'd do so, as signal handlers run asynchronously. Maybe I could put the results in a queue as they arrive, and fetch them in my calling code?

But even if it's possible, the only solutions I can think of seem pretty complex. Is there a simpler approach I'm missing?

[coleifer]

Here's a simple solution that (more or less) polls the list: 4737bdf

To avoid having to do this, you would probably want to put your results into their own queue/data-structure (rather than the result-store) and read from that in your application code.

For illustrative purposes only, you might store task results in a redis list:

@huey.task()
def produce_value(n):
    time.sleep(n)
    redis.lpush('producevalue.results', n)

# Then in your application:
produce_value.map(random.shuffle(range(10)))
i = 0
while i < 10:
    yield redis.blpop('producevalue.results')

[pfmoore]

Awesome - thanks for the quick and helpful response.

I presume that if I use an application queue, it would need to be something persistent, as the values would be written in the worker process but consumed in the client application? I'm using sqlite (because it comes with Python, and doesn't need a server) so I guess I could write something similar for that. Would I be right in assuming that creating custom tables in the huey database itself would be a very bad idea? I don't know enough about redis to be sure, but I assume the producevalue.results queue in your example isn't in the huey database?

[coleifer]

I presume that if I use an application queue, it would need to be something persistent, as the values would be written in the worker process but consumed in the client application?

Correct.

I'm using sqlite (because it comes with Python, and doesn't need a server) so I guess I could write something similar for that.

I don't think that would be any different than what we're doing in the patch that adds as_completed. Sqlite does not have a way to notify listeners when results are ready, so you'd have to poll your sqlite database in any case.

Would I be right in assuming that creating custom tables in the huey database itself would be a very bad idea?

The only issue with that would be running into the 1 writer issue. You probably better leave the huey sqlite db alone if for no other reason than to avoid locking it up with your application code.

but I assume the producevalue.results queue in your example isn't in the huey database?

That would just be any ordinary redis list in any redis database, including possibly the one used by huey. It doesn't make a difference for Redis.