Add way to store data returned from task

[tomdottom]

For example when a job created from a task such as:

@app.task(pass_context=True)
async def hello_world(ctx):
    return "Hello World"

finishes successfully, then the "Hello World" should be able to be looked up using the job id.

[ewjoachim]

As of today, the task result is logged, but not saved because this would mean the cleaning of finished tasks, which up until now was just a matter of maintenance, would become a matter of functionality too (you mustn't delete a finished task until its result was read, otherwise it's lost)

Your tasks are expected to have a side effect. But in your project, it's perfectly ok to define your own decorator that will store tasks results (and subsequently take care of the deletion) on your terms.

It might be interesting to expose a simpler way to define a "middleware" though.

[tomdottom]

Thanks for the explanation @ewjoachim

I'll look at righting some decorators for storing returned values into a custom table.

[ewjoachim]

We might do something more straightforward, but until then: https://procrastinate.readthedocs.io/en/stable/howto/middleware.html

[ewjoachim]

Cf #370 : maybe we could revisit this decision some day. It may depend on whether there are volunteers who want to take a shot at implementing this.

[tomdottom]

@ewjoachim I quite like the idea of a decorator/middleware and associated helper to retrieve results. So maybe a simple API like:

@app.task
@store_result
async def hello_world():
    return "Hello World!"

...
   
with app.open():
    result = await fetch_result(await hello_world.defer_async())

I'm guessing a task should fail if storage fails, but am not 100% sure about this.

[ewjoachim]

I quite like the idea of a decorator/middleware and associated helper to retrieve results

Do you mean you'd like Procrastinate to implement such a decorator, or do you like the idea of being able to do this on your own in your app outside of the scope of procrastinate itself ?

I'm guessing a task should fail if storage fails, but am not 100% sure about this.

I guess this is application-dependent :) Because storage specifics can be very custom. I think that's partly why I'm divided regarding having Procrastinate do this by itself.

[canadaduane]

Just wanted to add that I misunderstood what Procrastinate can/can't do and after implementing Procrastinate in my app, I'm probably rolling back to (pre-existing) celery implementation with trunk due to its ability to wait until the task completes and then yield the result back to the caller.

In our case, we don't want jobs just "logging" that they finish, but, rather providing an ID back that tells us where to go to get the result of that particular job.

[ewjoachim]

Ok, sorry to see you go. If we end up steering toward this direction, I'll make sure to drop you a line :)

[canadaduane]

Thanks!