Huey+Django: log executed tasks in django admin

[jedie]

I would like to have a log of async actions in the django admin. The goal would be: A list of the executed tasks with the following information:

• args+kwargs
• start time
• duration
• return value (if available)
• traceback if available: "task failed" else: "task successes"

I think that's the most interesting things, right?

Is there interest to integrate it directly into huey? Then I work on it in a fork and send a pull request.
If not, I will make a separate project out of it.

I ask myself how to do this most effectively. One idea would be to create new decorators similar to db_periodic_task, db_task. So the user has the choice to activate db-logging or not, only by changing the task decorator.

My suggestion is here: https://gist.github.com/jedie/e5bab0bc197d4aa51cceaccfd7316649

Remarks / Comments?

[jedie]

Next question: How to store the return value of the task? Just accept only None and String? Or use repr() or pprint.pformat() or json or pickle ?

In celery is the "result_serializer" settings: http://docs.celeryproject.org/en/latest/userguide/configuration.html#result-serializer and used json as default and has built-in support for JSON, pickle, YAML and msgpack: http://docs.celeryproject.org/en/latest/userguide/calling.html#calling-serializers

This flexibility in celery is nice, but that's why it's so complex ;) That's why I think there should be no setting for it.

The target in the database is a text field. It would be good to have human readable entries in it.

So I would say spontaneously to json: But then you have the problem that json can't serialize everything. Pickle could, but it's not human readable.

So pprint.pformat() but that would be a little unusual, wouldn't it?
So: just allow a string. (and the task may serialized it, if needed?)

[coleifer]

Is there interest to integrate it directly into huey?

Not right now, no.

[coleifer]

Huey serializes task args/kwargs with pickle before serializing them and writing them to the queue. Similarly, return values are pickled and stored in the result-store, then unpickled when read. So the APIs you interact with deal with python objects, but the actual storage layer is dealing with plain old bytes. For a higher-level view/task reporting I'd anticipate you'd be interested in dealing with the unpickled Python results.

You can hook into huey in a number of ways...by decorating the task functions -- which will add whatever work you're doing (db connections, inserting rows into a db, whatever) to every task executed by the worker process. Keep in mind if you're writing to a db your worker processes will all need to manage their db connections / transactional state / etc.

There's also the event streaming API which can be used to monitor events from the consumer in real-time, but that's not what you're after I don't think.

Any kind of advanced reporting or persistence beyond the normal logging done by the consumer, and the event streaming, is not something I want to add to huey.

[jedie]

OK, i start as separate project ;)

[vrbrito]

OK, i start as separate project ;)

Any updates on that? I would be interested in seeing what you've got.

[ppawlak]

OK, i start as separate project ;)

Any updates on that? I would be interested in seeing what you've got.

Same here.

@jedie Were you able to made a project for logging tasks in Django ?

[coleifer]

I think you all might consider using the log-files and other unix tools built around line-oriented text files.

[jwaschkau]

OK, i start as separate project ;)

Any updates on that? I would be interested in seeing what you've got.

Same here.

@jedie Were you able to made a project for logging tasks in Django ?

See https://github.com/boxine/django-huey-monitor