trigger an operator when a task fails

[smarden1]

What's the best way to trigger an operator when a task fails? I'd like to be able to specify different tasks or even different branches when a task fails. For example, it would be nice to write to a random db using a hook when one of my tasks fails -- is this currently possible or would the operator api have to be modified?

Ideally, I'd like to be able to trigger any operator on retry or failure instead of just sending a single email.

[mistercrunch]

Unfortunately there's no easy way to do this now. The absolute easiest would be to add a on_failure_callback and on on_retry_callback to BaseOperator. These would be expected to be python callables (not operator objects) to which I could pass the context (much like the operators execute method receive).

Would that work for you?

[mistercrunch]

Here's a first draft, I'm going to add a few unittests and this will make it into 1.3
#151

[smarden1]

I left some comments. Thanks for doing this so fast! I really appreciate it

[mistercrunch]

It's been on my mind for a while and just thought of a very easy implementation that covers pretty much all use cases.

[r39132]

I could also use such an operator! +1

[mistercrunch]

The implementation ended up being a set of callbacks (it's not an operator, and does not become an actual node in the DAG) that run atomically with the task. One on failure, one on retries, and I threw one in on success. You could use that to for any purpose really, from very specific use cases to very high level and across the board like gathering specific operational metadata or integrating with other systems.

[smarden1]

👍

Thanks for the quick implementation!