Use logging system for failure logging

[defect]

When using log from within tasks we have a pretty decent logging system set up that allows for pluggable loggers (syslog + sending logs to collins for instance), but this is not being used by the task runner. This means that task failures only shows up on stdout since we just puts them.

This changes the framework to use the logging system instead for failure logging to ensure that that the reason for the failures, as well as retries, gets logged the same way log messages from the tasks does.

@tumblr/systems

[roymarantz]

👍
do you want to also log task is being skipped messages or would that be too noisy? (I think it is too noisy)
Does the task name get included (correctly) in the log message? (not that this PR should fix that)

[defect]

Yeah, i agree on that being too noisy. Perhaps if we had the notion of a severity level or something, but at that point it's starting to get pretty complex.

The log method prefixes the message with the task name. It looks something like this when failing in the run block:

FailTask :: task is attempting run #1...
FailTask :: Run #1 caused error: <error message here>
FailTask :: task is sleeping for 0 seconds...
FailTask :: task is attempting run #2...

and similarly for the precondition, init and condition blocks. I removed the task name from their error messages to avoid duplication.

[byxorna]

I like it! Great job @defect! I would say dont worry about squashing noisy logs.... Leave it up to the log ingester to hide/drop noise like debug, etc. That already is the case with collins (default views on overview page only shows warning and higher severity).

[william-richard]

LGTM

[defect]

@byxorna The problem is that the logging system in genesis does't have different "levels" of logs. When using the collins logger for instance everything is logged as INFORMATIONAL.

It might make more sense to send everything through the logging system if/when we do have different levels, but right now I mostly want to get actual failures logged to something else than stdout. I'll open a ticket to investigate implementing a more proper logging system with different levels/severities though.