[celery] add retry reason metadata to spans

[Kyle-Verhoog]

Background

The method of retrying a task in celery raises retry exceptions. These appear to be getting caught and mishandled as errors in the tracer.

Overview

This PR aims to gracefully handle celery retry exceptions.

Current Status

Some investigation has been done and there is a signal API provided to trace retries. However I have been unable to replicate the retry exception being marked as an error in the tracer.

[palazzem]

Related to #603

[brettlangdon]

The reason these aren't failing is because you are explicitly raising a Retry exception.

This is something that Celery will catch and handle on it's own.

The other case we need to be able to support is when you set a retry policy for the task and another type of exception is raised.

@self.app.task(autotry_for(Exception, ))
def fn_exception():
    raise Exception('oh no, what happened?!')

[brettlangdon]

I think the cases we need to handle are:

# Success \0/
@app.task
def fn_success():
    pass


# This should be reported as an error
@app.task
def fn_exception():
    raise Exception()


# This should be reported as success (the desired action of retrying occurred)
# We should add metadata to the span to show that it will be retried
# We need to make sure we handle when the max number of
#   retries have occurred (this should then be an error)
@app.task
def fn_retry():
    raise Retry()


# Same as above
@app.task(bind=True)
def fn_retry(self):
    raise self.retry()


# This feels like it should fall under the same as the previous,
#   mark as success unless max retires have occurred
@app.task(autoretry_for=(Exception, ))
def fn_retry_exception():
    raise Exception()

[brettlangdon]

@Kyle-Verhoog stealing this PR from you :)

Changed the base to 0.15-dev.

This PR now is focused on adding celery task retry reason as metadata to the span.

[brettlangdon]

We don't want this added as an exception, retrying isn't necessarily an error.

We could potentially add other metadata here, like the type of exception that was raised, and traceback info, but reason alone should be good to get started?

[Kyle-Verhoog]

I can't officially review the PR because I created it, but this looks good to me 👍

[labbati]

LGTM 👍