fix(tracing): Fix celery custom headers propagation

[beezz]

This fixes the broken propagation of Celery custom headers that are used by our distributed tracing.

The bug seems to be in Celery worker Request class which is not propagating the request headers properly (self._request_dict["headers"]).

When the task is later executed using Request.execute_using_pool, message.headers is not passed (nor message) which makes me think it's a bug since this way message.headers are completely ignored.

I'm opening this PR here to illustrate the issue so we can decide how to properly fix it upstream and in sentry-sdk.

[untitaker]

so I guess what I am confused about is why the sdk testsuite passes? There is one test that tests this against the Redis backend with a real Worker that forks.

Is this reproducible with redis and rabbitmq?

[beezz]

I don't think this is specific to a backend but I will check the sdk tests.

[beezz]

Tests are passing because they always use the Celery proto2 and here it makes headers=message.headers. See https://github.com/celery/celery/blob/master/celery/worker/strategy.py#L132-L135

If I instruct the test with Redis to use celery.conf.task_protocol=1 it fails.

Sentry is using task_protocol=1 ~ https://github.com/getsentry/sentry/blob/master/src/sentry/conf/server.py#L502