Better error info for failure when task has no request stack.

[stevecj]

Checklist

• I have checked the issues list
  for similar or identical enhancement to an existing feature.
• I have checked the pull requests list
  for existing proposed enhancements.
• I have checked the commit log
  to find out if the if the same enhancement was already implemented in the
  master branch.
• I have included all related issues and possible duplicate issues in this issue
  (If there are none, check this box anyway).

Brief Summary

Handle the exception that occurs in build_tracer() in celery/app/trace.py when a task has None as the value of its request_stack attribute, and raise an exception with a message that says what task had the failure.

The reason I am asking for this is that I just had to fight with a problem during an upgrade to celery 5.2.1 in a Django app. I had thought that all of our tasks were defined using @shared_task, but it turns out one of them was defined as a subclass of Task and then registered. The way that was done does not work right in celery 5.2.1, and celery crashed as a result. The error output was useless, and I had to edit the installed celery code to add instrumentation to help me track it down.

The code that I patched in celery/concurrency/prefork.py to tell me what task was the problem:

    # rebuild execution handler for all tasks.
    from celery.app.trace import build_tracer
    for name, task in app.tasks.items():
        #########
        logger.warn(f'name: {name}')
        #########
        task.__trace__ = build_tracer(name, task, app.loader, hostname,
                                      app=app)

Once I knew which task it was, I was able to solve the problem. It would be nice not to have to hand-edit the celery code to get that information though.

[open-collective-bot]

Hey @stevecj 👋,
Thank you for opening an issue. We will get back to you as soon as we can.
Also, check out our Open Collective and consider backing us - every little helps!

We also offer priority support for our sponsors.
If you require immediate assistance please consider sponsoring us.

[jpatters]

@stevecj how did you solve the problem? I am experiencing this as well. All of my tasks are shared tasks, however the ones that are causing this are being autodiscovered from another package.

[thedrow]

Since Celery is crashing, we should treat this as a bug.

Could you please provide the relevant lines that caused the issue?

[jpatters]

On further investigation, this is not crashing celery and I am not sure that I am experiencing the exact same issue. Here's a stack trace in case anyone thinks otherwise:

IndexError: list index out of range
  File "celery/app/trace.py", line 468, in trace_task
    I, R, state, retval = on_error(task_request, exc, uuid)
  File "celery/app/trace.py", line 380, in on_error
    task, request, eager=eager, call_errbacks=call_errbacks,
  File "celery/app/trace.py", line 183, in handle_error_state
    call_errbacks=call_errbacks)
  File "celery/app/trace.py", line 237, in handle_failure
    self._log_error(task, req, einfo)
  File "celery/app/trace.py", line 267, in _log_error
    extra={'data': context})
  File "__init__.py", line 1444, in log
    self._log(level, msg, args, **kwargs)
  File "__init__.py", line 1514, in _log
    self.handle(record)
  File "__init__.py", line 1524, in handle
    self.callHandlers(record)
  File "__init__.py", line 1586, in callHandlers
    hdlr.handle(record)
  File "__init__.py", line 894, in handle
    self.emit(record)
  File "django/utils/log.py", line 120, in emit
    message = "%s\n\n%s" % (self.format(no_exc_record), reporter.get_traceback_text())
  File "django/views/debug.py", line 337, in get_traceback_text
    c = Context(self.get_traceback_data(), autoescape=False, use_l10n=False)
  File "django/views/debug.py", line 303, in get_traceback_data
    'settings': self.filter.get_safe_settings(),
  File "django/views/debug.py", line 116, in get_safe_settings
    settings_dict[k] = self.cleanse_setting(k, getattr(settings, k))
  File "django/conf/__init__.py", line 142, in PASSWORD_RESET_TIMEOUT_DAYS
    filename, _, _, _ = stack[-2]

Something is causing traceback.extract_stack() to return an empty stack and the problem seems to originate in the same place within celery.

[kbehlers]

Here's a minimal reproducible example:

from celery import Celery, Task

app = Celery()

class CustomTask(Task):
    name = "custom_task"
    def run(self):
        return 'hello'

app.tasks.register(CustomTask())  # broken, AttributeError: 'NoneType' object has no attribute 'push'
# app.register_task(CustomTask())  # fixed, no error

The bug/regression appears for the first time in v5.2.0b1 and is not present in 5.1.2 for me. Again it only applies to class-based tasks, and it has everything to do with what method is used to register the task.