"Maximum recursion depth exceeded" causes datastructures.py to fail and almost impossible to debug the error

[honzajavorek]

I had a task which caused error "maximum recursion depth exceeded". It contained unexpected and unwanted infinite loop of function calls, which were "invisible" in code by eyes. They took place only when task was run directly in Celery, so it was necessary to debug it inside Celery.

However, this exception caused Celery's exception handling to fail in the same way -- in datastructures.py while creating ExceptionInfo it again caused it's own "maximum recursion depth exceeded". This was very frustrating, as I couldn't determine in any way what originally caused the problem (and in fact it was a pretty trivial thing with calling babel translating filter in email template). So I spent 3 hours by commenting out try/except blocks in Celery code and determining what really happened.

I am not skilled in handling tracebacks, but as I was finally able to perform traceback.print_tb(tb) in my injected code and it successfully printed traceback of the original exception, I assume this is problem which is possible to fix and therefore it is a bug in Celery.

[ask]

You need to add an example able to reproduce the problem, or at least a traceback, as without that I'm not able to fix or even verify your issue.

[honzajavorek]

It's trivial, just raise "maximum recursion depth exceeded". E.g. use Celerybeat with task like this:

@celery.task(ignore_result=True, name='recursion')
def recursion_task():
    sum = lambda n: n <= 1 or n + sum(n - 1)
    sum(1000)

...with settings:

CELERYBEAT_SCHEDULE = {
    'recursion-test': {'task': 'recursion', 'schedule': crontab(minute='*')}
}

I used Celerybeat, because it's my exact use case from original project, however I doubt standard Celery would behave differently. With standard Python recursion depth, you get the same thing as I describe above.