New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

The "retry_backoff" task argument is not working in 4.1.0 version #4341

Closed
JoeZ99 opened this Issue Oct 25, 2017 · 14 comments

Comments

Projects
None yet
@JoeZ99

JoeZ99 commented Oct 25, 2017

Checklist

  • [ x] I have included the output of celery -A proj report in the issue.
    (if you are not able to do this, then at least specify the Celery
    version affected).
software -> celery:4.1.0 (latentcall) kombu:4.1.0 py:2.7.12
            billiard:3.5.0.2 py-amqp:2.1.4
platform -> system:Linux arch:64bit, ELF imp:CPython
loader   -> celery.loaders.app.AppLoader
settings -> transport:amqp results:cache+memcached://memcached:11211/
  • I have verified that the issue exists against the master branch of Celery.
    4.1

Steps to reproduce

create a simple task
from requests.exceptions import RequestException

@app.task(autoretry_for=(RequestException,), retry_backoff=True)
def x():
    if x.request.retries < 3:
        raise RequestException
    return "F#@! SUCCESS!!"

Expected behavior

exponential delayed retries. 1s, 2s, 4s

celery_1           | [2017-10-25 17:02:31,150: INFO/MainProcess] Received task: helpck.custom_results.tasks.x[58036219-9bdd-4017-a36c-3c65fc1fa3b2]  ETA:[2017-10-25 17:05:31.069687+00:00] 
celery_1           | [2017-10-25 17:02:31,152: INFO/PoolWorker-4] Task helpck.custom_results.tasks.x[58036219-9bdd-4017-a36c-3c65fc1fa3b2] retry: Retry in 1s: RequestException()
celery_1           | [2017-10-25 17:02:32,372: INFO/PoolWorker-5] Task helpck.custom_results.tasks.x[58036219-9bdd-4017-a36c-3c65fc1fa3b2] retry: Retry in 2s: RequestException()
celery_1           | [2017-10-25 17:02:34,374: INFO/MainProcess] Received task: helpck.custom_results.tasks.x[58036219-9bdd-4017-a36c-3c65fc1fa3b2]  ETA:[2017-10-25 17:08:31.346456+00:00] 
celery_1           | [2017-10-25 17:08:31,656: INFO/PoolWorker-6] Task helpck.custom_results.tasks.x[58036219-9bdd-4017-a36c-3c65fc1fa3b2] succeeded in 0.00370980799198s: 'F SUCCESS'

Actual behavior

retries every 180 seconds

celery_1           | [2017-10-25 17:02:31,150: INFO/MainProcess] Received task: helpck.custom_results.tasks.x[58036219-9bdd-4017-a36c-3c65fc1fa3b2]  ETA:[2017-10-25 17:05:31.069687+00:00] 
celery_1           | [2017-10-25 17:02:31,152: INFO/PoolWorker-4] Task helpck.custom_results.tasks.x[58036219-9bdd-4017-a36c-3c65fc1fa3b2] retry: Retry in 180s: RequestException()
celery_1           | [2017-10-25 17:05:31,372: INFO/PoolWorker-5] Task helpck.custom_results.tasks.x[58036219-9bdd-4017-a36c-3c65fc1fa3b2] retry: Retry in 180s: RequestException()
celery_1           | [2017-10-25 17:05:31,374: INFO/MainProcess] Received task: helpck.custom_results.tasks.x[58036219-9bdd-4017-a36c-3c65fc1fa3b2]  ETA:[2017-10-25 17:08:31.346456+00:00] 
celery_1           | [2017-10-25 17:08:31,656: INFO/PoolWorker-6] Task helpck.custom_results.tasks.x[58036219-9bdd-4017-a36c-3c65fc1fa3b2] succeeded in 0.00370980799198s: 'FG SUCCESS'
@JoeZ99

This comment has been minimized.

JoeZ99 commented Oct 25, 2017

if you check

https://github.com/celery/celery/blob/v4.1.0/celery/app/base.py#L438

you will see there's no code related to the backoff alorithm. there is no get_exponential_backoff

but in

http://docs.celeryproject.org/en/latest/_modules/celery/app/base.html

(which claims it's for 4.1)

there is a get_exponential_backoff function defined which takes into account all those arguments.

looks like the backoff_retry feature is from versions newer than 4.1.0???

@JoeZ99 JoeZ99 changed the title from The "retry_backoff" task argument is not working to The "retry_backoff" task argument is not working in 4.1.0 version Oct 25, 2017

@sanjioh

This comment has been minimized.

sanjioh commented Oct 26, 2017

Confirmed here with Celery 4.1

@Fingel

This comment has been minimized.

Fingel commented Nov 16, 2017

I'm also confused by this. The documentation here says it's "new in 4.1":
http://docs.celeryproject.org/en/latest/userguide/tasks.html#automatic-retry-for-known-exceptions

But I can't seem to get it to work. I thought I must have been making some mistake, then I found this issue.

@ChrisCalculus

This comment has been minimized.

ChrisCalculus commented Nov 22, 2017

Running into this issue as well (running 4.1).

@mmcardle

This comment has been minimized.

mmcardle commented Nov 24, 2017

Also seeing this issue (4.1)

@shihanwan

This comment has been minimized.

shihanwan commented Dec 18, 2017

Confirming this issue on 4.1.0

@auvipy

This comment has been minimized.

Member

auvipy commented Dec 18, 2017

does it exist in master too?

@georgepsarakis

This comment has been minimized.

Member

georgepsarakis commented Dec 18, 2017

@auvipy the PR with this feature was merged after 4.1.0 was released.

@thedrow

This comment has been minimized.

Member

thedrow commented Dec 19, 2017

Should we close then?

@auvipy auvipy added this to the v4.2 milestone Dec 19, 2017

@auvipy auvipy closed this Dec 19, 2017

knightjoel added a commit to knightjoel/zpark that referenced this issue Jan 9, 2018

Adjust retry/backoff timers due to a Celery issue
Despite what the Celery 4.1 docs say, it appears that the exponential
backoff feature isn't actually present in the 4.1 release. Just use a
regular fixed delay and step for now.

See issue celery/celery#4341
@divad

This comment has been minimized.

divad commented Jan 22, 2018

Can you update the docs or release a new version? This is confusing and gives new users the idea they can't trust Celery docs...

@thedrow

This comment has been minimized.

Member

thedrow commented Jan 23, 2018

Please follow #4387.
We just need the release notes and resolve another issue.

@WoLfulus

This comment has been minimized.

WoLfulus commented Feb 8, 2018

I thought I was doing something wrong :(

@Ashlett

This comment has been minimized.

Ashlett commented Mar 21, 2018

This is not cool... people are wasting their time trying to figure out why this feature is not working, only to find out it is not actually implemented in the current version. Can you please remove it from the docs until it's released?

@celery celery locked as off topic and limited conversation to collaborators Mar 22, 2018

@georgepsarakis

This comment has been minimized.

Member

georgepsarakis commented Mar 22, 2018

@Ashlett sorry for facing issues with this, the next version (4.2.0) will be released soon. See #4387 for details.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.