Celery Multi creates pid and log files in the wrong directory #6017

maxmalysh · 2020-03-31T14:01:20Z

Here is what docs say:

CELERYD_PID_FILE — Full path to the PID file. Default is /var/run/celery/%n.pid
CELERYD_LOG_FILE — Full path to the worker log file. Default is /var/log/celery/%n%I.log Note: Using %I is important when using the prefork pool as having multiple processes share the same log file will lead to race conditions.

In reality, celery multi creates logs and pid files in the working directory instead of /var/run and /var/log:

--logfile=default%I.log --pidfile=default.pid

My systemd service:

# /etc/systemd/system/celery.service

[Unit]
Description=Celery Service
After=network.target

[Service]
Type=forking
User=myuser
Restart=always
EnvironmentFile=-/etc/conf.d/celery
WorkingDirectory=/home/myuser/myproject

ExecStart=/bin/bash -c 'source /etc/environment; \
  /home/myuser/venv/bin/celery multi start ${CELERYD_NODES} \
  -A ${CELERY_APP} ${CELERYD_OPTS}'

ExecStop=/bin/bash -c 'source /etc/environment; \
  /home/myuser/venv/bin/celery multi stopwait ${CELERYD_NODES}'

ExecReload=/bin/bash -c 'source /etc/environment; \
  /home/myuser/venv/bin/celery multi restart ${CELERYD_NODES} \
  -A ${CELERY_APP} --loglevel="${CELERYD_LOG_LEVEL}" ${CELERYD_OPTS}'

[Install]
WantedBy=multi-user.target

My /etc/conf.d/celery:

# /etc/conf.d/celery
CELERYD_NODES="default slow"
CELERY_APP="pool.celery:app"
CELERYD_MULTI="multi"
CELERYD_OPTS="-Q:1 celery -Q:2 slow --time-limit=15 --concurrency=1 -O fair -E"

Systemd status report:

root@pool1:~# sudo systemctl status celery
● celery.service - Celery Service
   Loaded: loaded (/etc/systemd/system/celery.service; enabled; vendor preset: enabled)
   Active: active (running) since Tue 2020-03-31 13:38:02 UTC; 1min 23s ago
  Process: 2648 ExecStop=/bin/bash -c source /etc/environment;    /home/myuser/venv/bin/celery multi stopwait ${CELERYD_NODES} (code=exited, status=0/SUCCESS)
  Process: 2661 ExecStart=/bin/bash -c source /etc/environment;    /home/myuser/venv/bin/celery multi start ${CELERYD_NODES} -A ${CELERY_APP} ${CELERYD_OPTS} (code=exited, status=0/SUCCESS)
 Main PID: 27107 (code=exited, status=1/FAILURE)
    Tasks: 4 (limit: 4915)
   CGroup: /system.slice/celery.service
           ├─2680 /home/myuser/venv/bin/python3.8 -m celery worker -A pool.celery:app --time-limit=15 --concurrency=1 -O fair -E -Q celery --logfile=default%I.log --pidfile=default.pid --hostname=default@pool1.mycorp.
           ├─2688 /home/myuser/venv/bin/python3.8 -m celery worker -A pool.celery:app --time-limit=15 --concurrency=1 -O fair -E -Q slow --logfile=slow%I.log --pidfile=slow.pid --hostname=slow@pool1.mycorp.org
           ├─2694 /home/myuser/venv/bin/python3.8 -m celery worker -A pool.celery:app --time-limit=15 --concurrency=1 -O fair -E -Q celery --logfile=default%I.log --pidfile=default.pid --hostname=default@pool1.mycorp.
           └─2695 /home/myuser/venv/bin/python3.8 -m celery worker -A pool.celery:app --time-limit=15 --concurrency=1 -O fair -E -Q slow --logfile=slow%I.log --pidfile=slow.pid --hostname=slow@pool1.mycorp.org

There are three options:
a. There's a bug in Celery 4.4.2
b. There's a mistake in docs
c. I'm missing something and everything works as expected

I'm using Celery 4.4.2 on Ubuntu 18.04.

The text was updated successfully, but these errors were encountered:

Changed the default paths for logs and pid files to be '/var/log/celery' and '/var/run/celery' Handled by creating the respective paths if not exist. Used os.makedir(path,if_exists=True)

Changed the default paths for logs and pid files to be '/var/log/celery' and '/var/run/celery' Handled by creating the respective paths if not exist. Used os.makedir(path,if_exists=True) Updated the .travis.yml file.

--> Added Multi default logfiles and pidfiles paths [Description]: --> Changed the default paths for log files & pid files to be '/var/log/celery' and '/var/run/celery' --> Handled by creating the respective paths if not exist. --> Used os.makedir(path,if_exists=True) [Unit Test Added]: --> .travis.yml - config updated with 'before install'. --> t/unit/apps/test_multi.py - Changed the default log files & pid files paths wherever required.

mchataigner · 2020-05-17T21:12:53Z

I'm sorry but this is not cross-platform, I'm currently working at fixing Windows build.
If Windows is still supported by celery, this needs some improvements as the commits contains platform specific paths…

it relates to celery#6017

auvipy · 2020-05-18T01:33:31Z

thanks for noticing it!

* do not load memcache nor couchbase lib during windows build those libraries depends on native libraries libcouchbase and libmemcached that are not installed on Appveyor. As only unit tests runs on Appveyor, it should be fine * Add python 3.8 workaround for app trap * skip tests file_descriptor_safety tests on windows AsyncPool is not supported on Windows so Pool does have _fileno_to_outq attribute, making the test fail * Fix crossplatform log and pid files in multi mode it relates to #6017 * Use tox to build and test on windows * remove tox_install_command * drop python 2.7 from windows build

auvipy · 2020-06-01T13:32:43Z

Here is what docs say:

* `CELERYD_PID_FILE` — Full path to the PID file. Default is `/var/run/celery/%n.pid`

* `CELERYD_LOG_FILE` — Full path to the worker log file. Default is `/var/log/celery/%n%I.log` Note: Using %I is important when using the prefork pool as having multiple processes share the same log file will lead to race conditions.

In reality, celery multi creates logs and pid files in the working directory instead of /var/run and /var/log:

--logfile=default%I.log --pidfile=default.pid

My systemd service:

# /etc/systemd/system/celery.service

[Unit]
Description=Celery Service
After=network.target

[Service]
Type=forking
User=myuser
Restart=always
EnvironmentFile=-/etc/conf.d/celery
WorkingDirectory=/home/myuser/myproject

ExecStart=/bin/bash -c 'source /etc/environment; \
  /home/myuser/venv/bin/celery multi start ${CELERYD_NODES} \
  -A ${CELERY_APP} ${CELERYD_OPTS}'

ExecStop=/bin/bash -c 'source /etc/environment; \
  /home/myuser/venv/bin/celery multi stopwait ${CELERYD_NODES}'

ExecReload=/bin/bash -c 'source /etc/environment; \
  /home/myuser/venv/bin/celery multi restart ${CELERYD_NODES} \
  -A ${CELERY_APP} --loglevel="${CELERYD_LOG_LEVEL}" ${CELERYD_OPTS}'

[Install]
WantedBy=multi-user.target

My /etc/conf.d/celery:

# /etc/conf.d/celery
CELERYD_NODES="default slow"
CELERY_APP="pool.celery:app"
CELERYD_MULTI="multi"
CELERYD_OPTS="-Q:1 celery -Q:2 slow --time-limit=15 --concurrency=1 -O fair -E"

Systemd status report:

root@pool1:~# sudo systemctl status celery
● celery.service - Celery Service
   Loaded: loaded (/etc/systemd/system/celery.service; enabled; vendor preset: enabled)
   Active: active (running) since Tue 2020-03-31 13:38:02 UTC; 1min 23s ago
  Process: 2648 ExecStop=/bin/bash -c source /etc/environment;    /home/myuser/venv/bin/celery multi stopwait ${CELERYD_NODES} (code=exited, status=0/SUCCESS)
  Process: 2661 ExecStart=/bin/bash -c source /etc/environment;    /home/myuser/venv/bin/celery multi start ${CELERYD_NODES} -A ${CELERY_APP} ${CELERYD_OPTS} (code=exited, status=0/SUCCESS)
 Main PID: 27107 (code=exited, status=1/FAILURE)
    Tasks: 4 (limit: 4915)
   CGroup: /system.slice/celery.service
           ├─2680 /home/myuser/venv/bin/python3.8 -m celery worker -A pool.celery:app --time-limit=15 --concurrency=1 -O fair -E -Q celery --logfile=default%I.log --pidfile=default.pid --hostname=default@pool1.mycorp.
           ├─2688 /home/myuser/venv/bin/python3.8 -m celery worker -A pool.celery:app --time-limit=15 --concurrency=1 -O fair -E -Q slow --logfile=slow%I.log --pidfile=slow.pid --hostname=slow@pool1.mycorp.org
           ├─2694 /home/myuser/venv/bin/python3.8 -m celery worker -A pool.celery:app --time-limit=15 --concurrency=1 -O fair -E -Q celery --logfile=default%I.log --pidfile=default.pid --hostname=default@pool1.mycorp.
           └─2695 /home/myuser/venv/bin/python3.8 -m celery worker -A pool.celery:app --time-limit=15 --concurrency=1 -O fair -E -Q slow --logfile=slow%I.log --pidfile=slow.pid --hostname=slow@pool1.mycorp.org

There are three options:
a. There's a bug in Celery 4.4.2
b. There's a mistake in docs
c. I'm missing something and everything works as expected

I'm using Celery 4.4.2 on Ubuntu 18.04.

can you check this #6136?

@task

* Added integration coverage for link_error (#5373) * Added coverage for link_error. * Use pytest-rerunfailed plugin instead of rolling our own custom implementation. * Added link_error with retries. This currently fails. * Remove unused import. * Fix import on Python 2.7. * retries in link_error do not hang the worker anymore. * Run error callbacks eagerly when the task itself is run eagerly. Fixes #4899. * Adjust unit tests accordingly. * Grammar in documentation (#5780) * Grammar in documentation * Address review. * pypy 7.2 matrix (#5790) * removed extra slashes in CELERY_BROKER_URL (#5792) The Celery broker URL in settings.py had 2 slashes in the end which are not required and can be misleading. so I changed :- CELERY_BROKER_URL = 'amqp://guest:guest@localhost//' to CELERY_BROKER_URL = 'amqp://guest:guest@localhost' * Fix #5772 task_default_exchange & task_default_exchange_type not work (#5773) * Fix #5772 task_default_exchange & task_default_exchange_type not work * Add unit test: test_setting_default_exchange * Move default_exchange test to standalone class * Run integration suite with memcached results backend. (#5739) * Fix hanging forever when fetching results from a group(chain(group)) canvas. (#5744) PR #5739 uncovered multiple problems with the cache backend. This PR should resolve one of them. PR #5638 fixed the same test case for our async results backends that support native join. However, it did not fix the test case for sync results backends that support native join. * Fix regression in PR #5681. (#5753) See comment in the diff for details. * Grammatical fix to CONTRIBUTING.rst doc (#5794) * Fix #5734 Celery does not consider authMechanism on mongodb backend URLs (#5795) * Fix #5734 Celery does not consider authMechanism on mongodb backend URLs * Add unit test: test_get_connection_with_authmechanism * Add unit test: test_get_connection_with_authmechanism_no_username * Fix errors in Python 2.7 Remove "," after "**" operator * Revert "Revert "Revert "Added handle of SIGTERM in BaseTask in celery/task.py to prevent kill the task" (#5577)" (#5586)" (#5797) This reverts commit f79894e. * Add Python 3.8 Support (#5785) * Added Python 3.8 to the build matrix. * Ensure a supported tblib version is installed for Python 3.8 and above. In addition, modernize the relevant tests. * Workaround patching problem in test. * py 3.8 in clasifier * ubuntu bionic (#5799) * ubuntu bionic * fast finish * sync bumversion with pypi release * Dev.req (#5803) * update docker config * undo hardpin * devr req install from github master * update docker config (#5801) * update docker config * make dockerfile to install from github master dev branch by default * update download link * Isort. * Grammatical & punctuation fixes for CONTRIBUTING.rst document (#5804) * update dockerfile * switched to ubuntu bionic * update docker * keep it empty until we reconfigure it again with autopep8 * Fixed Dockerfile (#5809) * Update document CONTRIBUTING.rst & fix Dockerfile typo (#5813) * Added an issue template for minor releases. * reference gocelery Go Client/Server for Celery (#5815) * Add enterprise language (#5818) * Fix/correct minor doc typos (#5825) * Correct a small typo * Correct bad contributing documentation links * Preserve the task priority in case of a retry (#5820) * Preserve the task priority in case of a retry * Created test case for retried tasks with priority * Implement an integration test for retried tasks with priorities * bump kombu * basic changelog for celery 4.4.0rc4 * bump celery 4.4.0rc4 * events bootstep disabled if no events (#5807) * events bootstep disabled if no events * Added unit tests. * update bug report template * fixing ascii art to look nicer (#5831) * Only rerun flaky tests when failures can be intermediary. * Rename Changelog to Changelog.rst * The test_nested_group_chain test can run without native_join support. (#5838) * Run integration tests with Cassandra (#5834) * Run integration tests with Cassandra. * Configure cassandra result backend * Pre-create keyspace and table * Fix deprecation warning. * Fix path to cqlsh. * Increase connection timeout. * Wait until the cluster is available. * SQS - Reject on failure (#5843) * reject on failure * add documentation * test fix * test fix * test fix * Add a concurrency model with ThreadPoolExecutor (#5099) * Add a concurrency model with ThreadPoolExecutor * thread model test for pypy * Chain primitive's code example fix in canvas documentation (Regression PR#4444) (#5845) * Changed multi-line string (#5846) This string wasn't rendering properly and was printing the python statement too. Although the change isn't as pretty code-wise, it gets rid of an annoyance for the user. * Add auto expiry for DynamoDB backend (#5805) * Add auto expiry for DynamoDB backend This adds auto-expire support for the DynamoDB backend, via the DynamoDB Time to Live feature. * Require boto3>=1.9.178 for DynamoDB TTL support boto3 version 1.9.178 requires botocore>=1.12.178. botocore version 1.12.178 introduces support for the DynamoDB UpdateTimeToLive call. The UpdateTimeToLive call is used by the DynamoDB backend to enable TTL support on a newly created table. * Separate TTL handling from table creation Handle TTL enabling/disabling separately from the table get-or-create function. Improve handling of cases where the TTL is already set to the desired state. DynamoDB only allows a single TTL update action within a fairly long time window, so some problematic cases (changing the TTL attribute, enabling/disabling TTL when it was recently modified) will raise exceptions that have to be dealt with. * Handle older boto3 versions If the boto3 TTL methods are not found, log an informative error. If the user wants to enable TTL, raise an exception; if TTL should be disabled, simply return. * Improve logging - Handle exceptions by logging the error and re-raising - Log (level debug) when the desired TTL state is already in place * Add and use _has_ttl() convenience method Additional changes: - Handle exceptions when calling boto3's describe_time_to_live() - Fix test cases for missing TTL methods * Update ttl_seconds documentation * Log invalid TTL; catch and raise ValueError * Separate method _get_table_ttl_description * Separate ttl method validation function * Clarify tri-state TTL value * Improve test coverage * Fix minor typo in comment * Mark test as xfail when using the cache backend. (#5851) * [Fix #5436] Store extending result in all backends (#5661) * [Fix #5436] Store extending result in all backends * Fix sqlalchemy * More fixu * Fixing tests * removing not necessary import * Removing debug code * Removing debug code * Add tests for get_result_meta in base and database * Revert "Add auto expiry for DynamoDB backend (#5805)" (#5855) This reverts commit f7f5bcf. * Revert "Mark test as xfail when using the cache backend. (#5851)" (#5854) This reverts commit 1b303c2. * docs: Document Redis commands used by celery (#5853) * remove cache back end integrtion test. (#5856) * Fix a race condition when publishing a very large chord header (#5850) * Added a test case which artificially introduces a delay to group.save(). * Fix race condition by delaying the task only after saving the group. * update tox * Remove duplicate boto dependency. (#5858) * Revert "remove cache back end integrtion test. (#5856)" (#5859) This reverts commit e0ac7a1. * Revert "Revert "Add auto expiry for DynamoDB backend (#5805)" (#5855)" (#5857) This reverts commit 4ddc605. * Revert "update tox" This reverts commit 49427f5. * Fix the test_simple_chord_with_a_delay_in_group_save test. * Revert "Revert "Skip unsupported canvas when using the cache backend"" (#5860) * Revert "Revert "Mark test as xfail when using the cache backend. (#5851)" (#5854)" This reverts commit fc101c6. * Make the xfail condition stricter. * Fix the xfail condition. * Linters should use Python 3.8. * Move pypy unit tests to the correct stage. * Temporarily allow PyPy to fail since it is unavailable in Travis. * Remove unused variables. * Fix unused imports. * Fix pydocstyle errors in dynamodb. * Fix pydocstyle errors in redis backend. * bump kombu to 4.6.7 * celery 4.4.0rc5 changelog * celery 4.4.0rc5 * rm redundant code (#5864) * isort. * Document the threads task pool in the CLI. * Removed the paragraph about using librabbitmq. Refer to #5872 (#5873) * Task class definitions can have retry attributes (#5869) * autoretry_for * retry_kwargs * retry_backoff * retry_backoff_max * retry_jitter can now be defined as cls attributes. All of these can be overriden from the @task decorator #4684 * whatsnew in Celery 4.4 as per projects standard (#5817) * 4.4 whatsnew * update * update * Move old whatsnew to history. * Remove old news & fix markers. * Added a section notifying Python 3.4 has been dropped. * Added a note about ElasticSearch basic auth. * Added a note about being able to replace eagerly run tasks. * Update index. * Address comment. * Described boto3 version updates. * Fix heading. * More news. * Thread pool. * Add Django and Config changes * Bump version 4.4.0 * upate readme * Update docs regarding Redis Message Priorities (#5874) * Update docs regarding Redis Message Priorities * fixup! Update docs regarding Redis Message Priorities * Update 4.4.0 docs (#5875) * Update 4.4 release changelog * Update whatsnew-4.4 * Update tasks docs * Fix recent tasks doc file update (#5879) * Include renamed Changelog.rst in source releases. (#5880) Changelog.rst was renamed from Changelog in fd023ec but MANIFEST.in was not updated to include the new name. This fixes the file name so Changelog.rst will show up in future source releases again. * Reorganised project_urls and classifiers. (#5884) * Use safequote in SQS Getting Started doc (#5885) * Have appveyor build relevant versions of Python. (#5887) * Have appveyor build relevant and buildable versions of Python. * Appveyor is missing CI requirements to build. * Pin pycurl to version that will build with appveyor (because wheels files exist) * Restrict python 2.7 64 bit version of python-dateutil for parse. * Use is_alive instead of isAlive for Python 3.9 compatibility. (#5898) * Very minor tweak to commen to improve docs (#5900) As discussed here: https://stackoverflow.com/questions/58816271/celery-task-asyncresult-takes-task-id-but-is-documented-to-get-asyncresult-inst this comment seems to flow to a very confusing and misleading piece of documentation here: https://docs.celeryproject.org/en/latest/reference/celery.app.task.html#celery.app.task.Task.AsyncResult * Support configuring schema of a PostgreSQL database (#5910) * Support configuring schema of a PostgreSQL database * Add unit test * Remove blank line * Fix raise issue to make exception message more friendly (#5912) Signed-off-by: Chenyang Yan <memory.yancy@gmail.com> * Add progress for retry connections (#5915) This will show current retry progress so it will clear confusion about how many retries will be tried for connecting to broker. Closes #4556 * chg: change xrange to range (#5926) * update docs for json serializer and add note for int keys serialization (#5932) * fix indentation for note block in calling.rst (#5933) * Added links to other issue trackers. (#5939) * Add labels automatically for issues. (#5938) * remove redundant raise from docstring (#5941) `throw` is True by default so the Retry exception will already get raised by calling `self.retry(countdown=60 * 5, exc=exc)` * Run pyupgrade. * Fix typo (#5943) * Remove fallback code for Python 2 support. * docs: fixes Rabbits and Warrens link in routing userguide (#4007) (#5949) * Fix labels on Github issue templates. (#5955) Use quotation marks to escape labels on Github issue templates. This prevents the colon from breaking the template. * added retry_on_timeout and socket_keepalive to config and doc (#5952) * Fixed event capture from building infinite list (#5870) * Fix error propagation example (#5966) * update range (#5971) * update setup.cfg * bump billiard to 3.6.3.0 * Update __init__.py (#5951) * Update __init__.py Fixed issue for object with result_backend=True (decode fails on multiple None request) * Update __init__.py suggested changeds * Update __init__.py * Use configured db schema also for sequences (#5972) * Added a default value for retries in worker.strategy. (#5945) * Added a default value for retries in worker.strategy. I was facing an issue when adding tasks directly to rabbitmq using pika instead of calling task.apply_async. The issue was the self.retry mechanisum was failing. In app/tasks.py the line `retries = request.retries + 1` was causing the issue. On further tracing I figured out that it was because the default .get value (None) was getting passed through this function and was raising TypeError: unsupported operand type(s) for +: 'NoneType' and 'int' * Add test cases for default and custom retries value * pypy 7.3 (#5980) * Pass `interval` to `get_many` (#5931) * Pass `interval` to `get_many` * Fix: Syntax error for py2.7 * Fix: Syntax error for py2.7 * Fixed problem with conflicting autoretry_for task parameter and Task.replace() (#5934) * Fix #5917 (#5918) * Fix changelog (#5881) * merge in place the apps beat schedule in the default Schedule class. (#5908) * Handle Redis connection errors in result consumer (#5921) * Handle Redis connection errors in result consumer * Closes #5919. * Use context manager for Redis conusmer reconnect * Log error when result backend reconnection fails * Fix inspect_command documentation (#5983) * Use gevent and eventlet wait() functions to remove busy-wait (#5974) * Use gevent and eventlet wait() functions to remove busy-wait Fixes issue #4999. Calling AsyncResult.get() in a gevent context would cause the async Drainer to repeatedly call wait_for until the result was completed. I've updated the code to have a specific implementation for gevent and eventlet that will cause wait_for to only return every "timeout" # of seconds, rather than repeatedly returning. Some things I'd like some feedback on: * Where's the best place to add test coverage for this? It doesn't look like there are any tests that directly exercised the Drainer yet so I would probably look to add some of these to the backends/ unit tests. * The way I did this for the Eventlet interface was to rely on the private _exit_event member of the GreenThread instance; to do this without relying on a private member would require some additional changes to the backend Drainer interface so that we could wait for an eventlet-specific event in wait_for(). I can do this, just wanted to get some feedback before. * Add unit tests for Drainer classes In order for this to work without monkeypatching in the tests, I needed to call sleep(0) to let the gevent/eventlet greenlets to yield control back to the calling thread. I also made the check interval configurable in the drainer so that we didn't need to sleep multiples of 1 second in the tests. * Weaken asserts since they don't pass on CI * Fix eventlet auto-patching DNS resolver module on import By default it looks like "import eventlet" imports the greendns module unless the environment EVENTLET_NO_GREENDNS is set to true. This broke a pymongo test. * Add tests ensuring that the greenlet loop isn't blocked These tests make sure that while drain_events_until is running that other gevent/eventlet concurrency can run. * Clean up tests and make sure they wait for all the threads to stop * Fix chords with chained groups (#5947) * kombu 4.6.8 * update setup * updated version 4.4.1 * Fix: Accept and swallow `kwargs` to handle unexpected keyword arguments * Allow boto to look for credentials in S3Backend * add reference to Rusty Celery * Update document of revoke method in Control class * Fix copy-paste error in result_compression docs * Make 'socket_keepalive' optional variable (#6000) * update connection params - socket_keepalive is optional now * update readme - added versionadded 4.4.1 and fixed `redis_socket_keepalive` * added check of socket_keepalive in arguments for UnixSocketConnect * Fixed incorrect setting name in documentation (#6002) * updated version 4.4.2 * Fix backend utf-8 encoding in s3 backend Celery backend uses utf-8 to deserialize results, which would fail for some serializations like pickle. * Fix typo in celery.bin.multi document * Upgraded pycurl to the latest version that supports wheel. * pytest 5.3.5 max * Add uptime to the stats inspect command * Doc tweaks: mostly grammar and punctuation (#6016) * Fix a bunch of comma splices in the docs * Remove some unnecessary words from next-steps doc * Tweak awkward wording; fix bad em-dash * Fix a bunch more comma splices in next-steps doc * Miscellaneous grammar/punctuation/wording fixes * Link to task options in task decorator docs * Fixing issue #6019: unable to use mysql SSL parameters when getting mysql engine (#6020) * Fixing issue #6019: unable to use mysql SSL parametes in create_engine() * adding test for get_engine when self.forked is False and engine args are passed in for create_engine() * Clean TraceBack to reduce memory leaks for exception task (#6024) * Clean TraceBack to reduce memory leaks * add unit test * add unit test * reject unittest * Patch For Python 2.7 compatibility * update unittest * Register to the garbage collector by explicitly referring to f_locals. * need more check * update code coverage * update Missing unit test * 3.4 -> 3.5 Co-authored-by: heedong.jung <heedong.jung@samsung.com> * exceptions: NotRegistered: fix up language Minor fix to the language. * Note about autodiscover_tasks and periodic tasks This is particularly important for Django projects that put periodic tasks into each app's `tasks.py` and want to use one as a periodic task. By the time `autodiscover_tasks()` loads those tasks, the `on_after_configure` Signal has already come and gone, so anything decorated with `@app.on_after_finalize.connect` will never be called. If there's other documentation on this subject, I could not find it. * Avoid PyModulelevel, deprecated in Sphinx 4 Use `PyFunction` instead of `PyModulelevel` to avoid this deprecation warning: RemovedInSphinx40Warning: PyModulelevel is deprecated. Please check the implementation of <class 'celery.contrib.sphinx.TaskDirective'> This replacement is one of the options listed in the Sphinx docs (https://www.sphinx-doc.org/en/master/extdev/deprecated.html). * Give up sending a worker-offline message if transport is not connected (#6039) * If worker-offline event fails to send, give up and die peacefully * Add test for retry= and msgs in heartbeat * Fix the build and all documentation warnings. I finally upgraded our theme to 2.0. As a result we've upgraded Sphinx to 2.0. Work to upgrade Sphinx to 3.0 will proceed in a different PR. This upgrade also fixes our build issues caused by #6032. We don't support Sphinx 1.x as a result of that patch. I've also included the missing 4.3 changelog to our history. * Support both Sphinx 2 and 3. * Add Task to __all__ in celery.__init__.py * Add missing parenthesis to example in docs * Ensure a single chain object in a chain does not raise MaximumRecursionError. Previously chain([chain(sig)]) would crash. We now ensure it doesn't. Fixes #5973. * update setup.py * fix typo missing quote at the end of line * Fix a typo in monitoring doc * update travis * update ubuntu to focal foss 20.04 LTS * Fix autoscale when prefetch_multiplier is 1 * Allow start_worker to function without ping task * Update celeryd.conf Move the directory of the program before the execution of the command/script * Add documentation for "predefined_queue_urls" * [Fix #6074]: Add missing documentation for MongoDB as result backend. * update funding * 🐛 Correctly handle configuring the serializer for always_eager mode. (#6079) * 🐛 Correctly handle configuring the serializer for always_eager mode. options['serializer'] will always exist, because it is initialized from an mattrgetter. Even if unset, it will be present in the options with a value of None. * 🐛 Add a test for new always_eager + task_serializer behavior. * ✏️ Whoops missed a : * Remove doubling of prefetch_count increase when prefetch_multiplier gt 1 (#6081) * try ubuntu focal (#6088) * Fix eager function not returning result after retries. Using apply function does not return correct results after at least one retry because the return value of successive call is not going back to the original caller. * return retry result if not throw and is_eager if throw is false, we would be interested by the result of retry and not the current result which will be an exception. This way it does not break the logic of `raise self.retry` This should be used like `return self.retry(..., throw=False)` in an except statement. * revert formatting change * Add tests for eager retry without throw * update predefined-queues documentation Suggested version of configuration does not work. Additionally I'd like to mention, that `access_key_id` and `secret_access_key` are mandatory fields and not allowing you to go with defaults AWS_* env variables. I can contribute for this variables to be optional Also I'm not sure if security token will apply, could you please advice how to do it? * Fix couchbase version < 3.0.0 as API changed * Remove reference to -O fair in optimizations -O fair was made the default in Celery 4.0 https://docs.celeryproject.org/en/stable/history/whatsnew-4.0.html#ofair-is-now-the-default-scheduling-strategy * pytest ranges * pypy3 * revert to bionic * do not load docs.txt requirements for python 2.7 As it requires Sphinx >= 2.0.0 and there is no such version compatible with python 2.7 * update cassandra travis integration test configuration cassandra:latest docker image changed location of cqlsh program * pin cassandra-driver CI get stuck after all cassandra integration tests * Fix all flake8 lint errors * Fix all pydocstyle lint errors * Fix all configcheck lint errors * Always requeue while worker lost regardless of the redelivered flag (#6103) * #5598 fix, always redelivery while WorkerLostError * fix, change the requeue flag so the task will remain PENDING * Allow relative paths in the filesystem backend (#6070) * Allow relative paths in the filesystem backend * fix order of if statements * [Fixed Issue #6017] --> Added Multi default logfiles and pidfiles paths [Description]: --> Changed the default paths for log files & pid files to be '/var/log/celery' and '/var/run/celery' --> Handled by creating the respective paths if not exist. --> Used os.makedir(path,if_exists=True) [Unit Test Added]: --> .travis.yml - config updated with 'before install'. --> t/unit/apps/test_multi.py - Changed the default log files & pid files paths wherever required. * Avoid race condition due to task duplication. In some circumstances like a network partitioning, some tasks might be duplicated. Sometimes, this lead to a race condition where a lost task overwrites the result of the last successful task in the backend. In order to avoid this race condition we prevent updating the result if it's already in successful state. This fix has been done for KV backends only and therefore won't work with other backends. * adding tests * Exceptions must be old-style classes or derived from BaseException, but here self.result may not subclass of BaseException. * update fund link * Fix windows build (#6104) * do not load memcache nor couchbase lib during windows build those libraries depends on native libraries libcouchbase and libmemcached that are not installed on Appveyor. As only unit tests runs on Appveyor, it should be fine * Add python 3.8 workaround for app trap * skip tests file_descriptor_safety tests on windows AsyncPool is not supported on Windows so Pool does have _fileno_to_outq attribute, making the test fail * Fix crossplatform log and pid files in multi mode it relates to #6017 * Use tox to build and test on windows * remove tox_install_command * drop python 2.7 from windows build * Add encode to meta task in base.py (#5894) * Add encode to base.py meta result Fix bug with impossibility to load None from task meta * Add tests for None. Remove exceed encode. * Update base.py Add return payload if None * Update time.py to solve the microsecond issues (#5199) When `relative` is set to True, the day, hour, minutes second will be round to the nearest one, however, the original program do not update the microsecond (reset it). As a result, the run-time offset on the microsecond will then be accumulated. For example, given the interval is 15s and relative is set to True 1. 2018-11-27T15:01:30.123236+08:00 2. 2018-11-27T15:01:45.372687+08:00 3. 2018-11-27T15:02:00.712601+08:00 4. 2018-11-27T15:02:15.987720+08:00 5. 2018-11-27T15:02:31.023670+08:00 * Change backend _ensure_not_eager error to warning * Add priority support for 'celery.chord_unlock' task (#5766) * Change eager retry behaviour even with raise self.retry, it should return the eventual value or MaxRetriesExceededError. if return value of eager apply is Retry exception, retry eagerly the task signature * Order supported Python versions * Avoid race condition in elasticsearch backend if a task is retried, the task retry may work concurrently to current task. store_result may come out of order. it may cause a non ready state (Retry) to override a ready state (Success, Failure). If this happens, it will block indefinitely pending any chord depending on this task. this change makes document updates safe for concurrent writes. https://www.elastic.co/guide/en/elasticsearch/reference/current/optimistic-concurrency-control.html * backends base get_many pass READY_STATES arg * test backends base get_many pass READY_STATES arg * Add integration tests for Elasticsearch and fix _update * Revert "revert to bionic" This reverts commit 6e09157. * remove jython check * feat(backend): Adds cleanup to ArangoDB backend * Delete Document Known Issue with CONN_MAX_AGE in 4.3 * issue 6108 fix filesystem backend cannot not be serialized by picked (#6120) * issue 6108 fix filesystem backend cannot not be serialized by picked #6108 * issue-6108 fix unit test failure * issue-6108 fix flake8 warning Co-authored-by: Murphy Meng <mmeng@mirriad.com> * kombu==4.6.9 (#6133) * changelog for 4.4.3 * v 4.4.3 * remove un supported classifier * Fix autoretry_for with explicit retry (#6138) * Add tests for eager task retry * Fixes #6135 If autoretry_for is set too broad on Exception, then autoretry may get a Retry if that's the case, rethrow directly instead of wrapping it in another Retry to avoid loosing new args * Use Django DB max age connection setting (fixes #4116) * Add retry on recoverable exception for the backend (#6122) * Add state to KeyValueStoreBackend.set method This way, a backend implementation is able to take decisions based on current state to store meta in case of failures. * Add retry on recoverable exception for the backend acks.late makes celery acknowledge messages only after processing and storing result on the backend. However, in case of backend unreachable, it will shadow a Retry exception and put the task as failed in the backend not retrying the task and acknoledging it on the broker. With this new result_backend_always_retry setting, if the backend exception is recoverable (to be defined per backend implementation), it will retry the backend operation with an exponential backoff. * Make elasticsearch backward compatible with 6.x * Make ES retry storing updates in a better way if existing value in the backend is success, then do nothing. if it is a ready status, then update it only if new value is a ready status as well. else update it. This way, a SUCCESS cannot be overriden so that we do not loose results but any ready state other than success (FAILURE, REVOKED) can be overriden by another ready status (i.e. a SUCCESS) * Add test for value not found in ES backend * Fix random distribution of jitter for exponential backoff random.randrange should be called with the actual so that all numbers have equivalent probability, otherwise maximum value does have a way higher probability of occuring. * fix unit test if extra modules are not present * ElasticSearch: add setting to save meta as json * fix #6136. celery 4.4.3 always trying create /var/run/celery directory (#6142) * fix #6136. celery 4.4.3 always trying create /var/run/celery directory, even if it's not needed. * fix #6136. cleanup * Add task_internal_error signal (#6049) * Add internal_error signal There is no special signal for an out of body error which can be the result of a bad result backend. * Fix syntax error. * Document the task_internal_error signal. Co-authored-by: Laurentiu Dragan <ldragan@bloomberg.net> * changelog for v4.4.4 * kombu 4.6.10 (#6144) * v4.4.4 * Add missing dependency on future (#6146) Fixes #6145 * ElasticSearch: Retry index if document was deleted between index and update (#6140) * ElasticSearch: Retry index if document was deleted between index and update * Elasticsearch increase coverage to 100% * Fix pydocstyle * Specify minimum version of Sphinx for Celery extension (#6150) The Sphinx extension requires Sphinx 2 or later due to #6032. * fix windows build * fix flake8 error * fix multi tests in local Mock os.mkdir and os.makedirs to avoid creating /var/run/celery and /var/log/celery during unit tests if run without root priviledges * Customize the retry interval of chord_unlock tasks * changelog v4.4.5 * v4.4.5 * Fix typo in comment. * Remove autoscale force_scale methods (#6085) * Remove autoscale force_scale methods * Remove unused variable in test * Pass ping destination to request The destination argument worked fine from CLI but didn't get used when calling ping from Python. * Fix autoscale test * chord: merge init options with run options * put back KeyValueStoreBackend.set method without state It turns out it was breaking some other projects. wrapping set method with _set_with_state, this way it will not break existing Backend. while enabling this feature for other Backend. Currently, only ElasticsearchBackend supports this feature. It protects concurrent update to corrupt state in the backend. Existing success cannot be overriden, nor a ready state by a non ready state. i.e. a Retry state cannot override a Success or Failure. As a result, chord_unlock task will not loop forever due to missing ready state on the backend. * added --range-prefix option to `celery multi` (#6180) * added --range-prefix option to `celery multi` Added option for overriding default range prefix when running multiple workers prividing range with `celery multy` command. * covered multi --range-prefix with tests * fixed --range-prefix test * Added as_list function to AsyncResult class (#6179) * Add as_list method to return task IDs as a list * Add a test for as_list method * Add docstring for as_list method * Fix CassandraBackend error in threads or gevent pool (#6147) * Fix CassandraBackend error in threads or gevent pool * remove CassandraBackend.process_cleanup * Add test case * Add test case * Add comments test_as_uri Co-authored-by: baixue <baixue@wecash.net> * changelog for v4.4.6 * v4.4.6 * Update Wiki link in "resources" In the page linked below, the link to wiki is outdated. Fixed that. https://docs.celeryproject.org/en/stable/getting-started/resources.html * test_canvas: Add test for chord-in-chain Add test case for the issue where a chord in a chain does not work when using .apply(). This works fine with .apply_async(). * Trying to fix flaky tests in ci * fix pydocstyle errors * fix pydocstyle * Drainer tests, put a lower constraint on number of intervals liveness should iterate 10 times per interval while drain_events only once. However, as it may use thread that may be scheduled out of order, we may end up in some situation where liveness and drain_events were called the same amount of time. Lowering the constraint from < to <= to avoid failing the tests. * pyupgrade. * Fix merge error. Co-authored-by: Борис Верховский <boris.verk@gmail.com> Co-authored-by: Asif Saif Uddin <auvipy@gmail.com> Co-authored-by: Jainal Gosaliya <jainal09gosalia@gmail.com> Co-authored-by: gsfish <root@grassfish.net> Co-authored-by: Dipankar Achinta <di.punk.car19@gmail.com> Co-authored-by: spengjie <spengjie@sina.com> Co-authored-by: Chris Griffin <chris-griffin@users.noreply.github.com> Co-authored-by: Muhammad Hewedy <mhewedy@gmail.com> Co-authored-by: Blaine Bublitz <blaine.bublitz@gmail.com> Co-authored-by: Tamu <tamsanh@gmail.com> Co-authored-by: Erik Tews <erik@datenzone.de> Co-authored-by: abhinav nilaratna <anilaratna2@bloomberg.net> Co-authored-by: Wyatt Paul <wpaul@hearsaycorp.com> Co-authored-by: gal cohen <gal.nevis@gmail.com> Co-authored-by: whuji <alfred@huji.fr> Co-authored-by: Param Kapur <paramkapur2002@gmail.com> Co-authored-by: Sven Ulland <sven.ulland@gmail.com> Co-authored-by: Safwan Rahman <safwan.rahman15@gmail.com> Co-authored-by: Aissaoui Anouar <tobia@crossbone.cc> Co-authored-by: Neal Wang <qdzzyb2015@gmail.com> Co-authored-by: Alireza Amouzadeh <alireza@amouzadeh.net> Co-authored-by: Marcos Moyano <marcos@anue.biz> Co-authored-by: Stepan Henek <stepan+github@henek.name> Co-authored-by: Andrew Sklyarov <AndrewPix@users.noreply.github.com> Co-authored-by: Michael Fladischer <michael@fladi.at> Co-authored-by: Dejan Lekic <dejan.lekic@gmail.com> Co-authored-by: Yannick Schuchmann <yannick.schuchmann@googlemail.com> Co-authored-by: Matt Davis <matteius@gmail.com> Co-authored-by: Xtreak <tir.karthi@gmail.com> Co-authored-by: Bernd Wechner <bernd-wechner@users.noreply.github.com> Co-authored-by: Sören Oldag <soeren_oldag@freenet.de> Co-authored-by: uddmorningsun <memory.yancy@gmail.com> Co-authored-by: Amar Fadil <34912365+marfgold1@users.noreply.github.com> Co-authored-by: woodenrobot <woodenrobot1993@gmail.com> Co-authored-by: Sardorbek Imomaliev <sardorbek.imomaliev@gmail.com> Co-authored-by: Alex Riina <alex.riina@gmail.com> Co-authored-by: Joon Hwan 김준환 <xncbf12@gmail.com> Co-authored-by: Prabakaran Kumaresshan <k_prabakaran+github@hotmail.com> Co-authored-by: Martey Dodoo <martey@mobolic.com> Co-authored-by: Konstantin Seleznev <4374093+Seleznev-nvkz@users.noreply.github.com> Co-authored-by: Prodge <Prodge@users.noreply.github.com> Co-authored-by: Abdelhadi Dyouri <raryat@gmail.com> Co-authored-by: Ixiodor <Ixiodor@users.noreply.github.com> Co-authored-by: abhishekakamai <47558404+abhishekakamai@users.noreply.github.com> Co-authored-by: Allan Lei <allanlei@helveticode.com> Co-authored-by: M1ha Shvn <work_shvein_mihail@mail.ru> Co-authored-by: Salih Caglar Ispirli <caglarispirli@gmail.com> Co-authored-by: Micha Moskovic <michamos@gmail.com> Co-authored-by: Chris Burr <chrisburr@users.noreply.github.com> Co-authored-by: Dave King <tildedave@gmail.com> Co-authored-by: Dmitry Nikulin <v.dmitry.nikulin@gmail.com> Co-authored-by: Michael Gaddis <mgaddis@ancestry.com> Co-authored-by: epwalsh <epwalsh10@gmail.com> Co-authored-by: TalRoni <tal.inon.16@gmail.com> Co-authored-by: Leo Singer <leo.singer@ligo.org> Co-authored-by: Stephen Tomkinson <neonbunny@users.noreply.github.com> Co-authored-by: Abhishek <abhishek.shukla@xeneta.com> Co-authored-by: theirix <theirix@gmail.com> Co-authored-by: yukihira1992 <hirayama@cancerscan.jp> Co-authored-by: jpays <jerome.pays@cnp.fr> Co-authored-by: Greg Ward <greg@gerg.ca> Co-authored-by: Alexa Griffith <agriffith@bluecore.com> Co-authored-by: heedong <63043496+heedong-jung@users.noreply.github.com> Co-authored-by: heedong.jung <heedong.jung@samsung.com> Co-authored-by: Shreyansh Khajanchi <shreyanshk@users.noreply.github.com> Co-authored-by: Sam Thompson <georgedorn@users.noreply.github.com> Co-authored-by: Alphadelta14 <alpha@pokesplash.net> Co-authored-by: Azimjon Pulatov <azimjohn@yahoo.com> Co-authored-by: ysde <ysde108@gmail.com> Co-authored-by: AmirMohammad Ziaei <amir_zia@outlook.com> Co-authored-by: Ben Nadler <ben.nadler@gusto.com> Co-authored-by: Harald Nezbeda <hn@nezhar.com> Co-authored-by: Chris Frisina <github@specialorange.org> Co-authored-by: Adam Eijdenberg <adam.eijdenberg@digital.gov.au> Co-authored-by: rafaelreuber <rafaelreuber@gmail.com> Co-authored-by: Noah Kantrowitz <noah@coderanger.net> Co-authored-by: Ben Nadler <jbennadler@gmail.com> Co-authored-by: Clement Michaud <c.michaud@criteo.com> Co-authored-by: Mathieu Chataigner <m.chataigner@criteo.com> Co-authored-by: eugeneyalansky <65346459+eugeneyalansky@users.noreply.github.com> Co-authored-by: Leonard Lu <leonard@socialcodeinc.com> Co-authored-by: XinYang <yangxinhust@hotmail.com> Co-authored-by: Ingolf Becker <ingolf.becker@googlemail.com> Co-authored-by: Anuj Chauhan <an12ch98@gmail.com> Co-authored-by: shaoziwei <ziwei.shao@cloudchef.io> Co-authored-by: Mathieu Chataigner <mathieu.chataigner@gmail.com> Co-authored-by: Anakael <shemeldima2@yandex.ru> Co-authored-by: Danny Chan <danny.chan@c-k.dev> Co-authored-by: Sebastiaan ten Pas <sebastiaan@diggimedia.nl> Co-authored-by: David TILLOY <d.tilloy@criteo.com> Co-authored-by: Anthony N. Simon <anthonynsimon@users.noreply.github.com> Co-authored-by: lironhl <liron.lavy@gmail.com> Co-authored-by: Raphael Cohen <raphael.cohen@sekoia.fr> Co-authored-by: JaeyoungHeo <jay.jaeyoung@gmail.com> Co-authored-by: singlaive <singlaive@gmail.com> Co-authored-by: Murphy Meng <mmeng@mirriad.com> Co-authored-by: Wu Haotian <whtsky@gmail.com> Co-authored-by: Kwist <velnik@gmail.com> Co-authored-by: Laurentiu Dragan <ldragan@bloomberg.net> Co-authored-by: Michal Čihař <michal@cihar.com> Co-authored-by: Radim Sückr <radim.suckr@gmail.com> Co-authored-by: Artem Vasilyev <artem.v.vasilyev@gmail.com> Co-authored-by: kakakikikeke-fork <kakakikikeke_new@yahoo.co.jp> Co-authored-by: Pysaoke <pysaoke@gmail.com> Co-authored-by: baixue <baixue@wecash.net> Co-authored-by: Prashant Sinha <prashantsinha@outlook.com> Co-authored-by: AbdealiJK <abdealikothari@gmail.com>

* 'abstractproperty' is deprecated. Use 'property' with 'abstractmethod' instead * Fix #2849 - Initial work of celery 5.0.0 alpha1 series by dropping python below 3.6 from marix & remove import from __future__ (#5684) * initial work of celery 5.0.0 alpha-1 series by dropping python below 3.6 * i-5651(ut): add ut for ResultSet.join_native (#5679) * dropped python versions below 3.6 from tox * dropped python versions below 3.6 from travis * dropped python versions below 3.6 from appveyor * dropped python2 compat __future__ imports from tests * Fixed a bug where canvases with a group and tasks in the middle followed by a group fails to complete and indefinitely hangs. (#5681) Fixes #5512, fixes #5354, fixes #2573. * dropped python2 compat __future__ imports from celery * dropped python2 compat code from init * revert readme change about version * removed python 2 object inheritance (#5687) * removed python 2 object inheritance * removed python 2 object inheritance * removed python 2 compatibility decorator (#5689) * removed python 2 compatibility decorator * removed python 2 compatibility decorator * removed python 2 compatibility decorator * removed python 2 compatibility decorator * Remove unused imports. * Remove unused imports of python_2_unicode_compatible. Also removed leftover useage of them where they were still used. * Run pyupgrade on codebase (#5726) * Run pyupgrade on codebase. * Use format strings where possible. * pyupgrade examples. * pyupgrade on celerydocs extension. * pyupgrade on updated code. * Address code review comments. * Address code review comments. * Remove unused imports. * Fix indentation. * Address code review comments. * Fix syntax error. * Fix syntax error. * Fix syntax error. * pytest 5.x for celery 5 (#5791) * Port latest changes from master to v5-dev (#5942) * Fix serialization and deserialization of nested exception classes (#5717) * Fix #5597: chain priority (#5759) * adding `worker_process_shutdown` to __all__ (#5762) * Fix typo (#5769) * Reformat code. * Simplify commands to looking for celery worker processes (#5778) * update doc- celery supports storage list. (#5776) * Update introduction.rst * Update introduction.rst * Fail xfailed tests if the failure is unexpected. * Added integration coverage for link_error (#5373) * Added coverage for link_error. * Use pytest-rerunfailed plugin instead of rolling our own custom implementation. * Added link_error with retries. This currently fails. * Remove unused import. * Fix import on Python 2.7. * retries in link_error do not hang the worker anymore. * Run error callbacks eagerly when the task itself is run eagerly. Fixes #4899. * Adjust unit tests accordingly. * Grammar in documentation (#5780) * Grammar in documentation * Address review. * pypy 7.2 matrix (#5790) * removed extra slashes in CELERY_BROKER_URL (#5792) The Celery broker URL in settings.py had 2 slashes in the end which are not required and can be misleading. so I changed :- CELERY_BROKER_URL = 'amqp://guest:guest@localhost//' to CELERY_BROKER_URL = 'amqp://guest:guest@localhost' * Fix #5772 task_default_exchange & task_default_exchange_type not work (#5773) * Fix #5772 task_default_exchange & task_default_exchange_type not work * Add unit test: test_setting_default_exchange * Move default_exchange test to standalone class * Run integration suite with memcached results backend. (#5739) * Fix hanging forever when fetching results from a group(chain(group)) canvas. (#5744) PR #5739 uncovered multiple problems with the cache backend. This PR should resolve one of them. PR #5638 fixed the same test case for our async results backends that support native join. However, it did not fix the test case for sync results backends that support native join. * Fix regression in PR #5681. (#5753) See comment in the diff for details. * Grammatical fix to CONTRIBUTING.rst doc (#5794) * Fix #5734 Celery does not consider authMechanism on mongodb backend URLs (#5795) * Fix #5734 Celery does not consider authMechanism on mongodb backend URLs * Add unit test: test_get_connection_with_authmechanism * Add unit test: test_get_connection_with_authmechanism_no_username * Fix errors in Python 2.7 Remove "," after "**" operator * Revert "Revert "Revert "Added handle of SIGTERM in BaseTask in celery/task.py to prevent kill the task" (#5577)" (#5586)" (#5797) This reverts commit f79894e0a2c7156fd0ca5e8e3b652b6a46a7e8e7. * Add Python 3.8 Support (#5785) * Added Python 3.8 to the build matrix. * Ensure a supported tblib version is installed for Python 3.8 and above. In addition, modernize the relevant tests. * Workaround patching problem in test. * py 3.8 in clasifier * ubuntu bionic (#5799) * ubuntu bionic * fast finish * sync bumversion with pypi release * Dev.req (#5803) * update docker config * undo hardpin * devr req install from github master * update docker config (#5801) * update docker config * make dockerfile to install from github master dev branch by default * update download link * Isort. * Grammatical & punctuation fixes for CONTRIBUTING.rst document (#5804) * update dockerfile * switched to ubuntu bionic * update docker * keep it empty until we reconfigure it again with autopep8 * Fixed Dockerfile (#5809) * Update document CONTRIBUTING.rst & fix Dockerfile typo (#5813) * Added an issue template for minor releases. * reference gocelery Go Client/Server for Celery (#5815) * Add enterprise language (#5818) * Fix/correct minor doc typos (#5825) * Correct a small typo * Correct bad contributing documentation links * Preserve the task priority in case of a retry (#5820) * Preserve the task priority in case of a retry * Created test case for retried tasks with priority * Implement an integration test for retried tasks with priorities * bump kombu * basic changelog for celery 4.4.0rc4 * bump celery 4.4.0rc4 * events bootstep disabled if no events (#5807) * events bootstep disabled if no events * Added unit tests. * update bug report template * fixing ascii art to look nicer (#5831) * Only rerun flaky tests when failures can be intermediary. * Rename Changelog to Changelog.rst * The test_nested_group_chain test can run without native_join support. (#5838) * Run integration tests with Cassandra (#5834) * Run integration tests with Cassandra. * Configure cassandra result backend * Pre-create keyspace and table * Fix deprecation warning. * Fix path to cqlsh. * Increase connection timeout. * Wait until the cluster is available. * SQS - Reject on failure (#5843) * reject on failure * add documentation * test fix * test fix * test fix * Add a concurrency model with ThreadPoolExecutor (#5099) * Add a concurrency model with ThreadPoolExecutor * thread model test for pypy * Chain primitive's code example fix in canvas documentation (Regression PR#4444) (#5845) * Changed multi-line string (#5846) This string wasn't rendering properly and was printing the python statement too. Although the change isn't as pretty code-wise, it gets rid of an annoyance for the user. * Add auto expiry for DynamoDB backend (#5805) * Add auto expiry for DynamoDB backend This adds auto-expire support for the DynamoDB backend, via the DynamoDB Time to Live feature. * Require boto3>=1.9.178 for DynamoDB TTL support boto3 version 1.9.178 requires botocore>=1.12.178. botocore version 1.12.178 introduces support for the DynamoDB UpdateTimeToLive call. The UpdateTimeToLive call is used by the DynamoDB backend to enable TTL support on a newly created table. * Separate TTL handling from table creation Handle TTL enabling/disabling separately from the table get-or-create function. Improve handling of cases where the TTL is already set to the desired state. DynamoDB only allows a single TTL update action within a fairly long time window, so some problematic cases (changing the TTL attribute, enabling/disabling TTL when it was recently modified) will raise exceptions that have to be dealt with. * Handle older boto3 versions If the boto3 TTL methods are not found, log an informative error. If the user wants to enable TTL, raise an exception; if TTL should be disabled, simply return. * Improve logging - Handle exceptions by logging the error and re-raising - Log (level debug) when the desired TTL state is already in place * Add and use _has_ttl() convenience method Additional changes: - Handle exceptions when calling boto3's describe_time_to_live() - Fix test cases for missing TTL methods * Update ttl_seconds documentation * Log invalid TTL; catch and raise ValueError * Separate method _get_table_ttl_description * Separate ttl method validation function * Clarify tri-state TTL value * Improve test coverage * Fix minor typo in comment * Mark test as xfail when using the cache backend. (#5851) * [Fix #5436] Store extending result in all backends (#5661) * [Fix #5436] Store extending result in all backends * Fix sqlalchemy * More fixu * Fixing tests * removing not necessary import * Removing debug code * Removing debug code * Add tests for get_result_meta in base and database * Revert "Add auto expiry for DynamoDB backend (#5805)" (#5855) This reverts commit f7f5bcfceca692d0e78c742a7c09c424f53d915b. * Revert "Mark test as xfail when using the cache backend. (#5851)" (#5854) This reverts commit 1b303c2968836245aaa43c3d0ff9249dd8bf9ed2. * docs: Document Redis commands used by celery (#5853) * remove cache back end integrtion test. (#5856) * Fix a race condition when publishing a very large chord header (#5850) * Added a test case which artificially introduces a delay to group.save(). * Fix race condition by delaying the task only after saving the group. * update tox * Remove duplicate boto dependency. (#5858) * Revert "remove cache back end integrtion test. (#5856)" (#5859) This reverts commit e0ac7a19a745dd5a52a615c1330bd67f2cef4d00. * Revert "Revert "Add auto expiry for DynamoDB backend (#5805)" (#5855)" (#5857) This reverts commit 4ddc605392d7694760f23069c34ede34b3e582c3. * Revert "update tox" This reverts commit 49427f51049073e38439ea9b3413978784a24999. * Fix the test_simple_chord_with_a_delay_in_group_save test. * Revert "Revert "Skip unsupported canvas when using the cache backend"" (#5860) * Revert "Revert "Mark test as xfail when using the cache backend. (#5851)" (#5854)" This reverts commit fc101c61c1912c4dafa661981f8b865c011e8a55. * Make the xfail condition stricter. * Fix the xfail condition. * Linters should use Python 3.8. * Move pypy unit tests to the correct stage. * Temporarily allow PyPy to fail since it is unavailable in Travis. * Remove unused variables. * Fix unused imports. * Fix pydocstyle errors in dynamodb. * Fix pydocstyle errors in redis backend. * bump kombu to 4.6.7 * celery 4.4.0rc5 changelog * celery 4.4.0rc5 * rm redundant code (#5864) * isort. * Document the threads task pool in the CLI. * Removed the paragraph about using librabbitmq. Refer to #5872 (#5873) * Task class definitions can have retry attributes (#5869) * autoretry_for * retry_kwargs * retry_backoff * retry_backoff_max * retry_jitter can now be defined as cls attributes. All of these can be overriden from the @task decorator https://github.com/celery/celery/issues/4684 * whatsnew in Celery 4.4 as per projects standard (#5817) * 4.4 whatsnew * update * update * Move old whatsnew to history. * Remove old news & fix markers. * Added a section notifying Python 3.4 has been dropped. * Added a note about ElasticSearch basic auth. * Added a note about being able to replace eagerly run tasks. * Update index. * Address comment. * Described boto3 version updates. * Fix heading. * More news. * Thread pool. * Add Django and Config changes * Bump version 4.4.0 * upate readme * Update docs regarding Redis Message Priorities (#5874) * Update docs regarding Redis Message Priorities * fixup! Update docs regarding Redis Message Priorities * Update 4.4.0 docs (#5875) * Update 4.4 release changelog * Update whatsnew-4.4 * Update tasks docs * Fix recent tasks doc file update (#5879) * Include renamed Changelog.rst in source releases. (#5880) Changelog.rst was renamed from Changelog in fd023ec174bedc2dc65c63a0dc7c85e425ac00c6 but MANIFEST.in was not updated to include the new name. This fixes the file name so Changelog.rst will show up in future source releases again. * Reorganised project_urls and classifiers. (#5884) * Use safequote in SQS Getting Started doc (#5885) * Have appveyor build relevant versions of Python. (#5887) * Have appveyor build relevant and buildable versions of Python. * Appveyor is missing CI requirements to build. * Pin pycurl to version that will build with appveyor (because wheels files exist) * Restrict python 2.7 64 bit version of python-dateutil for parse. * Use is_alive instead of isAlive for Python 3.9 compatibility. (#5898) * Very minor tweak to commen to improve docs (#5900) As discussed here: https://stackoverflow.com/questions/58816271/celery-task-asyncresult-takes-task-id-but-is-documented-to-get-asyncresult-inst this comment seems to flow to a very confusing and misleading piece of documentation here: https://docs.celeryproject.org/en/latest/reference/celery.app.task.html#celery.app.task.Task.AsyncResult * Support configuring schema of a PostgreSQL database (#5910) * Support configuring schema of a PostgreSQL database * Add unit test * Remove blank line * Fix raise issue to make exception message more friendly (#5912) Signed-off-by: Chenyang Yan <memory.yancy@gmail.com> * Add progress for retry connections (#5915) This will show current retry progress so it will clear confusion about how many retries will be tried for connecting to broker. Closes #4556 * chg: change xrange to range (#5926) * update docs for json serializer and add note for int keys serialization (#5932) * fix indentation for note block in calling.rst (#5933) * Added links to other issue trackers. (#5939) * Add labels automatically for issues. (#5938) * Run pyupgrade. Co-authored-by: Michal Čihař <michal@cihar.com> Co-authored-by: ptitpoulpe <ptitpoulpe@ptitpoulpe.fr> Co-authored-by: Didi Bar-Zev <didi@hiredscore.com> Co-authored-by: Santos Solorzano <santosjavier22@gmail.com> Co-authored-by: manlix <manlix@yandex.ru> Co-authored-by: Jimmy <54828848+sckhg1367@users.noreply.github.com> Co-authored-by: Борис Верховский <boris.verk@gmail.com> Co-authored-by: Asif Saif Uddin <auvipy@gmail.com> Co-authored-by: Jainal Gosaliya <jainal09gosalia@gmail.com> Co-authored-by: gsfish <caoyu97@hotmail.com> Co-authored-by: Dipankar Achinta <di.punk.car19@gmail.com> Co-authored-by: Pengjie Song (宋鹏捷) <spengjie@sina.com> Co-authored-by: Chris Griffin <chris-griffin@users.noreply.github.com> Co-authored-by: Muhammad Hewedy <mhewedy@gmail.com> Co-authored-by: Blaine Bublitz <blaine.bublitz@gmail.com> Co-authored-by: Tamu <tamsanh@gmail.com> Co-authored-by: Erik Tews <erik@datenzone.de> Co-authored-by: abhinav nilaratna <anilaratna2@bloomberg.net> Co-authored-by: Wyatt Paul <wpaul@hearsaycorp.com> Co-authored-by: gal cohen <gal.nevis@gmail.com> Co-authored-by: as <alfred@huji.fr> Co-authored-by: Param Kapur <paramkapur2002@gmail.com> Co-authored-by: Sven Ulland <sven.ulland@gmail.com> Co-authored-by: Safwan Rahman <safwan.rahman15@gmail.com> Co-authored-by: Aissaoui Anouar <tobia@crossbone.cc> Co-authored-by: Neal Wang <qdzzyb2015@gmail.com> Co-authored-by: Alireza Amouzadeh <alireza@amouzadeh.net> Co-authored-by: Marcos Moyano <marcos@anue.biz> Co-authored-by: Stepan Henek <stepan+github@henek.name> Co-authored-by: Andrew Sklyarov <AndrewPix@users.noreply.github.com> Co-authored-by: Michael Fladischer <michael@fladi.at> Co-authored-by: Dejan Lekic <dejan.lekic@gmail.com> Co-authored-by: Yannick Schuchmann <yannick.schuchmann@googlemail.com> Co-authored-by: Matt Davis <matteius@gmail.com> Co-authored-by: Karthikeyan Singaravelan <tir.karthi@gmail.com> Co-authored-by: Bernd Wechner <bernd-wechner@users.noreply.github.com> Co-authored-by: Sören Oldag <soeren_oldag@freenet.de> Co-authored-by: uddmorningsun <memory.yancy@gmail.com> Co-authored-by: Amar Fadil <34912365+marfgold1@users.noreply.github.com> Co-authored-by: woodenrobot <woodenrobot1993@gmail.com> Co-authored-by: Sardorbek Imomaliev <sardorbek.imomaliev@gmail.com> * Remove fallback code for Python 2 support marked with TODOs. (#5953) Co-authored-by: Asif Saif Uddin <auvipy@gmail.com> * Remove PY3 conditionals (#5954) * Added integration coverage for link_error (#5373) * Added coverage for link_error. * Use pytest-rerunfailed plugin instead of rolling our own custom implementation. * Added link_error with retries. This currently fails. * Remove unused import. * Fix import on Python 2.7. * retries in link_error do not hang the worker anymore. * Run error callbacks eagerly when the task itself is run eagerly. Fixes #4899. * Adjust unit tests accordingly. * Grammar in documentation (#5780) * Grammar in documentation * Address review. * pypy 7.2 matrix (#5790) * removed extra slashes in CELERY_BROKER_URL (#5792) The Celery broker URL in settings.py had 2 slashes in the end which are not required and can be misleading. so I changed :- CELERY_BROKER_URL = 'amqp://guest:guest@localhost//' to CELERY_BROKER_URL = 'amqp://guest:guest@localhost' * Fix #5772 task_default_exchange & task_default_exchange_type not work (#5773) * Fix #5772 task_default_exchange & task_default_exchange_type not work * Add unit test: test_setting_default_exchange * Move default_exchange test to standalone class * Run integration suite with memcached results backend. (#5739) * Fix hanging forever when fetching results from a group(chain(group)) canvas. (#5744) PR #5739 uncovered multiple problems with the cache backend. This PR should resolve one of them. PR #5638 fixed the same test case for our async results backends that support native join. However, it did not fix the test case for sync results backends that support native join. * Fix regression in PR #5681. (#5753) See comment in the diff for details. * Grammatical fix to CONTRIBUTING.rst doc (#5794) * Fix #5734 Celery does not consider authMechanism on mongodb backend URLs (#5795) * Fix #5734 Celery does not consider authMechanism on mongodb backend URLs * Add unit test: test_get_connection_with_authmechanism * Add unit test: test_get_connection_with_authmechanism_no_username * Fix errors in Python 2.7 Remove "," after "**" operator * Revert "Revert "Revert "Added handle of SIGTERM in BaseTask in celery/task.py to prevent kill the task" (#5577)" (#5586)" (#5797) This reverts commit f79894e0a2c7156fd0ca5e8e3b652b6a46a7e8e7. * Add Python 3.8 Support (#5785) * Added Python 3.8 to the build matrix. * Ensure a supported tblib version is installed for Python 3.8 and above. In addition, modernize the relevant tests. * Workaround patching problem in test. * py 3.8 in clasifier * ubuntu bionic (#5799) * ubuntu bionic * fast finish * sync bumversion with pypi release * Dev.req (#5803) * update docker config * undo hardpin * devr req install from github master * update docker config (#5801) * update docker config * make dockerfile to install from github master dev branch by default * update download link * Isort. * Grammatical & punctuation fixes for CONTRIBUTING.rst document (#5804) * update dockerfile * switched to ubuntu bionic * update docker * keep it empty until we reconfigure it again with autopep8 * Fixed Dockerfile (#5809) * Update document CONTRIBUTING.rst & fix Dockerfile typo (#5813) * Added an issue template for minor releases. * reference gocelery Go Client/Server for Celery (#5815) * Add enterprise language (#5818) * Fix/correct minor doc typos (#5825) * Correct a small typo * Correct bad contributing documentation links * Preserve the task priority in case of a retry (#5820) * Preserve the task priority in case of a retry * Created test case for retried tasks with priority * Implement an integration test for retried tasks with priorities * bump kombu * basic changelog for celery 4.4.0rc4 * bump celery 4.4.0rc4 * events bootstep disabled if no events (#5807) * events bootstep disabled if no events * Added unit tests. * update bug report template * fixing ascii art to look nicer (#5831) * Only rerun flaky tests when failures can be intermediary. * Rename Changelog to Changelog.rst * The test_nested_group_chain test can run without native_join support. (#5838) * Run integration tests with Cassandra (#5834) * Run integration tests with Cassandra. * Configure cassandra result backend * Pre-create keyspace and table * Fix deprecation warning. * Fix path to cqlsh. * Increase connection timeout. * Wait until the cluster is available. * SQS - Reject on failure (#5843) * reject on failure * add documentation * test fix * test fix * test fix * Add a concurrency model with ThreadPoolExecutor (#5099) * Add a concurrency model with ThreadPoolExecutor * thread model test for pypy * Chain primitive's code example fix in canvas documentation (Regression PR#4444) (#5845) * Changed multi-line string (#5846) This string wasn't rendering properly and was printing the python statement too. Although the change isn't as pretty code-wise, it gets rid of an annoyance for the user. * Add auto expiry for DynamoDB backend (#5805) * Add auto expiry for DynamoDB backend This adds auto-expire support for the DynamoDB backend, via the DynamoDB Time to Live feature. * Require boto3>=1.9.178 for DynamoDB TTL support boto3 version 1.9.178 requires botocore>=1.12.178. botocore version 1.12.178 introduces support for the DynamoDB UpdateTimeToLive call. The UpdateTimeToLive call is used by the DynamoDB backend to enable TTL support on a newly created table. * Separate TTL handling from table creation Handle TTL enabling/disabling separately from the table get-or-create function. Improve handling of cases where the TTL is already set to the desired state. DynamoDB only allows a single TTL update action within a fairly long time window, so some problematic cases (changing the TTL attribute, enabling/disabling TTL when it was recently modified) will raise exceptions that have to be dealt with. * Handle older boto3 versions If the boto3 TTL methods are not found, log an informative error. If the user wants to enable TTL, raise an exception; if TTL should be disabled, simply return. * Improve logging - Handle exceptions by logging the error and re-raising - Log (level debug) when the desired TTL state is already in place * Add and use _has_ttl() convenience method Additional changes: - Handle exceptions when calling boto3's describe_time_to_live() - Fix test cases for missing TTL methods * Update ttl_seconds documentation * Log invalid TTL; catch and raise ValueError * Separate method _get_table_ttl_description * Separate ttl method validation function * Clarify tri-state TTL value * Improve test coverage * Fix minor typo in comment * Mark test as xfail when using the cache backend. (#5851) * [Fix #5436] Store extending result in all backends (#5661) * [Fix #5436] Store extending result in all backends * Fix sqlalchemy * More fixu * Fixing tests * removing not necessary import * Removing debug code * Removing debug code * Add tests for get_result_meta in base and database * Revert "Add auto expiry for DynamoDB backend (#5805)" (#5855) This reverts commit f7f5bcfceca692d0e78c742a7c09c424f53d915b. * Revert "Mark test as xfail when using the cache backend. (#5851)" (#5854) This reverts commit 1b303c2968836245aaa43c3d0ff9249dd8bf9ed2. * docs: Document Redis commands used by celery (#5853) * remove cache back end integrtion test. (#5856) * Fix a race condition when publishing a very large chord header (#5850) * Added a test case which artificially introduces a delay to group.save(). * Fix race condition by delaying the task only after saving the group. * update tox * Remove duplicate boto dependency. (#5858) * Revert "remove cache back end integrtion test. (#5856)" (#5859) This reverts commit e0ac7a19a745dd5a52a615c1330bd67f2cef4d00. * Revert "Revert "Add auto expiry for DynamoDB backend (#5805)" (#5855)" (#5857) This reverts commit 4ddc605392d7694760f23069c34ede34b3e582c3. * Revert "update tox" This reverts commit 49427f51049073e38439ea9b3413978784a24999. * Fix the test_simple_chord_with_a_delay_in_group_save test. * Revert "Revert "Skip unsupported canvas when using the cache backend"" (#5860) * Revert "Revert "Mark test as xfail when using the cache backend. (#5851)" (#5854)" This reverts commit fc101c61c1912c4dafa661981f8b865c011e8a55. * Make the xfail condition stricter. * Fix the xfail condition. * Linters should use Python 3.8. * Move pypy unit tests to the correct stage. * Temporarily allow PyPy to fail since it is unavailable in Travis. * Remove unused variables. * Fix unused imports. * Fix pydocstyle errors in dynamodb. * Fix pydocstyle errors in redis backend. * bump kombu to 4.6.7 * celery 4.4.0rc5 changelog * celery 4.4.0rc5 * rm redundant code (#5864) * isort. * Document the threads task pool in the CLI. * Removed the paragraph about using librabbitmq. Refer to #5872 (#5873) * Task class definitions can have retry attributes (#5869) * autoretry_for * retry_kwargs * retry_backoff * retry_backoff_max * retry_jitter can now be defined as cls attributes. All of these can be overriden from the @task decorator https://github.com/celery/celery/issues/4684 * whatsnew in Celery 4.4 as per projects standard (#5817) * 4.4 whatsnew * update * update * Move old whatsnew to history. * Remove old news & fix markers. * Added a section notifying Python 3.4 has been dropped. * Added a note about ElasticSearch basic auth. * Added a note about being able to replace eagerly run tasks. * Update index. * Address comment. * Described boto3 version updates. * Fix heading. * More news. * Thread pool. * Add Django and Config changes * Bump version 4.4.0 * upate readme * Update docs regarding Redis Message Priorities (#5874) * Update docs regarding Redis Message Priorities * fixup! Update docs regarding Redis Message Priorities * Update 4.4.0 docs (#5875) * Update 4.4 release changelog * Update whatsnew-4.4 * Update tasks docs * Fix recent tasks doc file update (#5879) * Include renamed Changelog.rst in source releases. (#5880) Changelog.rst was renamed from Changelog in fd023ec174bedc2dc65c63a0dc7c85e425ac00c6 but MANIFEST.in was not updated to include the new name. This fixes the file name so Changelog.rst will show up in future source releases again. * Reorganised project_urls and classifiers. (#5884) * Use safequote in SQS Getting Started doc (#5885) * Have appveyor build relevant versions of Python. (#5887) * Have appveyor build relevant and buildable versions of Python. * Appveyor is missing CI requirements to build. * Pin pycurl to version that will build with appveyor (because wheels files exist) * Restrict python 2.7 64 bit version of python-dateutil for parse. * Use is_alive instead of isAlive for Python 3.9 compatibility. (#5898) * Very minor tweak to commen to improve docs (#5900) As discussed here: https://stackoverflow.com/questions/58816271/celery-task-asyncresult-takes-task-id-but-is-documented-to-get-asyncresult-inst this comment seems to flow to a very confusing and misleading piece of documentation here: https://docs.celeryproject.org/en/latest/reference/celery.app.task.html#celery.app.task.Task.AsyncResult * Support configuring schema of a PostgreSQL database (#5910) * Support configuring schema of a PostgreSQL database * Add unit test * Remove blank line * Fix raise issue to make exception message more friendly (#5912) Signed-off-by: Chenyang Yan <memory.yancy@gmail.com> * Add progress for retry connections (#5915) This will show current retry progress so it will clear confusion about how many retries will be tried for connecting to broker. Closes #4556 * chg: change xrange to range (#5926) * update docs for json serializer and add note for int keys serialization (#5932) * fix indentation for note block in calling.rst (#5933) * Added links to other issue trackers. (#5939) * Add labels automatically for issues. (#5938) * remove redundant raise from docstring (#5941) `throw` is True by default so the Retry exception will already get raised by calling `self.retry(countdown=60 * 5, exc=exc)` * Run pyupgrade. * Fix typo (#5943) * Remove fallback code for Python 2 support. * docs: fixes Rabbits and Warrens link in routing userguide (#4007) (#5949) * Fix labels on Github issue templates. (#5955) Use quotation marks to escape labels on Github issue templates. This prevents the colon from breaking the template. * added retry_on_timeout and socket_keepalive to config and doc (#5952) * Fixed event capture from building infinite list (#5870) * Fix error propagation example (#5966) * update range (#5971) * update setup.cfg * bump billiard to 3.6.3.0 * Update __init__.py (#5951) * Update __init__.py Fixed issue for object with result_backend=True (decode fails on multiple None request) * Update __init__.py suggested changeds * Update __init__.py * Use configured db schema also for sequences (#5972) * Added a default value for retries in worker.strategy. (#5945) * Added a default value for retries in worker.strategy. I was facing an issue when adding tasks directly to rabbitmq using pika instead of calling task.apply_async. The issue was the self.retry mechanisum was failing. In app/tasks.py the line `retries = request.retries + 1` was causing the issue. On further tracing I figured out that it was because the default .get value (None) was getting passed through this function and was raising TypeError: unsupported operand type(s) for +: 'NoneType' and 'int' * Add test cases for default and custom retries value * pypy 7.3 (#5980) * Pass `interval` to `get_many` (#5931) * Pass `interval` to `get_many` * Fix: Syntax error for py2.7 * Fix: Syntax error for py2.7 * Fixed problem with conflicting autoretry_for task parameter and Task.replace() (#5934) * Fix #5917 (#5918) * Fix changelog (#5881) * merge in place the apps beat schedule in the default Schedule class. (#5908) * Handle Redis connection errors in result consumer (#5921) * Handle Redis connection errors in result consumer * Closes #5919. * Use context manager for Redis conusmer reconnect * Log error when result backend reconnection fails * Fix inspect_command documentation (#5983) * Use gevent and eventlet wait() functions to remove busy-wait (#5974) * Use gevent and eventlet wait() functions to remove busy-wait Fixes issue #4999. Calling AsyncResult.get() in a gevent context would cause the async Drainer to repeatedly call wait_for until the result was completed. I've updated the code to have a specific implementation for gevent and eventlet that will cause wait_for to only return every "timeout" # of seconds, rather than repeatedly returning. Some things I'd like some feedback on: * Where's the best place to add test coverage for this? It doesn't look like there are any tests that directly exercised the Drainer yet so I would probably look to add some of these to the backends/ unit tests. * The way I did this for the Eventlet interface was to rely on the private _exit_event member of the GreenThread instance; to do this without relying on a private member would require some additional changes to the backend Drainer interface so that we could wait for an eventlet-specific event in wait_for(). I can do this, just wanted to get some feedback before. * Add unit tests for Drainer classes In order for this to work without monkeypatching in the tests, I needed to call sleep(0) to let the gevent/eventlet greenlets to yield control back to the calling thread. I also made the check interval configurable in the drainer so that we didn't need to sleep multiples of 1 second in the tests. * Weaken asserts since they don't pass on CI * Fix eventlet auto-patching DNS resolver module on import By default it looks like "import eventlet" imports the greendns module unless the environment EVENTLET_NO_GREENDNS is set to true. This broke a pymongo test. * Add tests ensuring that the greenlet loop isn't blocked These tests make sure that while drain_events_until is running that other gevent/eventlet concurrency can run. * Clean up tests and make sure they wait for all the threads to stop * Fix chords with chained groups (#5947) * kombu 4.6.8 * update setup * updated version 4.4.1 * Fix: Accept and swallow `kwargs` to handle unexpected keyword arguments * Allow boto to look for credentials in S3Backend * add reference to Rusty Celery * Update document of revoke method in Control class * Fix copy-paste error in result_compression docs * Make 'socket_keepalive' optional variable (#6000) * update connection params - socket_keepalive is optional now * update readme - added versionadded 4.4.1 and fixed `redis_socket_keepalive` * added check of socket_keepalive in arguments for UnixSocketConnect * Fixed incorrect setting name in documentation (#6002) * updated version 4.4.2 * Fix backend utf-8 encoding in s3 backend Celery backend uses utf-8 to deserialize results, which would fail for some serializations like pickle. * Fix typo in celery.bin.multi document * Upgraded pycurl to the latest version that supports wheel. * pytest 5.3.5 max * Add uptime to the stats inspect command * Doc tweaks: mostly grammar and punctuation (#6016) * Fix a bunch of comma splices in the docs * Remove some unnecessary words from next-steps doc * Tweak awkward wording; fix bad em-dash * Fix a bunch more comma splices in next-steps doc * Miscellaneous grammar/punctuation/wording fixes * Link to task options in task decorator docs * Fixing issue #6019: unable to use mysql SSL parameters when getting mysql engine (#6020) * Fixing issue #6019: unable to use mysql SSL parametes in create_engine() * adding test for get_engine when self.forked is False and engine args are passed in for create_engine() * Clean TraceBack to reduce memory leaks for exception task (#6024) * Clean TraceBack to reduce memory leaks * add unit test * add unit test * reject unittest * Patch For Python 2.7 compatibility * update unittest * Register to the garbage collector by explicitly referring to f_locals. * need more check * update code coverage * update Missing unit test * 3.4 -> 3.5 Co-authored-by: heedong.jung <heedong.jung@samsung.com> * exceptions: NotRegistered: fix up language Minor fix to the language. * Note about autodiscover_tasks and periodic tasks This is particularly important for Django projects that put periodic tasks into each app's `tasks.py` and want to use one as a periodic task. By the time `autodiscover_tasks()` loads those tasks, the `on_after_configure` Signal has already come and gone, so anything decorated with `@app.on_after_finalize.connect` will never be called. If there's other documentation on this subject, I could not find it. * Avoid PyModulelevel, deprecated in Sphinx 4 Use `PyFunction` instead of `PyModulelevel` to avoid this deprecation warning: RemovedInSphinx40Warning: PyModulelevel is deprecated. Please check the implementation of <class 'celery.contrib.sphinx.TaskDirective'> This replacement is one of the options listed in the Sphinx docs (https://www.sphinx-doc.org/en/master/extdev/deprecated.html). * Give up sending a worker-offline message if transport is not connected (#6039) * If worker-offline event fails to send, give up and die peacefully * Add test for retry= and msgs in heartbeat * Fix the build and all documentation warnings. I finally upgraded our theme to 2.0. As a result we've upgraded Sphinx to 2.0. Work to upgrade Sphinx to 3.0 will proceed in a different PR. This upgrade also fixes our build issues caused by #6032. We don't support Sphinx 1.x as a result of that patch. I've also included the missing 4.3 changelog to our history. * Support both Sphinx 2 and 3. * Add Task to __all__ in celery.__init__.py * Add missing parenthesis to example in docs * Ensure a single chain object in a chain does not raise MaximumRecursionError. Previously chain([chain(sig)]) would crash. We now ensure it doesn't. Fixes #5973. * update setup.py * fix typo missing quote at the end of line * Fix a typo in monitoring doc * update travis * update ubuntu to focal foss 20.04 LTS * Fix autoscale when prefetch_multiplier is 1 * Allow start_worker to function without ping task * Update celeryd.conf Move the directory of the program before the execution of the command/script * Add documentation for "predefined_queue_urls" * [Fix #6074]: Add missing documentation for MongoDB as result backend. * update funding * 🐛 Correctly handle configuring the serializer for always_eager mode. (#6079) * 🐛 Correctly handle configuring the serializer for always_eager mode. options['serializer'] will always exist, because it is initialized from an mattrgetter. Even if unset, it will be present in the options with a value of None. * 🐛 Add a test for new always_eager + task_serializer behavior. * ✏️ Whoops missed a : * Remove doubling of prefetch_count increase when prefetch_multiplier gt 1 (#6081) * try ubuntu focal (#6088) * Fix eager function not returning result after retries. Using apply function does not return correct results after at least one retry because the return value of successive call is not going back to the original caller. * return retry result if not throw and is_eager if throw is false, we would be interested by the result of retry and not the current result which will be an exception. This way it does not break the logic of `raise self.retry` This should be used like `return self.retry(..., throw=False)` in an except statement. * revert formatting change * Add tests for eager retry without throw * update predefined-queues documentation Suggested version of configuration does not work. Additionally I'd like to mention, that `access_key_id` and `secret_access_key` are mandatory fields and not allowing you to go with defaults AWS_* env variables. I can contribute for this variables to be optional Also I'm not sure if security token will apply, could you please advice how to do it? * Fix couchbase version < 3.0.0 as API changed * Remove reference to -O fair in optimizations -O fair was made the default in Celery 4.0 https://docs.celeryproject.org/en/stable/history/whatsnew-4.0.html#ofair-is-now-the-default-scheduling-strategy * pytest ranges * pypy3 * revert to bionic * do not load docs.txt requirements for python 2.7 As it requires Sphinx >= 2.0.0 and there is no such version compatible with python 2.7 * update cassandra travis integration test configuration cassandra:latest docker image changed location of cqlsh program * pin cassandra-driver CI get stuck after all cassandra integration tests * Fix all flake8 lint errors * Fix all pydocstyle lint errors * Fix all configcheck lint errors * Always requeue while worker lost regardless of the redelivered flag (#6103) * #5598 fix, always redelivery while WorkerLostError * fix, change the requeue flag so the task will remain PENDING * Allow relative paths in the filesystem backend (#6070) * Allow relative paths in the filesystem backend * fix order of if statements * [Fixed Issue #6017] --> Added Multi default logfiles and pidfiles paths [Description]: --> Changed the default paths for log files & pid files to be '/var/log/celery' and '/var/run/celery' --> Handled by creating the respective paths if not exist. --> Used os.makedir(path,if_exists=True) [Unit Test Added]: --> .travis.yml - config updated with 'before install'. --> t/unit/apps/test_multi.py - Changed the default log files & pid files paths wherever required. * Avoid race condition due to task duplication. In some circumstances like a network partitioning, some tasks might be duplicated. Sometimes, this lead to a race condition where a lost task overwrites the result of the last successful task in the backend. In order to avoid this race condition we prevent updating the result if it's already in successful state. This fix has been done for KV backends only and therefore won't work with other backends. * adding tests * Exceptions must be old-style classes or derived from BaseException, but here self.result may not subclass of BaseException. * update fund link * Fix windows build (#6104) * do not load memcache nor couchbase lib during windows build those libraries depends on native libraries libcouchbase and libmemcached that are not installed on Appveyor. As only unit tests runs on Appveyor, it should be fine * Add python 3.8 workaround for app trap * skip tests file_descriptor_safety tests on windows AsyncPool is not supported on Windows so Pool does have _fileno_to_outq attribute, making the test fail * Fix crossplatform log and pid files in multi mode it relates to #6017 * Use tox to build and test on windows * remove tox_install_command * drop python 2.7 from windows build * Add encode to meta task in base.py (#5894) * Add encode to base.py meta result Fix bug with impossibility to load None from task meta * Add tests for None. Remove exceed encode. * Update base.py Add return payload if None * Update time.py to solve the microsecond issues (#5199) When `relative` is set to True, the day, hour, minutes second will be round to the nearest one, however, the original program do not update the microsecond (reset it). As a result, the run-time offset on the microsecond will then be accumulated. For example, given the interval is 15s and relative is set to True 1. 2018-11-27T15:01:30.123236+08:00 2. 2018-11-27T15:01:45.372687+08:00 3. 2018-11-27T15:02:00.712601+08:00 4. 2018-11-27T15:02:15.987720+08:00 5. 2018-11-27T15:02:31.023670+08:00 * Change backend _ensure_not_eager error to warning * Add priority support for 'celery.chord_unlock' task (#5766) * Change eager retry behaviour even with raise self.retry, it should return the eventual value or MaxRetriesExceededError. if return value of eager apply is Retry exception, retry eagerly the task signature * Order supported Python versions * Avoid race condition in elasticsearch backend if a task is retried, the task retry may work concurrently to current task. store_result may come out of order. it may cause a non ready state (Retry) to override a ready state (Success, Failure). If this happens, it will block indefinitely pending any chord depending on this task. this change makes document updates safe for concurrent writes. https://www.elastic.co/guide/en/elasticsearch/reference/current/optimistic-concurrency-control.html * backends base get_many pass READY_STATES arg * test backends base get_many pass READY_STATES arg * Add integration tests for Elasticsearch and fix _update * Revert "revert to bionic" This reverts commit 6e091573f2ab0d0989b8d7c26b677c80377c1721. * remove jython check * feat(backend): Adds cleanup to ArangoDB backend * Delete Document Known Issue with CONN_MAX_AGE in 4.3 * issue 6108 fix filesystem backend cannot not be serialized by picked (#6120) * issue 6108 fix filesystem backend cannot not be serialized by picked https://github.com/celery/celery/issues/6108 * issue-6108 fix unit test failure * issue-6108 fix flake8 warning Co-authored-by: Murphy Meng <mmeng@mirriad.com> * kombu==4.6.9 (#6133) * changelog for 4.4.3 * v 4.4.3 * remove un supported classifier * Fix autoretry_for with explicit retry (#6138) * Add tests for eager task retry * Fixes #6135 If autoretry_for is set too broad on Exception, then autoretry may get a Retry if that's the case, rethrow directly instead of wrapping it in another Retry to avoid loosing new args * Use Django DB max age connection setting (fixes #4116) * Add retry on recoverable exception for the backend (#6122) * Add state to KeyValueStoreBackend.set method This way, a backend implementation is able to take decisions based on current state to store meta in case of failures. * Add retry on recoverable exception for the backend acks.late makes celery acknowledge messages only after processing and storing result on the backend. However, in case of backend unreachable, it will shadow a Retry exception and put the task as failed in the backend not retrying the task and acknoledging it on the broker. With this new result_backend_always_retry setting, if the backend exception is recoverable (to be defined per backend implementation), it will retry the backend operation with an exponential backoff. * Make elasticsearch backward compatible with 6.x * Make ES retry storing updates in a better way if existing value in the backend is success, then do nothing. if it is a ready status, then update it only if new value is a ready status as well. else update it. This way, a SUCCESS cannot be overriden so that we do not loose results but any ready state other than success (FAILURE, REVOKED) can be overriden by another ready status (i.e. a SUCCESS) * Add test for value not found in ES backend * Fix random distribution of jitter for exponential backoff random.randrange should be called with the actual so that all numbers have equivalent probability, otherwise maximum value does have a way higher probability of occuring. * fix unit test if extra modules are not present * ElasticSearch: add setting to save meta as json * fix #6136. celery 4.4.3 always trying create /var/run/celery directory (#6142) * fix #6136. celery 4.4.3 always trying create /var/run/celery directory, even if it's not needed. * fix #6136. cleanup * Add task_internal_error signal (#6049) * Add internal_error signal There is no special signal for an out of body error which can be the result of a bad result backend. * Fix syntax error. * Document the task_internal_error signal. Co-authored-by: Laurentiu Dragan <ldragan@bloomberg.net> * changelog for v4.4.4 * kombu 4.6.10 (#6144) * v4.4.4 * Add missing dependency on future (#6146) Fixes #6145 * ElasticSearch: Retry index if document was deleted between index and update (#6140) * ElasticSearch: Retry index if document was deleted between index and update * Elasticsearch increase coverage to 100% * Fix pydocstyle * Specify minimum version of Sphinx for Celery extension (#6150) The Sphinx extension requires Sphinx 2 or later due to #6032. * fix windows build * fix flake8 error * fix multi tests in local Mock os.mkdir and os.makedirs to avoid creating /var/run/celery and /var/log/celery during unit tests if run without root priviledges * Customize the retry interval of chord_unlock tasks * changelog v4.4.5 * v4.4.5 * Fix typo in comment. * Remove autoscale force_scale methods (#6085) * Remove autoscale force_scale methods * Remove unused variable in test * Pass ping destination to request The destination argument worked fine from CLI but didn't get used when calling ping from Python. * Fix autoscale test * chord: merge init options with run options * put back KeyValueStoreBackend.set method without state It turns out it was breaking some other projects. wrapping set method with _set_with_state, this way it will not break existing Backend. while enabling this feature for other Backend. Currently, only ElasticsearchBackend supports this feature. It protects concurrent update to corrupt state in the backend. Existing success cannot be overriden, nor a ready state by a non ready state. i.e. a Retry state cannot override a Success or Failure. As a result, chord_unlock task will not loop forever due to missing ready state on the backend. * added --range-prefix option to `celery multi` (#6180) * added --range-prefix option to `celery multi` Added option for overriding default range prefix when running multiple workers prividing range with `celery multy` command. * covered multi --range-prefix with tests * fixed --range-prefix test * Added as_list function to AsyncResult class (#6179) * Add as_list method to return task IDs as a list * Add a test for as_list method * Add docstring for as_list method * Fix CassandraBackend error in threads or gevent pool (#6147) * Fix CassandraBackend error in threads or gevent pool * remove CassandraBackend.process_cleanup * Add test case * Add test case * Add comments test_as_uri Co-authored-by: baixue <baixue@wecash.net> * changelog for v4.4.6 * v4.4.6 * Update Wiki link in "resources" In the page linked below, the link to wiki is outdated. Fixed that. https://docs.celeryproject.org/en/stable/getting-started/resources.html * test_canvas: Add test for chord-in-chain Add test case for the issue where a chord in a chain does not work when using .apply(). This works fine with .apply_async(). * Trying to fix flaky tests in ci * fix pydocstyle errors * fix pydocstyle * Drainer tests, put a lower constraint on number of intervals liveness should iterate 10 times per interval while drain_events only once. However, as it may use thread that may be scheduled out of order, we may end up in some situation where liveness and drain_events were called the same amount of time. Lowering the constraint from < to <= to avoid failing the tests. * pyupgrade. * Fix merge error. Co-authored-by: Борис Верховский <boris.verk@gmail.com> Co-authored-by: Asif Saif Uddin <auvipy@gmail.com> Co-authored-by: Jainal Gosaliya <jainal09gosalia@gmail.com> Co-authored-by: gsfish <root@grassfish.net> Co-authored-by: Dipankar Achinta <di.punk.car19@gmail.com> Co-authored-by: spengjie <spengjie@sina.com> Co-authored-by: Chris Griffin <chris-griffin@users.noreply.github.com> Co-authored-by: Muhammad Hewedy <mhewedy@gmail.com> Co-authored-by: Blaine Bublitz <blaine.bublitz@gmail.com> Co-authored-by: Tamu <tamsanh@gmail.com> Co-authored-by: Erik Tews <erik@datenzone.de> Co-authored-by: abhinav nilaratna <anilaratna2@bloomberg.net> Co-authored-by: Wyatt Paul <wpaul@hearsaycorp.com> Co-authored-by: gal cohen <gal.nevis@gmail.com> Co-authored-by: whuji <alfred@huji.fr> Co-authored-by: Param Kapur <paramkapur2002@gmail.com> Co-authored-by: Sven Ulland <sven.ulland@gmail.com> Co-authored-by: Safwan Rahman <safwan.rahman15@gmail.com> Co-authored-by: Aissaoui Anouar <tobia@crossbone.cc> Co-authored-by: Neal Wang <qdzzyb2015@gmail.com> Co-authored-by: Alireza Amouzadeh <alireza@amouzadeh.net> Co-authored-by: Marcos Moyano <marcos@anue.biz> Co-authored-by: Stepan Henek <stepan+github@henek.name> Co-authored-by: Andrew Sklyarov <AndrewPix@users.noreply.github.com> Co-authored-by: Michael Fladischer <michael@fladi.at> Co-authored-by: Dejan Lekic <dejan.lekic@gmail.com> Co-authored-by: Yannick Schuchmann <yannick.schuchmann@googlemail.com> Co-authored-by: Matt Davis <matteius@gmail.com> Co-authored-by: Xtreak <tir.karthi@gmail.com> Co-authored-by: Bernd Wechner <bernd-wechner@users.noreply.github.com> Co-authored-by: Sören Oldag <soeren_oldag@freenet.de> Co-authored-by: uddmorningsun <memory.yancy@gmail.com> Co-authored-by: Amar Fadil <34912365+marfgold1@users.noreply.github.com> Co-authored-by: woodenrobot <woodenrobot1993@gmail.com> Co-authored-by: Sardorbek Imomaliev <sardorbek.imomaliev@gmail.com> Co-authored-by: Alex Riina <alex.riina@gmail.com> Co-authored-by: Joon Hwan 김준환 <xncbf12@gmail.com> Co-authored-by: Prabakaran Kumaresshan <k_prabakaran+github@hotmail.com> Co-authored-by: Martey Dodoo <martey@mobolic.com> Co-authored-by: Konstantin Seleznev <4374093+Seleznev-nvkz@users.noreply.github.com> Co-authored-by: Prodge <Prodge@users.noreply.github.com> Co-authored-by: Abdelhadi Dyouri <raryat@gmail.com> Co-authored-by: Ixiodor <Ixiodor@users.noreply.github.com> Co-authored-by: abhishekakamai <47558404+abhishekakamai@users.noreply.github.com> Co-authored-by: Allan Lei <allanlei@helveticode.com> Co-authored-by: M1ha Shvn <work_shvein_mihail@mail.ru> Co-authored-by: Salih Caglar Ispirli <caglarispirli@gmail.com> Co-authored-by: Micha Moskovic <michamos@gmail.com> Co-authored-by: Chris Burr <chrisburr@users.noreply.github.com> Co-authored-by: Dave King <tildedave@gmail.com> Co-authored-by: Dmitry Nikulin <v.dmitry.nikulin@gmail.com> Co-authored-by: Michael Gaddis <mgaddis@ancestry.com> Co-authored-by: epwalsh <epwalsh10@gmail.com> Co-authored-by: TalRoni <tal.inon.16@gmail.com> Co-authored-by: Leo Singer <leo.singer@ligo.org> Co-authored-by: Stephen Tomkinson <neonbunny@users.noreply.github.com> Co-authored-by: Abhishek <abhishek.shukla@xeneta.com> Co-authored-by: theirix <theirix@gmail.com> Co-authored-by: yukihira1992 <hirayama@cancerscan.jp> Co-authored-by: jpays <jerome.pays@cnp.fr> Co-authored-by: Greg Ward <greg@gerg.ca> Co-authored-by: Alexa Griffith <agriffith@bluecore.com> Co-authored-by: heedong <63043496+heedong-jung@users.noreply.github.com> Co-authored-by: heedong.jung <heedong.jung@samsung.com> Co-authored-by: Shreyansh Khajanchi <shreyanshk@users.noreply.github.com> Co-authored-by: Sam Thompson <georgedorn@users.noreply.github.com> Co-authored-by: Alphadelta14 <alpha@pokesplash.net> Co-authored-by: Azimjon Pulatov <azimjohn@yahoo.com> Co-authored-by: ysde <ysde108@gmail.com> Co-authored-by: AmirMohammad Ziaei <amir_zia@outlook.com> Co-authored-by: Ben Nadler <ben.nadler@gusto.com> Co-authored-by: Harald Nezbeda <hn@nezhar.com> Co-authored-by: Chris Frisina <github@specialorange.org> Co-authored-by: Adam Eijdenberg <adam.eijdenberg@digital.gov.au> Co-authored-by: rafaelreuber <rafaelreuber@gmail.com> Co-authored-by: Noah Kantrowitz <noah@coderanger.net> Co-authored-by: Ben Nadler <jbennadler@gmail.com> Co-authored-by: Clement Michaud <c.michaud@criteo.com> Co-authored-by: Mathieu Chataigner <m.chataigner@criteo.com> Co-authored-by: eugeneyalansky <65346459+eugeneyalansky@users.noreply.github.com> Co-authored-by: Leonard Lu <leonard@socialcodeinc.com> Co-authored-by: XinYang <yangxinhust@hotmail.com> Co-authored-by: Ingolf Becker <ingolf.becker@googlemail.com> Co-authored-by: Anuj Chauhan <an12ch98@gmail.com> Co-authored-by: shaoziwei <ziwei.shao@cloudchef.io> Co-authored-by: Mathieu Chataigner <mathieu.chataigner@gmail.com> Co-authored-by: Anakael <shemeldima2@yandex.ru> Co-authored-by: Danny Chan <danny.chan@c-k.dev> Co-authored-by: Sebastiaan ten Pas <sebastiaan@diggimedia.nl> Co-authored-by: David TILLOY <d.tilloy@criteo.com> Co-authored-by: Anthony N. Simon <anthonynsimon@users.noreply.github.com> Co-authored-by: lironhl <liron.lavy@gmail.com> Co-authored-by: Raphael Cohen <raphael.cohen@sekoia.fr> Co-authored-by: JaeyoungHeo <jay.jaeyoung@gmail.com> Co-authored-by: singlaive <singlaive@gmail.com> Co-authored-by: Murphy Meng <mmeng@mirriad.com> Co-authored-by: Wu Haotian <whtsky@gmail.com> Co-authored-by: Kwist <velnik@gmail.com> Co-authored-by: Laurentiu Dragan <ldragan@bloomberg.net> Co-authored-by: Michal Čihař <michal@cihar.com> Co-authored-by: Radim Sückr <radim.suckr@gmail.com> Co-authored-by: Artem Vasilyev <artem.v.vasilyev@gmail.com> Co-authored-by: kakakikikeke-fork <kakakikikeke_new@yahoo.co.jp> Co-authored-by: Pysaoke <pysaoke@gmail.com> Co-authored-by: baixue <baixue@wecash.net> Co-authored-by: Prashant Sinha <prashantsinha@outlook.com> Co-authored-by: AbdealiJK <abdealikothari@gmail.com> * Remove Python 2 compatibility code from Celery (#6221) * Remove five from celery/__init__.py * Remove five from celery/beat.py * Remove five from celery/bootsteps.py * Remove five from celery/exceptions.py * Remove five from celery/local.py * Remove five from celery/platforms.py * Remove five from celery/result.py * Remove five from celery/schedules.py * Remove five from celery/app/amqp.py * Remove five from celery/app/annotations.py * Remove five from celery/app/backends.py * Remove five from celery/app/base.py * Remove five from celery/app/control.py * Remove five from celery/app/defaults.py * Remove five from celery/app/log.py * Remove five from celery/app/registry.py * Remove five from celery/app/routes.py * Remove five from celery/app/task.py * Remove five from celery/app/trace.py * Remove five from celery/app/utils.py * Remove five from celery/apps/beat.py * Remove five from celery/apps/multi.py * Remove five from celery/apps/worker.py * Remove five from celery/backends/database/__init__.py * Remove five from celery/backends/amqp.py * Remove five from celery/backends/asynchronous.py * Remove five from celery/backends/base.py * Remove five from celery/backends/dynamodb.py * Remove five from celery/backends/elasticsearch.py * Remove five from celery/backends/mongodb.py * Remove five from celery/backends/redis.py * Remove five from celery/backends/rpc.py * Remove five from celery/concurrency/asynpool.py * Remove five from celery/concurrency/base.py * Remove five from celery/concurrency/prefork.py * Remove five from celery/contrib/testing/manager.py * Remove five from celery/contrib/migrate.py * Remove five from celery/contrib/rdb.py * Remove five from celery/events/cursesmon.py * Remove five from celery/events/dispatcher.py * Remove five from celery/events/state.py * Remove five from celery/loaders/base.py * Remove five from celery/security/certificate.py * Remove five from celery/security/utils.py * Remove five from celery/task/base.py * Remove five from celery/utils/dispatch/signal.py * Remove five from celery/utils/abstract.py * Remove five from celery/utils/collections.py * Remove five from celery/utils/debug.py * Remove five from celery/utils/functional.py * Remove five from celery/utils/graph.py * Remove five from celery/utils/imports.py * Remove five from celery/utils/log.py * Remove five from celery/utils/saferepr.py * Remove five from celery/utils/serialization.py * Remove five from celery/utils/term.py * Remove five from celery/utils/text.py * Remove five from celery/utils/threads.py * Remove five from celery/utils/time.py * Remove five from celery/utils/timer2.py * Remove five from celery/consumer/consumer.py * Remove five from celery/consumer/gossip.py * Remove five from celery/consumer/mingle.py * Remove five from celery/worker/autoscale.py * Remove five from celery/worker/components.py * Remove five from celery/worker/control.py * Remove five from celery/worker/request.py * Remove five from celery/worker/state.py * Remove five from celery/worker/worker.py * Remove five from celery/t/benchmarks/bench_worker.py * Remove five from celery/t/integration/test_canvas.py * Remove five from celery/t/unit/app * Remove five from celery/t/unit/backends * Remove five from celery/t/unit/compat_modules * Remove five from celery/t/unit/concurrency * Remove five from celery/t/unit/contrib * Remove five from celery/t/unit/events * Remove five from celery/t/unit/security * Remove five from celery/t/unit/tasks * Remove five from celery/t/unit/utils * Remove five from celery/t/unit/worker * Sort imports. * Comment out PyPy for now. * Remove flakeplus. * Happify linter. * Fix merge problems. * Delete backport. * Remove unused import. * Remove logic that notifies user that the Python version isn't supported from setup.py. pip already does that for us. * Add a trove classifier to indicate Celery only supports Python 3. * Restore usage of `reraise` for consistency with the kombu port. * Drop Python 2 compatibility code from our Sphinx extension. * Remove mention of flakeplus from tox.ini. * Remove mention of flakeplus from our CONTRIBUTING guide. * Bump Sphinx requirement. * Remove Python 2 compatibility code from our custom Sphinx extension. * Resolve Sphinx warning due to removed section in 32ff7b45aa3d78aedca61b6554a9db39122924fd. * Remove pydocstyle from build matrix as it was removed from master. See #6278. * Bump version: 4.4.7 → 5.0.0-alpha1 * Final touches. * Fix README. * Bump Kombu to 5.0.0. * Bump version: 5.0.0-alpha1 → 5.0.0a2 * Fix wrong version. * Remove autodoc for removed module. * Remove documentation for removed methods. * Remove the riak backend since riak is no longer maintained. Co-authored-by: tothegump <tothegump@gmail.com> Co-authored-by: Omer Katz <omer.drow@gmail.com> Co-authored-by: Michal Čihař <michal@cihar.com> Co-authored-by: ptitpoulpe <ptitpoulpe@ptitpoulpe.fr> Co-authored-by: Didi Bar-Zev <didi@hiredscore.com> Co-authored-by: Santos Solorzano <santosjavier22@gmail.com> Co-authored-by: manlix <manlix@yandex.ru> Co-authored-by: Jimmy <54828848+sckhg1367@users.noreply.github.com> Co-authored-by: Борис Верховский <boris.verk@gmail.com> Co-authored-by: Jainal Gosaliya <jainal09gosalia@gmail.com> Co-authored-by: gsfish <caoyu97@hotmail.com> Co-authored-by: Dipankar Achinta <di.punk.car19@gmail.com> Co-authored-by: Pengjie Song (宋鹏捷) <spengjie@sina.com> Co-authored-by: Chris Griffin <chris-griffin@users.noreply.github.com> Co-authored-by: Muhammad Hewedy <mhewedy@gmail.com> Co-authored-by: Blaine Bublitz <blaine.bublitz@gmail.com> Co-authored-by: Tamu <tamsanh@gmail.com> Co-authored-by: Erik Tews <erik@datenzone.de> Co-authored-by: abhinav nilaratna <anilaratna2@bloomberg.net> Co-authored-by: Wyatt Paul <wpaul@hearsaycorp.com> Co-authored-by: gal cohen <gal.nevis@gmail.com> Co-authored-by: as <alfred@huji.fr> Co-authored-by: Param Kapur <paramkapur2002@gmail.com> Co-authored-by: Sven Ulland <sven.ulland@gmail.com> Co-authored-by: Safwan Rahman <safwan.rahman15@gmail.com> Co-authored-by: Aissaoui Anouar <tobia@crossbone.cc> Co-authored-by: Neal Wang <qdzzyb2015@gmail.com> Co-authored-by: Alireza Amouzadeh <alireza@amouzadeh.net> Co-authored-by: Marcos Moyano <marcos@anue.biz> Co-authored-by: Stepan Henek <stepan+github@henek.name> Co-authored-by: Andrew Sklyarov <AndrewPix@users.noreply.github.com> Co-authored-by: Michael Fladischer <michael@fladi.at> Co-authored-by: Dejan Lekic <dejan.lekic@gmail.com> Co-authored-by: Yannick Schuchmann <yannick.schuchmann@googlemail.com> Co-authored-by: Matt Davis <matteius@gmail.com> Co-authored-by: Karthikeyan Singaravelan <tir.karthi@gmail.com> Co-authored-by: Bernd Wechner <bernd-wechner@users.noreply.github.com> Co-authored-by: Sören Oldag <soeren_oldag@freenet.de> Co-authored-by: uddmorningsun <memory.yancy@gmail.com> Co-authored-by: Amar Fadil <34912365+marfgold1@users.noreply.github.com> Co-authored-by: woodenrobot <woodenrobot1993@gmail.com> Co-authored-by: Sardorbek Imomaliev <sardorbek.imomaliev@gmail.com> Co-authored-by: gsfish <root@grassfish.net> Co-authored-by: Alex Riina <alex.riina@gmail.com> Co-authored-by: Joon Hwan 김준환 <xncbf12@gmail.com> Co-authored-by: Prabakaran Kumaresshan <k_prabakaran+github@hotmail.com> Co-authored-by: Martey Dodoo <martey@mobolic.com> Co-authored-by: Konstantin Seleznev <4374093+Seleznev-nvkz@users.noreply.github.com> Co-authored-by: Prodge <Prodge@users.noreply.github.com> Co-authored-by: Abdelhadi Dyouri <raryat@gmail.com> Co-authored-by: Ixiodor <Ixiodor@users.noreply.github.com> Co-authored-by: abhishekakamai <47558404+abhishekakamai@users.noreply.github.com> Co-authored-by: Allan Lei <allanlei@h…

…ork (#5718) * 'abstractproperty' is deprecated. Use 'property' with 'abstractmethod' instead * Fix #2849 - Initial work of celery 5.0.0 alpha1 series by dropping python below 3.6 from marix & remove import from __future__ (#5684) * initial work of celery 5.0.0 alpha-1 series by dropping python below 3.6 * i-5651(ut): add ut for ResultSet.join_native (#5679) * dropped python versions below 3.6 from tox * dropped python versions below 3.6 from travis * dropped python versions below 3.6 from appveyor * dropped python2 compat __future__ imports from tests * Fixed a bug where canvases with a group and tasks in the middle followed by a group fails to complete and indefinitely hangs. (#5681) Fixes #5512, fixes #5354, fixes #2573. * dropped python2 compat __future__ imports from celery * dropped python2 compat code from init * revert readme change about version * removed python 2 object inheritance (#5687) * removed python 2 object inheritance * removed python 2 object inheritance * removed python 2 compatibility decorator (#5689) * removed python 2 compatibility decorator * removed python 2 compatibility decorator * removed python 2 compatibility decorator * removed python 2 compatibility decorator * Remove unused imports. * Remove unused imports of python_2_unicode_compatible. Also removed leftover useage of them where they were still used. * Run pyupgrade on codebase (#5726) * Run pyupgrade on codebase. * Use format strings where possible. * pyupgrade examples. * pyupgrade on celerydocs extension. * pyupgrade on updated code. * Address code review comments. * Address code review comments. * Remove unused imports. * Fix indentation. * Address code review comments. * Fix syntax error. * Fix syntax error. * Fix syntax error. * pytest 5.x for celery 5 (#5791) * Port latest changes from master to v5-dev (#5942) * Fix serialization and deserialization of nested exception classes (#5717) * Fix #5597: chain priority (#5759) * adding `worker_process_shutdown` to __all__ (#5762) * Fix typo (#5769) * Reformat code. * Simplify commands to looking for celery worker processes (#5778) * update doc- celery supports storage list. (#5776) * Update introduction.rst * Update introduction.rst * Fail xfailed tests if the failure is unexpected. * Added integration coverage for link_error (#5373) * Added coverage for link_error. * Use pytest-rerunfailed plugin instead of rolling our own custom implementation. * Added link_error with retries. This currently fails. * Remove unused import. * Fix import on Python 2.7. * retries in link_error do not hang the worker anymore. * Run error callbacks eagerly when the task itself is run eagerly. Fixes #4899. * Adjust unit tests accordingly. * Grammar in documentation (#5780) * Grammar in documentation * Address review. * pypy 7.2 matrix (#5790) * removed extra slashes in CELERY_BROKER_URL (#5792) The Celery broker URL in settings.py had 2 slashes in the end which are not required and can be misleading. so I changed :- CELERY_BROKER_URL = 'amqp://guest:guest@localhost//' to CELERY_BROKER_URL = 'amqp://guest:guest@localhost' * Fix #5772 task_default_exchange & task_default_exchange_type not work (#5773) * Fix #5772 task_default_exchange & task_default_exchange_type not work * Add unit test: test_setting_default_exchange * Move default_exchange test to standalone class * Run integration suite with memcached results backend. (#5739) * Fix hanging forever when fetching results from a group(chain(group)) canvas. (#5744) PR #5739 uncovered multiple problems with the cache backend. This PR should resolve one of them. PR #5638 fixed the same test case for our async results backends that support native join. However, it did not fix the test case for sync results backends that support native join. * Fix regression in PR #5681. (#5753) See comment in the diff for details. * Grammatical fix to CONTRIBUTING.rst doc (#5794) * Fix #5734 Celery does not consider authMechanism on mongodb backend URLs (#5795) * Fix #5734 Celery does not consider authMechanism on mongodb backend URLs * Add unit test: test_get_connection_with_authmechanism * Add unit test: test_get_connection_with_authmechanism_no_username * Fix errors in Python 2.7 Remove "," after "**" operator * Revert "Revert "Revert "Added handle of SIGTERM in BaseTask in celery/task.py to prevent kill the task" (#5577)" (#5586)" (#5797) This reverts commit f79894e0a2c7156fd0ca5e8e3b652b6a46a7e8e7. * Add Python 3.8 Support (#5785) * Added Python 3.8 to the build matrix. * Ensure a supported tblib version is installed for Python 3.8 and above. In addition, modernize the relevant tests. * Workaround patching problem in test. * py 3.8 in clasifier * ubuntu bionic (#5799) * ubuntu bionic * fast finish * sync bumversion with pypi release * Dev.req (#5803) * update docker config * undo hardpin * devr req install from github master * update docker config (#5801) * update docker config * make dockerfile to install from github master dev branch by default * update download link * Isort. * Grammatical & punctuation fixes for CONTRIBUTING.rst document (#5804) * update dockerfile * switched to ubuntu bionic * update docker * keep it empty until we reconfigure it again with autopep8 * Fixed Dockerfile (#5809) * Update document CONTRIBUTING.rst & fix Dockerfile typo (#5813) * Added an issue template for minor releases. * reference gocelery Go Client/Server for Celery (#5815) * Add enterprise language (#5818) * Fix/correct minor doc typos (#5825) * Correct a small typo * Correct bad contributing documentation links * Preserve the task priority in case of a retry (#5820) * Preserve the task priority in case of a retry * Created test case for retried tasks with priority * Implement an integration test for retried tasks with priorities * bump kombu * basic changelog for celery 4.4.0rc4 * bump celery 4.4.0rc4 * events bootstep disabled if no events (#5807) * events bootstep disabled if no events * Added unit tests. * update bug report template * fixing ascii art to look nicer (#5831) * Only rerun flaky tests when failures can be intermediary. * Rename Changelog to Changelog.rst * The test_nested_group_chain test can run without native_join support. (#5838) * Run integration tests with Cassandra (#5834) * Run integration tests with Cassandra. * Configure cassandra result backend * Pre-create keyspace and table * Fix deprecation warning. * Fix path to cqlsh. * Increase connection timeout. * Wait until the cluster is available. * SQS - Reject on failure (#5843) * reject on failure * add documentation * test fix * test fix * test fix * Add a concurrency model with ThreadPoolExecutor (#5099) * Add a concurrency model with ThreadPoolExecutor * thread model test for pypy * Chain primitive's code example fix in canvas documentation (Regression PR#4444) (#5845) * Changed multi-line string (#5846) This string wasn't rendering properly and was printing the python statement too. Although the change isn't as pretty code-wise, it gets rid of an annoyance for the user. * Add auto expiry for DynamoDB backend (#5805) * Add auto expiry for DynamoDB backend This adds auto-expire support for the DynamoDB backend, via the DynamoDB Time to Live feature. * Require boto3>=1.9.178 for DynamoDB TTL support boto3 version 1.9.178 requires botocore>=1.12.178. botocore version 1.12.178 introduces support for the DynamoDB UpdateTimeToLive call. The UpdateTimeToLive call is used by the DynamoDB backend to enable TTL support on a newly created table. * Separate TTL handling from table creation Handle TTL enabling/disabling separately from the table get-or-create function. Improve handling of cases where the TTL is already set to the desired state. DynamoDB only allows a single TTL update action within a fairly long time window, so some problematic cases (changing the TTL attribute, enabling/disabling TTL when it was recently modified) will raise exceptions that have to be dealt with. * Handle older boto3 versions If the boto3 TTL methods are not found, log an informative error. If the user wants to enable TTL, raise an exception; if TTL should be disabled, simply return. * Improve logging - Handle exceptions by logging the error and re-raising - Log (level debug) when the desired TTL state is already in place * Add and use _has_ttl() convenience method Additional changes: - Handle exceptions when calling boto3's describe_time_to_live() - Fix test cases for missing TTL methods * Update ttl_seconds documentation * Log invalid TTL; catch and raise ValueError * Separate method _get_table_ttl_description * Separate ttl method validation function * Clarify tri-state TTL value * Improve test coverage * Fix minor typo in comment * Mark test as xfail when using the cache backend. (#5851) * [Fix #5436] Store extending result in all backends (#5661) * [Fix #5436] Store extending result in all backends * Fix sqlalchemy * More fixu * Fixing tests * removing not necessary import * Removing debug code * Removing debug code * Add tests for get_result_meta in base and database * Revert "Add auto expiry for DynamoDB backend (#5805)" (#5855) This reverts commit f7f5bcfceca692d0e78c742a7c09c424f53d915b. * Revert "Mark test as xfail when using the cache backend. (#5851)" (#5854) This reverts commit 1b303c2968836245aaa43c3d0ff9249dd8bf9ed2. * docs: Document Redis commands used by celery (#5853) * remove cache back end integrtion test. (#5856) * Fix a race condition when publishing a very large chord header (#5850) * Added a test case which artificially introduces a delay to group.save(). * Fix race condition by delaying the task only after saving the group. * update tox * Remove duplicate boto dependency. (#5858) * Revert "remove cache back end integrtion test. (#5856)" (#5859) This reverts commit e0ac7a19a745dd5a52a615c1330bd67f2cef4d00. * Revert "Revert "Add auto expiry for DynamoDB backend (#5805)" (#5855)" (#5857) This reverts commit 4ddc605392d7694760f23069c34ede34b3e582c3. * Revert "update tox" This reverts commit 49427f51049073e38439ea9b3413978784a24999. * Fix the test_simple_chord_with_a_delay_in_group_save test. * Revert "Revert "Skip unsupported canvas when using the cache backend"" (#5860) * Revert "Revert "Mark test as xfail when using the cache backend. (#5851)" (#5854)" This reverts commit fc101c61c1912c4dafa661981f8b865c011e8a55. * Make the xfail condition stricter. * Fix the xfail condition. * Linters should use Python 3.8. * Move pypy unit tests to the correct stage. * Temporarily allow PyPy to fail since it is unavailable in Travis. * Remove unused variables. * Fix unused imports. * Fix pydocstyle errors in dynamodb. * Fix pydocstyle errors in redis backend. * bump kombu to 4.6.7 * celery 4.4.0rc5 changelog * celery 4.4.0rc5 * rm redundant code (#5864) * isort. * Document the threads task pool in the CLI. * Removed the paragraph about using librabbitmq. Refer to #5872 (#5873) * Task class definitions can have retry attributes (#5869) * autoretry_for * retry_kwargs * retry_backoff * retry_backoff_max * retry_jitter can now be defined as cls attributes. All of these can be overriden from the @task decorator https://github.com/celery/celery/issues/4684 * whatsnew in Celery 4.4 as per projects standard (#5817) * 4.4 whatsnew * update * update * Move old whatsnew to history. * Remove old news & fix markers. * Added a section notifying Python 3.4 has been dropped. * Added a note about ElasticSearch basic auth. * Added a note about being able to replace eagerly run tasks. * Update index. * Address comment. * Described boto3 version updates. * Fix heading. * More news. * Thread pool. * Add Django and Config changes * Bump version 4.4.0 * upate readme * Update docs regarding Redis Message Priorities (#5874) * Update docs regarding Redis Message Priorities * fixup! Update docs regarding Redis Message Priorities * Update 4.4.0 docs (#5875) * Update 4.4 release changelog * Update whatsnew-4.4 * Update tasks docs * Fix recent tasks doc file update (#5879) * Include renamed Changelog.rst in source releases. (#5880) Changelog.rst was renamed from Changelog in fd023ec174bedc2dc65c63a0dc7c85e425ac00c6 but MANIFEST.in was not updated to include the new name. This fixes the file name so Changelog.rst will show up in future source releases again. * Reorganised project_urls and classifiers. (#5884) * Use safequote in SQS Getting Started doc (#5885) * Have appveyor build relevant versions of Python. (#5887) * Have appveyor build relevant and buildable versions of Python. * Appveyor is missing CI requirements to build. * Pin pycurl to version that will build with appveyor (because wheels files exist) * Restrict python 2.7 64 bit version of python-dateutil for parse. * Use is_alive instead of isAlive for Python 3.9 compatibility. (#5898) * Very minor tweak to commen to improve docs (#5900) As discussed here: https://stackoverflow.com/questions/58816271/celery-task-asyncresult-takes-task-id-but-is-documented-to-get-asyncresult-inst this comment seems to flow to a very confusing and misleading piece of documentation here: https://docs.celeryproject.org/en/latest/reference/celery.app.task.html#celery.app.task.Task.AsyncResult * Support configuring schema of a PostgreSQL database (#5910) * Support configuring schema of a PostgreSQL database * Add unit test * Remove blank line * Fix raise issue to make exception message more friendly (#5912) Signed-off-by: Chenyang Yan <memory.yancy@gmail.com> * Add progress for retry connections (#5915) This will show current retry progress so it will clear confusion about how many retries will be tried for connecting to broker. Closes #4556 * chg: change xrange to range (#5926) * update docs for json serializer and add note for int keys serialization (#5932) * fix indentation for note block in calling.rst (#5933) * Added links to other issue trackers. (#5939) * Add labels automatically for issues. (#5938) * Run pyupgrade. Co-authored-by: Michal Čihař <michal@cihar.com> Co-authored-by: ptitpoulpe <ptitpoulpe@ptitpoulpe.fr> Co-authored-by: Didi Bar-Zev <didi@hiredscore.com> Co-authored-by: Santos Solorzano <santosjavier22@gmail.com> Co-authored-by: manlix <manlix@yandex.ru> Co-authored-by: Jimmy <54828848+sckhg1367@users.noreply.github.com> Co-authored-by: Борис Верховский <boris.verk@gmail.com> Co-authored-by: Asif Saif Uddin <auvipy@gmail.com> Co-authored-by: Jainal Gosaliya <jainal09gosalia@gmail.com> Co-authored-by: gsfish <caoyu97@hotmail.com> Co-authored-by: Dipankar Achinta <di.punk.car19@gmail.com> Co-authored-by: Pengjie Song (宋鹏捷) <spengjie@sina.com> Co-authored-by: Chris Griffin <chris-griffin@users.noreply.github.com> Co-authored-by: Muhammad Hewedy <mhewedy@gmail.com> Co-authored-by: Blaine Bublitz <blaine.bublitz@gmail.com> Co-authored-by: Tamu <tamsanh@gmail.com> Co-authored-by: Erik Tews <erik@datenzone.de> Co-authored-by: abhinav nilaratna <anilaratna2@bloomberg.net> Co-authored-by: Wyatt Paul <wpaul@hearsaycorp.com> Co-authored-by: gal cohen <gal.nevis@gmail.com> Co-authored-by: as <alfred@huji.fr> Co-authored-by: Param Kapur <paramkapur2002@gmail.com> Co-authored-by: Sven Ulland <sven.ulland@gmail.com> Co-authored-by: Safwan Rahman <safwan.rahman15@gmail.com> Co-authored-by: Aissaoui Anouar <tobia@crossbone.cc> Co-authored-by: Neal Wang <qdzzyb2015@gmail.com> Co-authored-by: Alireza Amouzadeh <alireza@amouzadeh.net> Co-authored-by: Marcos Moyano <marcos@anue.biz> Co-authored-by: Stepan Henek <stepan+github@henek.name> Co-authored-by: Andrew Sklyarov <AndrewPix@users.noreply.github.com> Co-authored-by: Michael Fladischer <michael@fladi.at> Co-authored-by: Dejan Lekic <dejan.lekic@gmail.com> Co-authored-by: Yannick Schuchmann <yannick.schuchmann@googlemail.com> Co-authored-by: Matt Davis <matteius@gmail.com> Co-authored-by: Karthikeyan Singaravelan <tir.karthi@gmail.com> Co-authored-by: Bernd Wechner <bernd-wechner@users.noreply.github.com> Co-authored-by: Sören Oldag <soeren_oldag@freenet.de> Co-authored-by: uddmorningsun <memory.yancy@gmail.com> Co-authored-by: Amar Fadil <34912365+marfgold1@users.noreply.github.com> Co-authored-by: woodenrobot <woodenrobot1993@gmail.com> Co-authored-by: Sardorbek Imomaliev <sardorbek.imomaliev@gmail.com> * Remove fallback code for Python 2 support marked with TODOs. (#5953) Co-authored-by: Asif Saif Uddin <auvipy@gmail.com> * Remove PY3 conditionals (#5954) * Added integration coverage for link_error (#5373) * Added coverage for link_error. * Use pytest-rerunfailed plugin instead of rolling our own custom implementation. * Added link_error with retries. This currently fails. * Remove unused import. * Fix import on Python 2.7. * retries in link_error do not hang the worker anymore. * Run error callbacks eagerly when the task itself is run eagerly. Fixes #4899. * Adjust unit tests accordingly. * Grammar in documentation (#5780) * Grammar in documentation * Address review. * pypy 7.2 matrix (#5790) * removed extra slashes in CELERY_BROKER_URL (#5792) The Celery broker URL in settings.py had 2 slashes in the end which are not required and can be misleading. so I changed :- CELERY_BROKER_URL = 'amqp://guest:guest@localhost//' to CELERY_BROKER_URL = 'amqp://guest:guest@localhost' * Fix #5772 task_default_exchange & task_default_exchange_type not work (#5773) * Fix #5772 task_default_exchange & task_default_exchange_type not work * Add unit test: test_setting_default_exchange * Move default_exchange test to standalone class * Run integration suite with memcached results backend. (#5739) * Fix hanging forever when fetching results from a group(chain(group)) canvas. (#5744) PR #5739 uncovered multiple problems with the cache backend. This PR should resolve one of them. PR #5638 fixed the same test case for our async results backends that support native join. However, it did not fix the test case for sync results backends that support native join. * Fix regression in PR #5681. (#5753) See comment in the diff for details. * Grammatical fix to CONTRIBUTING.rst doc (#5794) * Fix #5734 Celery does not consider authMechanism on mongodb backend URLs (#5795) * Fix #5734 Celery does not consider authMechanism on mongodb backend URLs * Add unit test: test_get_connection_with_authmechanism * Add unit test: test_get_connection_with_authmechanism_no_username * Fix errors in Python 2.7 Remove "," after "**" operator * Revert "Revert "Revert "Added handle of SIGTERM in BaseTask in celery/task.py to prevent kill the task" (#5577)" (#5586)" (#5797) This reverts commit f79894e0a2c7156fd0ca5e8e3b652b6a46a7e8e7. * Add Python 3.8 Support (#5785) * Added Python 3.8 to the build matrix. * Ensure a supported tblib version is installed for Python 3.8 and above. In addition, modernize the relevant tests. * Workaround patching problem in test. * py 3.8 in clasifier * ubuntu bionic (#5799) * ubuntu bionic * fast finish * sync bumversion with pypi release * Dev.req (#5803) * update docker config * undo hardpin * devr req install from github master * update docker config (#5801) * update docker config * make dockerfile to install from github master dev branch by default * update download link * Isort. * Grammatical & punctuation fixes for CONTRIBUTING.rst document (#5804) * update dockerfile * switched to ubuntu bionic * update docker * keep it empty until we reconfigure it again with autopep8 * Fixed Dockerfile (#5809) * Update document CONTRIBUTING.rst & fix Dockerfile typo (#5813) * Added an issue template for minor releases. * reference gocelery Go Client/Server for Celery (#5815) * Add enterprise language (#5818) * Fix/correct minor doc typos (#5825) * Correct a small typo * Correct bad contributing documentation links * Preserve the task priority in case of a retry (#5820) * Preserve the task priority in case of a retry * Created test case for retried tasks with priority * Implement an integration test for retried tasks with priorities * bump kombu * basic changelog for celery 4.4.0rc4 * bump celery 4.4.0rc4 * events bootstep disabled if no events (#5807) * events bootstep disabled if no events * Added unit tests. * update bug report template * fixing ascii art to look nicer (#5831) * Only rerun flaky tests when failures can be intermediary. * Rename Changelog to Changelog.rst * The test_nested_group_chain test can run without native_join support. (#5838) * Run integration tests with Cassandra (#5834) * Run integration tests with Cassandra. * Configure cassandra result backend * Pre-create keyspace and table * Fix deprecation warning. * Fix path to cqlsh. * Increase connection timeout. * Wait until the cluster is available. * SQS - Reject on failure (#5843) * reject on failure * add documentation * test fix * test fix * test fix * Add a concurrency model with ThreadPoolExecutor (#5099) * Add a concurrency model with ThreadPoolExecutor * thread model test for pypy * Chain primitive's code example fix in canvas documentation (Regression PR#4444) (#5845) * Changed multi-line string (#5846) This string wasn't rendering properly and was printing the python statement too. Although the change isn't as pretty code-wise, it gets rid of an annoyance for the user. * Add auto expiry for DynamoDB backend (#5805) * Add auto expiry for DynamoDB backend This adds auto-expire support for the DynamoDB backend, via the DynamoDB Time to Live feature. * Require boto3>=1.9.178 for DynamoDB TTL support boto3 version 1.9.178 requires botocore>=1.12.178. botocore version 1.12.178 introduces support for the DynamoDB UpdateTimeToLive call. The UpdateTimeToLive call is used by the DynamoDB backend to enable TTL support on a newly created table. * Separate TTL handling from table creation Handle TTL enabling/disabling separately from the table get-or-create function. Improve handling of cases where the TTL is already set to the desired state. DynamoDB only allows a single TTL update action within a fairly long time window, so some problematic cases (changing the TTL attribute, enabling/disabling TTL when it was recently modified) will raise exceptions that have to be dealt with. * Handle older boto3 versions If the boto3 TTL methods are not found, log an informative error. If the user wants to enable TTL, raise an exception; if TTL should be disabled, simply return. * Improve logging - Handle exceptions by logging the error and re-raising - Log (level debug) when the desired TTL state is already in place * Add and use _has_ttl() convenience method Additional changes: - Handle exceptions when calling boto3's describe_time_to_live() - Fix test cases for missing TTL methods * Update ttl_seconds documentation * Log invalid TTL; catch and raise ValueError * Separate method _get_table_ttl_description * Separate ttl method validation function * Clarify tri-state TTL value * Improve test coverage * Fix minor typo in comment * Mark test as xfail when using the cache backend. (#5851) * [Fix #5436] Store extending result in all backends (#5661) * [Fix #5436] Store extending result in all backends * Fix sqlalchemy * More fixu * Fixing tests * removing not necessary import * Removing debug code * Removing debug code * Add tests for get_result_meta in base and database * Revert "Add auto expiry for DynamoDB backend (#5805)" (#5855) This reverts commit f7f5bcfceca692d0e78c742a7c09c424f53d915b. * Revert "Mark test as xfail when using the cache backend. (#5851)" (#5854) This reverts commit 1b303c2968836245aaa43c3d0ff9249dd8bf9ed2. * docs: Document Redis commands used by celery (#5853) * remove cache back end integrtion test. (#5856) * Fix a race condition when publishing a very large chord header (#5850) * Added a test case which artificially introduces a delay to group.save(). * Fix race condition by delaying the task only after saving the group. * update tox * Remove duplicate boto dependency. (#5858) * Revert "remove cache back end integrtion test. (#5856)" (#5859) This reverts commit e0ac7a19a745dd5a52a615c1330bd67f2cef4d00. * Revert "Revert "Add auto expiry for DynamoDB backend (#5805)" (#5855)" (#5857) This reverts commit 4ddc605392d7694760f23069c34ede34b3e582c3. * Revert "update tox" This reverts commit 49427f51049073e38439ea9b3413978784a24999. * Fix the test_simple_chord_with_a_delay_in_group_save test. * Revert "Revert "Skip unsupported canvas when using the cache backend"" (#5860) * Revert "Revert "Mark test as xfail when using the cache backend. (#5851)" (#5854)" This reverts commit fc101c61c1912c4dafa661981f8b865c011e8a55. * Make the xfail condition stricter. * Fix the xfail condition. * Linters should use Python 3.8. * Move pypy unit tests to the correct stage. * Temporarily allow PyPy to fail since it is unavailable in Travis. * Remove unused variables. * Fix unused imports. * Fix pydocstyle errors in dynamodb. * Fix pydocstyle errors in redis backend. * bump kombu to 4.6.7 * celery 4.4.0rc5 changelog * celery 4.4.0rc5 * rm redundant code (#5864) * isort. * Document the threads task pool in the CLI. * Removed the paragraph about using librabbitmq. Refer to #5872 (#5873) * Task class definitions can have retry attributes (#5869) * autoretry_for * retry_kwargs * retry_backoff * retry_backoff_max * retry_jitter can now be defined as cls attributes. All of these can be overriden from the @task decorator https://github.com/celery/celery/issues/4684 * whatsnew in Celery 4.4 as per projects standard (#5817) * 4.4 whatsnew * update * update * Move old whatsnew to history. * Remove old news & fix markers. * Added a section notifying Python 3.4 has been dropped. * Added a note about ElasticSearch basic auth. * Added a note about being able to replace eagerly run tasks. * Update index. * Address comment. * Described boto3 version updates. * Fix heading. * More news. * Thread pool. * Add Django and Config changes * Bump version 4.4.0 * upate readme * Update docs regarding Redis Message Priorities (#5874) * Update docs regarding Redis Message Priorities * fixup! Update docs regarding Redis Message Priorities * Update 4.4.0 docs (#5875) * Update 4.4 release changelog * Update whatsnew-4.4 * Update tasks docs * Fix recent tasks doc file update (#5879) * Include renamed Changelog.rst in source releases. (#5880) Changelog.rst was renamed from Changelog in fd023ec174bedc2dc65c63a0dc7c85e425ac00c6 but MANIFEST.in was not updated to include the new name. This fixes the file name so Changelog.rst will show up in future source releases again. * Reorganised project_urls and classifiers. (#5884) * Use safequote in SQS Getting Started doc (#5885) * Have appveyor build relevant versions of Python. (#5887) * Have appveyor build relevant and buildable versions of Python. * Appveyor is missing CI requirements to build. * Pin pycurl to version that will build with appveyor (because wheels files exist) * Restrict python 2.7 64 bit version of python-dateutil for parse. * Use is_alive instead of isAlive for Python 3.9 compatibility. (#5898) * Very minor tweak to commen to improve docs (#5900) As discussed here: https://stackoverflow.com/questions/58816271/celery-task-asyncresult-takes-task-id-but-is-documented-to-get-asyncresult-inst this comment seems to flow to a very confusing and misleading piece of documentation here: https://docs.celeryproject.org/en/latest/reference/celery.app.task.html#celery.app.task.Task.AsyncResult * Support configuring schema of a PostgreSQL database (#5910) * Support configuring schema of a PostgreSQL database * Add unit test * Remove blank line * Fix raise issue to make exception message more friendly (#5912) Signed-off-by: Chenyang Yan <memory.yancy@gmail.com> * Add progress for retry connections (#5915) This will show current retry progress so it will clear confusion about how many retries will be tried for connecting to broker. Closes #4556 * chg: change xrange to range (#5926) * update docs for json serializer and add note for int keys serialization (#5932) * fix indentation for note block in calling.rst (#5933) * Added links to other issue trackers. (#5939) * Add labels automatically for issues. (#5938) * remove redundant raise from docstring (#5941) `throw` is True by default so the Retry exception will already get raised by calling `self.retry(countdown=60 * 5, exc=exc)` * Run pyupgrade. * Fix typo (#5943) * Remove fallback code for Python 2 support. * docs: fixes Rabbits and Warrens link in routing userguide (#4007) (#5949) * Fix labels on Github issue templates. (#5955) Use quotation marks to escape labels on Github issue templates. This prevents the colon from breaking the template. * added retry_on_timeout and socket_keepalive to config and doc (#5952) * Fixed event capture from building infinite list (#5870) * Fix error propagation example (#5966) * update range (#5971) * update setup.cfg * bump billiard to 3.6.3.0 * Update __init__.py (#5951) * Update __init__.py Fixed issue for object with result_backend=True (decode fails on multiple None request) * Update __init__.py suggested changeds * Update __init__.py * Use configured db schema also for sequences (#5972) * Added a default value for retries in worker.strategy. (#5945) * Added a default value for retries in worker.strategy. I was facing an issue when adding tasks directly to rabbitmq using pika instead of calling task.apply_async. The issue was the self.retry mechanisum was failing. In app/tasks.py the line `retries = request.retries + 1` was causing the issue. On further tracing I figured out that it was because the default .get value (None) was getting passed through this function and was raising TypeError: unsupported operand type(s) for +: 'NoneType' and 'int' * Add test cases for default and custom retries value * pypy 7.3 (#5980) * Pass `interval` to `get_many` (#5931) * Pass `interval` to `get_many` * Fix: Syntax error for py2.7 * Fix: Syntax error for py2.7 * Fixed problem with conflicting autoretry_for task parameter and Task.replace() (#5934) * Fix #5917 (#5918) * Fix changelog (#5881) * merge in place the apps beat schedule in the default Schedule class. (#5908) * Handle Redis connection errors in result consumer (#5921) * Handle Redis connection errors in result consumer * Closes #5919. * Use context manager for Redis conusmer reconnect * Log error when result backend reconnection fails * Fix inspect_command documentation (#5983) * Use gevent and eventlet wait() functions to remove busy-wait (#5974) * Use gevent and eventlet wait() functions to remove busy-wait Fixes issue #4999. Calling AsyncResult.get() in a gevent context would cause the async Drainer to repeatedly call wait_for until the result was completed. I've updated the code to have a specific implementation for gevent and eventlet that will cause wait_for to only return every "timeout" # of seconds, rather than repeatedly returning. Some things I'd like some feedback on: * Where's the best place to add test coverage for this? It doesn't look like there are any tests that directly exercised the Drainer yet so I would probably look to add some of these to the backends/ unit tests. * The way I did this for the Eventlet interface was to rely on the private _exit_event member of the GreenThread instance; to do this without relying on a private member would require some additional changes to the backend Drainer interface so that we could wait for an eventlet-specific event in wait_for(). I can do this, just wanted to get some feedback before. * Add unit tests for Drainer classes In order for this to work without monkeypatching in the tests, I needed to call sleep(0) to let the gevent/eventlet greenlets to yield control back to the calling thread. I also made the check interval configurable in the drainer so that we didn't need to sleep multiples of 1 second in the tests. * Weaken asserts since they don't pass on CI * Fix eventlet auto-patching DNS resolver module on import By default it looks like "import eventlet" imports the greendns module unless the environment EVENTLET_NO_GREENDNS is set to true. This broke a pymongo test. * Add tests ensuring that the greenlet loop isn't blocked These tests make sure that while drain_events_until is running that other gevent/eventlet concurrency can run. * Clean up tests and make sure they wait for all the threads to stop * Fix chords with chained groups (#5947) * kombu 4.6.8 * update setup * updated version 4.4.1 * Fix: Accept and swallow `kwargs` to handle unexpected keyword arguments * Allow boto to look for credentials in S3Backend * add reference to Rusty Celery * Update document of revoke method in Control class * Fix copy-paste error in result_compression docs * Make 'socket_keepalive' optional variable (#6000) * update connection params - socket_keepalive is optional now * update readme - added versionadded 4.4.1 and fixed `redis_socket_keepalive` * added check of socket_keepalive in arguments for UnixSocketConnect * Fixed incorrect setting name in documentation (#6002) * updated version 4.4.2 * Fix backend utf-8 encoding in s3 backend Celery backend uses utf-8 to deserialize results, which would fail for some serializations like pickle. * Fix typo in celery.bin.multi document * Upgraded pycurl to the latest version that supports wheel. * pytest 5.3.5 max * Add uptime to the stats inspect command * Doc tweaks: mostly grammar and punctuation (#6016) * Fix a bunch of comma splices in the docs * Remove some unnecessary words from next-steps doc * Tweak awkward wording; fix bad em-dash * Fix a bunch more comma splices in next-steps doc * Miscellaneous grammar/punctuation/wording fixes * Link to task options in task decorator docs * Fixing issue #6019: unable to use mysql SSL parameters when getting mysql engine (#6020) * Fixing issue #6019: unable to use mysql SSL parametes in create_engine() * adding test for get_engine when self.forked is False and engine args are passed in for create_engine() * Clean TraceBack to reduce memory leaks for exception task (#6024) * Clean TraceBack to reduce memory leaks * add unit test * add unit test * reject unittest * Patch For Python 2.7 compatibility * update unittest * Register to the garbage collector by explicitly referring to f_locals. * need more check * update code coverage * update Missing unit test * 3.4 -> 3.5 Co-authored-by: heedong.jung <heedong.jung@samsung.com> * exceptions: NotRegistered: fix up language Minor fix to the language. * Note about autodiscover_tasks and periodic tasks This is particularly important for Django projects that put periodic tasks into each app's `tasks.py` and want to use one as a periodic task. By the time `autodiscover_tasks()` loads those tasks, the `on_after_configure` Signal has already come and gone, so anything decorated with `@app.on_after_finalize.connect` will never be called. If there's other documentation on this subject, I could not find it. * Avoid PyModulelevel, deprecated in Sphinx 4 Use `PyFunction` instead of `PyModulelevel` to avoid this deprecation warning: RemovedInSphinx40Warning: PyModulelevel is deprecated. Please check the implementation of <class 'celery.contrib.sphinx.TaskDirective'> This replacement is one of the options listed in the Sphinx docs (https://www.sphinx-doc.org/en/master/extdev/deprecated.html). * Give up sending a worker-offline message if transport is not connected (#6039) * If worker-offline event fails to send, give up and die peacefully * Add test for retry= and msgs in heartbeat * Fix the build and all documentation warnings. I finally upgraded our theme to 2.0. As a result we've upgraded Sphinx to 2.0. Work to upgrade Sphinx to 3.0 will proceed in a different PR. This upgrade also fixes our build issues caused by #6032. We don't support Sphinx 1.x as a result of that patch. I've also included the missing 4.3 changelog to our history. * Support both Sphinx 2 and 3. * Add Task to __all__ in celery.__init__.py * Add missing parenthesis to example in docs * Ensure a single chain object in a chain does not raise MaximumRecursionError. Previously chain([chain(sig)]) would crash. We now ensure it doesn't. Fixes #5973. * update setup.py * fix typo missing quote at the end of line * Fix a typo in monitoring doc * update travis * update ubuntu to focal foss 20.04 LTS * Fix autoscale when prefetch_multiplier is 1 * Allow start_worker to function without ping task * Update celeryd.conf Move the directory of the program before the execution of the command/script * Add documentation for "predefined_queue_urls" * [Fix #6074]: Add missing documentation for MongoDB as result backend. * update funding * 🐛 Correctly handle configuring the serializer for always_eager mode. (#6079) * 🐛 Correctly handle configuring the serializer for always_eager mode. options['serializer'] will always exist, because it is initialized from an mattrgetter. Even if unset, it will be present in the options with a value of None. * 🐛 Add a test for new always_eager + task_serializer behavior. * ✏️ Whoops missed a : * Remove doubling of prefetch_count increase when prefetch_multiplier gt 1 (#6081) * try ubuntu focal (#6088) * Fix eager function not returning result after retries. Using apply function does not return correct results after at least one retry because the return value of successive call is not going back to the original caller. * return retry result if not throw and is_eager if throw is false, we would be interested by the result of retry and not the current result which will be an exception. This way it does not break the logic of `raise self.retry` This should be used like `return self.retry(..., throw=False)` in an except statement. * revert formatting change * Add tests for eager retry without throw * update predefined-queues documentation Suggested version of configuration does not work. Additionally I'd like to mention, that `access_key_id` and `secret_access_key` are mandatory fields and not allowing you to go with defaults AWS_* env variables. I can contribute for this variables to be optional Also I'm not sure if security token will apply, could you please advice how to do it? * Fix couchbase version < 3.0.0 as API changed * Remove reference to -O fair in optimizations -O fair was made the default in Celery 4.0 https://docs.celeryproject.org/en/stable/history/whatsnew-4.0.html#ofair-is-now-the-default-scheduling-strategy * pytest ranges * pypy3 * revert to bionic * do not load docs.txt requirements for python 2.7 As it requires Sphinx >= 2.0.0 and there is no such version compatible with python 2.7 * update cassandra travis integration test configuration cassandra:latest docker image changed location of cqlsh program * pin cassandra-driver CI get stuck after all cassandra integration tests * Fix all flake8 lint errors * Fix all pydocstyle lint errors * Fix all configcheck lint errors * Always requeue while worker lost regardless of the redelivered flag (#6103) * #5598 fix, always redelivery while WorkerLostError * fix, change the requeue flag so the task will remain PENDING * Allow relative paths in the filesystem backend (#6070) * Allow relative paths in the filesystem backend * fix order of if statements * [Fixed Issue #6017] --> Added Multi default logfiles and pidfiles paths [Description]: --> Changed the default paths for log files & pid files to be '/var/log/celery' and '/var/run/celery' --> Handled by creating the respective paths if not exist. --> Used os.makedir(path,if_exists=True) [Unit Test Added]: --> .travis.yml - config updated with 'before install'. --> t/unit/apps/test_multi.py - Changed the default log files & pid files paths wherever required. * Avoid race condition due to task duplication. In some circumstances like a network partitioning, some tasks might be duplicated. Sometimes, this lead to a race condition where a lost task overwrites the result of the last successful task in the backend. In order to avoid this race condition we prevent updating the result if it's already in successful state. This fix has been done for KV backends only and therefore won't work with other backends. * adding tests * Exceptions must be old-style classes or derived from BaseException, but here self.result may not subclass of BaseException. * update fund link * Fix windows build (#6104) * do not load memcache nor couchbase lib during windows build those libraries depends on native libraries libcouchbase and libmemcached that are not installed on Appveyor. As only unit tests runs on Appveyor, it should be fine * Add python 3.8 workaround for app trap * skip tests file_descriptor_safety tests on windows AsyncPool is not supported on Windows so Pool does have _fileno_to_outq attribute, making the test fail * Fix crossplatform log and pid files in multi mode it relates to #6017 * Use tox to build and test on windows * remove tox_install_command * drop python 2.7 from windows build * Add encode to meta task in base.py (#5894) * Add encode to base.py meta result Fix bug with impossibility to load None from task meta * Add tests for None. Remove exceed encode. * Update base.py Add return payload if None * Update time.py to solve the microsecond issues (#5199) When `relative` is set to True, the day, hour, minutes second will be round to the nearest one, however, the original program do not update the microsecond (reset it). As a result, the run-time offset on the microsecond will then be accumulated. For example, given the interval is 15s and relative is set to True 1. 2018-11-27T15:01:30.123236+08:00 2. 2018-11-27T15:01:45.372687+08:00 3. 2018-11-27T15:02:00.712601+08:00 4. 2018-11-27T15:02:15.987720+08:00 5. 2018-11-27T15:02:31.023670+08:00 * Change backend _ensure_not_eager error to warning * Add priority support for 'celery.chord_unlock' task (#5766) * Change eager retry behaviour even with raise self.retry, it should return the eventual value or MaxRetriesExceededError. if return value of eager apply is Retry exception, retry eagerly the task signature * Order supported Python versions * Avoid race condition in elasticsearch backend if a task is retried, the task retry may work concurrently to current task. store_result may come out of order. it may cause a non ready state (Retry) to override a ready state (Success, Failure). If this happens, it will block indefinitely pending any chord depending on this task. this change makes document updates safe for concurrent writes. https://www.elastic.co/guide/en/elasticsearch/reference/current/optimistic-concurrency-control.html * backends base get_many pass READY_STATES arg * test backends base get_many pass READY_STATES arg * Add integration tests for Elasticsearch and fix _update * Revert "revert to bionic" This reverts commit 6e091573f2ab0d0989b8d7c26b677c80377c1721. * remove jython check * feat(backend): Adds cleanup to ArangoDB backend * Delete Document Known Issue with CONN_MAX_AGE in 4.3 * issue 6108 fix filesystem backend cannot not be serialized by picked (#6120) * issue 6108 fix filesystem backend cannot not be serialized by picked https://github.com/celery/celery/issues/6108 * issue-6108 fix unit test failure * issue-6108 fix flake8 warning Co-authored-by: Murphy Meng <mmeng@mirriad.com> * kombu==4.6.9 (#6133) * changelog for 4.4.3 * v 4.4.3 * remove un supported classifier * Fix autoretry_for with explicit retry (#6138) * Add tests for eager task retry * Fixes #6135 If autoretry_for is set too broad on Exception, then autoretry may get a Retry if that's the case, rethrow directly instead of wrapping it in another Retry to avoid loosing new args * Use Django DB max age connection setting (fixes #4116) * Add retry on recoverable exception for the backend (#6122) * Add state to KeyValueStoreBackend.set method This way, a backend implementation is able to take decisions based on current state to store meta in case of failures. * Add retry on recoverable exception for the backend acks.late makes celery acknowledge messages only after processing and storing result on the backend. However, in case of backend unreachable, it will shadow a Retry exception and put the task as failed in the backend not retrying the task and acknoledging it on the broker. With this new result_backend_always_retry setting, if the backend exception is recoverable (to be defined per backend implementation), it will retry the backend operation with an exponential backoff. * Make elasticsearch backward compatible with 6.x * Make ES retry storing updates in a better way if existing value in the backend is success, then do nothing. if it is a ready status, then update it only if new value is a ready status as well. else update it. This way, a SUCCESS cannot be overriden so that we do not loose results but any ready state other than success (FAILURE, REVOKED) can be overriden by another ready status (i.e. a SUCCESS) * Add test for value not found in ES backend * Fix random distribution of jitter for exponential backoff random.randrange should be called with the actual so that all numbers have equivalent probability, otherwise maximum value does have a way higher probability of occuring. * fix unit test if extra modules are not present * ElasticSearch: add setting to save meta as json * fix #6136. celery 4.4.3 always trying create /var/run/celery directory (#6142) * fix #6136. celery 4.4.3 always trying create /var/run/celery directory, even if it's not needed. * fix #6136. cleanup * Add task_internal_error signal (#6049) * Add internal_error signal There is no special signal for an out of body error which can be the result of a bad result backend. * Fix syntax error. * Document the task_internal_error signal. Co-authored-by: Laurentiu Dragan <ldragan@bloomberg.net> * changelog for v4.4.4 * kombu 4.6.10 (#6144) * v4.4.4 * Add missing dependency on future (#6146) Fixes #6145 * ElasticSearch: Retry index if document was deleted between index and update (#6140) * ElasticSearch: Retry index if document was deleted between index and update * Elasticsearch increase coverage to 100% * Fix pydocstyle * Specify minimum version of Sphinx for Celery extension (#6150) The Sphinx extension requires Sphinx 2 or later due to #6032. * fix windows build * fix flake8 error * fix multi tests in local Mock os.mkdir and os.makedirs to avoid creating /var/run/celery and /var/log/celery during unit tests if run without root priviledges * Customize the retry interval of chord_unlock tasks * changelog v4.4.5 * v4.4.5 * Fix typo in comment. * Remove autoscale force_scale methods (#6085) * Remove autoscale force_scale methods * Remove unused variable in test * Pass ping destination to request The destination argument worked fine from CLI but didn't get used when calling ping from Python. * Fix autoscale test * chord: merge init options with run options * put back KeyValueStoreBackend.set method without state It turns out it was breaking some other projects. wrapping set method with _set_with_state, this way it will not break existing Backend. while enabling this feature for other Backend. Currently, only ElasticsearchBackend supports this feature. It protects concurrent update to corrupt state in the backend. Existing success cannot be overriden, nor a ready state by a non ready state. i.e. a Retry state cannot override a Success or Failure. As a result, chord_unlock task will not loop forever due to missing ready state on the backend. * added --range-prefix option to `celery multi` (#6180) * added --range-prefix option to `celery multi` Added option for overriding default range prefix when running multiple workers prividing range with `celery multy` command. * covered multi --range-prefix with tests * fixed --range-prefix test * Added as_list function to AsyncResult class (#6179) * Add as_list method to return task IDs as a list * Add a test for as_list method * Add docstring for as_list method * Fix CassandraBackend error in threads or gevent pool (#6147) * Fix CassandraBackend error in threads or gevent pool * remove CassandraBackend.process_cleanup * Add test case * Add test case * Add comments test_as_uri Co-authored-by: baixue <baixue@wecash.net> * changelog for v4.4.6 * v4.4.6 * Update Wiki link in "resources" In the page linked below, the link to wiki is outdated. Fixed that. https://docs.celeryproject.org/en/stable/getting-started/resources.html * test_canvas: Add test for chord-in-chain Add test case for the issue where a chord in a chain does not work when using .apply(). This works fine with .apply_async(). * Trying to fix flaky tests in ci * fix pydocstyle errors * fix pydocstyle * Drainer tests, put a lower constraint on number of intervals liveness should iterate 10 times per interval while drain_events only once. However, as it may use thread that may be scheduled out of order, we may end up in some situation where liveness and drain_events were called the same amount of time. Lowering the constraint from < to <= to avoid failing the tests. * pyupgrade. * Fix merge error. Co-authored-by: Борис Верховский <boris.verk@gmail.com> Co-authored-by: Asif Saif Uddin <auvipy@gmail.com> Co-authored-by: Jainal Gosaliya <jainal09gosalia@gmail.com> Co-authored-by: gsfish <root@grassfish.net> Co-authored-by: Dipankar Achinta <di.punk.car19@gmail.com> Co-authored-by: spengjie <spengjie@sina.com> Co-authored-by: Chris Griffin <chris-griffin@users.noreply.github.com> Co-authored-by: Muhammad Hewedy <mhewedy@gmail.com> Co-authored-by: Blaine Bublitz <blaine.bublitz@gmail.com> Co-authored-by: Tamu <tamsanh@gmail.com> Co-authored-by: Erik Tews <erik@datenzone.de> Co-authored-by: abhinav nilaratna <anilaratna2@bloomberg.net> Co-authored-by: Wyatt Paul <wpaul@hearsaycorp.com> Co-authored-by: gal cohen <gal.nevis@gmail.com> Co-authored-by: whuji <alfred@huji.fr> Co-authored-by: Param Kapur <paramkapur2002@gmail.com> Co-authored-by: Sven Ulland <sven.ulland@gmail.com> Co-authored-by: Safwan Rahman <safwan.rahman15@gmail.com> Co-authored-by: Aissaoui Anouar <tobia@crossbone.cc> Co-authored-by: Neal Wang <qdzzyb2015@gmail.com> Co-authored-by: Alireza Amouzadeh <alireza@amouzadeh.net> Co-authored-by: Marcos Moyano <marcos@anue.biz> Co-authored-by: Stepan Henek <stepan+github@henek.name> Co-authored-by: Andrew Sklyarov <AndrewPix@users.noreply.github.com> Co-authored-by: Michael Fladischer <michael@fladi.at> Co-authored-by: Dejan Lekic <dejan.lekic@gmail.com> Co-authored-by: Yannick Schuchmann <yannick.schuchmann@googlemail.com> Co-authored-by: Matt Davis <matteius@gmail.com> Co-authored-by: Xtreak <tir.karthi@gmail.com> Co-authored-by: Bernd Wechner <bernd-wechner@users.noreply.github.com> Co-authored-by: Sören Oldag <soeren_oldag@freenet.de> Co-authored-by: uddmorningsun <memory.yancy@gmail.com> Co-authored-by: Amar Fadil <34912365+marfgold1@users.noreply.github.com> Co-authored-by: woodenrobot <woodenrobot1993@gmail.com> Co-authored-by: Sardorbek Imomaliev <sardorbek.imomaliev@gmail.com> Co-authored-by: Alex Riina <alex.riina@gmail.com> Co-authored-by: Joon Hwan 김준환 <xncbf12@gmail.com> Co-authored-by: Prabakaran Kumaresshan <k_prabakaran+github@hotmail.com> Co-authored-by: Martey Dodoo <martey@mobolic.com> Co-authored-by: Konstantin Seleznev <4374093+Seleznev-nvkz@users.noreply.github.com> Co-authored-by: Prodge <Prodge@users.noreply.github.com> Co-authored-by: Abdelhadi Dyouri <raryat@gmail.com> Co-authored-by: Ixiodor <Ixiodor@users.noreply.github.com> Co-authored-by: abhishekakamai <47558404+abhishekakamai@users.noreply.github.com> Co-authored-by: Allan Lei <allanlei@helveticode.com> Co-authored-by: M1ha Shvn <work_shvein_mihail@mail.ru> Co-authored-by: Salih Caglar Ispirli <caglarispirli@gmail.com> Co-authored-by: Micha Moskovic <michamos@gmail.com> Co-authored-by: Chris Burr <chrisburr@users.noreply.github.com> Co-authored-by: Dave King <tildedave@gmail.com> Co-authored-by: Dmitry Nikulin <v.dmitry.nikulin@gmail.com> Co-authored-by: Michael Gaddis <mgaddis@ancestry.com> Co-authored-by: epwalsh <epwalsh10@gmail.com> Co-authored-by: TalRoni <tal.inon.16@gmail.com> Co-authored-by: Leo Singer <leo.singer@ligo.org> Co-authored-by: Stephen Tomkinson <neonbunny@users.noreply.github.com> Co-authored-by: Abhishek <abhishek.shukla@xeneta.com> Co-authored-by: theirix <theirix@gmail.com> Co-authored-by: yukihira1992 <hirayama@cancerscan.jp> Co-authored-by: jpays <jerome.pays@cnp.fr> Co-authored-by: Greg Ward <greg@gerg.ca> Co-authored-by: Alexa Griffith <agriffith@bluecore.com> Co-authored-by: heedong <63043496+heedong-jung@users.noreply.github.com> Co-authored-by: heedong.jung <heedong.jung@samsung.com> Co-authored-by: Shreyansh Khajanchi <shreyanshk@users.noreply.github.com> Co-authored-by: Sam Thompson <georgedorn@users.noreply.github.com> Co-authored-by: Alphadelta14 <alpha@pokesplash.net> Co-authored-by: Azimjon Pulatov <azimjohn@yahoo.com> Co-authored-by: ysde <ysde108@gmail.com> Co-authored-by: AmirMohammad Ziaei <amir_zia@outlook.com> Co-authored-by: Ben Nadler <ben.nadler@gusto.com> Co-authored-by: Harald Nezbeda <hn@nezhar.com> Co-authored-by: Chris Frisina <github@specialorange.org> Co-authored-by: Adam Eijdenberg <adam.eijdenberg@digital.gov.au> Co-authored-by: rafaelreuber <rafaelreuber@gmail.com> Co-authored-by: Noah Kantrowitz <noah@coderanger.net> Co-authored-by: Ben Nadler <jbennadler@gmail.com> Co-authored-by: Clement Michaud <c.michaud@criteo.com> Co-authored-by: Mathieu Chataigner <m.chataigner@criteo.com> Co-authored-by: eugeneyalansky <65346459+eugeneyalansky@users.noreply.github.com> Co-authored-by: Leonard Lu <leonard@socialcodeinc.com> Co-authored-by: XinYang <yangxinhust@hotmail.com> Co-authored-by: Ingolf Becker <ingolf.becker@googlemail.com> Co-authored-by: Anuj Chauhan <an12ch98@gmail.com> Co-authored-by: shaoziwei <ziwei.shao@cloudchef.io> Co-authored-by: Mathieu Chataigner <mathieu.chataigner@gmail.com> Co-authored-by: Anakael <shemeldima2@yandex.ru> Co-authored-by: Danny Chan <danny.chan@c-k.dev> Co-authored-by: Sebastiaan ten Pas <sebastiaan@diggimedia.nl> Co-authored-by: David TILLOY <d.tilloy@criteo.com> Co-authored-by: Anthony N. Simon <anthonynsimon@users.noreply.github.com> Co-authored-by: lironhl <liron.lavy@gmail.com> Co-authored-by: Raphael Cohen <raphael.cohen@sekoia.fr> Co-authored-by: JaeyoungHeo <jay.jaeyoung@gmail.com> Co-authored-by: singlaive <singlaive@gmail.com> Co-authored-by: Murphy Meng <mmeng@mirriad.com> Co-authored-by: Wu Haotian <whtsky@gmail.com> Co-authored-by: Kwist <velnik@gmail.com> Co-authored-by: Laurentiu Dragan <ldragan@bloomberg.net> Co-authored-by: Michal Čihař <michal@cihar.com> Co-authored-by: Radim Sückr <radim.suckr@gmail.com> Co-authored-by: Artem Vasilyev <artem.v.vasilyev@gmail.com> Co-authored-by: kakakikikeke-fork <kakakikikeke_new@yahoo.co.jp> Co-authored-by: Pysaoke <pysaoke@gmail.com> Co-authored-by: baixue <baixue@wecash.net> Co-authored-by: Prashant Sinha <prashantsinha@outlook.com> Co-authored-by: AbdealiJK <abdealikothari@gmail.com> * Remove Python 2 compatibility code from Celery (#6221) * Remove five from celery/__init__.py * Remove five from celery/beat.py * Remove five from celery/bootsteps.py * Remove five from celery/exceptions.py * Remove five from celery/local.py * Remove five from celery/platforms.py * Remove five from celery/result.py * Remove five from celery/schedules.py * Remove five from celery/app/amqp.py * Remove five from celery/app/annotations.py * Remove five from celery/app/backends.py * Remove five from celery/app/base.py * Remove five from celery/app/control.py * Remove five from celery/app/defaults.py * Remove five from celery/app/log.py * Remove five from celery/app/registry.py * Remove five from celery/app/routes.py * Remove five from celery/app/task.py * Remove five from celery/app/trace.py * Remove five from celery/app/utils.py * Remove five from celery/apps/beat.py * Remove five from celery/apps/multi.py * Remove five from celery/apps/worker.py * Remove five from celery/backends/database/__init__.py * Remove five from celery/backends/amqp.py * Remove five from celery/backends/asynchronous.py * Remove five from celery/backends/base.py * Remove five from celery/backends/dynamodb.py * Remove five from celery/backends/elasticsearch.py * Remove five from celery/backends/mongodb.py * Remove five from celery/backends/redis.py * Remove five from celery/backends/rpc.py * Remove five from celery/concurrency/asynpool.py * Remove five from celery/concurrency/base.py * Remove five from celery/concurrency/prefork.py * Remove five from celery/contrib/testing/manager.py * Remove five from celery/contrib/migrate.py * Remove five from celery/contrib/rdb.py * Remove five from celery/events/cursesmon.py * Remove five from celery/events/dispatcher.py * Remove five from celery/events/state.py * Remove five from celery/loaders/base.py * Remove five from celery/security/certificate.py * Remove five from celery/security/utils.py * Remove five from celery/task/base.py * Remove five from celery/utils/dispatch/signal.py * Remove five from celery/utils/abstract.py * Remove five from celery/utils/collections.py * Remove five from celery/utils/debug.py * Remove five from celery/utils/functional.py * Remove five from celery/utils/graph.py * Remove five from celery/utils/imports.py * Remove five from celery/utils/log.py * Remove five from celery/utils/saferepr.py * Remove five from celery/utils/serialization.py * Remove five from celery/utils/term.py * Remove five from celery/utils/text.py * Remove five from celery/utils/threads.py * Remove five from celery/utils/time.py * Remove five from celery/utils/timer2.py * Remove five from celery/consumer/consumer.py * Remove five from celery/consumer/gossip.py * Remove five from celery/consumer/mingle.py * Remove five from celery/worker/autoscale.py * Remove five from celery/worker/components.py * Remove five from celery/worker/control.py * Remove five from celery/worker/request.py * Remove five from celery/worker/state.py * Remove five from celery/worker/worker.py * Remove five from celery/t/benchmarks/bench_worker.py * Remove five from celery/t/integration/test_canvas.py * Remove five from celery/t/unit/app * Remove five from celery/t/unit/backends * Remove five from celery/t/unit/compat_modules * Remove five from celery/t/unit/concurrency * Remove five from celery/t/unit/contrib * Remove five from celery/t/unit/events * Remove five from celery/t/unit/security * Remove five from celery/t/unit/tasks * Remove five from celery/t/unit/utils * Remove five from celery/t/unit/worker * Sort imports. * Comment out PyPy for now. * Remove flakeplus. * Happify linter. * Fix merge problems. * Delete backport. * Remove unused import. * Remove logic that notifies user that the Python version isn't supported from setup.py. pip already does that for us. * Add a trove classifier to indicate Celery only supports Python 3. * Restore usage of `reraise` for consistency with the kombu port. * Drop Python 2 compatibility code from our Sphinx extension. * Remove mention of flakeplus from tox.ini. * Remove mention of flakeplus from our CONTRIBUTING guide. * Bump Sphinx requirement. * Remove Python 2 compatibility code from our custom Sphinx extension. * Resolve Sphinx warning due to removed section in 32ff7b45aa3d78aedca61b6554a9db39122924fd. * Remove pydocstyle from build matrix as it was removed from master. See #6278. * Bump version: 4.4.7 → 5.0.0-alpha1 * Final touches. * Fix README. * Bump Kombu to 5.0.0. * Bump version: 5.0.0-alpha1 → 5.0.0a2 * Fix wrong version. * Remove autodoc for removed module. * Remove documentation for removed methods. * Remove the riak backend since riak is no longer maintained. * Remove riak backend since riak is no longer maintained. * Start fresh. * Added all arguments for the celery worker command. Still needs more documentation and improvements... * Load the application and execute a worker. * Added the rest of the global options. If an app is not specified we now use the default app. In addition, we now exit with the correct status code. * Extract validation into parameter types. * Restructure and document. * Allow to pass worker configuration options from command line. * Implement the beat command. * Allow to configure celery options through the CLI. * Implement the `celery call` command. * Implement the `celery list bindings` command. * Implement the `celery purge` command. * Implement the `celery result` command. * Implement the `celery migrate` task. * Implemented the celery@thedrow: OK 1 node online. command. * Take --no-color in consideration when outputting to stdout. * Ensure `celery worker` takes `--no-color` into consideration. * Use the preformatted OK string. * Adopt the NO_COLOR standard. See https://no-color.org/ for details. * Split commands into separate files. * Added 'did you mean' messages. * Implement the `celery events` command. * Text style should take --no-color into consideration as well. * Implement the basic `celery inspect` command. * Improve UI. * Organize the code. * Implement the `celery graph bootsteps` command. * Implement the `celery graph workers` command. * Implement the `celery upgrade settings` command. * Implement the `celery report` command. * Delete former unit tests. * Implement the `celery logtool` command. * Pass the quiet argument to the CLI context. * Limit inspect to existing actions. * Implement the `celery control` command. * Basic scaffold for the `celery amqp` shell command. * Start implementing the shell commands. * Implement basic.publish and basic.get. * Echo OK after acknowledgement. * Reformat Code. * Implement the exchange.declare command. * Implement the exchange.delete command. * Implement the queue.bind command. * Implement the queue.declare command. * Implement the queue.delete command. * Echo queue.declare result to screen. * Echo queue.delete result to screen. * Implement the queue.purge command. * Fix color support for error(). * Report errors and continue. * Handle connection errors and reconnect on error. * Refactor. * Implement the `celery shell` command. * Isort. * Add documentation. * Correct argument types. * Implement detach for `celery worker`. * Documentation. * Implement detach for `celery beat`. * Documentation. * Implement the `celery multi` command. * Documentation. * Implement user options. * Collect command actions from the correct registry. * Isort. * Fix access to app. * Match arguments for control. * Start fres…

--> Added Multi default logfiles and pidfiles paths [Description]: --> Changed the default paths for log files & pid files to be '/var/log/celery' and '/var/run/celery' --> Handled by creating the respective paths if not exist. --> Used os.makedir(path,if_exists=True) [Unit Test Added]: --> .travis.yml - config updated with 'before install'. --> t/unit/apps/test_multi.py - Changed the default log files & pid files paths wherever required.

* do not load memcache nor couchbase lib during windows build those libraries depends on native libraries libcouchbase and libmemcached that are not installed on Appveyor. As only unit tests runs on Appveyor, it should be fine * Add python 3.8 workaround for app trap * skip tests file_descriptor_safety tests on windows AsyncPool is not supported on Windows so Pool does have _fileno_to_outq attribute, making the test fail * Fix crossplatform log and pid files in multi mode it relates to celery#6017 * Use tox to build and test on windows * remove tox_install_command * drop python 2.7 from windows build

* 'abstractproperty' is deprecated. Use 'property' with 'abstractmethod' instead * Fix #2849 - Initial work of celery 5.0.0 alpha1 series by dropping python below 3.6 from marix & remove import from __future__ (#5684) * initial work of celery 5.0.0 alpha-1 series by dropping python below 3.6 * i-5651(ut): add ut for ResultSet.join_native (#5679) * dropped python versions below 3.6 from tox * dropped python versions below 3.6 from travis * dropped python versions below 3.6 from appveyor * dropped python2 compat __future__ imports from tests * Fixed a bug where canvases with a group and tasks in the middle followed by a group fails to complete and indefinitely hangs. (#5681) Fixes #5512, fixes #5354, fixes #2573. * dropped python2 compat __future__ imports from celery * dropped python2 compat code from init * revert readme change about version * removed python 2 object inheritance (#5687) * removed python 2 object inheritance * removed python 2 object inheritance * removed python 2 compatibility decorator (#5689) * removed python 2 compatibility decorator * removed python 2 compatibility decorator * removed python 2 compatibility decorator * removed python 2 compatibility decorator * Remove unused imports. * Remove unused imports of python_2_unicode_compatible. Also removed leftover useage of them where they were still used. * Run pyupgrade on codebase (#5726) * Run pyupgrade on codebase. * Use format strings where possible. * pyupgrade examples. * pyupgrade on celerydocs extension. * pyupgrade on updated code. * Address code review comments. * Address code review comments. * Remove unused imports. * Fix indentation. * Address code review comments. * Fix syntax error. * Fix syntax error. * Fix syntax error. * pytest 5.x for celery 5 (#5791) * Port latest changes from master to v5-dev (#5942) * Fix serialization and deserialization of nested exception classes (#5717) * Fix #5597: chain priority (#5759) * adding `worker_process_shutdown` to __all__ (#5762) * Fix typo (#5769) * Reformat code. * Simplify commands to looking for celery worker processes (#5778) * update doc- celery supports storage list. (#5776) * Update introduction.rst * Update introduction.rst * Fail xfailed tests if the failure is unexpected. * Added integration coverage for link_error (#5373) * Added coverage for link_error. * Use pytest-rerunfailed plugin instead of rolling our own custom implementation. * Added link_error with retries. This currently fails. * Remove unused import. * Fix import on Python 2.7. * retries in link_error do not hang the worker anymore. * Run error callbacks eagerly when the task itself is run eagerly. Fixes #4899. * Adjust unit tests accordingly. * Grammar in documentation (#5780) * Grammar in documentation * Address review. * pypy 7.2 matrix (#5790) * removed extra slashes in CELERY_BROKER_URL (#5792) The Celery broker URL in settings.py had 2 slashes in the end which are not required and can be misleading. so I changed :- CELERY_BROKER_URL = 'amqp://guest:guest@localhost//' to CELERY_BROKER_URL = 'amqp://guest:guest@localhost' * Fix #5772 task_default_exchange & task_default_exchange_type not work (#5773) * Fix #5772 task_default_exchange & task_default_exchange_type not work * Add unit test: test_setting_default_exchange * Move default_exchange test to standalone class * Run integration suite with memcached results backend. (#5739) * Fix hanging forever when fetching results from a group(chain(group)) canvas. (#5744) PR #5739 uncovered multiple problems with the cache backend. This PR should resolve one of them. PR #5638 fixed the same test case for our async results backends that support native join. However, it did not fix the test case for sync results backends that support native join. * Fix regression in PR #5681. (#5753) See comment in the diff for details. * Grammatical fix to CONTRIBUTING.rst doc (#5794) * Fix #5734 Celery does not consider authMechanism on mongodb backend URLs (#5795) * Fix #5734 Celery does not consider authMechanism on mongodb backend URLs * Add unit test: test_get_connection_with_authmechanism * Add unit test: test_get_connection_with_authmechanism_no_username * Fix errors in Python 2.7 Remove "," after "**" operator * Revert "Revert "Revert "Added handle of SIGTERM in BaseTask in celery/task.py to prevent kill the task" (#5577)" (#5586)" (#5797) This reverts commit f79894e0a2c7156fd0ca5e8e3b652b6a46a7e8e7. * Add Python 3.8 Support (#5785) * Added Python 3.8 to the build matrix. * Ensure a supported tblib version is installed for Python 3.8 and above. In addition, modernize the relevant tests. * Workaround patching problem in test. * py 3.8 in clasifier * ubuntu bionic (#5799) * ubuntu bionic * fast finish * sync bumversion with pypi release * Dev.req (#5803) * update docker config * undo hardpin * devr req install from github master * update docker config (#5801) * update docker config * make dockerfile to install from github master dev branch by default * update download link * Isort. * Grammatical & punctuation fixes for CONTRIBUTING.rst document (#5804) * update dockerfile * switched to ubuntu bionic * update docker * keep it empty until we reconfigure it again with autopep8 * Fixed Dockerfile (#5809) * Update document CONTRIBUTING.rst & fix Dockerfile typo (#5813) * Added an issue template for minor releases. * reference gocelery Go Client/Server for Celery (#5815) * Add enterprise language (#5818) * Fix/correct minor doc typos (#5825) * Correct a small typo * Correct bad contributing documentation links * Preserve the task priority in case of a retry (#5820) * Preserve the task priority in case of a retry * Created test case for retried tasks with priority * Implement an integration test for retried tasks with priorities * bump kombu * basic changelog for celery 4.4.0rc4 * bump celery 4.4.0rc4 * events bootstep disabled if no events (#5807) * events bootstep disabled if no events * Added unit tests. * update bug report template * fixing ascii art to look nicer (#5831) * Only rerun flaky tests when failures can be intermediary. * Rename Changelog to Changelog.rst * The test_nested_group_chain test can run without native_join support. (#5838) * Run integration tests with Cassandra (#5834) * Run integration tests with Cassandra. * Configure cassandra result backend * Pre-create keyspace and table * Fix deprecation warning. * Fix path to cqlsh. * Increase connection timeout. * Wait until the cluster is available. * SQS - Reject on failure (#5843) * reject on failure * add documentation * test fix * test fix * test fix * Add a concurrency model with ThreadPoolExecutor (#5099) * Add a concurrency model with ThreadPoolExecutor * thread model test for pypy * Chain primitive's code example fix in canvas documentation (Regression PR#4444) (#5845) * Changed multi-line string (#5846) This string wasn't rendering properly and was printing the python statement too. Although the change isn't as pretty code-wise, it gets rid of an annoyance for the user. * Add auto expiry for DynamoDB backend (#5805) * Add auto expiry for DynamoDB backend This adds auto-expire support for the DynamoDB backend, via the DynamoDB Time to Live feature. * Require boto3>=1.9.178 for DynamoDB TTL support boto3 version 1.9.178 requires botocore>=1.12.178. botocore version 1.12.178 introduces support for the DynamoDB UpdateTimeToLive call. The UpdateTimeToLive call is used by the DynamoDB backend to enable TTL support on a newly created table. * Separate TTL handling from table creation Handle TTL enabling/disabling separately from the table get-or-create function. Improve handling of cases where the TTL is already set to the desired state. DynamoDB only allows a single TTL update action within a fairly long time window, so some problematic cases (changing the TTL attribute, enabling/disabling TTL when it was recently modified) will raise exceptions that have to be dealt with. * Handle older boto3 versions If the boto3 TTL methods are not found, log an informative error. If the user wants to enable TTL, raise an exception; if TTL should be disabled, simply return. * Improve logging - Handle exceptions by logging the error and re-raising - Log (level debug) when the desired TTL state is already in place * Add and use _has_ttl() convenience method Additional changes: - Handle exceptions when calling boto3's describe_time_to_live() - Fix test cases for missing TTL methods * Update ttl_seconds documentation * Log invalid TTL; catch and raise ValueError * Separate method _get_table_ttl_description * Separate ttl method validation function * Clarify tri-state TTL value * Improve test coverage * Fix minor typo in comment * Mark test as xfail when using the cache backend. (#5851) * [Fix #5436] Store extending result in all backends (#5661) * [Fix #5436] Store extending result in all backends * Fix sqlalchemy * More fixu * Fixing tests * removing not necessary import * Removing debug code * Removing debug code * Add tests for get_result_meta in base and database * Revert "Add auto expiry for DynamoDB backend (#5805)" (#5855) This reverts commit f7f5bcfceca692d0e78c742a7c09c424f53d915b. * Revert "Mark test as xfail when using the cache backend. (#5851)" (#5854) This reverts commit 1b303c2968836245aaa43c3d0ff9249dd8bf9ed2. * docs: Document Redis commands used by celery (#5853) * remove cache back end integrtion test. (#5856) * Fix a race condition when publishing a very large chord header (#5850) * Added a test case which artificially introduces a delay to group.save(). * Fix race condition by delaying the task only after saving the group. * update tox * Remove duplicate boto dependency. (#5858) * Revert "remove cache back end integrtion test. (#5856)" (#5859) This reverts commit e0ac7a19a745dd5a52a615c1330bd67f2cef4d00. * Revert "Revert "Add auto expiry for DynamoDB backend (#5805)" (#5855)" (#5857) This reverts commit 4ddc605392d7694760f23069c34ede34b3e582c3. * Revert "update tox" This reverts commit 49427f51049073e38439ea9b3413978784a24999. * Fix the test_simple_chord_with_a_delay_in_group_save test. * Revert "Revert "Skip unsupported canvas when using the cache backend"" (#5860) * Revert "Revert "Mark test as xfail when using the cache backend. (#5851)" (#5854)" This reverts commit fc101c61c1912c4dafa661981f8b865c011e8a55. * Make the xfail condition stricter. * Fix the xfail condition. * Linters should use Python 3.8. * Move pypy unit tests to the correct stage. * Temporarily allow PyPy to fail since it is unavailable in Travis. * Remove unused variables. * Fix unused imports. * Fix pydocstyle errors in dynamodb. * Fix pydocstyle errors in redis backend. * bump kombu to 4.6.7 * celery 4.4.0rc5 changelog * celery 4.4.0rc5 * rm redundant code (#5864) * isort. * Document the threads task pool in the CLI. * Removed the paragraph about using librabbitmq. Refer to #5872 (#5873) * Task class definitions can have retry attributes (#5869) * autoretry_for * retry_kwargs * retry_backoff * retry_backoff_max * retry_jitter can now be defined as cls attributes. All of these can be overriden from the @task decorator https://github.com/celery/celery/issues/4684 * whatsnew in Celery 4.4 as per projects standard (#5817) * 4.4 whatsnew * update * update * Move old whatsnew to history. * Remove old news & fix markers. * Added a section notifying Python 3.4 has been dropped. * Added a note about ElasticSearch basic auth. * Added a note about being able to replace eagerly run tasks. * Update index. * Address comment. * Described boto3 version updates. * Fix heading. * More news. * Thread pool. * Add Django and Config changes * Bump version 4.4.0 * upate readme * Update docs regarding Redis Message Priorities (#5874) * Update docs regarding Redis Message Priorities * fixup! Update docs regarding Redis Message Priorities * Update 4.4.0 docs (#5875) * Update 4.4 release changelog * Update whatsnew-4.4 * Update tasks docs * Fix recent tasks doc file update (#5879) * Include renamed Changelog.rst in source releases. (#5880) Changelog.rst was renamed from Changelog in fd023ec174bedc2dc65c63a0dc7c85e425ac00c6 but MANIFEST.in was not updated to include the new name. This fixes the file name so Changelog.rst will show up in future source releases again. * Reorganised project_urls and classifiers. (#5884) * Use safequote in SQS Getting Started doc (#5885) * Have appveyor build relevant versions of Python. (#5887) * Have appveyor build relevant and buildable versions of Python. * Appveyor is missing CI requirements to build. * Pin pycurl to version that will build with appveyor (because wheels files exist) * Restrict python 2.7 64 bit version of python-dateutil for parse. * Use is_alive instead of isAlive for Python 3.9 compatibility. (#5898) * Very minor tweak to commen to improve docs (#5900) As discussed here: https://stackoverflow.com/questions/58816271/celery-task-asyncresult-takes-task-id-but-is-documented-to-get-asyncresult-inst this comment seems to flow to a very confusing and misleading piece of documentation here: https://docs.celeryproject.org/en/latest/reference/celery.app.task.html#celery.app.task.Task.AsyncResult * Support configuring schema of a PostgreSQL database (#5910) * Support configuring schema of a PostgreSQL database * Add unit test * Remove blank line * Fix raise issue to make exception message more friendly (#5912) Signed-off-by: Chenyang Yan <memory.yancy@gmail.com> * Add progress for retry connections (#5915) This will show current retry progress so it will clear confusion about how many retries will be tried for connecting to broker. Closes #4556 * chg: change xrange to range (#5926) * update docs for json serializer and add note for int keys serialization (#5932) * fix indentation for note block in calling.rst (#5933) * Added links to other issue trackers. (#5939) * Add labels automatically for issues. (#5938) * Run pyupgrade. Co-authored-by: Michal Čihař <michal@cihar.com> Co-authored-by: ptitpoulpe <ptitpoulpe@ptitpoulpe.fr> Co-authored-by: Didi Bar-Zev <didi@hiredscore.com> Co-authored-by: Santos Solorzano <santosjavier22@gmail.com> Co-authored-by: manlix <manlix@yandex.ru> Co-authored-by: Jimmy <54828848+sckhg1367@users.noreply.github.com> Co-authored-by: Борис Верховский <boris.verk@gmail.com> Co-authored-by: Asif Saif Uddin <auvipy@gmail.com> Co-authored-by: Jainal Gosaliya <jainal09gosalia@gmail.com> Co-authored-by: gsfish <caoyu97@hotmail.com> Co-authored-by: Dipankar Achinta <di.punk.car19@gmail.com> Co-authored-by: Pengjie Song (宋鹏捷) <spengjie@sina.com> Co-authored-by: Chris Griffin <chris-griffin@users.noreply.github.com> Co-authored-by: Muhammad Hewedy <mhewedy@gmail.com> Co-authored-by: Blaine Bublitz <blaine.bublitz@gmail.com> Co-authored-by: Tamu <tamsanh@gmail.com> Co-authored-by: Erik Tews <erik@datenzone.de> Co-authored-by: abhinav nilaratna <anilaratna2@bloomberg.net> Co-authored-by: Wyatt Paul <wpaul@hearsaycorp.com> Co-authored-by: gal cohen <gal.nevis@gmail.com> Co-authored-by: as <alfred@huji.fr> Co-authored-by: Param Kapur <paramkapur2002@gmail.com> Co-authored-by: Sven Ulland <sven.ulland@gmail.com> Co-authored-by: Safwan Rahman <safwan.rahman15@gmail.com> Co-authored-by: Aissaoui Anouar <tobia@crossbone.cc> Co-authored-by: Neal Wang <qdzzyb2015@gmail.com> Co-authored-by: Alireza Amouzadeh <alireza@amouzadeh.net> Co-authored-by: Marcos Moyano <marcos@anue.biz> Co-authored-by: Stepan Henek <stepan+github@henek.name> Co-authored-by: Andrew Sklyarov <AndrewPix@users.noreply.github.com> Co-authored-by: Michael Fladischer <michael@fladi.at> Co-authored-by: Dejan Lekic <dejan.lekic@gmail.com> Co-authored-by: Yannick Schuchmann <yannick.schuchmann@googlemail.com> Co-authored-by: Matt Davis <matteius@gmail.com> Co-authored-by: Karthikeyan Singaravelan <tir.karthi@gmail.com> Co-authored-by: Bernd Wechner <bernd-wechner@users.noreply.github.com> Co-authored-by: Sören Oldag <soeren_oldag@freenet.de> Co-authored-by: uddmorningsun <memory.yancy@gmail.com> Co-authored-by: Amar Fadil <34912365+marfgold1@users.noreply.github.com> Co-authored-by: woodenrobot <woodenrobot1993@gmail.com> Co-authored-by: Sardorbek Imomaliev <sardorbek.imomaliev@gmail.com> * Remove fallback code for Python 2 support marked with TODOs. (#5953) Co-authored-by: Asif Saif Uddin <auvipy@gmail.com> * Remove PY3 conditionals (#5954) * Added integration coverage for link_error (#5373) * Added coverage for link_error. * Use pytest-rerunfailed plugin instead of rolling our own custom implementation. * Added link_error with retries. This currently fails. * Remove unused import. * Fix import on Python 2.7. * retries in link_error do not hang the worker anymore. * Run error callbacks eagerly when the task itself is run eagerly. Fixes #4899. * Adjust unit tests accordingly. * Grammar in documentation (#5780) * Grammar in documentation * Address review. * pypy 7.2 matrix (#5790) * removed extra slashes in CELERY_BROKER_URL (#5792) The Celery broker URL in settings.py had 2 slashes in the end which are not required and can be misleading. so I changed :- CELERY_BROKER_URL = 'amqp://guest:guest@localhost//' to CELERY_BROKER_URL = 'amqp://guest:guest@localhost' * Fix #5772 task_default_exchange & task_default_exchange_type not work (#5773) * Fix #5772 task_default_exchange & task_default_exchange_type not work * Add unit test: test_setting_default_exchange * Move default_exchange test to standalone class * Run integration suite with memcached results backend. (#5739) * Fix hanging forever when fetching results from a group(chain(group)) canvas. (#5744) PR #5739 uncovered multiple problems with the cache backend. This PR should resolve one of them. PR #5638 fixed the same test case for our async results backends that support native join. However, it did not fix the test case for sync results backends that support native join. * Fix regression in PR #5681. (#5753) See comment in the diff for details. * Grammatical fix to CONTRIBUTING.rst doc (#5794) * Fix #5734 Celery does not consider authMechanism on mongodb backend URLs (#5795) * Fix #5734 Celery does not consider authMechanism on mongodb backend URLs * Add unit test: test_get_connection_with_authmechanism * Add unit test: test_get_connection_with_authmechanism_no_username * Fix errors in Python 2.7 Remove "," after "**" operator * Revert "Revert "Revert "Added handle of SIGTERM in BaseTask in celery/task.py to prevent kill the task" (#5577)" (#5586)" (#5797) This reverts commit f79894e0a2c7156fd0ca5e8e3b652b6a46a7e8e7. * Add Python 3.8 Support (#5785) * Added Python 3.8 to the build matrix. * Ensure a supported tblib version is installed for Python 3.8 and above. In addition, modernize the relevant tests. * Workaround patching problem in test. * py 3.8 in clasifier * ubuntu bionic (#5799) * ubuntu bionic * fast finish * sync bumversion with pypi release * Dev.req (#5803) * update docker config * undo hardpin * devr req install from github master * update docker config (#5801) * update docker config * make dockerfile to install from github master dev branch by default * update download link * Isort. * Grammatical & punctuation fixes for CONTRIBUTING.rst document (#5804) * update dockerfile * switched to ubuntu bionic * update docker * keep it empty until we reconfigure it again with autopep8 * Fixed Dockerfile (#5809) * Update document CONTRIBUTING.rst & fix Dockerfile typo (#5813) * Added an issue template for minor releases. * reference gocelery Go Client/Server for Celery (#5815) * Add enterprise language (#5818) * Fix/correct minor doc typos (#5825) * Correct a small typo * Correct bad contributing documentation links * Preserve the task priority in case of a retry (#5820) * Preserve the task priority in case of a retry * Created test case for retried tasks with priority * Implement an integration test for retried tasks with priorities * bump kombu * basic changelog for celery 4.4.0rc4 * bump celery 4.4.0rc4 * events bootstep disabled if no events (#5807) * events bootstep disabled if no events * Added unit tests. * update bug report template * fixing ascii art to look nicer (#5831) * Only rerun flaky tests when failures can be intermediary. * Rename Changelog to Changelog.rst * The test_nested_group_chain test can run without native_join support. (#5838) * Run integration tests with Cassandra (#5834) * Run integration tests with Cassandra. * Configure cassandra result backend * Pre-create keyspace and table * Fix deprecation warning. * Fix path to cqlsh. * Increase connection timeout. * Wait until the cluster is available. * SQS - Reject on failure (#5843) * reject on failure * add documentation * test fix * test fix * test fix * Add a concurrency model with ThreadPoolExecutor (#5099) * Add a concurrency model with ThreadPoolExecutor * thread model test for pypy * Chain primitive's code example fix in canvas documentation (Regression PR#4444) (#5845) * Changed multi-line string (#5846) This string wasn't rendering properly and was printing the python statement too. Although the change isn't as pretty code-wise, it gets rid of an annoyance for the user. * Add auto expiry for DynamoDB backend (#5805) * Add auto expiry for DynamoDB backend This adds auto-expire support for the DynamoDB backend, via the DynamoDB Time to Live feature. * Require boto3>=1.9.178 for DynamoDB TTL support boto3 version 1.9.178 requires botocore>=1.12.178. botocore version 1.12.178 introduces support for the DynamoDB UpdateTimeToLive call. The UpdateTimeToLive call is used by the DynamoDB backend to enable TTL support on a newly created table. * Separate TTL handling from table creation Handle TTL enabling/disabling separately from the table get-or-create function. Improve handling of cases where the TTL is already set to the desired state. DynamoDB only allows a single TTL update action within a fairly long time window, so some problematic cases (changing the TTL attribute, enabling/disabling TTL when it was recently modified) will raise exceptions that have to be dealt with. * Handle older boto3 versions If the boto3 TTL methods are not found, log an informative error. If the user wants to enable TTL, raise an exception; if TTL should be disabled, simply return. * Improve logging - Handle exceptions by logging the error and re-raising - Log (level debug) when the desired TTL state is already in place * Add and use _has_ttl() convenience method Additional changes: - Handle exceptions when calling boto3's describe_time_to_live() - Fix test cases for missing TTL methods * Update ttl_seconds documentation * Log invalid TTL; catch and raise ValueError * Separate method _get_table_ttl_description * Separate ttl method validation function * Clarify tri-state TTL value * Improve test coverage * Fix minor typo in comment * Mark test as xfail when using the cache backend. (#5851) * [Fix #5436] Store extending result in all backends (#5661) * [Fix #5436] Store extending result in all backends * Fix sqlalchemy * More fixu * Fixing tests * removing not necessary import * Removing debug code * Removing debug code * Add tests for get_result_meta in base and database * Revert "Add auto expiry for DynamoDB backend (#5805)" (#5855) This reverts commit f7f5bcfceca692d0e78c742a7c09c424f53d915b. * Revert "Mark test as xfail when using the cache backend. (#5851)" (#5854) This reverts commit 1b303c2968836245aaa43c3d0ff9249dd8bf9ed2. * docs: Document Redis commands used by celery (#5853) * remove cache back end integrtion test. (#5856) * Fix a race condition when publishing a very large chord header (#5850) * Added a test case which artificially introduces a delay to group.save(). * Fix race condition by delaying the task only after saving the group. * update tox * Remove duplicate boto dependency. (#5858) * Revert "remove cache back end integrtion test. (#5856)" (#5859) This reverts commit e0ac7a19a745dd5a52a615c1330bd67f2cef4d00. * Revert "Revert "Add auto expiry for DynamoDB backend (#5805)" (#5855)" (#5857) This reverts commit 4ddc605392d7694760f23069c34ede34b3e582c3. * Revert "update tox" This reverts commit 49427f51049073e38439ea9b3413978784a24999. * Fix the test_simple_chord_with_a_delay_in_group_save test. * Revert "Revert "Skip unsupported canvas when using the cache backend"" (#5860) * Revert "Revert "Mark test as xfail when using the cache backend. (#5851)" (#5854)" This reverts commit fc101c61c1912c4dafa661981f8b865c011e8a55. * Make the xfail condition stricter. * Fix the xfail condition. * Linters should use Python 3.8. * Move pypy unit tests to the correct stage. * Temporarily allow PyPy to fail since it is unavailable in Travis. * Remove unused variables. * Fix unused imports. * Fix pydocstyle errors in dynamodb. * Fix pydocstyle errors in redis backend. * bump kombu to 4.6.7 * celery 4.4.0rc5 changelog * celery 4.4.0rc5 * rm redundant code (#5864) * isort. * Document the threads task pool in the CLI. * Removed the paragraph about using librabbitmq. Refer to #5872 (#5873) * Task class definitions can have retry attributes (#5869) * autoretry_for * retry_kwargs * retry_backoff * retry_backoff_max * retry_jitter can now be defined as cls attributes. All of these can be overriden from the @task decorator https://github.com/celery/celery/issues/4684 * whatsnew in Celery 4.4 as per projects standard (#5817) * 4.4 whatsnew * update * update * Move old whatsnew to history. * Remove old news & fix markers. * Added a section notifying Python 3.4 has been dropped. * Added a note about ElasticSearch basic auth. * Added a note about being able to replace eagerly run tasks. * Update index. * Address comment. * Described boto3 version updates. * Fix heading. * More news. * Thread pool. * Add Django and Config changes * Bump version 4.4.0 * upate readme * Update docs regarding Redis Message Priorities (#5874) * Update docs regarding Redis Message Priorities * fixup! Update docs regarding Redis Message Priorities * Update 4.4.0 docs (#5875) * Update 4.4 release changelog * Update whatsnew-4.4 * Update tasks docs * Fix recent tasks doc file update (#5879) * Include renamed Changelog.rst in source releases. (#5880) Changelog.rst was renamed from Changelog in fd023ec174bedc2dc65c63a0dc7c85e425ac00c6 but MANIFEST.in was not updated to include the new name. This fixes the file name so Changelog.rst will show up in future source releases again. * Reorganised project_urls and classifiers. (#5884) * Use safequote in SQS Getting Started doc (#5885) * Have appveyor build relevant versions of Python. (#5887) * Have appveyor build relevant and buildable versions of Python. * Appveyor is missing CI requirements to build. * Pin pycurl to version that will build with appveyor (because wheels files exist) * Restrict python 2.7 64 bit version of python-dateutil for parse. * Use is_alive instead of isAlive for Python 3.9 compatibility. (#5898) * Very minor tweak to commen to improve docs (#5900) As discussed here: https://stackoverflow.com/questions/58816271/celery-task-asyncresult-takes-task-id-but-is-documented-to-get-asyncresult-inst this comment seems to flow to a very confusing and misleading piece of documentation here: https://docs.celeryproject.org/en/latest/reference/celery.app.task.html#celery.app.task.Task.AsyncResult * Support configuring schema of a PostgreSQL database (#5910) * Support configuring schema of a PostgreSQL database * Add unit test * Remove blank line * Fix raise issue to make exception message more friendly (#5912) Signed-off-by: Chenyang Yan <memory.yancy@gmail.com> * Add progress for retry connections (#5915) This will show current retry progress so it will clear confusion about how many retries will be tried for connecting to broker. Closes #4556 * chg: change xrange to range (#5926) * update docs for json serializer and add note for int keys serialization (#5932) * fix indentation for note block in calling.rst (#5933) * Added links to other issue trackers. (#5939) * Add labels automatically for issues. (#5938) * remove redundant raise from docstring (#5941) `throw` is True by default so the Retry exception will already get raised by calling `self.retry(countdown=60 * 5, exc=exc)` * Run pyupgrade. * Fix typo (#5943) * Remove fallback code for Python 2 support. * docs: fixes Rabbits and Warrens link in routing userguide (#4007) (#5949) * Fix labels on Github issue templates. (#5955) Use quotation marks to escape labels on Github issue templates. This prevents the colon from breaking the template. * added retry_on_timeout and socket_keepalive to config and doc (#5952) * Fixed event capture from building infinite list (#5870) * Fix error propagation example (#5966) * update range (#5971) * update setup.cfg * bump billiard to 3.6.3.0 * Update __init__.py (#5951) * Update __init__.py Fixed issue for object with result_backend=True (decode fails on multiple None request) * Update __init__.py suggested changeds * Update __init__.py * Use configured db schema also for sequences (#5972) * Added a default value for retries in worker.strategy. (#5945) * Added a default value for retries in worker.strategy. I was facing an issue when adding tasks directly to rabbitmq using pika instead of calling task.apply_async. The issue was the self.retry mechanisum was failing. In app/tasks.py the line `retries = request.retries + 1` was causing the issue. On further tracing I figured out that it was because the default .get value (None) was getting passed through this function and was raising TypeError: unsupported operand type(s) for +: 'NoneType' and 'int' * Add test cases for default and custom retries value * pypy 7.3 (#5980) * Pass `interval` to `get_many` (#5931) * Pass `interval` to `get_many` * Fix: Syntax error for py2.7 * Fix: Syntax error for py2.7 * Fixed problem with conflicting autoretry_for task parameter and Task.replace() (#5934) * Fix #5917 (#5918) * Fix changelog (#5881) * merge in place the apps beat schedule in the default Schedule class. (#5908) * Handle Redis connection errors in result consumer (#5921) * Handle Redis connection errors in result consumer * Closes #5919. * Use context manager for Redis conusmer reconnect * Log error when result backend reconnection fails * Fix inspect_command documentation (#5983) * Use gevent and eventlet wait() functions to remove busy-wait (#5974) * Use gevent and eventlet wait() functions to remove busy-wait Fixes issue #4999. Calling AsyncResult.get() in a gevent context would cause the async Drainer to repeatedly call wait_for until the result was completed. I've updated the code to have a specific implementation for gevent and eventlet that will cause wait_for to only return every "timeout" # of seconds, rather than repeatedly returning. Some things I'd like some feedback on: * Where's the best place to add test coverage for this? It doesn't look like there are any tests that directly exercised the Drainer yet so I would probably look to add some of these to the backends/ unit tests. * The way I did this for the Eventlet interface was to rely on the private _exit_event member of the GreenThread instance; to do this without relying on a private member would require some additional changes to the backend Drainer interface so that we could wait for an eventlet-specific event in wait_for(). I can do this, just wanted to get some feedback before. * Add unit tests for Drainer classes In order for this to work without monkeypatching in the tests, I needed to call sleep(0) to let the gevent/eventlet greenlets to yield control back to the calling thread. I also made the check interval configurable in the drainer so that we didn't need to sleep multiples of 1 second in the tests. * Weaken asserts since they don't pass on CI * Fix eventlet auto-patching DNS resolver module on import By default it looks like "import eventlet" imports the greendns module unless the environment EVENTLET_NO_GREENDNS is set to true. This broke a pymongo test. * Add tests ensuring that the greenlet loop isn't blocked These tests make sure that while drain_events_until is running that other gevent/eventlet concurrency can run. * Clean up tests and make sure they wait for all the threads to stop * Fix chords with chained groups (#5947) * kombu 4.6.8 * update setup * updated version 4.4.1 * Fix: Accept and swallow `kwargs` to handle unexpected keyword arguments * Allow boto to look for credentials in S3Backend * add reference to Rusty Celery * Update document of revoke method in Control class * Fix copy-paste error in result_compression docs * Make 'socket_keepalive' optional variable (#6000) * update connection params - socket_keepalive is optional now * update readme - added versionadded 4.4.1 and fixed `redis_socket_keepalive` * added check of socket_keepalive in arguments for UnixSocketConnect * Fixed incorrect setting name in documentation (#6002) * updated version 4.4.2 * Fix backend utf-8 encoding in s3 backend Celery backend uses utf-8 to deserialize results, which would fail for some serializations like pickle. * Fix typo in celery.bin.multi document * Upgraded pycurl to the latest version that supports wheel. * pytest 5.3.5 max * Add uptime to the stats inspect command * Doc tweaks: mostly grammar and punctuation (#6016) * Fix a bunch of comma splices in the docs * Remove some unnecessary words from next-steps doc * Tweak awkward wording; fix bad em-dash * Fix a bunch more comma splices in next-steps doc * Miscellaneous grammar/punctuation/wording fixes * Link to task options in task decorator docs * Fixing issue #6019: unable to use mysql SSL parameters when getting mysql engine (#6020) * Fixing issue #6019: unable to use mysql SSL parametes in create_engine() * adding test for get_engine when self.forked is False and engine args are passed in for create_engine() * Clean TraceBack to reduce memory leaks for exception task (#6024) * Clean TraceBack to reduce memory leaks * add unit test * add unit test * reject unittest * Patch For Python 2.7 compatibility * update unittest * Register to the garbage collector by explicitly referring to f_locals. * need more check * update code coverage * update Missing unit test * 3.4 -> 3.5 Co-authored-by: heedong.jung <heedong.jung@samsung.com> * exceptions: NotRegistered: fix up language Minor fix to the language. * Note about autodiscover_tasks and periodic tasks This is particularly important for Django projects that put periodic tasks into each app's `tasks.py` and want to use one as a periodic task. By the time `autodiscover_tasks()` loads those tasks, the `on_after_configure` Signal has already come and gone, so anything decorated with `@app.on_after_finalize.connect` will never be called. If there's other documentation on this subject, I could not find it. * Avoid PyModulelevel, deprecated in Sphinx 4 Use `PyFunction` instead of `PyModulelevel` to avoid this deprecation warning: RemovedInSphinx40Warning: PyModulelevel is deprecated. Please check the implementation of <class 'celery.contrib.sphinx.TaskDirective'> This replacement is one of the options listed in the Sphinx docs (https://www.sphinx-doc.org/en/master/extdev/deprecated.html). * Give up sending a worker-offline message if transport is not connected (#6039) * If worker-offline event fails to send, give up and die peacefully * Add test for retry= and msgs in heartbeat * Fix the build and all documentation warnings. I finally upgraded our theme to 2.0. As a result we've upgraded Sphinx to 2.0. Work to upgrade Sphinx to 3.0 will proceed in a different PR. This upgrade also fixes our build issues caused by #6032. We don't support Sphinx 1.x as a result of that patch. I've also included the missing 4.3 changelog to our history. * Support both Sphinx 2 and 3. * Add Task to __all__ in celery.__init__.py * Add missing parenthesis to example in docs * Ensure a single chain object in a chain does not raise MaximumRecursionError. Previously chain([chain(sig)]) would crash. We now ensure it doesn't. Fixes #5973. * update setup.py * fix typo missing quote at the end of line * Fix a typo in monitoring doc * update travis * update ubuntu to focal foss 20.04 LTS * Fix autoscale when prefetch_multiplier is 1 * Allow start_worker to function without ping task * Update celeryd.conf Move the directory of the program before the execution of the command/script * Add documentation for "predefined_queue_urls" * [Fix #6074]: Add missing documentation for MongoDB as result backend. * update funding * 🐛 Correctly handle configuring the serializer for always_eager mode. (#6079) * 🐛 Correctly handle configuring the serializer for always_eager mode. options['serializer'] will always exist, because it is initialized from an mattrgetter. Even if unset, it will be present in the options with a value of None. * 🐛 Add a test for new always_eager + task_serializer behavior. * ✏️ Whoops missed a : * Remove doubling of prefetch_count increase when prefetch_multiplier gt 1 (#6081) * try ubuntu focal (#6088) * Fix eager function not returning result after retries. Using apply function does not return correct results after at least one retry because the return value of successive call is not going back to the original caller. * return retry result if not throw and is_eager if throw is false, we would be interested by the result of retry and not the current result which will be an exception. This way it does not break the logic of `raise self.retry` This should be used like `return self.retry(..., throw=False)` in an except statement. * revert formatting change * Add tests for eager retry without throw * update predefined-queues documentation Suggested version of configuration does not work. Additionally I'd like to mention, that `access_key_id` and `secret_access_key` are mandatory fields and not allowing you to go with defaults AWS_* env variables. I can contribute for this variables to be optional Also I'm not sure if security token will apply, could you please advice how to do it? * Fix couchbase version < 3.0.0 as API changed * Remove reference to -O fair in optimizations -O fair was made the default in Celery 4.0 https://docs.celeryproject.org/en/stable/history/whatsnew-4.0.html#ofair-is-now-the-default-scheduling-strategy * pytest ranges * pypy3 * revert to bionic * do not load docs.txt requirements for python 2.7 As it requires Sphinx >= 2.0.0 and there is no such version compatible with python 2.7 * update cassandra travis integration test configuration cassandra:latest docker image changed location of cqlsh program * pin cassandra-driver CI get stuck after all cassandra integration tests * Fix all flake8 lint errors * Fix all pydocstyle lint errors * Fix all configcheck lint errors * Always requeue while worker lost regardless of the redelivered flag (#6103) * #5598 fix, always redelivery while WorkerLostError * fix, change the requeue flag so the task will remain PENDING * Allow relative paths in the filesystem backend (#6070) * Allow relative paths in the filesystem backend * fix order of if statements * [Fixed Issue #6017] --> Added Multi default logfiles and pidfiles paths [Description]: --> Changed the default paths for log files & pid files to be '/var/log/celery' and '/var/run/celery' --> Handled by creating the respective paths if not exist. --> Used os.makedir(path,if_exists=True) [Unit Test Added]: --> .travis.yml - config updated with 'before install'. --> t/unit/apps/test_multi.py - Changed the default log files & pid files paths wherever required. * Avoid race condition due to task duplication. In some circumstances like a network partitioning, some tasks might be duplicated. Sometimes, this lead to a race condition where a lost task overwrites the result of the last successful task in the backend. In order to avoid this race condition we prevent updating the result if it's already in successful state. This fix has been done for KV backends only and therefore won't work with other backends. * adding tests * Exceptions must be old-style classes or derived from BaseException, but here self.result may not subclass of BaseException. * update fund link * Fix windows build (#6104) * do not load memcache nor couchbase lib during windows build those libraries depends on native libraries libcouchbase and libmemcached that are not installed on Appveyor. As only unit tests runs on Appveyor, it should be fine * Add python 3.8 workaround for app trap * skip tests file_descriptor_safety tests on windows AsyncPool is not supported on Windows so Pool does have _fileno_to_outq attribute, making the test fail * Fix crossplatform log and pid files in multi mode it relates to #6017 * Use tox to build and test on windows * remove tox_install_command * drop python 2.7 from windows build * Add encode to meta task in base.py (#5894) * Add encode to base.py meta result Fix bug with impossibility to load None from task meta * Add tests for None. Remove exceed encode. * Update base.py Add return payload if None * Update time.py to solve the microsecond issues (#5199) When `relative` is set to True, the day, hour, minutes second will be round to the nearest one, however, the original program do not update the microsecond (reset it). As a result, the run-time offset on the microsecond will then be accumulated. For example, given the interval is 15s and relative is set to True 1. 2018-11-27T15:01:30.123236+08:00 2. 2018-11-27T15:01:45.372687+08:00 3. 2018-11-27T15:02:00.712601+08:00 4. 2018-11-27T15:02:15.987720+08:00 5. 2018-11-27T15:02:31.023670+08:00 * Change backend _ensure_not_eager error to warning * Add priority support for 'celery.chord_unlock' task (#5766) * Change eager retry behaviour even with raise self.retry, it should return the eventual value or MaxRetriesExceededError. if return value of eager apply is Retry exception, retry eagerly the task signature * Order supported Python versions * Avoid race condition in elasticsearch backend if a task is retried, the task retry may work concurrently to current task. store_result may come out of order. it may cause a non ready state (Retry) to override a ready state (Success, Failure). If this happens, it will block indefinitely pending any chord depending on this task. this change makes document updates safe for concurrent writes. https://www.elastic.co/guide/en/elasticsearch/reference/current/optimistic-concurrency-control.html * backends base get_many pass READY_STATES arg * test backends base get_many pass READY_STATES arg * Add integration tests for Elasticsearch and fix _update * Revert "revert to bionic" This reverts commit 6e091573f2ab0d0989b8d7c26b677c80377c1721. * remove jython check * feat(backend): Adds cleanup to ArangoDB backend * Delete Document Known Issue with CONN_MAX_AGE in 4.3 * issue 6108 fix filesystem backend cannot not be serialized by picked (#6120) * issue 6108 fix filesystem backend cannot not be serialized by picked https://github.com/celery/celery/issues/6108 * issue-6108 fix unit test failure * issue-6108 fix flake8 warning Co-authored-by: Murphy Meng <mmeng@mirriad.com> * kombu==4.6.9 (#6133) * changelog for 4.4.3 * v 4.4.3 * remove un supported classifier * Fix autoretry_for with explicit retry (#6138) * Add tests for eager task retry * Fixes #6135 If autoretry_for is set too broad on Exception, then autoretry may get a Retry if that's the case, rethrow directly instead of wrapping it in another Retry to avoid loosing new args * Use Django DB max age connection setting (fixes #4116) * Add retry on recoverable exception for the backend (#6122) * Add state to KeyValueStoreBackend.set method This way, a backend implementation is able to take decisions based on current state to store meta in case of failures. * Add retry on recoverable exception for the backend acks.late makes celery acknowledge messages only after processing and storing result on the backend. However, in case of backend unreachable, it will shadow a Retry exception and put the task as failed in the backend not retrying the task and acknoledging it on the broker. With this new result_backend_always_retry setting, if the backend exception is recoverable (to be defined per backend implementation), it will retry the backend operation with an exponential backoff. * Make elasticsearch backward compatible with 6.x * Make ES retry storing updates in a better way if existing value in the backend is success, then do nothing. if it is a ready status, then update it only if new value is a ready status as well. else update it. This way, a SUCCESS cannot be overriden so that we do not loose results but any ready state other than success (FAILURE, REVOKED) can be overriden by another ready status (i.e. a SUCCESS) * Add test for value not found in ES backend * Fix random distribution of jitter for exponential backoff random.randrange should be called with the actual so that all numbers have equivalent probability, otherwise maximum value does have a way higher probability of occuring. * fix unit test if extra modules are not present * ElasticSearch: add setting to save meta as json * fix #6136. celery 4.4.3 always trying create /var/run/celery directory (#6142) * fix #6136. celery 4.4.3 always trying create /var/run/celery directory, even if it's not needed. * fix #6136. cleanup * Add task_internal_error signal (#6049) * Add internal_error signal There is no special signal for an out of body error which can be the result of a bad result backend. * Fix syntax error. * Document the task_internal_error signal. Co-authored-by: Laurentiu Dragan <ldragan@bloomberg.net> * changelog for v4.4.4 * kombu 4.6.10 (#6144) * v4.4.4 * Add missing dependency on future (#6146) Fixes #6145 * ElasticSearch: Retry index if document was deleted between index and update (#6140) * ElasticSearch: Retry index if document was deleted between index and update * Elasticsearch increase coverage to 100% * Fix pydocstyle * Specify minimum version of Sphinx for Celery extension (#6150) The Sphinx extension requires Sphinx 2 or later due to #6032. * fix windows build * fix flake8 error * fix multi tests in local Mock os.mkdir and os.makedirs to avoid creating /var/run/celery and /var/log/celery during unit tests if run without root priviledges * Customize the retry interval of chord_unlock tasks * changelog v4.4.5 * v4.4.5 * Fix typo in comment. * Remove autoscale force_scale methods (#6085) * Remove autoscale force_scale methods * Remove unused variable in test * Pass ping destination to request The destination argument worked fine from CLI but didn't get used when calling ping from Python. * Fix autoscale test * chord: merge init options with run options * put back KeyValueStoreBackend.set method without state It turns out it was breaking some other projects. wrapping set method with _set_with_state, this way it will not break existing Backend. while enabling this feature for other Backend. Currently, only ElasticsearchBackend supports this feature. It protects concurrent update to corrupt state in the backend. Existing success cannot be overriden, nor a ready state by a non ready state. i.e. a Retry state cannot override a Success or Failure. As a result, chord_unlock task will not loop forever due to missing ready state on the backend. * added --range-prefix option to `celery multi` (#6180) * added --range-prefix option to `celery multi` Added option for overriding default range prefix when running multiple workers prividing range with `celery multy` command. * covered multi --range-prefix with tests * fixed --range-prefix test * Added as_list function to AsyncResult class (#6179) * Add as_list method to return task IDs as a list * Add a test for as_list method * Add docstring for as_list method * Fix CassandraBackend error in threads or gevent pool (#6147) * Fix CassandraBackend error in threads or gevent pool * remove CassandraBackend.process_cleanup * Add test case * Add test case * Add comments test_as_uri Co-authored-by: baixue <baixue@wecash.net> * changelog for v4.4.6 * v4.4.6 * Update Wiki link in "resources" In the page linked below, the link to wiki is outdated. Fixed that. https://docs.celeryproject.org/en/stable/getting-started/resources.html * test_canvas: Add test for chord-in-chain Add test case for the issue where a chord in a chain does not work when using .apply(). This works fine with .apply_async(). * Trying to fix flaky tests in ci * fix pydocstyle errors * fix pydocstyle * Drainer tests, put a lower constraint on number of intervals liveness should iterate 10 times per interval while drain_events only once. However, as it may use thread that may be scheduled out of order, we may end up in some situation where liveness and drain_events were called the same amount of time. Lowering the constraint from < to <= to avoid failing the tests. * pyupgrade. * Fix merge error. Co-authored-by: Борис Верховский <boris.verk@gmail.com> Co-authored-by: Asif Saif Uddin <auvipy@gmail.com> Co-authored-by: Jainal Gosaliya <jainal09gosalia@gmail.com> Co-authored-by: gsfish <root@grassfish.net> Co-authored-by: Dipankar Achinta <di.punk.car19@gmail.com> Co-authored-by: spengjie <spengjie@sina.com> Co-authored-by: Chris Griffin <chris-griffin@users.noreply.github.com> Co-authored-by: Muhammad Hewedy <mhewedy@gmail.com> Co-authored-by: Blaine Bublitz <blaine.bublitz@gmail.com> Co-authored-by: Tamu <tamsanh@gmail.com> Co-authored-by: Erik Tews <erik@datenzone.de> Co-authored-by: abhinav nilaratna <anilaratna2@bloomberg.net> Co-authored-by: Wyatt Paul <wpaul@hearsaycorp.com> Co-authored-by: gal cohen <gal.nevis@gmail.com> Co-authored-by: whuji <alfred@huji.fr> Co-authored-by: Param Kapur <paramkapur2002@gmail.com> Co-authored-by: Sven Ulland <sven.ulland@gmail.com> Co-authored-by: Safwan Rahman <safwan.rahman15@gmail.com> Co-authored-by: Aissaoui Anouar <tobia@crossbone.cc> Co-authored-by: Neal Wang <qdzzyb2015@gmail.com> Co-authored-by: Alireza Amouzadeh <alireza@amouzadeh.net> Co-authored-by: Marcos Moyano <marcos@anue.biz> Co-authored-by: Stepan Henek <stepan+github@henek.name> Co-authored-by: Andrew Sklyarov <AndrewPix@users.noreply.github.com> Co-authored-by: Michael Fladischer <michael@fladi.at> Co-authored-by: Dejan Lekic <dejan.lekic@gmail.com> Co-authored-by: Yannick Schuchmann <yannick.schuchmann@googlemail.com> Co-authored-by: Matt Davis <matteius@gmail.com> Co-authored-by: Xtreak <tir.karthi@gmail.com> Co-authored-by: Bernd Wechner <bernd-wechner@users.noreply.github.com> Co-authored-by: Sören Oldag <soeren_oldag@freenet.de> Co-authored-by: uddmorningsun <memory.yancy@gmail.com> Co-authored-by: Amar Fadil <34912365+marfgold1@users.noreply.github.com> Co-authored-by: woodenrobot <woodenrobot1993@gmail.com> Co-authored-by: Sardorbek Imomaliev <sardorbek.imomaliev@gmail.com> Co-authored-by: Alex Riina <alex.riina@gmail.com> Co-authored-by: Joon Hwan 김준환 <xncbf12@gmail.com> Co-authored-by: Prabakaran Kumaresshan <k_prabakaran+github@hotmail.com> Co-authored-by: Martey Dodoo <martey@mobolic.com> Co-authored-by: Konstantin Seleznev <4374093+Seleznev-nvkz@users.noreply.github.com> Co-authored-by: Prodge <Prodge@users.noreply.github.com> Co-authored-by: Abdelhadi Dyouri <raryat@gmail.com> Co-authored-by: Ixiodor <Ixiodor@users.noreply.github.com> Co-authored-by: abhishekakamai <47558404+abhishekakamai@users.noreply.github.com> Co-authored-by: Allan Lei <allanlei@helveticode.com> Co-authored-by: M1ha Shvn <work_shvein_mihail@mail.ru> Co-authored-by: Salih Caglar Ispirli <caglarispirli@gmail.com> Co-authored-by: Micha Moskovic <michamos@gmail.com> Co-authored-by: Chris Burr <chrisburr@users.noreply.github.com> Co-authored-by: Dave King <tildedave@gmail.com> Co-authored-by: Dmitry Nikulin <v.dmitry.nikulin@gmail.com> Co-authored-by: Michael Gaddis <mgaddis@ancestry.com> Co-authored-by: epwalsh <epwalsh10@gmail.com> Co-authored-by: TalRoni <tal.inon.16@gmail.com> Co-authored-by: Leo Singer <leo.singer@ligo.org> Co-authored-by: Stephen Tomkinson <neonbunny@users.noreply.github.com> Co-authored-by: Abhishek <abhishek.shukla@xeneta.com> Co-authored-by: theirix <theirix@gmail.com> Co-authored-by: yukihira1992 <hirayama@cancerscan.jp> Co-authored-by: jpays <jerome.pays@cnp.fr> Co-authored-by: Greg Ward <greg@gerg.ca> Co-authored-by: Alexa Griffith <agriffith@bluecore.com> Co-authored-by: heedong <63043496+heedong-jung@users.noreply.github.com> Co-authored-by: heedong.jung <heedong.jung@samsung.com> Co-authored-by: Shreyansh Khajanchi <shreyanshk@users.noreply.github.com> Co-authored-by: Sam Thompson <georgedorn@users.noreply.github.com> Co-authored-by: Alphadelta14 <alpha@pokesplash.net> Co-authored-by: Azimjon Pulatov <azimjohn@yahoo.com> Co-authored-by: ysde <ysde108@gmail.com> Co-authored-by: AmirMohammad Ziaei <amir_zia@outlook.com> Co-authored-by: Ben Nadler <ben.nadler@gusto.com> Co-authored-by: Harald Nezbeda <hn@nezhar.com> Co-authored-by: Chris Frisina <github@specialorange.org> Co-authored-by: Adam Eijdenberg <adam.eijdenberg@digital.gov.au> Co-authored-by: rafaelreuber <rafaelreuber@gmail.com> Co-authored-by: Noah Kantrowitz <noah@coderanger.net> Co-authored-by: Ben Nadler <jbennadler@gmail.com> Co-authored-by: Clement Michaud <c.michaud@criteo.com> Co-authored-by: Mathieu Chataigner <m.chataigner@criteo.com> Co-authored-by: eugeneyalansky <65346459+eugeneyalansky@users.noreply.github.com> Co-authored-by: Leonard Lu <leonard@socialcodeinc.com> Co-authored-by: XinYang <yangxinhust@hotmail.com> Co-authored-by: Ingolf Becker <ingolf.becker@googlemail.com> Co-authored-by: Anuj Chauhan <an12ch98@gmail.com> Co-authored-by: shaoziwei <ziwei.shao@cloudchef.io> Co-authored-by: Mathieu Chataigner <mathieu.chataigner@gmail.com> Co-authored-by: Anakael <shemeldima2@yandex.ru> Co-authored-by: Danny Chan <danny.chan@c-k.dev> Co-authored-by: Sebastiaan ten Pas <sebastiaan@diggimedia.nl> Co-authored-by: David TILLOY <d.tilloy@criteo.com> Co-authored-by: Anthony N. Simon <anthonynsimon@users.noreply.github.com> Co-authored-by: lironhl <liron.lavy@gmail.com> Co-authored-by: Raphael Cohen <raphael.cohen@sekoia.fr> Co-authored-by: JaeyoungHeo <jay.jaeyoung@gmail.com> Co-authored-by: singlaive <singlaive@gmail.com> Co-authored-by: Murphy Meng <mmeng@mirriad.com> Co-authored-by: Wu Haotian <whtsky@gmail.com> Co-authored-by: Kwist <velnik@gmail.com> Co-authored-by: Laurentiu Dragan <ldragan@bloomberg.net> Co-authored-by: Michal Čihař <michal@cihar.com> Co-authored-by: Radim Sückr <radim.suckr@gmail.com> Co-authored-by: Artem Vasilyev <artem.v.vasilyev@gmail.com> Co-authored-by: kakakikikeke-fork <kakakikikeke_new@yahoo.co.jp> Co-authored-by: Pysaoke <pysaoke@gmail.com> Co-authored-by: baixue <baixue@wecash.net> Co-authored-by: Prashant Sinha <prashantsinha@outlook.com> Co-authored-by: AbdealiJK <abdealikothari@gmail.com> * Remove Python 2 compatibility code from Celery (#6221) * Remove five from celery/__init__.py * Remove five from celery/beat.py * Remove five from celery/bootsteps.py * Remove five from celery/exceptions.py * Remove five from celery/local.py * Remove five from celery/platforms.py * Remove five from celery/result.py * Remove five from celery/schedules.py * Remove five from celery/app/amqp.py * Remove five from celery/app/annotations.py * Remove five from celery/app/backends.py * Remove five from celery/app/base.py * Remove five from celery/app/control.py * Remove five from celery/app/defaults.py * Remove five from celery/app/log.py * Remove five from celery/app/registry.py * Remove five from celery/app/routes.py * Remove five from celery/app/task.py * Remove five from celery/app/trace.py * Remove five from celery/app/utils.py * Remove five from celery/apps/beat.py * Remove five from celery/apps/multi.py * Remove five from celery/apps/worker.py * Remove five from celery/backends/database/__init__.py * Remove five from celery/backends/amqp.py * Remove five from celery/backends/asynchronous.py * Remove five from celery/backends/base.py * Remove five from celery/backends/dynamodb.py * Remove five from celery/backends/elasticsearch.py * Remove five from celery/backends/mongodb.py * Remove five from celery/backends/redis.py * Remove five from celery/backends/rpc.py * Remove five from celery/concurrency/asynpool.py * Remove five from celery/concurrency/base.py * Remove five from celery/concurrency/prefork.py * Remove five from celery/contrib/testing/manager.py * Remove five from celery/contrib/migrate.py * Remove five from celery/contrib/rdb.py * Remove five from celery/events/cursesmon.py * Remove five from celery/events/dispatcher.py * Remove five from celery/events/state.py * Remove five from celery/loaders/base.py * Remove five from celery/security/certificate.py * Remove five from celery/security/utils.py * Remove five from celery/task/base.py * Remove five from celery/utils/dispatch/signal.py * Remove five from celery/utils/abstract.py * Remove five from celery/utils/collections.py * Remove five from celery/utils/debug.py * Remove five from celery/utils/functional.py * Remove five from celery/utils/graph.py * Remove five from celery/utils/imports.py * Remove five from celery/utils/log.py * Remove five from celery/utils/saferepr.py * Remove five from celery/utils/serialization.py * Remove five from celery/utils/term.py * Remove five from celery/utils/text.py * Remove five from celery/utils/threads.py * Remove five from celery/utils/time.py * Remove five from celery/utils/timer2.py * Remove five from celery/consumer/consumer.py * Remove five from celery/consumer/gossip.py * Remove five from celery/consumer/mingle.py * Remove five from celery/worker/autoscale.py * Remove five from celery/worker/components.py * Remove five from celery/worker/control.py * Remove five from celery/worker/request.py * Remove five from celery/worker/state.py * Remove five from celery/worker/worker.py * Remove five from celery/t/benchmarks/bench_worker.py * Remove five from celery/t/integration/test_canvas.py * Remove five from celery/t/unit/app * Remove five from celery/t/unit/backends * Remove five from celery/t/unit/compat_modules * Remove five from celery/t/unit/concurrency * Remove five from celery/t/unit/contrib * Remove five from celery/t/unit/events * Remove five from celery/t/unit/security * Remove five from celery/t/unit/tasks * Remove five from celery/t/unit/utils * Remove five from celery/t/unit/worker * Sort imports. * Comment out PyPy for now. * Remove flakeplus. * Happify linter. * Fix merge problems. * Delete backport. * Remove unused import. * Remove logic that notifies user that the Python version isn't supported from setup.py. pip already does that for us. * Add a trove classifier to indicate Celery only supports Python 3. * Restore usage of `reraise` for consistency with the kombu port. * Drop Python 2 compatibility code from our Sphinx extension. * Remove mention of flakeplus from tox.ini. * Remove mention of flakeplus from our CONTRIBUTING guide. * Bump Sphinx requirement. * Remove Python 2 compatibility code from our custom Sphinx extension. * Resolve Sphinx warning due to removed section in 32ff7b45aa3d78aedca61b6554a9db39122924fd. * Remove pydocstyle from build matrix as it was removed from master. See #6278. * Bump version: 4.4.7 → 5.0.0-alpha1 * Final touches. * Fix README. * Bump Kombu to 5.0.0. * Bump version: 5.0.0-alpha1 → 5.0.0a2 * Fix wrong version. * Remove autodoc for removed module. * Remove documentation for removed methods. * Remove the riak backend since riak is no longer maintained. Co-authored-by: tothegump <tothegump@gmail.com> Co-authored-by: Omer Katz <omer.drow@gmail.com> Co-authored-by: Michal Čihař <michal@cihar.com> Co-authored-by: ptitpoulpe <ptitpoulpe@ptitpoulpe.fr> Co-authored-by: Didi Bar-Zev <didi@hiredscore.com> Co-authored-by: Santos Solorzano <santosjavier22@gmail.com> Co-authored-by: manlix <manlix@yandex.ru> Co-authored-by: Jimmy <54828848+sckhg1367@users.noreply.github.com> Co-authored-by: Борис Верховский <boris.verk@gmail.com> Co-authored-by: Jainal Gosaliya <jainal09gosalia@gmail.com> Co-authored-by: gsfish <caoyu97@hotmail.com> Co-authored-by: Dipankar Achinta <di.punk.car19@gmail.com> Co-authored-by: Pengjie Song (宋鹏捷) <spengjie@sina.com> Co-authored-by: Chris Griffin <chris-griffin@users.noreply.github.com> Co-authored-by: Muhammad Hewedy <mhewedy@gmail.com> Co-authored-by: Blaine Bublitz <blaine.bublitz@gmail.com> Co-authored-by: Tamu <tamsanh@gmail.com> Co-authored-by: Erik Tews <erik@datenzone.de> Co-authored-by: abhinav nilaratna <anilaratna2@bloomberg.net> Co-authored-by: Wyatt Paul <wpaul@hearsaycorp.com> Co-authored-by: gal cohen <gal.nevis@gmail.com> Co-authored-by: as <alfred@huji.fr> Co-authored-by: Param Kapur <paramkapur2002@gmail.com> Co-authored-by: Sven Ulland <sven.ulland@gmail.com> Co-authored-by: Safwan Rahman <safwan.rahman15@gmail.com> Co-authored-by: Aissaoui Anouar <tobia@crossbone.cc> Co-authored-by: Neal Wang <qdzzyb2015@gmail.com> Co-authored-by: Alireza Amouzadeh <alireza@amouzadeh.net> Co-authored-by: Marcos Moyano <marcos@anue.biz> Co-authored-by: Stepan Henek <stepan+github@henek.name> Co-authored-by: Andrew Sklyarov <AndrewPix@users.noreply.github.com> Co-authored-by: Michael Fladischer <michael@fladi.at> Co-authored-by: Dejan Lekic <dejan.lekic@gmail.com> Co-authored-by: Yannick Schuchmann <yannick.schuchmann@googlemail.com> Co-authored-by: Matt Davis <matteius@gmail.com> Co-authored-by: Karthikeyan Singaravelan <tir.karthi@gmail.com> Co-authored-by: Bernd Wechner <bernd-wechner@users.noreply.github.com> Co-authored-by: Sören Oldag <soeren_oldag@freenet.de> Co-authored-by: uddmorningsun <memory.yancy@gmail.com> Co-authored-by: Amar Fadil <34912365+marfgold1@users.noreply.github.com> Co-authored-by: woodenrobot <woodenrobot1993@gmail.com> Co-authored-by: Sardorbek Imomaliev <sardorbek.imomaliev@gmail.com> Co-authored-by: gsfish <root@grassfish.net> Co-authored-by: Alex Riina <alex.riina@gmail.com> Co-authored-by: Joon Hwan 김준환 <xncbf12@gmail.com> Co-authored-by: Prabakaran Kumaresshan <k_prabakaran+github@hotmail.com> Co-authored-by: Martey Dodoo <martey@mobolic.com> Co-authored-by: Konstantin Seleznev <4374093+Seleznev-nvkz@users.noreply.github.com> Co-authored-by: Prodge <Prodge@users.noreply.github.com> Co-authored-by: Abdelhadi Dyouri <raryat@gmail.com> Co-authored-by: Ixiodor <Ixiodor@users.noreply.github.com> Co-authored-by: abhishekakamai <47558404+abhishekakamai@users.noreply.github.com> Co-authored-by: Allan Lei <allanlei@h…

…ork (celery#5718) * 'abstractproperty' is deprecated. Use 'property' with 'abstractmethod' instead * Fix #2849 - Initial work of celery 5.0.0 alpha1 series by dropping python below 3.6 from marix & remove import from __future__ (#5684) * initial work of celery 5.0.0 alpha-1 series by dropping python below 3.6 * i-5651(ut): add ut for ResultSet.join_native (#5679) * dropped python versions below 3.6 from tox * dropped python versions below 3.6 from travis * dropped python versions below 3.6 from appveyor * dropped python2 compat __future__ imports from tests * Fixed a bug where canvases with a group and tasks in the middle followed by a group fails to complete and indefinitely hangs. (#5681) Fixes #5512, fixes #5354, fixes #2573. * dropped python2 compat __future__ imports from celery * dropped python2 compat code from init * revert readme change about version * removed python 2 object inheritance (#5687) * removed python 2 object inheritance * removed python 2 object inheritance * removed python 2 compatibility decorator (#5689) * removed python 2 compatibility decorator * removed python 2 compatibility decorator * removed python 2 compatibility decorator * removed python 2 compatibility decorator * Remove unused imports. * Remove unused imports of python_2_unicode_compatible. Also removed leftover useage of them where they were still used. * Run pyupgrade on codebase (#5726) * Run pyupgrade on codebase. * Use format strings where possible. * pyupgrade examples. * pyupgrade on celerydocs extension. * pyupgrade on updated code. * Address code review comments. * Address code review comments. * Remove unused imports. * Fix indentation. * Address code review comments. * Fix syntax error. * Fix syntax error. * Fix syntax error. * pytest 5.x for celery 5 (#5791) * Port latest changes from master to v5-dev (#5942) * Fix serialization and deserialization of nested exception classes (#5717) * Fix #5597: chain priority (#5759) * adding `worker_process_shutdown` to __all__ (#5762) * Fix typo (#5769) * Reformat code. * Simplify commands to looking for celery worker processes (#5778) * update doc- celery supports storage list. (#5776) * Update introduction.rst * Update introduction.rst * Fail xfailed tests if the failure is unexpected. * Added integration coverage for link_error (#5373) * Added coverage for link_error. * Use pytest-rerunfailed plugin instead of rolling our own custom implementation. * Added link_error with retries. This currently fails. * Remove unused import. * Fix import on Python 2.7. * retries in link_error do not hang the worker anymore. * Run error callbacks eagerly when the task itself is run eagerly. Fixes #4899. * Adjust unit tests accordingly. * Grammar in documentation (#5780) * Grammar in documentation * Address review. * pypy 7.2 matrix (#5790) * removed extra slashes in CELERY_BROKER_URL (#5792) The Celery broker URL in settings.py had 2 slashes in the end which are not required and can be misleading. so I changed :- CELERY_BROKER_URL = 'amqp://guest:guest@localhost//' to CELERY_BROKER_URL = 'amqp://guest:guest@localhost' * Fix #5772 task_default_exchange & task_default_exchange_type not work (#5773) * Fix #5772 task_default_exchange & task_default_exchange_type not work * Add unit test: test_setting_default_exchange * Move default_exchange test to standalone class * Run integration suite with memcached results backend. (#5739) * Fix hanging forever when fetching results from a group(chain(group)) canvas. (#5744) PR #5739 uncovered multiple problems with the cache backend. This PR should resolve one of them. PR #5638 fixed the same test case for our async results backends that support native join. However, it did not fix the test case for sync results backends that support native join. * Fix regression in PR #5681. (#5753) See comment in the diff for details. * Grammatical fix to CONTRIBUTING.rst doc (#5794) * Fix #5734 Celery does not consider authMechanism on mongodb backend URLs (#5795) * Fix #5734 Celery does not consider authMechanism on mongodb backend URLs * Add unit test: test_get_connection_with_authmechanism * Add unit test: test_get_connection_with_authmechanism_no_username * Fix errors in Python 2.7 Remove "," after "**" operator * Revert "Revert "Revert "Added handle of SIGTERM in BaseTask in celery/task.py to prevent kill the task" (#5577)" (#5586)" (#5797) This reverts commit f79894e0a2c7156fd0ca5e8e3b652b6a46a7e8e7. * Add Python 3.8 Support (#5785) * Added Python 3.8 to the build matrix. * Ensure a supported tblib version is installed for Python 3.8 and above. In addition, modernize the relevant tests. * Workaround patching problem in test. * py 3.8 in clasifier * ubuntu bionic (#5799) * ubuntu bionic * fast finish * sync bumversion with pypi release * Dev.req (#5803) * update docker config * undo hardpin * devr req install from github master * update docker config (#5801) * update docker config * make dockerfile to install from github master dev branch by default * update download link * Isort. * Grammatical & punctuation fixes for CONTRIBUTING.rst document (#5804) * update dockerfile * switched to ubuntu bionic * update docker * keep it empty until we reconfigure it again with autopep8 * Fixed Dockerfile (#5809) * Update document CONTRIBUTING.rst & fix Dockerfile typo (#5813) * Added an issue template for minor releases. * reference gocelery Go Client/Server for Celery (#5815) * Add enterprise language (#5818) * Fix/correct minor doc typos (#5825) * Correct a small typo * Correct bad contributing documentation links * Preserve the task priority in case of a retry (#5820) * Preserve the task priority in case of a retry * Created test case for retried tasks with priority * Implement an integration test for retried tasks with priorities * bump kombu * basic changelog for celery 4.4.0rc4 * bump celery 4.4.0rc4 * events bootstep disabled if no events (#5807) * events bootstep disabled if no events * Added unit tests. * update bug report template * fixing ascii art to look nicer (#5831) * Only rerun flaky tests when failures can be intermediary. * Rename Changelog to Changelog.rst * The test_nested_group_chain test can run without native_join support. (#5838) * Run integration tests with Cassandra (#5834) * Run integration tests with Cassandra. * Configure cassandra result backend * Pre-create keyspace and table * Fix deprecation warning. * Fix path to cqlsh. * Increase connection timeout. * Wait until the cluster is available. * SQS - Reject on failure (#5843) * reject on failure * add documentation * test fix * test fix * test fix * Add a concurrency model with ThreadPoolExecutor (#5099) * Add a concurrency model with ThreadPoolExecutor * thread model test for pypy * Chain primitive's code example fix in canvas documentation (Regression PR#4444) (#5845) * Changed multi-line string (#5846) This string wasn't rendering properly and was printing the python statement too. Although the change isn't as pretty code-wise, it gets rid of an annoyance for the user. * Add auto expiry for DynamoDB backend (#5805) * Add auto expiry for DynamoDB backend This adds auto-expire support for the DynamoDB backend, via the DynamoDB Time to Live feature. * Require boto3>=1.9.178 for DynamoDB TTL support boto3 version 1.9.178 requires botocore>=1.12.178. botocore version 1.12.178 introduces support for the DynamoDB UpdateTimeToLive call. The UpdateTimeToLive call is used by the DynamoDB backend to enable TTL support on a newly created table. * Separate TTL handling from table creation Handle TTL enabling/disabling separately from the table get-or-create function. Improve handling of cases where the TTL is already set to the desired state. DynamoDB only allows a single TTL update action within a fairly long time window, so some problematic cases (changing the TTL attribute, enabling/disabling TTL when it was recently modified) will raise exceptions that have to be dealt with. * Handle older boto3 versions If the boto3 TTL methods are not found, log an informative error. If the user wants to enable TTL, raise an exception; if TTL should be disabled, simply return. * Improve logging - Handle exceptions by logging the error and re-raising - Log (level debug) when the desired TTL state is already in place * Add and use _has_ttl() convenience method Additional changes: - Handle exceptions when calling boto3's describe_time_to_live() - Fix test cases for missing TTL methods * Update ttl_seconds documentation * Log invalid TTL; catch and raise ValueError * Separate method _get_table_ttl_description * Separate ttl method validation function * Clarify tri-state TTL value * Improve test coverage * Fix minor typo in comment * Mark test as xfail when using the cache backend. (#5851) * [Fix #5436] Store extending result in all backends (#5661) * [Fix #5436] Store extending result in all backends * Fix sqlalchemy * More fixu * Fixing tests * removing not necessary import * Removing debug code * Removing debug code * Add tests for get_result_meta in base and database * Revert "Add auto expiry for DynamoDB backend (#5805)" (#5855) This reverts commit f7f5bcfceca692d0e78c742a7c09c424f53d915b. * Revert "Mark test as xfail when using the cache backend. (#5851)" (#5854) This reverts commit 1b303c2968836245aaa43c3d0ff9249dd8bf9ed2. * docs: Document Redis commands used by celery (#5853) * remove cache back end integrtion test. (#5856) * Fix a race condition when publishing a very large chord header (#5850) * Added a test case which artificially introduces a delay to group.save(). * Fix race condition by delaying the task only after saving the group. * update tox * Remove duplicate boto dependency. (#5858) * Revert "remove cache back end integrtion test. (#5856)" (#5859) This reverts commit e0ac7a19a745dd5a52a615c1330bd67f2cef4d00. * Revert "Revert "Add auto expiry for DynamoDB backend (#5805)" (#5855)" (#5857) This reverts commit 4ddc605392d7694760f23069c34ede34b3e582c3. * Revert "update tox" This reverts commit 49427f51049073e38439ea9b3413978784a24999. * Fix the test_simple_chord_with_a_delay_in_group_save test. * Revert "Revert "Skip unsupported canvas when using the cache backend"" (#5860) * Revert "Revert "Mark test as xfail when using the cache backend. (#5851)" (#5854)" This reverts commit fc101c61c1912c4dafa661981f8b865c011e8a55. * Make the xfail condition stricter. * Fix the xfail condition. * Linters should use Python 3.8. * Move pypy unit tests to the correct stage. * Temporarily allow PyPy to fail since it is unavailable in Travis. * Remove unused variables. * Fix unused imports. * Fix pydocstyle errors in dynamodb. * Fix pydocstyle errors in redis backend. * bump kombu to 4.6.7 * celery 4.4.0rc5 changelog * celery 4.4.0rc5 * rm redundant code (#5864) * isort. * Document the threads task pool in the CLI. * Removed the paragraph about using librabbitmq. Refer to #5872 (#5873) * Task class definitions can have retry attributes (#5869) * autoretry_for * retry_kwargs * retry_backoff * retry_backoff_max * retry_jitter can now be defined as cls attributes. All of these can be overriden from the @task decorator https://github.com/celery/celery/issues/4684 * whatsnew in Celery 4.4 as per projects standard (#5817) * 4.4 whatsnew * update * update * Move old whatsnew to history. * Remove old news & fix markers. * Added a section notifying Python 3.4 has been dropped. * Added a note about ElasticSearch basic auth. * Added a note about being able to replace eagerly run tasks. * Update index. * Address comment. * Described boto3 version updates. * Fix heading. * More news. * Thread pool. * Add Django and Config changes * Bump version 4.4.0 * upate readme * Update docs regarding Redis Message Priorities (#5874) * Update docs regarding Redis Message Priorities * fixup! Update docs regarding Redis Message Priorities * Update 4.4.0 docs (#5875) * Update 4.4 release changelog * Update whatsnew-4.4 * Update tasks docs * Fix recent tasks doc file update (#5879) * Include renamed Changelog.rst in source releases. (#5880) Changelog.rst was renamed from Changelog in fd023ec174bedc2dc65c63a0dc7c85e425ac00c6 but MANIFEST.in was not updated to include the new name. This fixes the file name so Changelog.rst will show up in future source releases again. * Reorganised project_urls and classifiers. (#5884) * Use safequote in SQS Getting Started doc (#5885) * Have appveyor build relevant versions of Python. (#5887) * Have appveyor build relevant and buildable versions of Python. * Appveyor is missing CI requirements to build. * Pin pycurl to version that will build with appveyor (because wheels files exist) * Restrict python 2.7 64 bit version of python-dateutil for parse. * Use is_alive instead of isAlive for Python 3.9 compatibility. (#5898) * Very minor tweak to commen to improve docs (#5900) As discussed here: https://stackoverflow.com/questions/58816271/celery-task-asyncresult-takes-task-id-but-is-documented-to-get-asyncresult-inst this comment seems to flow to a very confusing and misleading piece of documentation here: https://docs.celeryproject.org/en/latest/reference/celery.app.task.html#celery.app.task.Task.AsyncResult * Support configuring schema of a PostgreSQL database (#5910) * Support configuring schema of a PostgreSQL database * Add unit test * Remove blank line * Fix raise issue to make exception message more friendly (#5912) Signed-off-by: Chenyang Yan <memory.yancy@gmail.com> * Add progress for retry connections (#5915) This will show current retry progress so it will clear confusion about how many retries will be tried for connecting to broker. Closes #4556 * chg: change xrange to range (#5926) * update docs for json serializer and add note for int keys serialization (#5932) * fix indentation for note block in calling.rst (#5933) * Added links to other issue trackers. (#5939) * Add labels automatically for issues. (#5938) * Run pyupgrade. Co-authored-by: Michal Čihař <michal@cihar.com> Co-authored-by: ptitpoulpe <ptitpoulpe@ptitpoulpe.fr> Co-authored-by: Didi Bar-Zev <didi@hiredscore.com> Co-authored-by: Santos Solorzano <santosjavier22@gmail.com> Co-authored-by: manlix <manlix@yandex.ru> Co-authored-by: Jimmy <54828848+sckhg1367@users.noreply.github.com> Co-authored-by: Борис Верховский <boris.verk@gmail.com> Co-authored-by: Asif Saif Uddin <auvipy@gmail.com> Co-authored-by: Jainal Gosaliya <jainal09gosalia@gmail.com> Co-authored-by: gsfish <caoyu97@hotmail.com> Co-authored-by: Dipankar Achinta <di.punk.car19@gmail.com> Co-authored-by: Pengjie Song (宋鹏捷) <spengjie@sina.com> Co-authored-by: Chris Griffin <chris-griffin@users.noreply.github.com> Co-authored-by: Muhammad Hewedy <mhewedy@gmail.com> Co-authored-by: Blaine Bublitz <blaine.bublitz@gmail.com> Co-authored-by: Tamu <tamsanh@gmail.com> Co-authored-by: Erik Tews <erik@datenzone.de> Co-authored-by: abhinav nilaratna <anilaratna2@bloomberg.net> Co-authored-by: Wyatt Paul <wpaul@hearsaycorp.com> Co-authored-by: gal cohen <gal.nevis@gmail.com> Co-authored-by: as <alfred@huji.fr> Co-authored-by: Param Kapur <paramkapur2002@gmail.com> Co-authored-by: Sven Ulland <sven.ulland@gmail.com> Co-authored-by: Safwan Rahman <safwan.rahman15@gmail.com> Co-authored-by: Aissaoui Anouar <tobia@crossbone.cc> Co-authored-by: Neal Wang <qdzzyb2015@gmail.com> Co-authored-by: Alireza Amouzadeh <alireza@amouzadeh.net> Co-authored-by: Marcos Moyano <marcos@anue.biz> Co-authored-by: Stepan Henek <stepan+github@henek.name> Co-authored-by: Andrew Sklyarov <AndrewPix@users.noreply.github.com> Co-authored-by: Michael Fladischer <michael@fladi.at> Co-authored-by: Dejan Lekic <dejan.lekic@gmail.com> Co-authored-by: Yannick Schuchmann <yannick.schuchmann@googlemail.com> Co-authored-by: Matt Davis <matteius@gmail.com> Co-authored-by: Karthikeyan Singaravelan <tir.karthi@gmail.com> Co-authored-by: Bernd Wechner <bernd-wechner@users.noreply.github.com> Co-authored-by: Sören Oldag <soeren_oldag@freenet.de> Co-authored-by: uddmorningsun <memory.yancy@gmail.com> Co-authored-by: Amar Fadil <34912365+marfgold1@users.noreply.github.com> Co-authored-by: woodenrobot <woodenrobot1993@gmail.com> Co-authored-by: Sardorbek Imomaliev <sardorbek.imomaliev@gmail.com> * Remove fallback code for Python 2 support marked with TODOs. (#5953) Co-authored-by: Asif Saif Uddin <auvipy@gmail.com> * Remove PY3 conditionals (#5954) * Added integration coverage for link_error (#5373) * Added coverage for link_error. * Use pytest-rerunfailed plugin instead of rolling our own custom implementation. * Added link_error with retries. This currently fails. * Remove unused import. * Fix import on Python 2.7. * retries in link_error do not hang the worker anymore. * Run error callbacks eagerly when the task itself is run eagerly. Fixes #4899. * Adjust unit tests accordingly. * Grammar in documentation (#5780) * Grammar in documentation * Address review. * pypy 7.2 matrix (#5790) * removed extra slashes in CELERY_BROKER_URL (#5792) The Celery broker URL in settings.py had 2 slashes in the end which are not required and can be misleading. so I changed :- CELERY_BROKER_URL = 'amqp://guest:guest@localhost//' to CELERY_BROKER_URL = 'amqp://guest:guest@localhost' * Fix #5772 task_default_exchange & task_default_exchange_type not work (#5773) * Fix #5772 task_default_exchange & task_default_exchange_type not work * Add unit test: test_setting_default_exchange * Move default_exchange test to standalone class * Run integration suite with memcached results backend. (#5739) * Fix hanging forever when fetching results from a group(chain(group)) canvas. (#5744) PR #5739 uncovered multiple problems with the cache backend. This PR should resolve one of them. PR #5638 fixed the same test case for our async results backends that support native join. However, it did not fix the test case for sync results backends that support native join. * Fix regression in PR #5681. (#5753) See comment in the diff for details. * Grammatical fix to CONTRIBUTING.rst doc (#5794) * Fix #5734 Celery does not consider authMechanism on mongodb backend URLs (#5795) * Fix #5734 Celery does not consider authMechanism on mongodb backend URLs * Add unit test: test_get_connection_with_authmechanism * Add unit test: test_get_connection_with_authmechanism_no_username * Fix errors in Python 2.7 Remove "," after "**" operator * Revert "Revert "Revert "Added handle of SIGTERM in BaseTask in celery/task.py to prevent kill the task" (#5577)" (#5586)" (#5797) This reverts commit f79894e0a2c7156fd0ca5e8e3b652b6a46a7e8e7. * Add Python 3.8 Support (#5785) * Added Python 3.8 to the build matrix. * Ensure a supported tblib version is installed for Python 3.8 and above. In addition, modernize the relevant tests. * Workaround patching problem in test. * py 3.8 in clasifier * ubuntu bionic (#5799) * ubuntu bionic * fast finish * sync bumversion with pypi release * Dev.req (#5803) * update docker config * undo hardpin * devr req install from github master * update docker config (#5801) * update docker config * make dockerfile to install from github master dev branch by default * update download link * Isort. * Grammatical & punctuation fixes for CONTRIBUTING.rst document (#5804) * update dockerfile * switched to ubuntu bionic * update docker * keep it empty until we reconfigure it again with autopep8 * Fixed Dockerfile (#5809) * Update document CONTRIBUTING.rst & fix Dockerfile typo (#5813) * Added an issue template for minor releases. * reference gocelery Go Client/Server for Celery (#5815) * Add enterprise language (#5818) * Fix/correct minor doc typos (#5825) * Correct a small typo * Correct bad contributing documentation links * Preserve the task priority in case of a retry (#5820) * Preserve the task priority in case of a retry * Created test case for retried tasks with priority * Implement an integration test for retried tasks with priorities * bump kombu * basic changelog for celery 4.4.0rc4 * bump celery 4.4.0rc4 * events bootstep disabled if no events (#5807) * events bootstep disabled if no events * Added unit tests. * update bug report template * fixing ascii art to look nicer (#5831) * Only rerun flaky tests when failures can be intermediary. * Rename Changelog to Changelog.rst * The test_nested_group_chain test can run without native_join support. (#5838) * Run integration tests with Cassandra (#5834) * Run integration tests with Cassandra. * Configure cassandra result backend * Pre-create keyspace and table * Fix deprecation warning. * Fix path to cqlsh. * Increase connection timeout. * Wait until the cluster is available. * SQS - Reject on failure (#5843) * reject on failure * add documentation * test fix * test fix * test fix * Add a concurrency model with ThreadPoolExecutor (#5099) * Add a concurrency model with ThreadPoolExecutor * thread model test for pypy * Chain primitive's code example fix in canvas documentation (Regression PR#4444) (#5845) * Changed multi-line string (#5846) This string wasn't rendering properly and was printing the python statement too. Although the change isn't as pretty code-wise, it gets rid of an annoyance for the user. * Add auto expiry for DynamoDB backend (#5805) * Add auto expiry for DynamoDB backend This adds auto-expire support for the DynamoDB backend, via the DynamoDB Time to Live feature. * Require boto3>=1.9.178 for DynamoDB TTL support boto3 version 1.9.178 requires botocore>=1.12.178. botocore version 1.12.178 introduces support for the DynamoDB UpdateTimeToLive call. The UpdateTimeToLive call is used by the DynamoDB backend to enable TTL support on a newly created table. * Separate TTL handling from table creation Handle TTL enabling/disabling separately from the table get-or-create function. Improve handling of cases where the TTL is already set to the desired state. DynamoDB only allows a single TTL update action within a fairly long time window, so some problematic cases (changing the TTL attribute, enabling/disabling TTL when it was recently modified) will raise exceptions that have to be dealt with. * Handle older boto3 versions If the boto3 TTL methods are not found, log an informative error. If the user wants to enable TTL, raise an exception; if TTL should be disabled, simply return. * Improve logging - Handle exceptions by logging the error and re-raising - Log (level debug) when the desired TTL state is already in place * Add and use _has_ttl() convenience method Additional changes: - Handle exceptions when calling boto3's describe_time_to_live() - Fix test cases for missing TTL methods * Update ttl_seconds documentation * Log invalid TTL; catch and raise ValueError * Separate method _get_table_ttl_description * Separate ttl method validation function * Clarify tri-state TTL value * Improve test coverage * Fix minor typo in comment * Mark test as xfail when using the cache backend. (#5851) * [Fix #5436] Store extending result in all backends (#5661) * [Fix #5436] Store extending result in all backends * Fix sqlalchemy * More fixu * Fixing tests * removing not necessary import * Removing debug code * Removing debug code * Add tests for get_result_meta in base and database * Revert "Add auto expiry for DynamoDB backend (#5805)" (#5855) This reverts commit f7f5bcfceca692d0e78c742a7c09c424f53d915b. * Revert "Mark test as xfail when using the cache backend. (#5851)" (#5854) This reverts commit 1b303c2968836245aaa43c3d0ff9249dd8bf9ed2. * docs: Document Redis commands used by celery (#5853) * remove cache back end integrtion test. (#5856) * Fix a race condition when publishing a very large chord header (#5850) * Added a test case which artificially introduces a delay to group.save(). * Fix race condition by delaying the task only after saving the group. * update tox * Remove duplicate boto dependency. (#5858) * Revert "remove cache back end integrtion test. (#5856)" (#5859) This reverts commit e0ac7a19a745dd5a52a615c1330bd67f2cef4d00. * Revert "Revert "Add auto expiry for DynamoDB backend (#5805)" (#5855)" (#5857) This reverts commit 4ddc605392d7694760f23069c34ede34b3e582c3. * Revert "update tox" This reverts commit 49427f51049073e38439ea9b3413978784a24999. * Fix the test_simple_chord_with_a_delay_in_group_save test. * Revert "Revert "Skip unsupported canvas when using the cache backend"" (#5860) * Revert "Revert "Mark test as xfail when using the cache backend. (#5851)" (#5854)" This reverts commit fc101c61c1912c4dafa661981f8b865c011e8a55. * Make the xfail condition stricter. * Fix the xfail condition. * Linters should use Python 3.8. * Move pypy unit tests to the correct stage. * Temporarily allow PyPy to fail since it is unavailable in Travis. * Remove unused variables. * Fix unused imports. * Fix pydocstyle errors in dynamodb. * Fix pydocstyle errors in redis backend. * bump kombu to 4.6.7 * celery 4.4.0rc5 changelog * celery 4.4.0rc5 * rm redundant code (#5864) * isort. * Document the threads task pool in the CLI. * Removed the paragraph about using librabbitmq. Refer to #5872 (#5873) * Task class definitions can have retry attributes (#5869) * autoretry_for * retry_kwargs * retry_backoff * retry_backoff_max * retry_jitter can now be defined as cls attributes. All of these can be overriden from the @task decorator https://github.com/celery/celery/issues/4684 * whatsnew in Celery 4.4 as per projects standard (#5817) * 4.4 whatsnew * update * update * Move old whatsnew to history. * Remove old news & fix markers. * Added a section notifying Python 3.4 has been dropped. * Added a note about ElasticSearch basic auth. * Added a note about being able to replace eagerly run tasks. * Update index. * Address comment. * Described boto3 version updates. * Fix heading. * More news. * Thread pool. * Add Django and Config changes * Bump version 4.4.0 * upate readme * Update docs regarding Redis Message Priorities (#5874) * Update docs regarding Redis Message Priorities * fixup! Update docs regarding Redis Message Priorities * Update 4.4.0 docs (#5875) * Update 4.4 release changelog * Update whatsnew-4.4 * Update tasks docs * Fix recent tasks doc file update (#5879) * Include renamed Changelog.rst in source releases. (#5880) Changelog.rst was renamed from Changelog in fd023ec174bedc2dc65c63a0dc7c85e425ac00c6 but MANIFEST.in was not updated to include the new name. This fixes the file name so Changelog.rst will show up in future source releases again. * Reorganised project_urls and classifiers. (#5884) * Use safequote in SQS Getting Started doc (#5885) * Have appveyor build relevant versions of Python. (#5887) * Have appveyor build relevant and buildable versions of Python. * Appveyor is missing CI requirements to build. * Pin pycurl to version that will build with appveyor (because wheels files exist) * Restrict python 2.7 64 bit version of python-dateutil for parse. * Use is_alive instead of isAlive for Python 3.9 compatibility. (#5898) * Very minor tweak to commen to improve docs (#5900) As discussed here: https://stackoverflow.com/questions/58816271/celery-task-asyncresult-takes-task-id-but-is-documented-to-get-asyncresult-inst this comment seems to flow to a very confusing and misleading piece of documentation here: https://docs.celeryproject.org/en/latest/reference/celery.app.task.html#celery.app.task.Task.AsyncResult * Support configuring schema of a PostgreSQL database (#5910) * Support configuring schema of a PostgreSQL database * Add unit test * Remove blank line * Fix raise issue to make exception message more friendly (#5912) Signed-off-by: Chenyang Yan <memory.yancy@gmail.com> * Add progress for retry connections (#5915) This will show current retry progress so it will clear confusion about how many retries will be tried for connecting to broker. Closes #4556 * chg: change xrange to range (#5926) * update docs for json serializer and add note for int keys serialization (#5932) * fix indentation for note block in calling.rst (#5933) * Added links to other issue trackers. (#5939) * Add labels automatically for issues. (#5938) * remove redundant raise from docstring (#5941) `throw` is True by default so the Retry exception will already get raised by calling `self.retry(countdown=60 * 5, exc=exc)` * Run pyupgrade. * Fix typo (#5943) * Remove fallback code for Python 2 support. * docs: fixes Rabbits and Warrens link in routing userguide (#4007) (#5949) * Fix labels on Github issue templates. (#5955) Use quotation marks to escape labels on Github issue templates. This prevents the colon from breaking the template. * added retry_on_timeout and socket_keepalive to config and doc (#5952) * Fixed event capture from building infinite list (#5870) * Fix error propagation example (#5966) * update range (#5971) * update setup.cfg * bump billiard to 3.6.3.0 * Update __init__.py (#5951) * Update __init__.py Fixed issue for object with result_backend=True (decode fails on multiple None request) * Update __init__.py suggested changeds * Update __init__.py * Use configured db schema also for sequences (#5972) * Added a default value for retries in worker.strategy. (#5945) * Added a default value for retries in worker.strategy. I was facing an issue when adding tasks directly to rabbitmq using pika instead of calling task.apply_async. The issue was the self.retry mechanisum was failing. In app/tasks.py the line `retries = request.retries + 1` was causing the issue. On further tracing I figured out that it was because the default .get value (None) was getting passed through this function and was raising TypeError: unsupported operand type(s) for +: 'NoneType' and 'int' * Add test cases for default and custom retries value * pypy 7.3 (#5980) * Pass `interval` to `get_many` (#5931) * Pass `interval` to `get_many` * Fix: Syntax error for py2.7 * Fix: Syntax error for py2.7 * Fixed problem with conflicting autoretry_for task parameter and Task.replace() (#5934) * Fix #5917 (#5918) * Fix changelog (#5881) * merge in place the apps beat schedule in the default Schedule class. (#5908) * Handle Redis connection errors in result consumer (#5921) * Handle Redis connection errors in result consumer * Closes #5919. * Use context manager for Redis conusmer reconnect * Log error when result backend reconnection fails * Fix inspect_command documentation (#5983) * Use gevent and eventlet wait() functions to remove busy-wait (#5974) * Use gevent and eventlet wait() functions to remove busy-wait Fixes issue #4999. Calling AsyncResult.get() in a gevent context would cause the async Drainer to repeatedly call wait_for until the result was completed. I've updated the code to have a specific implementation for gevent and eventlet that will cause wait_for to only return every "timeout" # of seconds, rather than repeatedly returning. Some things I'd like some feedback on: * Where's the best place to add test coverage for this? It doesn't look like there are any tests that directly exercised the Drainer yet so I would probably look to add some of these to the backends/ unit tests. * The way I did this for the Eventlet interface was to rely on the private _exit_event member of the GreenThread instance; to do this without relying on a private member would require some additional changes to the backend Drainer interface so that we could wait for an eventlet-specific event in wait_for(). I can do this, just wanted to get some feedback before. * Add unit tests for Drainer classes In order for this to work without monkeypatching in the tests, I needed to call sleep(0) to let the gevent/eventlet greenlets to yield control back to the calling thread. I also made the check interval configurable in the drainer so that we didn't need to sleep multiples of 1 second in the tests. * Weaken asserts since they don't pass on CI * Fix eventlet auto-patching DNS resolver module on import By default it looks like "import eventlet" imports the greendns module unless the environment EVENTLET_NO_GREENDNS is set to true. This broke a pymongo test. * Add tests ensuring that the greenlet loop isn't blocked These tests make sure that while drain_events_until is running that other gevent/eventlet concurrency can run. * Clean up tests and make sure they wait for all the threads to stop * Fix chords with chained groups (#5947) * kombu 4.6.8 * update setup * updated version 4.4.1 * Fix: Accept and swallow `kwargs` to handle unexpected keyword arguments * Allow boto to look for credentials in S3Backend * add reference to Rusty Celery * Update document of revoke method in Control class * Fix copy-paste error in result_compression docs * Make 'socket_keepalive' optional variable (#6000) * update connection params - socket_keepalive is optional now * update readme - added versionadded 4.4.1 and fixed `redis_socket_keepalive` * added check of socket_keepalive in arguments for UnixSocketConnect * Fixed incorrect setting name in documentation (#6002) * updated version 4.4.2 * Fix backend utf-8 encoding in s3 backend Celery backend uses utf-8 to deserialize results, which would fail for some serializations like pickle. * Fix typo in celery.bin.multi document * Upgraded pycurl to the latest version that supports wheel. * pytest 5.3.5 max * Add uptime to the stats inspect command * Doc tweaks: mostly grammar and punctuation (#6016) * Fix a bunch of comma splices in the docs * Remove some unnecessary words from next-steps doc * Tweak awkward wording; fix bad em-dash * Fix a bunch more comma splices in next-steps doc * Miscellaneous grammar/punctuation/wording fixes * Link to task options in task decorator docs * Fixing issue #6019: unable to use mysql SSL parameters when getting mysql engine (#6020) * Fixing issue #6019: unable to use mysql SSL parametes in create_engine() * adding test for get_engine when self.forked is False and engine args are passed in for create_engine() * Clean TraceBack to reduce memory leaks for exception task (#6024) * Clean TraceBack to reduce memory leaks * add unit test * add unit test * reject unittest * Patch For Python 2.7 compatibility * update unittest * Register to the garbage collector by explicitly referring to f_locals. * need more check * update code coverage * update Missing unit test * 3.4 -> 3.5 Co-authored-by: heedong.jung <heedong.jung@samsung.com> * exceptions: NotRegistered: fix up language Minor fix to the language. * Note about autodiscover_tasks and periodic tasks This is particularly important for Django projects that put periodic tasks into each app's `tasks.py` and want to use one as a periodic task. By the time `autodiscover_tasks()` loads those tasks, the `on_after_configure` Signal has already come and gone, so anything decorated with `@app.on_after_finalize.connect` will never be called. If there's other documentation on this subject, I could not find it. * Avoid PyModulelevel, deprecated in Sphinx 4 Use `PyFunction` instead of `PyModulelevel` to avoid this deprecation warning: RemovedInSphinx40Warning: PyModulelevel is deprecated. Please check the implementation of <class 'celery.contrib.sphinx.TaskDirective'> This replacement is one of the options listed in the Sphinx docs (https://www.sphinx-doc.org/en/master/extdev/deprecated.html). * Give up sending a worker-offline message if transport is not connected (#6039) * If worker-offline event fails to send, give up and die peacefully * Add test for retry= and msgs in heartbeat * Fix the build and all documentation warnings. I finally upgraded our theme to 2.0. As a result we've upgraded Sphinx to 2.0. Work to upgrade Sphinx to 3.0 will proceed in a different PR. This upgrade also fixes our build issues caused by #6032. We don't support Sphinx 1.x as a result of that patch. I've also included the missing 4.3 changelog to our history. * Support both Sphinx 2 and 3. * Add Task to __all__ in celery.__init__.py * Add missing parenthesis to example in docs * Ensure a single chain object in a chain does not raise MaximumRecursionError. Previously chain([chain(sig)]) would crash. We now ensure it doesn't. Fixes #5973. * update setup.py * fix typo missing quote at the end of line * Fix a typo in monitoring doc * update travis * update ubuntu to focal foss 20.04 LTS * Fix autoscale when prefetch_multiplier is 1 * Allow start_worker to function without ping task * Update celeryd.conf Move the directory of the program before the execution of the command/script * Add documentation for "predefined_queue_urls" * [Fix #6074]: Add missing documentation for MongoDB as result backend. * update funding * 🐛 Correctly handle configuring the serializer for always_eager mode. (#6079) * 🐛 Correctly handle configuring the serializer for always_eager mode. options['serializer'] will always exist, because it is initialized from an mattrgetter. Even if unset, it will be present in the options with a value of None. * 🐛 Add a test for new always_eager + task_serializer behavior. * ✏️ Whoops missed a : * Remove doubling of prefetch_count increase when prefetch_multiplier gt 1 (#6081) * try ubuntu focal (#6088) * Fix eager function not returning result after retries. Using apply function does not return correct results after at least one retry because the return value of successive call is not going back to the original caller. * return retry result if not throw and is_eager if throw is false, we would be interested by the result of retry and not the current result which will be an exception. This way it does not break the logic of `raise self.retry` This should be used like `return self.retry(..., throw=False)` in an except statement. * revert formatting change * Add tests for eager retry without throw * update predefined-queues documentation Suggested version of configuration does not work. Additionally I'd like to mention, that `access_key_id` and `secret_access_key` are mandatory fields and not allowing you to go with defaults AWS_* env variables. I can contribute for this variables to be optional Also I'm not sure if security token will apply, could you please advice how to do it? * Fix couchbase version < 3.0.0 as API changed * Remove reference to -O fair in optimizations -O fair was made the default in Celery 4.0 https://docs.celeryproject.org/en/stable/history/whatsnew-4.0.html#ofair-is-now-the-default-scheduling-strategy * pytest ranges * pypy3 * revert to bionic * do not load docs.txt requirements for python 2.7 As it requires Sphinx >= 2.0.0 and there is no such version compatible with python 2.7 * update cassandra travis integration test configuration cassandra:latest docker image changed location of cqlsh program * pin cassandra-driver CI get stuck after all cassandra integration tests * Fix all flake8 lint errors * Fix all pydocstyle lint errors * Fix all configcheck lint errors * Always requeue while worker lost regardless of the redelivered flag (#6103) * #5598 fix, always redelivery while WorkerLostError * fix, change the requeue flag so the task will remain PENDING * Allow relative paths in the filesystem backend (#6070) * Allow relative paths in the filesystem backend * fix order of if statements * [Fixed Issue #6017] --> Added Multi default logfiles and pidfiles paths [Description]: --> Changed the default paths for log files & pid files to be '/var/log/celery' and '/var/run/celery' --> Handled by creating the respective paths if not exist. --> Used os.makedir(path,if_exists=True) [Unit Test Added]: --> .travis.yml - config updated with 'before install'. --> t/unit/apps/test_multi.py - Changed the default log files & pid files paths wherever required. * Avoid race condition due to task duplication. In some circumstances like a network partitioning, some tasks might be duplicated. Sometimes, this lead to a race condition where a lost task overwrites the result of the last successful task in the backend. In order to avoid this race condition we prevent updating the result if it's already in successful state. This fix has been done for KV backends only and therefore won't work with other backends. * adding tests * Exceptions must be old-style classes or derived from BaseException, but here self.result may not subclass of BaseException. * update fund link * Fix windows build (#6104) * do not load memcache nor couchbase lib during windows build those libraries depends on native libraries libcouchbase and libmemcached that are not installed on Appveyor. As only unit tests runs on Appveyor, it should be fine * Add python 3.8 workaround for app trap * skip tests file_descriptor_safety tests on windows AsyncPool is not supported on Windows so Pool does have _fileno_to_outq attribute, making the test fail * Fix crossplatform log and pid files in multi mode it relates to #6017 * Use tox to build and test on windows * remove tox_install_command * drop python 2.7 from windows build * Add encode to meta task in base.py (#5894) * Add encode to base.py meta result Fix bug with impossibility to load None from task meta * Add tests for None. Remove exceed encode. * Update base.py Add return payload if None * Update time.py to solve the microsecond issues (#5199) When `relative` is set to True, the day, hour, minutes second will be round to the nearest one, however, the original program do not update the microsecond (reset it). As a result, the run-time offset on the microsecond will then be accumulated. For example, given the interval is 15s and relative is set to True 1. 2018-11-27T15:01:30.123236+08:00 2. 2018-11-27T15:01:45.372687+08:00 3. 2018-11-27T15:02:00.712601+08:00 4. 2018-11-27T15:02:15.987720+08:00 5. 2018-11-27T15:02:31.023670+08:00 * Change backend _ensure_not_eager error to warning * Add priority support for 'celery.chord_unlock' task (#5766) * Change eager retry behaviour even with raise self.retry, it should return the eventual value or MaxRetriesExceededError. if return value of eager apply is Retry exception, retry eagerly the task signature * Order supported Python versions * Avoid race condition in elasticsearch backend if a task is retried, the task retry may work concurrently to current task. store_result may come out of order. it may cause a non ready state (Retry) to override a ready state (Success, Failure). If this happens, it will block indefinitely pending any chord depending on this task. this change makes document updates safe for concurrent writes. https://www.elastic.co/guide/en/elasticsearch/reference/current/optimistic-concurrency-control.html * backends base get_many pass READY_STATES arg * test backends base get_many pass READY_STATES arg * Add integration tests for Elasticsearch and fix _update * Revert "revert to bionic" This reverts commit 6e091573f2ab0d0989b8d7c26b677c80377c1721. * remove jython check * feat(backend): Adds cleanup to ArangoDB backend * Delete Document Known Issue with CONN_MAX_AGE in 4.3 * issue 6108 fix filesystem backend cannot not be serialized by picked (#6120) * issue 6108 fix filesystem backend cannot not be serialized by picked https://github.com/celery/celery/issues/6108 * issue-6108 fix unit test failure * issue-6108 fix flake8 warning Co-authored-by: Murphy Meng <mmeng@mirriad.com> * kombu==4.6.9 (#6133) * changelog for 4.4.3 * v 4.4.3 * remove un supported classifier * Fix autoretry_for with explicit retry (#6138) * Add tests for eager task retry * Fixes #6135 If autoretry_for is set too broad on Exception, then autoretry may get a Retry if that's the case, rethrow directly instead of wrapping it in another Retry to avoid loosing new args * Use Django DB max age connection setting (fixes #4116) * Add retry on recoverable exception for the backend (#6122) * Add state to KeyValueStoreBackend.set method This way, a backend implementation is able to take decisions based on current state to store meta in case of failures. * Add retry on recoverable exception for the backend acks.late makes celery acknowledge messages only after processing and storing result on the backend. However, in case of backend unreachable, it will shadow a Retry exception and put the task as failed in the backend not retrying the task and acknoledging it on the broker. With this new result_backend_always_retry setting, if the backend exception is recoverable (to be defined per backend implementation), it will retry the backend operation with an exponential backoff. * Make elasticsearch backward compatible with 6.x * Make ES retry storing updates in a better way if existing value in the backend is success, then do nothing. if it is a ready status, then update it only if new value is a ready status as well. else update it. This way, a SUCCESS cannot be overriden so that we do not loose results but any ready state other than success (FAILURE, REVOKED) can be overriden by another ready status (i.e. a SUCCESS) * Add test for value not found in ES backend * Fix random distribution of jitter for exponential backoff random.randrange should be called with the actual so that all numbers have equivalent probability, otherwise maximum value does have a way higher probability of occuring. * fix unit test if extra modules are not present * ElasticSearch: add setting to save meta as json * fix #6136. celery 4.4.3 always trying create /var/run/celery directory (#6142) * fix #6136. celery 4.4.3 always trying create /var/run/celery directory, even if it's not needed. * fix #6136. cleanup * Add task_internal_error signal (#6049) * Add internal_error signal There is no special signal for an out of body error which can be the result of a bad result backend. * Fix syntax error. * Document the task_internal_error signal. Co-authored-by: Laurentiu Dragan <ldragan@bloomberg.net> * changelog for v4.4.4 * kombu 4.6.10 (#6144) * v4.4.4 * Add missing dependency on future (#6146) Fixes #6145 * ElasticSearch: Retry index if document was deleted between index and update (#6140) * ElasticSearch: Retry index if document was deleted between index and update * Elasticsearch increase coverage to 100% * Fix pydocstyle * Specify minimum version of Sphinx for Celery extension (#6150) The Sphinx extension requires Sphinx 2 or later due to #6032. * fix windows build * fix flake8 error * fix multi tests in local Mock os.mkdir and os.makedirs to avoid creating /var/run/celery and /var/log/celery during unit tests if run without root priviledges * Customize the retry interval of chord_unlock tasks * changelog v4.4.5 * v4.4.5 * Fix typo in comment. * Remove autoscale force_scale methods (#6085) * Remove autoscale force_scale methods * Remove unused variable in test * Pass ping destination to request The destination argument worked fine from CLI but didn't get used when calling ping from Python. * Fix autoscale test * chord: merge init options with run options * put back KeyValueStoreBackend.set method without state It turns out it was breaking some other projects. wrapping set method with _set_with_state, this way it will not break existing Backend. while enabling this feature for other Backend. Currently, only ElasticsearchBackend supports this feature. It protects concurrent update to corrupt state in the backend. Existing success cannot be overriden, nor a ready state by a non ready state. i.e. a Retry state cannot override a Success or Failure. As a result, chord_unlock task will not loop forever due to missing ready state on the backend. * added --range-prefix option to `celery multi` (#6180) * added --range-prefix option to `celery multi` Added option for overriding default range prefix when running multiple workers prividing range with `celery multy` command. * covered multi --range-prefix with tests * fixed --range-prefix test * Added as_list function to AsyncResult class (#6179) * Add as_list method to return task IDs as a list * Add a test for as_list method * Add docstring for as_list method * Fix CassandraBackend error in threads or gevent pool (#6147) * Fix CassandraBackend error in threads or gevent pool * remove CassandraBackend.process_cleanup * Add test case * Add test case * Add comments test_as_uri Co-authored-by: baixue <baixue@wecash.net> * changelog for v4.4.6 * v4.4.6 * Update Wiki link in "resources" In the page linked below, the link to wiki is outdated. Fixed that. https://docs.celeryproject.org/en/stable/getting-started/resources.html * test_canvas: Add test for chord-in-chain Add test case for the issue where a chord in a chain does not work when using .apply(). This works fine with .apply_async(). * Trying to fix flaky tests in ci * fix pydocstyle errors * fix pydocstyle * Drainer tests, put a lower constraint on number of intervals liveness should iterate 10 times per interval while drain_events only once. However, as it may use thread that may be scheduled out of order, we may end up in some situation where liveness and drain_events were called the same amount of time. Lowering the constraint from < to <= to avoid failing the tests. * pyupgrade. * Fix merge error. Co-authored-by: Борис Верховский <boris.verk@gmail.com> Co-authored-by: Asif Saif Uddin <auvipy@gmail.com> Co-authored-by: Jainal Gosaliya <jainal09gosalia@gmail.com> Co-authored-by: gsfish <root@grassfish.net> Co-authored-by: Dipankar Achinta <di.punk.car19@gmail.com> Co-authored-by: spengjie <spengjie@sina.com> Co-authored-by: Chris Griffin <chris-griffin@users.noreply.github.com> Co-authored-by: Muhammad Hewedy <mhewedy@gmail.com> Co-authored-by: Blaine Bublitz <blaine.bublitz@gmail.com> Co-authored-by: Tamu <tamsanh@gmail.com> Co-authored-by: Erik Tews <erik@datenzone.de> Co-authored-by: abhinav nilaratna <anilaratna2@bloomberg.net> Co-authored-by: Wyatt Paul <wpaul@hearsaycorp.com> Co-authored-by: gal cohen <gal.nevis@gmail.com> Co-authored-by: whuji <alfred@huji.fr> Co-authored-by: Param Kapur <paramkapur2002@gmail.com> Co-authored-by: Sven Ulland <sven.ulland@gmail.com> Co-authored-by: Safwan Rahman <safwan.rahman15@gmail.com> Co-authored-by: Aissaoui Anouar <tobia@crossbone.cc> Co-authored-by: Neal Wang <qdzzyb2015@gmail.com> Co-authored-by: Alireza Amouzadeh <alireza@amouzadeh.net> Co-authored-by: Marcos Moyano <marcos@anue.biz> Co-authored-by: Stepan Henek <stepan+github@henek.name> Co-authored-by: Andrew Sklyarov <AndrewPix@users.noreply.github.com> Co-authored-by: Michael Fladischer <michael@fladi.at> Co-authored-by: Dejan Lekic <dejan.lekic@gmail.com> Co-authored-by: Yannick Schuchmann <yannick.schuchmann@googlemail.com> Co-authored-by: Matt Davis <matteius@gmail.com> Co-authored-by: Xtreak <tir.karthi@gmail.com> Co-authored-by: Bernd Wechner <bernd-wechner@users.noreply.github.com> Co-authored-by: Sören Oldag <soeren_oldag@freenet.de> Co-authored-by: uddmorningsun <memory.yancy@gmail.com> Co-authored-by: Amar Fadil <34912365+marfgold1@users.noreply.github.com> Co-authored-by: woodenrobot <woodenrobot1993@gmail.com> Co-authored-by: Sardorbek Imomaliev <sardorbek.imomaliev@gmail.com> Co-authored-by: Alex Riina <alex.riina@gmail.com> Co-authored-by: Joon Hwan 김준환 <xncbf12@gmail.com> Co-authored-by: Prabakaran Kumaresshan <k_prabakaran+github@hotmail.com> Co-authored-by: Martey Dodoo <martey@mobolic.com> Co-authored-by: Konstantin Seleznev <4374093+Seleznev-nvkz@users.noreply.github.com> Co-authored-by: Prodge <Prodge@users.noreply.github.com> Co-authored-by: Abdelhadi Dyouri <raryat@gmail.com> Co-authored-by: Ixiodor <Ixiodor@users.noreply.github.com> Co-authored-by: abhishekakamai <47558404+abhishekakamai@users.noreply.github.com> Co-authored-by: Allan Lei <allanlei@helveticode.com> Co-authored-by: M1ha Shvn <work_shvein_mihail@mail.ru> Co-authored-by: Salih Caglar Ispirli <caglarispirli@gmail.com> Co-authored-by: Micha Moskovic <michamos@gmail.com> Co-authored-by: Chris Burr <chrisburr@users.noreply.github.com> Co-authored-by: Dave King <tildedave@gmail.com> Co-authored-by: Dmitry Nikulin <v.dmitry.nikulin@gmail.com> Co-authored-by: Michael Gaddis <mgaddis@ancestry.com> Co-authored-by: epwalsh <epwalsh10@gmail.com> Co-authored-by: TalRoni <tal.inon.16@gmail.com> Co-authored-by: Leo Singer <leo.singer@ligo.org> Co-authored-by: Stephen Tomkinson <neonbunny@users.noreply.github.com> Co-authored-by: Abhishek <abhishek.shukla@xeneta.com> Co-authored-by: theirix <theirix@gmail.com> Co-authored-by: yukihira1992 <hirayama@cancerscan.jp> Co-authored-by: jpays <jerome.pays@cnp.fr> Co-authored-by: Greg Ward <greg@gerg.ca> Co-authored-by: Alexa Griffith <agriffith@bluecore.com> Co-authored-by: heedong <63043496+heedong-jung@users.noreply.github.com> Co-authored-by: heedong.jung <heedong.jung@samsung.com> Co-authored-by: Shreyansh Khajanchi <shreyanshk@users.noreply.github.com> Co-authored-by: Sam Thompson <georgedorn@users.noreply.github.com> Co-authored-by: Alphadelta14 <alpha@pokesplash.net> Co-authored-by: Azimjon Pulatov <azimjohn@yahoo.com> Co-authored-by: ysde <ysde108@gmail.com> Co-authored-by: AmirMohammad Ziaei <amir_zia@outlook.com> Co-authored-by: Ben Nadler <ben.nadler@gusto.com> Co-authored-by: Harald Nezbeda <hn@nezhar.com> Co-authored-by: Chris Frisina <github@specialorange.org> Co-authored-by: Adam Eijdenberg <adam.eijdenberg@digital.gov.au> Co-authored-by: rafaelreuber <rafaelreuber@gmail.com> Co-authored-by: Noah Kantrowitz <noah@coderanger.net> Co-authored-by: Ben Nadler <jbennadler@gmail.com> Co-authored-by: Clement Michaud <c.michaud@criteo.com> Co-authored-by: Mathieu Chataigner <m.chataigner@criteo.com> Co-authored-by: eugeneyalansky <65346459+eugeneyalansky@users.noreply.github.com> Co-authored-by: Leonard Lu <leonard@socialcodeinc.com> Co-authored-by: XinYang <yangxinhust@hotmail.com> Co-authored-by: Ingolf Becker <ingolf.becker@googlemail.com> Co-authored-by: Anuj Chauhan <an12ch98@gmail.com> Co-authored-by: shaoziwei <ziwei.shao@cloudchef.io> Co-authored-by: Mathieu Chataigner <mathieu.chataigner@gmail.com> Co-authored-by: Anakael <shemeldima2@yandex.ru> Co-authored-by: Danny Chan <danny.chan@c-k.dev> Co-authored-by: Sebastiaan ten Pas <sebastiaan@diggimedia.nl> Co-authored-by: David TILLOY <d.tilloy@criteo.com> Co-authored-by: Anthony N. Simon <anthonynsimon@users.noreply.github.com> Co-authored-by: lironhl <liron.lavy@gmail.com> Co-authored-by: Raphael Cohen <raphael.cohen@sekoia.fr> Co-authored-by: JaeyoungHeo <jay.jaeyoung@gmail.com> Co-authored-by: singlaive <singlaive@gmail.com> Co-authored-by: Murphy Meng <mmeng@mirriad.com> Co-authored-by: Wu Haotian <whtsky@gmail.com> Co-authored-by: Kwist <velnik@gmail.com> Co-authored-by: Laurentiu Dragan <ldragan@bloomberg.net> Co-authored-by: Michal Čihař <michal@cihar.com> Co-authored-by: Radim Sückr <radim.suckr@gmail.com> Co-authored-by: Artem Vasilyev <artem.v.vasilyev@gmail.com> Co-authored-by: kakakikikeke-fork <kakakikikeke_new@yahoo.co.jp> Co-authored-by: Pysaoke <pysaoke@gmail.com> Co-authored-by: baixue <baixue@wecash.net> Co-authored-by: Prashant Sinha <prashantsinha@outlook.com> Co-authored-by: AbdealiJK <abdealikothari@gmail.com> * Remove Python 2 compatibility code from Celery (#6221) * Remove five from celery/__init__.py * Remove five from celery/beat.py * Remove five from celery/bootsteps.py * Remove five from celery/exceptions.py * Remove five from celery/local.py * Remove five from celery/platforms.py * Remove five from celery/result.py * Remove five from celery/schedules.py * Remove five from celery/app/amqp.py * Remove five from celery/app/annotations.py * Remove five from celery/app/backends.py * Remove five from celery/app/base.py * Remove five from celery/app/control.py * Remove five from celery/app/defaults.py * Remove five from celery/app/log.py * Remove five from celery/app/registry.py * Remove five from celery/app/routes.py * Remove five from celery/app/task.py * Remove five from celery/app/trace.py * Remove five from celery/app/utils.py * Remove five from celery/apps/beat.py * Remove five from celery/apps/multi.py * Remove five from celery/apps/worker.py * Remove five from celery/backends/database/__init__.py * Remove five from celery/backends/amqp.py * Remove five from celery/backends/asynchronous.py * Remove five from celery/backends/base.py * Remove five from celery/backends/dynamodb.py * Remove five from celery/backends/elasticsearch.py * Remove five from celery/backends/mongodb.py * Remove five from celery/backends/redis.py * Remove five from celery/backends/rpc.py * Remove five from celery/concurrency/asynpool.py * Remove five from celery/concurrency/base.py * Remove five from celery/concurrency/prefork.py * Remove five from celery/contrib/testing/manager.py * Remove five from celery/contrib/migrate.py * Remove five from celery/contrib/rdb.py * Remove five from celery/events/cursesmon.py * Remove five from celery/events/dispatcher.py * Remove five from celery/events/state.py * Remove five from celery/loaders/base.py * Remove five from celery/security/certificate.py * Remove five from celery/security/utils.py * Remove five from celery/task/base.py * Remove five from celery/utils/dispatch/signal.py * Remove five from celery/utils/abstract.py * Remove five from celery/utils/collections.py * Remove five from celery/utils/debug.py * Remove five from celery/utils/functional.py * Remove five from celery/utils/graph.py * Remove five from celery/utils/imports.py * Remove five from celery/utils/log.py * Remove five from celery/utils/saferepr.py * Remove five from celery/utils/serialization.py * Remove five from celery/utils/term.py * Remove five from celery/utils/text.py * Remove five from celery/utils/threads.py * Remove five from celery/utils/time.py * Remove five from celery/utils/timer2.py * Remove five from celery/consumer/consumer.py * Remove five from celery/consumer/gossip.py * Remove five from celery/consumer/mingle.py * Remove five from celery/worker/autoscale.py * Remove five from celery/worker/components.py * Remove five from celery/worker/control.py * Remove five from celery/worker/request.py * Remove five from celery/worker/state.py * Remove five from celery/worker/worker.py * Remove five from celery/t/benchmarks/bench_worker.py * Remove five from celery/t/integration/test_canvas.py * Remove five from celery/t/unit/app * Remove five from celery/t/unit/backends * Remove five from celery/t/unit/compat_modules * Remove five from celery/t/unit/concurrency * Remove five from celery/t/unit/contrib * Remove five from celery/t/unit/events * Remove five from celery/t/unit/security * Remove five from celery/t/unit/tasks * Remove five from celery/t/unit/utils * Remove five from celery/t/unit/worker * Sort imports. * Comment out PyPy for now. * Remove flakeplus. * Happify linter. * Fix merge problems. * Delete backport. * Remove unused import. * Remove logic that notifies user that the Python version isn't supported from setup.py. pip already does that for us. * Add a trove classifier to indicate Celery only supports Python 3. * Restore usage of `reraise` for consistency with the kombu port. * Drop Python 2 compatibility code from our Sphinx extension. * Remove mention of flakeplus from tox.ini. * Remove mention of flakeplus from our CONTRIBUTING guide. * Bump Sphinx requirement. * Remove Python 2 compatibility code from our custom Sphinx extension. * Resolve Sphinx warning due to removed section in 32ff7b45aa3d78aedca61b6554a9db39122924fd. * Remove pydocstyle from build matrix as it was removed from master. See #6278. * Bump version: 4.4.7 → 5.0.0-alpha1 * Final touches. * Fix README. * Bump Kombu to 5.0.0. * Bump version: 5.0.0-alpha1 → 5.0.0a2 * Fix wrong version. * Remove autodoc for removed module. * Remove documentation for removed methods. * Remove the riak backend since riak is no longer maintained. * Remove riak backend since riak is no longer maintained. * Start fresh. * Added all arguments for the celery worker command. Still needs more documentation and improvements... * Load the application and execute a worker. * Added the rest of the global options. If an app is not specified we now use the default app. In addition, we now exit with the correct status code. * Extract validation into parameter types. * Restructure and document. * Allow to pass worker configuration options from command line. * Implement the beat command. * Allow to configure celery options through the CLI. * Implement the `celery call` command. * Implement the `celery list bindings` command. * Implement the `celery purge` command. * Implement the `celery result` command. * Implement the `celery migrate` task. * Implemented the celery@thedrow: OK 1 node online. command. * Take --no-color in consideration when outputting to stdout. * Ensure `celery worker` takes `--no-color` into consideration. * Use the preformatted OK string. * Adopt the NO_COLOR standard. See https://no-color.org/ for details. * Split commands into separate files. * Added 'did you mean' messages. * Implement the `celery events` command. * Text style should take --no-color into consideration as well. * Implement the basic `celery inspect` command. * Improve UI. * Organize the code. * Implement the `celery graph bootsteps` command. * Implement the `celery graph workers` command. * Implement the `celery upgrade settings` command. * Implement the `celery report` command. * Delete former unit tests. * Implement the `celery logtool` command. * Pass the quiet argument to the CLI context. * Limit inspect to existing actions. * Implement the `celery control` command. * Basic scaffold for the `celery amqp` shell command. * Start implementing the shell commands. * Implement basic.publish and basic.get. * Echo OK after acknowledgement. * Reformat Code. * Implement the exchange.declare command. * Implement the exchange.delete command. * Implement the queue.bind command. * Implement the queue.declare command. * Implement the queue.delete command. * Echo queue.declare result to screen. * Echo queue.delete result to screen. * Implement the queue.purge command. * Fix color support for error(). * Report errors and continue. * Handle connection errors and reconnect on error. * Refactor. * Implement the `celery shell` command. * Isort. * Add documentation. * Correct argument types. * Implement detach for `celery worker`. * Documentation. * Implement detach for `celery beat`. * Documentation. * Implement the `celery multi` command. * Documentation. * Implement user options. * Collect command actions from the correct registry. * Isort. * Fix access to app. * Match arguments for control. * Start fres…

* 'abstractproperty' is deprecated. Use 'property' with 'abstractmethod' instead * Fix #2849 - Initial work of celery 5.0.0 alpha1 series by dropping python below 3.6 from marix & remove import from __future__ (#5684) * initial work of celery 5.0.0 alpha-1 series by dropping python below 3.6 * i-5651(ut): add ut for ResultSet.join_native (#5679) * dropped python versions below 3.6 from tox * dropped python versions below 3.6 from travis * dropped python versions below 3.6 from appveyor * dropped python2 compat __future__ imports from tests * Fixed a bug where canvases with a group and tasks in the middle followed by a group fails to complete and indefinitely hangs. (#5681) Fixes #5512, fixes #5354, fixes #2573. * dropped python2 compat __future__ imports from celery * dropped python2 compat code from init * revert readme change about version * removed python 2 object inheritance (#5687) * removed python 2 object inheritance * removed python 2 object inheritance * removed python 2 compatibility decorator (#5689) * removed python 2 compatibility decorator * removed python 2 compatibility decorator * removed python 2 compatibility decorator * removed python 2 compatibility decorator * Remove unused imports. * Remove unused imports of python_2_unicode_compatible. Also removed leftover useage of them where they were still used. * Run pyupgrade on codebase (#5726) * Run pyupgrade on codebase. * Use format strings where possible. * pyupgrade examples. * pyupgrade on celerydocs extension. * pyupgrade on updated code. * Address code review comments. * Address code review comments. * Remove unused imports. * Fix indentation. * Address code review comments. * Fix syntax error. * Fix syntax error. * Fix syntax error. * pytest 5.x for celery 5 (#5791) * Port latest changes from master to v5-dev (#5942) * Fix serialization and deserialization of nested exception classes (#5717) * Fix #5597: chain priority (#5759) * adding `worker_process_shutdown` to __all__ (#5762) * Fix typo (#5769) * Reformat code. * Simplify commands to looking for celery worker processes (#5778) * update doc- celery supports storage list. (#5776) * Update introduction.rst * Update introduction.rst * Fail xfailed tests if the failure is unexpected. * Added integration coverage for link_error (#5373) * Added coverage for link_error. * Use pytest-rerunfailed plugin instead of rolling our own custom implementation. * Added link_error with retries. This currently fails. * Remove unused import. * Fix import on Python 2.7. * retries in link_error do not hang the worker anymore. * Run error callbacks eagerly when the task itself is run eagerly. Fixes #4899. * Adjust unit tests accordingly. * Grammar in documentation (#5780) * Grammar in documentation * Address review. * pypy 7.2 matrix (#5790) * removed extra slashes in CELERY_BROKER_URL (#5792) The Celery broker URL in settings.py had 2 slashes in the end which are not required and can be misleading. so I changed :- CELERY_BROKER_URL = 'amqp://guest:guest@localhost//' to CELERY_BROKER_URL = 'amqp://guest:guest@localhost' * Fix #5772 task_default_exchange & task_default_exchange_type not work (#5773) * Fix #5772 task_default_exchange & task_default_exchange_type not work * Add unit test: test_setting_default_exchange * Move default_exchange test to standalone class * Run integration suite with memcached results backend. (#5739) * Fix hanging forever when fetching results from a group(chain(group)) canvas. (#5744) PR #5739 uncovered multiple problems with the cache backend. This PR should resolve one of them. PR #5638 fixed the same test case for our async results backends that support native join. However, it did not fix the test case for sync results backends that support native join. * Fix regression in PR #5681. (#5753) See comment in the diff for details. * Grammatical fix to CONTRIBUTING.rst doc (#5794) * Fix #5734 Celery does not consider authMechanism on mongodb backend URLs (#5795) * Fix #5734 Celery does not consider authMechanism on mongodb backend URLs * Add unit test: test_get_connection_with_authmechanism * Add unit test: test_get_connection_with_authmechanism_no_username * Fix errors in Python 2.7 Remove "," after "**" operator * Revert "Revert "Revert "Added handle of SIGTERM in BaseTask in celery/task.py to prevent kill the task" (#5577)" (#5586)" (#5797) This reverts commit f79894e0a2c7156fd0ca5e8e3b652b6a46a7e8e7. * Add Python 3.8 Support (#5785) * Added Python 3.8 to the build matrix. * Ensure a supported tblib version is installed for Python 3.8 and above. In addition, modernize the relevant tests. * Workaround patching problem in test. * py 3.8 in clasifier * ubuntu bionic (#5799) * ubuntu bionic * fast finish * sync bumversion with pypi release * Dev.req (#5803) * update docker config * undo hardpin * devr req install from github master * update docker config (#5801) * update docker config * make dockerfile to install from github master dev branch by default * update download link * Isort. * Grammatical & punctuation fixes for CONTRIBUTING.rst document (#5804) * update dockerfile * switched to ubuntu bionic * update docker * keep it empty until we reconfigure it again with autopep8 * Fixed Dockerfile (#5809) * Update document CONTRIBUTING.rst & fix Dockerfile typo (#5813) * Added an issue template for minor releases. * reference gocelery Go Client/Server for Celery (#5815) * Add enterprise language (#5818) * Fix/correct minor doc typos (#5825) * Correct a small typo * Correct bad contributing documentation links * Preserve the task priority in case of a retry (#5820) * Preserve the task priority in case of a retry * Created test case for retried tasks with priority * Implement an integration test for retried tasks with priorities * bump kombu * basic changelog for celery 4.4.0rc4 * bump celery 4.4.0rc4 * events bootstep disabled if no events (#5807) * events bootstep disabled if no events * Added unit tests. * update bug report template * fixing ascii art to look nicer (#5831) * Only rerun flaky tests when failures can be intermediary. * Rename Changelog to Changelog.rst * The test_nested_group_chain test can run without native_join support. (#5838) * Run integration tests with Cassandra (#5834) * Run integration tests with Cassandra. * Configure cassandra result backend * Pre-create keyspace and table * Fix deprecation warning. * Fix path to cqlsh. * Increase connection timeout. * Wait until the cluster is available. * SQS - Reject on failure (#5843) * reject on failure * add documentation * test fix * test fix * test fix * Add a concurrency model with ThreadPoolExecutor (#5099) * Add a concurrency model with ThreadPoolExecutor * thread model test for pypy * Chain primitive's code example fix in canvas documentation (Regression PR#4444) (#5845) * Changed multi-line string (#5846) This string wasn't rendering properly and was printing the python statement too. Although the change isn't as pretty code-wise, it gets rid of an annoyance for the user. * Add auto expiry for DynamoDB backend (#5805) * Add auto expiry for DynamoDB backend This adds auto-expire support for the DynamoDB backend, via the DynamoDB Time to Live feature. * Require boto3>=1.9.178 for DynamoDB TTL support boto3 version 1.9.178 requires botocore>=1.12.178. botocore version 1.12.178 introduces support for the DynamoDB UpdateTimeToLive call. The UpdateTimeToLive call is used by the DynamoDB backend to enable TTL support on a newly created table. * Separate TTL handling from table creation Handle TTL enabling/disabling separately from the table get-or-create function. Improve handling of cases where the TTL is already set to the desired state. DynamoDB only allows a single TTL update action within a fairly long time window, so some problematic cases (changing the TTL attribute, enabling/disabling TTL when it was recently modified) will raise exceptions that have to be dealt with. * Handle older boto3 versions If the boto3 TTL methods are not found, log an informative error. If the user wants to enable TTL, raise an exception; if TTL should be disabled, simply return. * Improve logging - Handle exceptions by logging the error and re-raising - Log (level debug) when the desired TTL state is already in place * Add and use _has_ttl() convenience method Additional changes: - Handle exceptions when calling boto3's describe_time_to_live() - Fix test cases for missing TTL methods * Update ttl_seconds documentation * Log invalid TTL; catch and raise ValueError * Separate method _get_table_ttl_description * Separate ttl method validation function * Clarify tri-state TTL value * Improve test coverage * Fix minor typo in comment * Mark test as xfail when using the cache backend. (#5851) * [Fix #5436] Store extending result in all backends (#5661) * [Fix #5436] Store extending result in all backends * Fix sqlalchemy * More fixu * Fixing tests * removing not necessary import * Removing debug code * Removing debug code * Add tests for get_result_meta in base and database * Revert "Add auto expiry for DynamoDB backend (#5805)" (#5855) This reverts commit f7f5bcfceca692d0e78c742a7c09c424f53d915b. * Revert "Mark test as xfail when using the cache backend. (#5851)" (#5854) This reverts commit 1b303c2968836245aaa43c3d0ff9249dd8bf9ed2. * docs: Document Redis commands used by celery (#5853) * remove cache back end integrtion test. (#5856) * Fix a race condition when publishing a very large chord header (#5850) * Added a test case which artificially introduces a delay to group.save(). * Fix race condition by delaying the task only after saving the group. * update tox * Remove duplicate boto dependency. (#5858) * Revert "remove cache back end integrtion test. (#5856)" (#5859) This reverts commit e0ac7a19a745dd5a52a615c1330bd67f2cef4d00. * Revert "Revert "Add auto expiry for DynamoDB backend (#5805)" (#5855)" (#5857) This reverts commit 4ddc605392d7694760f23069c34ede34b3e582c3. * Revert "update tox" This reverts commit 49427f51049073e38439ea9b3413978784a24999. * Fix the test_simple_chord_with_a_delay_in_group_save test. * Revert "Revert "Skip unsupported canvas when using the cache backend"" (#5860) * Revert "Revert "Mark test as xfail when using the cache backend. (#5851)" (#5854)" This reverts commit fc101c61c1912c4dafa661981f8b865c011e8a55. * Make the xfail condition stricter. * Fix the xfail condition. * Linters should use Python 3.8. * Move pypy unit tests to the correct stage. * Temporarily allow PyPy to fail since it is unavailable in Travis. * Remove unused variables. * Fix unused imports. * Fix pydocstyle errors in dynamodb. * Fix pydocstyle errors in redis backend. * bump kombu to 4.6.7 * celery 4.4.0rc5 changelog * celery 4.4.0rc5 * rm redundant code (#5864) * isort. * Document the threads task pool in the CLI. * Removed the paragraph about using librabbitmq. Refer to #5872 (#5873) * Task class definitions can have retry attributes (#5869) * autoretry_for * retry_kwargs * retry_backoff * retry_backoff_max * retry_jitter can now be defined as cls attributes. All of these can be overriden from the @task decorator https://github.com/celery/celery/issues/4684 * whatsnew in Celery 4.4 as per projects standard (#5817) * 4.4 whatsnew * update * update * Move old whatsnew to history. * Remove old news & fix markers. * Added a section notifying Python 3.4 has been dropped. * Added a note about ElasticSearch basic auth. * Added a note about being able to replace eagerly run tasks. * Update index. * Address comment. * Described boto3 version updates. * Fix heading. * More news. * Thread pool. * Add Django and Config changes * Bump version 4.4.0 * upate readme * Update docs regarding Redis Message Priorities (#5874) * Update docs regarding Redis Message Priorities * fixup! Update docs regarding Redis Message Priorities * Update 4.4.0 docs (#5875) * Update 4.4 release changelog * Update whatsnew-4.4 * Update tasks docs * Fix recent tasks doc file update (#5879) * Include renamed Changelog.rst in source releases. (#5880) Changelog.rst was renamed from Changelog in fd023ec174bedc2dc65c63a0dc7c85e425ac00c6 but MANIFEST.in was not updated to include the new name. This fixes the file name so Changelog.rst will show up in future source releases again. * Reorganised project_urls and classifiers. (#5884) * Use safequote in SQS Getting Started doc (#5885) * Have appveyor build relevant versions of Python. (#5887) * Have appveyor build relevant and buildable versions of Python. * Appveyor is missing CI requirements to build. * Pin pycurl to version that will build with appveyor (because wheels files exist) * Restrict python 2.7 64 bit version of python-dateutil for parse. * Use is_alive instead of isAlive for Python 3.9 compatibility. (#5898) * Very minor tweak to commen to improve docs (#5900) As discussed here: https://stackoverflow.com/questions/58816271/celery-task-asyncresult-takes-task-id-but-is-documented-to-get-asyncresult-inst this comment seems to flow to a very confusing and misleading piece of documentation here: https://docs.celeryproject.org/en/latest/reference/celery.app.task.html#celery.app.task.Task.AsyncResult * Support configuring schema of a PostgreSQL database (#5910) * Support configuring schema of a PostgreSQL database * Add unit test * Remove blank line * Fix raise issue to make exception message more friendly (#5912) Signed-off-by: Chenyang Yan <memory.yancy@gmail.com> * Add progress for retry connections (#5915) This will show current retry progress so it will clear confusion about how many retries will be tried for connecting to broker. Closes #4556 * chg: change xrange to range (#5926) * update docs for json serializer and add note for int keys serialization (#5932) * fix indentation for note block in calling.rst (#5933) * Added links to other issue trackers. (#5939) * Add labels automatically for issues. (#5938) * Run pyupgrade. Co-authored-by: Michal Čihař <michal@cihar.com> Co-authored-by: ptitpoulpe <ptitpoulpe@ptitpoulpe.fr> Co-authored-by: Didi Bar-Zev <didi@hiredscore.com> Co-authored-by: Santos Solorzano <santosjavier22@gmail.com> Co-authored-by: manlix <manlix@yandex.ru> Co-authored-by: Jimmy <54828848+sckhg1367@users.noreply.github.com> Co-authored-by: Борис Верховский <boris.verk@gmail.com> Co-authored-by: Asif Saif Uddin <auvipy@gmail.com> Co-authored-by: Jainal Gosaliya <jainal09gosalia@gmail.com> Co-authored-by: gsfish <caoyu97@hotmail.com> Co-authored-by: Dipankar Achinta <di.punk.car19@gmail.com> Co-authored-by: Pengjie Song (宋鹏捷) <spengjie@sina.com> Co-authored-by: Chris Griffin <chris-griffin@users.noreply.github.com> Co-authored-by: Muhammad Hewedy <mhewedy@gmail.com> Co-authored-by: Blaine Bublitz <blaine.bublitz@gmail.com> Co-authored-by: Tamu <tamsanh@gmail.com> Co-authored-by: Erik Tews <erik@datenzone.de> Co-authored-by: abhinav nilaratna <anilaratna2@bloomberg.net> Co-authored-by: Wyatt Paul <wpaul@hearsaycorp.com> Co-authored-by: gal cohen <gal.nevis@gmail.com> Co-authored-by: as <alfred@huji.fr> Co-authored-by: Param Kapur <paramkapur2002@gmail.com> Co-authored-by: Sven Ulland <sven.ulland@gmail.com> Co-authored-by: Safwan Rahman <safwan.rahman15@gmail.com> Co-authored-by: Aissaoui Anouar <tobia@crossbone.cc> Co-authored-by: Neal Wang <qdzzyb2015@gmail.com> Co-authored-by: Alireza Amouzadeh <alireza@amouzadeh.net> Co-authored-by: Marcos Moyano <marcos@anue.biz> Co-authored-by: Stepan Henek <stepan+github@henek.name> Co-authored-by: Andrew Sklyarov <AndrewPix@users.noreply.github.com> Co-authored-by: Michael Fladischer <michael@fladi.at> Co-authored-by: Dejan Lekic <dejan.lekic@gmail.com> Co-authored-by: Yannick Schuchmann <yannick.schuchmann@googlemail.com> Co-authored-by: Matt Davis <matteius@gmail.com> Co-authored-by: Karthikeyan Singaravelan <tir.karthi@gmail.com> Co-authored-by: Bernd Wechner <bernd-wechner@users.noreply.github.com> Co-authored-by: Sören Oldag <soeren_oldag@freenet.de> Co-authored-by: uddmorningsun <memory.yancy@gmail.com> Co-authored-by: Amar Fadil <34912365+marfgold1@users.noreply.github.com> Co-authored-by: woodenrobot <woodenrobot1993@gmail.com> Co-authored-by: Sardorbek Imomaliev <sardorbek.imomaliev@gmail.com> * Remove fallback code for Python 2 support marked with TODOs. (#5953) Co-authored-by: Asif Saif Uddin <auvipy@gmail.com> * Remove PY3 conditionals (#5954) * Added integration coverage for link_error (#5373) * Added coverage for link_error. * Use pytest-rerunfailed plugin instead of rolling our own custom implementation. * Added link_error with retries. This currently fails. * Remove unused import. * Fix import on Python 2.7. * retries in link_error do not hang the worker anymore. * Run error callbacks eagerly when the task itself is run eagerly. Fixes #4899. * Adjust unit tests accordingly. * Grammar in documentation (#5780) * Grammar in documentation * Address review. * pypy 7.2 matrix (#5790) * removed extra slashes in CELERY_BROKER_URL (#5792) The Celery broker URL in settings.py had 2 slashes in the end which are not required and can be misleading. so I changed :- CELERY_BROKER_URL = 'amqp://guest:guest@localhost//' to CELERY_BROKER_URL = 'amqp://guest:guest@localhost' * Fix #5772 task_default_exchange & task_default_exchange_type not work (#5773) * Fix #5772 task_default_exchange & task_default_exchange_type not work * Add unit test: test_setting_default_exchange * Move default_exchange test to standalone class * Run integration suite with memcached results backend. (#5739) * Fix hanging forever when fetching results from a group(chain(group)) canvas. (#5744) PR #5739 uncovered multiple problems with the cache backend. This PR should resolve one of them. PR #5638 fixed the same test case for our async results backends that support native join. However, it did not fix the test case for sync results backends that support native join. * Fix regression in PR #5681. (#5753) See comment in the diff for details. * Grammatical fix to CONTRIBUTING.rst doc (#5794) * Fix #5734 Celery does not consider authMechanism on mongodb backend URLs (#5795) * Fix #5734 Celery does not consider authMechanism on mongodb backend URLs * Add unit test: test_get_connection_with_authmechanism * Add unit test: test_get_connection_with_authmechanism_no_username * Fix errors in Python 2.7 Remove "," after "**" operator * Revert "Revert "Revert "Added handle of SIGTERM in BaseTask in celery/task.py to prevent kill the task" (#5577)" (#5586)" (#5797) This reverts commit f79894e0a2c7156fd0ca5e8e3b652b6a46a7e8e7. * Add Python 3.8 Support (#5785) * Added Python 3.8 to the build matrix. * Ensure a supported tblib version is installed for Python 3.8 and above. In addition, modernize the relevant tests. * Workaround patching problem in test. * py 3.8 in clasifier * ubuntu bionic (#5799) * ubuntu bionic * fast finish * sync bumversion with pypi release * Dev.req (#5803) * update docker config * undo hardpin * devr req install from github master * update docker config (#5801) * update docker config * make dockerfile to install from github master dev branch by default * update download link * Isort. * Grammatical & punctuation fixes for CONTRIBUTING.rst document (#5804) * update dockerfile * switched to ubuntu bionic * update docker * keep it empty until we reconfigure it again with autopep8 * Fixed Dockerfile (#5809) * Update document CONTRIBUTING.rst & fix Dockerfile typo (#5813) * Added an issue template for minor releases. * reference gocelery Go Client/Server for Celery (#5815) * Add enterprise language (#5818) * Fix/correct minor doc typos (#5825) * Correct a small typo * Correct bad contributing documentation links * Preserve the task priority in case of a retry (#5820) * Preserve the task priority in case of a retry * Created test case for retried tasks with priority * Implement an integration test for retried tasks with priorities * bump kombu * basic changelog for celery 4.4.0rc4 * bump celery 4.4.0rc4 * events bootstep disabled if no events (#5807) * events bootstep disabled if no events * Added unit tests. * update bug report template * fixing ascii art to look nicer (#5831) * Only rerun flaky tests when failures can be intermediary. * Rename Changelog to Changelog.rst * The test_nested_group_chain test can run without native_join support. (#5838) * Run integration tests with Cassandra (#5834) * Run integration tests with Cassandra. * Configure cassandra result backend * Pre-create keyspace and table * Fix deprecation warning. * Fix path to cqlsh. * Increase connection timeout. * Wait until the cluster is available. * SQS - Reject on failure (#5843) * reject on failure * add documentation * test fix * test fix * test fix * Add a concurrency model with ThreadPoolExecutor (#5099) * Add a concurrency model with ThreadPoolExecutor * thread model test for pypy * Chain primitive's code example fix in canvas documentation (Regression PR#4444) (#5845) * Changed multi-line string (#5846) This string wasn't rendering properly and was printing the python statement too. Although the change isn't as pretty code-wise, it gets rid of an annoyance for the user. * Add auto expiry for DynamoDB backend (#5805) * Add auto expiry for DynamoDB backend This adds auto-expire support for the DynamoDB backend, via the DynamoDB Time to Live feature. * Require boto3>=1.9.178 for DynamoDB TTL support boto3 version 1.9.178 requires botocore>=1.12.178. botocore version 1.12.178 introduces support for the DynamoDB UpdateTimeToLive call. The UpdateTimeToLive call is used by the DynamoDB backend to enable TTL support on a newly created table. * Separate TTL handling from table creation Handle TTL enabling/disabling separately from the table get-or-create function. Improve handling of cases where the TTL is already set to the desired state. DynamoDB only allows a single TTL update action within a fairly long time window, so some problematic cases (changing the TTL attribute, enabling/disabling TTL when it was recently modified) will raise exceptions that have to be dealt with. * Handle older boto3 versions If the boto3 TTL methods are not found, log an informative error. If the user wants to enable TTL, raise an exception; if TTL should be disabled, simply return. * Improve logging - Handle exceptions by logging the error and re-raising - Log (level debug) when the desired TTL state is already in place * Add and use _has_ttl() convenience method Additional changes: - Handle exceptions when calling boto3's describe_time_to_live() - Fix test cases for missing TTL methods * Update ttl_seconds documentation * Log invalid TTL; catch and raise ValueError * Separate method _get_table_ttl_description * Separate ttl method validation function * Clarify tri-state TTL value * Improve test coverage * Fix minor typo in comment * Mark test as xfail when using the cache backend. (#5851) * [Fix #5436] Store extending result in all backends (#5661) * [Fix #5436] Store extending result in all backends * Fix sqlalchemy * More fixu * Fixing tests * removing not necessary import * Removing debug code * Removing debug code * Add tests for get_result_meta in base and database * Revert "Add auto expiry for DynamoDB backend (#5805)" (#5855) This reverts commit f7f5bcfceca692d0e78c742a7c09c424f53d915b. * Revert "Mark test as xfail when using the cache backend. (#5851)" (#5854) This reverts commit 1b303c2968836245aaa43c3d0ff9249dd8bf9ed2. * docs: Document Redis commands used by celery (#5853) * remove cache back end integrtion test. (#5856) * Fix a race condition when publishing a very large chord header (#5850) * Added a test case which artificially introduces a delay to group.save(). * Fix race condition by delaying the task only after saving the group. * update tox * Remove duplicate boto dependency. (#5858) * Revert "remove cache back end integrtion test. (#5856)" (#5859) This reverts commit e0ac7a19a745dd5a52a615c1330bd67f2cef4d00. * Revert "Revert "Add auto expiry for DynamoDB backend (#5805)" (#5855)" (#5857) This reverts commit 4ddc605392d7694760f23069c34ede34b3e582c3. * Revert "update tox" This reverts commit 49427f51049073e38439ea9b3413978784a24999. * Fix the test_simple_chord_with_a_delay_in_group_save test. * Revert "Revert "Skip unsupported canvas when using the cache backend"" (#5860) * Revert "Revert "Mark test as xfail when using the cache backend. (#5851)" (#5854)" This reverts commit fc101c61c1912c4dafa661981f8b865c011e8a55. * Make the xfail condition stricter. * Fix the xfail condition. * Linters should use Python 3.8. * Move pypy unit tests to the correct stage. * Temporarily allow PyPy to fail since it is unavailable in Travis. * Remove unused variables. * Fix unused imports. * Fix pydocstyle errors in dynamodb. * Fix pydocstyle errors in redis backend. * bump kombu to 4.6.7 * celery 4.4.0rc5 changelog * celery 4.4.0rc5 * rm redundant code (#5864) * isort. * Document the threads task pool in the CLI. * Removed the paragraph about using librabbitmq. Refer to #5872 (#5873) * Task class definitions can have retry attributes (#5869) * autoretry_for * retry_kwargs * retry_backoff * retry_backoff_max * retry_jitter can now be defined as cls attributes. All of these can be overriden from the @task decorator https://github.com/celery/celery/issues/4684 * whatsnew in Celery 4.4 as per projects standard (#5817) * 4.4 whatsnew * update * update * Move old whatsnew to history. * Remove old news & fix markers. * Added a section notifying Python 3.4 has been dropped. * Added a note about ElasticSearch basic auth. * Added a note about being able to replace eagerly run tasks. * Update index. * Address comment. * Described boto3 version updates. * Fix heading. * More news. * Thread pool. * Add Django and Config changes * Bump version 4.4.0 * upate readme * Update docs regarding Redis Message Priorities (#5874) * Update docs regarding Redis Message Priorities * fixup! Update docs regarding Redis Message Priorities * Update 4.4.0 docs (#5875) * Update 4.4 release changelog * Update whatsnew-4.4 * Update tasks docs * Fix recent tasks doc file update (#5879) * Include renamed Changelog.rst in source releases. (#5880) Changelog.rst was renamed from Changelog in fd023ec174bedc2dc65c63a0dc7c85e425ac00c6 but MANIFEST.in was not updated to include the new name. This fixes the file name so Changelog.rst will show up in future source releases again. * Reorganised project_urls and classifiers. (#5884) * Use safequote in SQS Getting Started doc (#5885) * Have appveyor build relevant versions of Python. (#5887) * Have appveyor build relevant and buildable versions of Python. * Appveyor is missing CI requirements to build. * Pin pycurl to version that will build with appveyor (because wheels files exist) * Restrict python 2.7 64 bit version of python-dateutil for parse. * Use is_alive instead of isAlive for Python 3.9 compatibility. (#5898) * Very minor tweak to commen to improve docs (#5900) As discussed here: https://stackoverflow.com/questions/58816271/celery-task-asyncresult-takes-task-id-but-is-documented-to-get-asyncresult-inst this comment seems to flow to a very confusing and misleading piece of documentation here: https://docs.celeryproject.org/en/latest/reference/celery.app.task.html#celery.app.task.Task.AsyncResult * Support configuring schema of a PostgreSQL database (#5910) * Support configuring schema of a PostgreSQL database * Add unit test * Remove blank line * Fix raise issue to make exception message more friendly (#5912) Signed-off-by: Chenyang Yan <memory.yancy@gmail.com> * Add progress for retry connections (#5915) This will show current retry progress so it will clear confusion about how many retries will be tried for connecting to broker. Closes #4556 * chg: change xrange to range (#5926) * update docs for json serializer and add note for int keys serialization (#5932) * fix indentation for note block in calling.rst (#5933) * Added links to other issue trackers. (#5939) * Add labels automatically for issues. (#5938) * remove redundant raise from docstring (#5941) `throw` is True by default so the Retry exception will already get raised by calling `self.retry(countdown=60 * 5, exc=exc)` * Run pyupgrade. * Fix typo (#5943) * Remove fallback code for Python 2 support. * docs: fixes Rabbits and Warrens link in routing userguide (#4007) (#5949) * Fix labels on Github issue templates. (#5955) Use quotation marks to escape labels on Github issue templates. This prevents the colon from breaking the template. * added retry_on_timeout and socket_keepalive to config and doc (#5952) * Fixed event capture from building infinite list (#5870) * Fix error propagation example (#5966) * update range (#5971) * update setup.cfg * bump billiard to 3.6.3.0 * Update __init__.py (#5951) * Update __init__.py Fixed issue for object with result_backend=True (decode fails on multiple None request) * Update __init__.py suggested changeds * Update __init__.py * Use configured db schema also for sequences (#5972) * Added a default value for retries in worker.strategy. (#5945) * Added a default value for retries in worker.strategy. I was facing an issue when adding tasks directly to rabbitmq using pika instead of calling task.apply_async. The issue was the self.retry mechanisum was failing. In app/tasks.py the line `retries = request.retries + 1` was causing the issue. On further tracing I figured out that it was because the default .get value (None) was getting passed through this function and was raising TypeError: unsupported operand type(s) for +: 'NoneType' and 'int' * Add test cases for default and custom retries value * pypy 7.3 (#5980) * Pass `interval` to `get_many` (#5931) * Pass `interval` to `get_many` * Fix: Syntax error for py2.7 * Fix: Syntax error for py2.7 * Fixed problem with conflicting autoretry_for task parameter and Task.replace() (#5934) * Fix #5917 (#5918) * Fix changelog (#5881) * merge in place the apps beat schedule in the default Schedule class. (#5908) * Handle Redis connection errors in result consumer (#5921) * Handle Redis connection errors in result consumer * Closes #5919. * Use context manager for Redis conusmer reconnect * Log error when result backend reconnection fails * Fix inspect_command documentation (#5983) * Use gevent and eventlet wait() functions to remove busy-wait (#5974) * Use gevent and eventlet wait() functions to remove busy-wait Fixes issue #4999. Calling AsyncResult.get() in a gevent context would cause the async Drainer to repeatedly call wait_for until the result was completed. I've updated the code to have a specific implementation for gevent and eventlet that will cause wait_for to only return every "timeout" # of seconds, rather than repeatedly returning. Some things I'd like some feedback on: * Where's the best place to add test coverage for this? It doesn't look like there are any tests that directly exercised the Drainer yet so I would probably look to add some of these to the backends/ unit tests. * The way I did this for the Eventlet interface was to rely on the private _exit_event member of the GreenThread instance; to do this without relying on a private member would require some additional changes to the backend Drainer interface so that we could wait for an eventlet-specific event in wait_for(). I can do this, just wanted to get some feedback before. * Add unit tests for Drainer classes In order for this to work without monkeypatching in the tests, I needed to call sleep(0) to let the gevent/eventlet greenlets to yield control back to the calling thread. I also made the check interval configurable in the drainer so that we didn't need to sleep multiples of 1 second in the tests. * Weaken asserts since they don't pass on CI * Fix eventlet auto-patching DNS resolver module on import By default it looks like "import eventlet" imports the greendns module unless the environment EVENTLET_NO_GREENDNS is set to true. This broke a pymongo test. * Add tests ensuring that the greenlet loop isn't blocked These tests make sure that while drain_events_until is running that other gevent/eventlet concurrency can run. * Clean up tests and make sure they wait for all the threads to stop * Fix chords with chained groups (#5947) * kombu 4.6.8 * update setup * updated version 4.4.1 * Fix: Accept and swallow `kwargs` to handle unexpected keyword arguments * Allow boto to look for credentials in S3Backend * add reference to Rusty Celery * Update document of revoke method in Control class * Fix copy-paste error in result_compression docs * Make 'socket_keepalive' optional variable (#6000) * update connection params - socket_keepalive is optional now * update readme - added versionadded 4.4.1 and fixed `redis_socket_keepalive` * added check of socket_keepalive in arguments for UnixSocketConnect * Fixed incorrect setting name in documentation (#6002) * updated version 4.4.2 * Fix backend utf-8 encoding in s3 backend Celery backend uses utf-8 to deserialize results, which would fail for some serializations like pickle. * Fix typo in celery.bin.multi document * Upgraded pycurl to the latest version that supports wheel. * pytest 5.3.5 max * Add uptime to the stats inspect command * Doc tweaks: mostly grammar and punctuation (#6016) * Fix a bunch of comma splices in the docs * Remove some unnecessary words from next-steps doc * Tweak awkward wording; fix bad em-dash * Fix a bunch more comma splices in next-steps doc * Miscellaneous grammar/punctuation/wording fixes * Link to task options in task decorator docs * Fixing issue #6019: unable to use mysql SSL parameters when getting mysql engine (#6020) * Fixing issue #6019: unable to use mysql SSL parametes in create_engine() * adding test for get_engine when self.forked is False and engine args are passed in for create_engine() * Clean TraceBack to reduce memory leaks for exception task (#6024) * Clean TraceBack to reduce memory leaks * add unit test * add unit test * reject unittest * Patch For Python 2.7 compatibility * update unittest * Register to the garbage collector by explicitly referring to f_locals. * need more check * update code coverage * update Missing unit test * 3.4 -> 3.5 Co-authored-by: heedong.jung <heedong.jung@samsung.com> * exceptions: NotRegistered: fix up language Minor fix to the language. * Note about autodiscover_tasks and periodic tasks This is particularly important for Django projects that put periodic tasks into each app's `tasks.py` and want to use one as a periodic task. By the time `autodiscover_tasks()` loads those tasks, the `on_after_configure` Signal has already come and gone, so anything decorated with `@app.on_after_finalize.connect` will never be called. If there's other documentation on this subject, I could not find it. * Avoid PyModulelevel, deprecated in Sphinx 4 Use `PyFunction` instead of `PyModulelevel` to avoid this deprecation warning: RemovedInSphinx40Warning: PyModulelevel is deprecated. Please check the implementation of <class 'celery.contrib.sphinx.TaskDirective'> This replacement is one of the options listed in the Sphinx docs (https://www.sphinx-doc.org/en/master/extdev/deprecated.html). * Give up sending a worker-offline message if transport is not connected (#6039) * If worker-offline event fails to send, give up and die peacefully * Add test for retry= and msgs in heartbeat * Fix the build and all documentation warnings. I finally upgraded our theme to 2.0. As a result we've upgraded Sphinx to 2.0. Work to upgrade Sphinx to 3.0 will proceed in a different PR. This upgrade also fixes our build issues caused by #6032. We don't support Sphinx 1.x as a result of that patch. I've also included the missing 4.3 changelog to our history. * Support both Sphinx 2 and 3. * Add Task to __all__ in celery.__init__.py * Add missing parenthesis to example in docs * Ensure a single chain object in a chain does not raise MaximumRecursionError. Previously chain([chain(sig)]) would crash. We now ensure it doesn't. Fixes #5973. * update setup.py * fix typo missing quote at the end of line * Fix a typo in monitoring doc * update travis * update ubuntu to focal foss 20.04 LTS * Fix autoscale when prefetch_multiplier is 1 * Allow start_worker to function without ping task * Update celeryd.conf Move the directory of the program before the execution of the command/script * Add documentation for "predefined_queue_urls" * [Fix #6074]: Add missing documentation for MongoDB as result backend. * update funding * 🐛 Correctly handle configuring the serializer for always_eager mode. (#6079) * 🐛 Correctly handle configuring the serializer for always_eager mode. options['serializer'] will always exist, because it is initialized from an mattrgetter. Even if unset, it will be present in the options with a value of None. * 🐛 Add a test for new always_eager + task_serializer behavior. * ✏️ Whoops missed a : * Remove doubling of prefetch_count increase when prefetch_multiplier gt 1 (#6081) * try ubuntu focal (#6088) * Fix eager function not returning result after retries. Using apply function does not return correct results after at least one retry because the return value of successive call is not going back to the original caller. * return retry result if not throw and is_eager if throw is false, we would be interested by the result of retry and not the current result which will be an exception. This way it does not break the logic of `raise self.retry` This should be used like `return self.retry(..., throw=False)` in an except statement. * revert formatting change * Add tests for eager retry without throw * update predefined-queues documentation Suggested version of configuration does not work. Additionally I'd like to mention, that `access_key_id` and `secret_access_key` are mandatory fields and not allowing you to go with defaults AWS_* env variables. I can contribute for this variables to be optional Also I'm not sure if security token will apply, could you please advice how to do it? * Fix couchbase version < 3.0.0 as API changed * Remove reference to -O fair in optimizations -O fair was made the default in Celery 4.0 https://docs.celeryproject.org/en/stable/history/whatsnew-4.0.html#ofair-is-now-the-default-scheduling-strategy * pytest ranges * pypy3 * revert to bionic * do not load docs.txt requirements for python 2.7 As it requires Sphinx >= 2.0.0 and there is no such version compatible with python 2.7 * update cassandra travis integration test configuration cassandra:latest docker image changed location of cqlsh program * pin cassandra-driver CI get stuck after all cassandra integration tests * Fix all flake8 lint errors * Fix all pydocstyle lint errors * Fix all configcheck lint errors * Always requeue while worker lost regardless of the redelivered flag (#6103) * #5598 fix, always redelivery while WorkerLostError * fix, change the requeue flag so the task will remain PENDING * Allow relative paths in the filesystem backend (#6070) * Allow relative paths in the filesystem backend * fix order of if statements * [Fixed Issue #6017] --> Added Multi default logfiles and pidfiles paths [Description]: --> Changed the default paths for log files & pid files to be '/var/log/celery' and '/var/run/celery' --> Handled by creating the respective paths if not exist. --> Used os.makedir(path,if_exists=True) [Unit Test Added]: --> .travis.yml - config updated with 'before install'. --> t/unit/apps/test_multi.py - Changed the default log files & pid files paths wherever required. * Avoid race condition due to task duplication. In some circumstances like a network partitioning, some tasks might be duplicated. Sometimes, this lead to a race condition where a lost task overwrites the result of the last successful task in the backend. In order to avoid this race condition we prevent updating the result if it's already in successful state. This fix has been done for KV backends only and therefore won't work with other backends. * adding tests * Exceptions must be old-style classes or derived from BaseException, but here self.result may not subclass of BaseException. * update fund link * Fix windows build (#6104) * do not load memcache nor couchbase lib during windows build those libraries depends on native libraries libcouchbase and libmemcached that are not installed on Appveyor. As only unit tests runs on Appveyor, it should be fine * Add python 3.8 workaround for app trap * skip tests file_descriptor_safety tests on windows AsyncPool is not supported on Windows so Pool does have _fileno_to_outq attribute, making the test fail * Fix crossplatform log and pid files in multi mode it relates to #6017 * Use tox to build and test on windows * remove tox_install_command * drop python 2.7 from windows build * Add encode to meta task in base.py (#5894) * Add encode to base.py meta result Fix bug with impossibility to load None from task meta * Add tests for None. Remove exceed encode. * Update base.py Add return payload if None * Update time.py to solve the microsecond issues (#5199) When `relative` is set to True, the day, hour, minutes second will be round to the nearest one, however, the original program do not update the microsecond (reset it). As a result, the run-time offset on the microsecond will then be accumulated. For example, given the interval is 15s and relative is set to True 1. 2018-11-27T15:01:30.123236+08:00 2. 2018-11-27T15:01:45.372687+08:00 3. 2018-11-27T15:02:00.712601+08:00 4. 2018-11-27T15:02:15.987720+08:00 5. 2018-11-27T15:02:31.023670+08:00 * Change backend _ensure_not_eager error to warning * Add priority support for 'celery.chord_unlock' task (#5766) * Change eager retry behaviour even with raise self.retry, it should return the eventual value or MaxRetriesExceededError. if return value of eager apply is Retry exception, retry eagerly the task signature * Order supported Python versions * Avoid race condition in elasticsearch backend if a task is retried, the task retry may work concurrently to current task. store_result may come out of order. it may cause a non ready state (Retry) to override a ready state (Success, Failure). If this happens, it will block indefinitely pending any chord depending on this task. this change makes document updates safe for concurrent writes. https://www.elastic.co/guide/en/elasticsearch/reference/current/optimistic-concurrency-control.html * backends base get_many pass READY_STATES arg * test backends base get_many pass READY_STATES arg * Add integration tests for Elasticsearch and fix _update * Revert "revert to bionic" This reverts commit 6e091573f2ab0d0989b8d7c26b677c80377c1721. * remove jython check * feat(backend): Adds cleanup to ArangoDB backend * Delete Document Known Issue with CONN_MAX_AGE in 4.3 * issue 6108 fix filesystem backend cannot not be serialized by picked (#6120) * issue 6108 fix filesystem backend cannot not be serialized by picked https://github.com/celery/celery/issues/6108 * issue-6108 fix unit test failure * issue-6108 fix flake8 warning Co-authored-by: Murphy Meng <mmeng@mirriad.com> * kombu==4.6.9 (#6133) * changelog for 4.4.3 * v 4.4.3 * remove un supported classifier * Fix autoretry_for with explicit retry (#6138) * Add tests for eager task retry * Fixes #6135 If autoretry_for is set too broad on Exception, then autoretry may get a Retry if that's the case, rethrow directly instead of wrapping it in another Retry to avoid loosing new args * Use Django DB max age connection setting (fixes #4116) * Add retry on recoverable exception for the backend (#6122) * Add state to KeyValueStoreBackend.set method This way, a backend implementation is able to take decisions based on current state to store meta in case of failures. * Add retry on recoverable exception for the backend acks.late makes celery acknowledge messages only after processing and storing result on the backend. However, in case of backend unreachable, it will shadow a Retry exception and put the task as failed in the backend not retrying the task and acknoledging it on the broker. With this new result_backend_always_retry setting, if the backend exception is recoverable (to be defined per backend implementation), it will retry the backend operation with an exponential backoff. * Make elasticsearch backward compatible with 6.x * Make ES retry storing updates in a better way if existing value in the backend is success, then do nothing. if it is a ready status, then update it only if new value is a ready status as well. else update it. This way, a SUCCESS cannot be overriden so that we do not loose results but any ready state other than success (FAILURE, REVOKED) can be overriden by another ready status (i.e. a SUCCESS) * Add test for value not found in ES backend * Fix random distribution of jitter for exponential backoff random.randrange should be called with the actual so that all numbers have equivalent probability, otherwise maximum value does have a way higher probability of occuring. * fix unit test if extra modules are not present * ElasticSearch: add setting to save meta as json * fix #6136. celery 4.4.3 always trying create /var/run/celery directory (#6142) * fix #6136. celery 4.4.3 always trying create /var/run/celery directory, even if it's not needed. * fix #6136. cleanup * Add task_internal_error signal (#6049) * Add internal_error signal There is no special signal for an out of body error which can be the result of a bad result backend. * Fix syntax error. * Document the task_internal_error signal. Co-authored-by: Laurentiu Dragan <ldragan@bloomberg.net> * changelog for v4.4.4 * kombu 4.6.10 (#6144) * v4.4.4 * Add missing dependency on future (#6146) Fixes #6145 * ElasticSearch: Retry index if document was deleted between index and update (#6140) * ElasticSearch: Retry index if document was deleted between index and update * Elasticsearch increase coverage to 100% * Fix pydocstyle * Specify minimum version of Sphinx for Celery extension (#6150) The Sphinx extension requires Sphinx 2 or later due to #6032. * fix windows build * fix flake8 error * fix multi tests in local Mock os.mkdir and os.makedirs to avoid creating /var/run/celery and /var/log/celery during unit tests if run without root priviledges * Customize the retry interval of chord_unlock tasks * changelog v4.4.5 * v4.4.5 * Fix typo in comment. * Remove autoscale force_scale methods (#6085) * Remove autoscale force_scale methods * Remove unused variable in test * Pass ping destination to request The destination argument worked fine from CLI but didn't get used when calling ping from Python. * Fix autoscale test * chord: merge init options with run options * put back KeyValueStoreBackend.set method without state It turns out it was breaking some other projects. wrapping set method with _set_with_state, this way it will not break existing Backend. while enabling this feature for other Backend. Currently, only ElasticsearchBackend supports this feature. It protects concurrent update to corrupt state in the backend. Existing success cannot be overriden, nor a ready state by a non ready state. i.e. a Retry state cannot override a Success or Failure. As a result, chord_unlock task will not loop forever due to missing ready state on the backend. * added --range-prefix option to `celery multi` (#6180) * added --range-prefix option to `celery multi` Added option for overriding default range prefix when running multiple workers prividing range with `celery multy` command. * covered multi --range-prefix with tests * fixed --range-prefix test * Added as_list function to AsyncResult class (#6179) * Add as_list method to return task IDs as a list * Add a test for as_list method * Add docstring for as_list method * Fix CassandraBackend error in threads or gevent pool (#6147) * Fix CassandraBackend error in threads or gevent pool * remove CassandraBackend.process_cleanup * Add test case * Add test case * Add comments test_as_uri Co-authored-by: baixue <baixue@wecash.net> * changelog for v4.4.6 * v4.4.6 * Update Wiki link in "resources" In the page linked below, the link to wiki is outdated. Fixed that. https://docs.celeryproject.org/en/stable/getting-started/resources.html * test_canvas: Add test for chord-in-chain Add test case for the issue where a chord in a chain does not work when using .apply(). This works fine with .apply_async(). * Trying to fix flaky tests in ci * fix pydocstyle errors * fix pydocstyle * Drainer tests, put a lower constraint on number of intervals liveness should iterate 10 times per interval while drain_events only once. However, as it may use thread that may be scheduled out of order, we may end up in some situation where liveness and drain_events were called the same amount of time. Lowering the constraint from < to <= to avoid failing the tests. * pyupgrade. * Fix merge error. Co-authored-by: Борис Верховский <boris.verk@gmail.com> Co-authored-by: Asif Saif Uddin <auvipy@gmail.com> Co-authored-by: Jainal Gosaliya <jainal09gosalia@gmail.com> Co-authored-by: gsfish <root@grassfish.net> Co-authored-by: Dipankar Achinta <di.punk.car19@gmail.com> Co-authored-by: spengjie <spengjie@sina.com> Co-authored-by: Chris Griffin <chris-griffin@users.noreply.github.com> Co-authored-by: Muhammad Hewedy <mhewedy@gmail.com> Co-authored-by: Blaine Bublitz <blaine.bublitz@gmail.com> Co-authored-by: Tamu <tamsanh@gmail.com> Co-authored-by: Erik Tews <erik@datenzone.de> Co-authored-by: abhinav nilaratna <anilaratna2@bloomberg.net> Co-authored-by: Wyatt Paul <wpaul@hearsaycorp.com> Co-authored-by: gal cohen <gal.nevis@gmail.com> Co-authored-by: whuji <alfred@huji.fr> Co-authored-by: Param Kapur <paramkapur2002@gmail.com> Co-authored-by: Sven Ulland <sven.ulland@gmail.com> Co-authored-by: Safwan Rahman <safwan.rahman15@gmail.com> Co-authored-by: Aissaoui Anouar <tobia@crossbone.cc> Co-authored-by: Neal Wang <qdzzyb2015@gmail.com> Co-authored-by: Alireza Amouzadeh <alireza@amouzadeh.net> Co-authored-by: Marcos Moyano <marcos@anue.biz> Co-authored-by: Stepan Henek <stepan+github@henek.name> Co-authored-by: Andrew Sklyarov <AndrewPix@users.noreply.github.com> Co-authored-by: Michael Fladischer <michael@fladi.at> Co-authored-by: Dejan Lekic <dejan.lekic@gmail.com> Co-authored-by: Yannick Schuchmann <yannick.schuchmann@googlemail.com> Co-authored-by: Matt Davis <matteius@gmail.com> Co-authored-by: Xtreak <tir.karthi@gmail.com> Co-authored-by: Bernd Wechner <bernd-wechner@users.noreply.github.com> Co-authored-by: Sören Oldag <soeren_oldag@freenet.de> Co-authored-by: uddmorningsun <memory.yancy@gmail.com> Co-authored-by: Amar Fadil <34912365+marfgold1@users.noreply.github.com> Co-authored-by: woodenrobot <woodenrobot1993@gmail.com> Co-authored-by: Sardorbek Imomaliev <sardorbek.imomaliev@gmail.com> Co-authored-by: Alex Riina <alex.riina@gmail.com> Co-authored-by: Joon Hwan 김준환 <xncbf12@gmail.com> Co-authored-by: Prabakaran Kumaresshan <k_prabakaran+github@hotmail.com> Co-authored-by: Martey Dodoo <martey@mobolic.com> Co-authored-by: Konstantin Seleznev <4374093+Seleznev-nvkz@users.noreply.github.com> Co-authored-by: Prodge <Prodge@users.noreply.github.com> Co-authored-by: Abdelhadi Dyouri <raryat@gmail.com> Co-authored-by: Ixiodor <Ixiodor@users.noreply.github.com> Co-authored-by: abhishekakamai <47558404+abhishekakamai@users.noreply.github.com> Co-authored-by: Allan Lei <allanlei@helveticode.com> Co-authored-by: M1ha Shvn <work_shvein_mihail@mail.ru> Co-authored-by: Salih Caglar Ispirli <caglarispirli@gmail.com> Co-authored-by: Micha Moskovic <michamos@gmail.com> Co-authored-by: Chris Burr <chrisburr@users.noreply.github.com> Co-authored-by: Dave King <tildedave@gmail.com> Co-authored-by: Dmitry Nikulin <v.dmitry.nikulin@gmail.com> Co-authored-by: Michael Gaddis <mgaddis@ancestry.com> Co-authored-by: epwalsh <epwalsh10@gmail.com> Co-authored-by: TalRoni <tal.inon.16@gmail.com> Co-authored-by: Leo Singer <leo.singer@ligo.org> Co-authored-by: Stephen Tomkinson <neonbunny@users.noreply.github.com> Co-authored-by: Abhishek <abhishek.shukla@xeneta.com> Co-authored-by: theirix <theirix@gmail.com> Co-authored-by: yukihira1992 <hirayama@cancerscan.jp> Co-authored-by: jpays <jerome.pays@cnp.fr> Co-authored-by: Greg Ward <greg@gerg.ca> Co-authored-by: Alexa Griffith <agriffith@bluecore.com> Co-authored-by: heedong <63043496+heedong-jung@users.noreply.github.com> Co-authored-by: heedong.jung <heedong.jung@samsung.com> Co-authored-by: Shreyansh Khajanchi <shreyanshk@users.noreply.github.com> Co-authored-by: Sam Thompson <georgedorn@users.noreply.github.com> Co-authored-by: Alphadelta14 <alpha@pokesplash.net> Co-authored-by: Azimjon Pulatov <azimjohn@yahoo.com> Co-authored-by: ysde <ysde108@gmail.com> Co-authored-by: AmirMohammad Ziaei <amir_zia@outlook.com> Co-authored-by: Ben Nadler <ben.nadler@gusto.com> Co-authored-by: Harald Nezbeda <hn@nezhar.com> Co-authored-by: Chris Frisina <github@specialorange.org> Co-authored-by: Adam Eijdenberg <adam.eijdenberg@digital.gov.au> Co-authored-by: rafaelreuber <rafaelreuber@gmail.com> Co-authored-by: Noah Kantrowitz <noah@coderanger.net> Co-authored-by: Ben Nadler <jbennadler@gmail.com> Co-authored-by: Clement Michaud <c.michaud@criteo.com> Co-authored-by: Mathieu Chataigner <m.chataigner@criteo.com> Co-authored-by: eugeneyalansky <65346459+eugeneyalansky@users.noreply.github.com> Co-authored-by: Leonard Lu <leonard@socialcodeinc.com> Co-authored-by: XinYang <yangxinhust@hotmail.com> Co-authored-by: Ingolf Becker <ingolf.becker@googlemail.com> Co-authored-by: Anuj Chauhan <an12ch98@gmail.com> Co-authored-by: shaoziwei <ziwei.shao@cloudchef.io> Co-authored-by: Mathieu Chataigner <mathieu.chataigner@gmail.com> Co-authored-by: Anakael <shemeldima2@yandex.ru> Co-authored-by: Danny Chan <danny.chan@c-k.dev> Co-authored-by: Sebastiaan ten Pas <sebastiaan@diggimedia.nl> Co-authored-by: David TILLOY <d.tilloy@criteo.com> Co-authored-by: Anthony N. Simon <anthonynsimon@users.noreply.github.com> Co-authored-by: lironhl <liron.lavy@gmail.com> Co-authored-by: Raphael Cohen <raphael.cohen@sekoia.fr> Co-authored-by: JaeyoungHeo <jay.jaeyoung@gmail.com> Co-authored-by: singlaive <singlaive@gmail.com> Co-authored-by: Murphy Meng <mmeng@mirriad.com> Co-authored-by: Wu Haotian <whtsky@gmail.com> Co-authored-by: Kwist <velnik@gmail.com> Co-authored-by: Laurentiu Dragan <ldragan@bloomberg.net> Co-authored-by: Michal Čihař <michal@cihar.com> Co-authored-by: Radim Sückr <radim.suckr@gmail.com> Co-authored-by: Artem Vasilyev <artem.v.vasilyev@gmail.com> Co-authored-by: kakakikikeke-fork <kakakikikeke_new@yahoo.co.jp> Co-authored-by: Pysaoke <pysaoke@gmail.com> Co-authored-by: baixue <baixue@wecash.net> Co-authored-by: Prashant Sinha <prashantsinha@outlook.com> Co-authored-by: AbdealiJK <abdealikothari@gmail.com> * Remove Python 2 compatibility code from Celery (#6221) * Remove five from celery/__init__.py * Remove five from celery/beat.py * Remove five from celery/bootsteps.py * Remove five from celery/exceptions.py * Remove five from celery/local.py * Remove five from celery/platforms.py * Remove five from celery/result.py * Remove five from celery/schedules.py * Remove five from celery/app/amqp.py * Remove five from celery/app/annotations.py * Remove five from celery/app/backends.py * Remove five from celery/app/base.py * Remove five from celery/app/control.py * Remove five from celery/app/defaults.py * Remove five from celery/app/log.py * Remove five from celery/app/registry.py * Remove five from celery/app/routes.py * Remove five from celery/app/task.py * Remove five from celery/app/trace.py * Remove five from celery/app/utils.py * Remove five from celery/apps/beat.py * Remove five from celery/apps/multi.py * Remove five from celery/apps/worker.py * Remove five from celery/backends/database/__init__.py * Remove five from celery/backends/amqp.py * Remove five from celery/backends/asynchronous.py * Remove five from celery/backends/base.py * Remove five from celery/backends/dynamodb.py * Remove five from celery/backends/elasticsearch.py * Remove five from celery/backends/mongodb.py * Remove five from celery/backends/redis.py * Remove five from celery/backends/rpc.py * Remove five from celery/concurrency/asynpool.py * Remove five from celery/concurrency/base.py * Remove five from celery/concurrency/prefork.py * Remove five from celery/contrib/testing/manager.py * Remove five from celery/contrib/migrate.py * Remove five from celery/contrib/rdb.py * Remove five from celery/events/cursesmon.py * Remove five from celery/events/dispatcher.py * Remove five from celery/events/state.py * Remove five from celery/loaders/base.py * Remove five from celery/security/certificate.py * Remove five from celery/security/utils.py * Remove five from celery/task/base.py * Remove five from celery/utils/dispatch/signal.py * Remove five from celery/utils/abstract.py * Remove five from celery/utils/collections.py * Remove five from celery/utils/debug.py * Remove five from celery/utils/functional.py * Remove five from celery/utils/graph.py * Remove five from celery/utils/imports.py * Remove five from celery/utils/log.py * Remove five from celery/utils/saferepr.py * Remove five from celery/utils/serialization.py * Remove five from celery/utils/term.py * Remove five from celery/utils/text.py * Remove five from celery/utils/threads.py * Remove five from celery/utils/time.py * Remove five from celery/utils/timer2.py * Remove five from celery/consumer/consumer.py * Remove five from celery/consumer/gossip.py * Remove five from celery/consumer/mingle.py * Remove five from celery/worker/autoscale.py * Remove five from celery/worker/components.py * Remove five from celery/worker/control.py * Remove five from celery/worker/request.py * Remove five from celery/worker/state.py * Remove five from celery/worker/worker.py * Remove five from celery/t/benchmarks/bench_worker.py * Remove five from celery/t/integration/test_canvas.py * Remove five from celery/t/unit/app * Remove five from celery/t/unit/backends * Remove five from celery/t/unit/compat_modules * Remove five from celery/t/unit/concurrency * Remove five from celery/t/unit/contrib * Remove five from celery/t/unit/events * Remove five from celery/t/unit/security * Remove five from celery/t/unit/tasks * Remove five from celery/t/unit/utils * Remove five from celery/t/unit/worker * Sort imports. * Comment out PyPy for now. * Remove flakeplus. * Happify linter. * Fix merge problems. * Delete backport. * Remove unused import. * Remove logic that notifies user that the Python version isn't supported from setup.py. pip already does that for us. * Add a trove classifier to indicate Celery only supports Python 3. * Restore usage of `reraise` for consistency with the kombu port. * Drop Python 2 compatibility code from our Sphinx extension. * Remove mention of flakeplus from tox.ini. * Remove mention of flakeplus from our CONTRIBUTING guide. * Bump Sphinx requirement. * Remove Python 2 compatibility code from our custom Sphinx extension. * Resolve Sphinx warning due to removed section in 32ff7b45aa3d78aedca61b6554a9db39122924fd. * Remove pydocstyle from build matrix as it was removed from master. See #6278. * Bump version: 4.4.7 → 5.0.0-alpha1 * Final touches. * Fix README. * Bump Kombu to 5.0.0. * Bump version: 5.0.0-alpha1 → 5.0.0a2 * Fix wrong version. * Remove autodoc for removed module. * Remove documentation for removed methods. * Remove the riak backend since riak is no longer maintained. Co-authored-by: tothegump <tothegump@gmail.com> Co-authored-by: Omer Katz <omer.drow@gmail.com> Co-authored-by: Michal Čihař <michal@cihar.com> Co-authored-by: ptitpoulpe <ptitpoulpe@ptitpoulpe.fr> Co-authored-by: Didi Bar-Zev <didi@hiredscore.com> Co-authored-by: Santos Solorzano <santosjavier22@gmail.com> Co-authored-by: manlix <manlix@yandex.ru> Co-authored-by: Jimmy <54828848+sckhg1367@users.noreply.github.com> Co-authored-by: Борис Верховский <boris.verk@gmail.com> Co-authored-by: Jainal Gosaliya <jainal09gosalia@gmail.com> Co-authored-by: gsfish <caoyu97@hotmail.com> Co-authored-by: Dipankar Achinta <di.punk.car19@gmail.com> Co-authored-by: Pengjie Song (宋鹏捷) <spengjie@sina.com> Co-authored-by: Chris Griffin <chris-griffin@users.noreply.github.com> Co-authored-by: Muhammad Hewedy <mhewedy@gmail.com> Co-authored-by: Blaine Bublitz <blaine.bublitz@gmail.com> Co-authored-by: Tamu <tamsanh@gmail.com> Co-authored-by: Erik Tews <erik@datenzone.de> Co-authored-by: abhinav nilaratna <anilaratna2@bloomberg.net> Co-authored-by: Wyatt Paul <wpaul@hearsaycorp.com> Co-authored-by: gal cohen <gal.nevis@gmail.com> Co-authored-by: as <alfred@huji.fr> Co-authored-by: Param Kapur <paramkapur2002@gmail.com> Co-authored-by: Sven Ulland <sven.ulland@gmail.com> Co-authored-by: Safwan Rahman <safwan.rahman15@gmail.com> Co-authored-by: Aissaoui Anouar <tobia@crossbone.cc> Co-authored-by: Neal Wang <qdzzyb2015@gmail.com> Co-authored-by: Alireza Amouzadeh <alireza@amouzadeh.net> Co-authored-by: Marcos Moyano <marcos@anue.biz> Co-authored-by: Stepan Henek <stepan+github@henek.name> Co-authored-by: Andrew Sklyarov <AndrewPix@users.noreply.github.com> Co-authored-by: Michael Fladischer <michael@fladi.at> Co-authored-by: Dejan Lekic <dejan.lekic@gmail.com> Co-authored-by: Yannick Schuchmann <yannick.schuchmann@googlemail.com> Co-authored-by: Matt Davis <matteius@gmail.com> Co-authored-by: Karthikeyan Singaravelan <tir.karthi@gmail.com> Co-authored-by: Bernd Wechner <bernd-wechner@users.noreply.github.com> Co-authored-by: Sören Oldag <soeren_oldag@freenet.de> Co-authored-by: uddmorningsun <memory.yancy@gmail.com> Co-authored-by: Amar Fadil <34912365+marfgold1@users.noreply.github.com> Co-authored-by: woodenrobot <woodenrobot1993@gmail.com> Co-authored-by: Sardorbek Imomaliev <sardorbek.imomaliev@gmail.com> Co-authored-by: gsfish <root@grassfish.net> Co-authored-by: Alex Riina <alex.riina@gmail.com> Co-authored-by: Joon Hwan 김준환 <xncbf12@gmail.com> Co-authored-by: Prabakaran Kumaresshan <k_prabakaran+github@hotmail.com> Co-authored-by: Martey Dodoo <martey@mobolic.com> Co-authored-by: Konstantin Seleznev <4374093+Seleznev-nvkz@users.noreply.github.com> Co-authored-by: Prodge <Prodge@users.noreply.github.com> Co-authored-by: Abdelhadi Dyouri <raryat@gmail.com> Co-authored-by: Ixiodor <Ixiodor@users.noreply.github.com> Co-authored-by: abhishekakamai <47558404+abhishekakamai@users.noreply.github.com> Co-authored-by: Allan Lei <allanlei@helveticode.com> Co-authored-by: M1ha Shvn <work_shvein_mihail@mail.ru> Co-authored-by: Salih Caglar Ispirli <caglarispirli@gmail.com> Co-authored-by: Micha Moskovic <michamos@gmail.com> Co-authored-by: Chris Burr <chrisburr@users.noreply.github.com> Co-authored-by: Dave King <tildedave@gmail.com> Co-authored-by: Dmitry Nikulin <v.dmitry.nikulin@gmail.com> Co-authored-by: Michael Gaddis <mgaddis@ancestry.com> Co-authored-by: epwalsh <epwalsh10@gmail.com> Co-authored-by: TalRoni <tal.inon.16@gmail.com> Co-authored-by: Leo Singer <leo.singer@ligo.org> Co-authored-by: Stephen Tomkinson <neonbunny@users.noreply.github.com> Co-authored-by: Abhishek <abhishek.shukla@xeneta.com> Co-authored-by: theirix <theirix@gmail.com> Co-authored-by: yukihira1992 <hirayama@cancerscan.jp> Co-authored-by: jpays <jerome.pays@cnp.fr> Co-authored-by: Greg Ward <greg@gerg.ca> Co-authored-by: Alexa Griffith <agriffith@bluecore.com> Co-authored-by: heedong <63043496+heedong-jung@users.noreply.github.com> Co-authored-by: heedong.jung <heedong.jung@samsung.com> Co-authored-by: Shreyansh Khajanchi <shreyanshk@users.noreply.github.com> Co-authored-by: Sam Thompson <georgedorn@users.noreply.github.com> Co-authored-by: Alphadelta14 <alpha@pokesplash.net> Co-authored-by: Azimjon Pulatov <azimjohn@yahoo.com> Co-authored-by: ysde <ysde108@gmail.com> Co-authored-by: AmirMohammad Ziaei <amir_zia@outlook.com> Co-authored-by: Ben Nadler <ben.nadler@gusto.com> Co-authored-by: Harald Nezbeda <hn@nezhar.com> Co-authored-by: Chris Frisina <github@specialorange.org> Co-authored-by: Adam Eijdenberg <adam.eijdenberg@digital.gov.au> Co-authored-by: rafaelreuber <rafaelreuber@gmail.com> Co-authored-by: Noah Kantrowitz <noah@coderanger.net> Co-authored-by: Ben Nadler <jbennadler@gmail.com> Co-authored-by: Clement Michaud <c.michaud@criteo.com> Co-authored-by: Mathieu Chataigner <m.chataigner@criteo.com> Co-authored-by: eugeneyalansky <65346459+eugeneyalansky@users.noreply.github.com> Co-authored-by: Leonard Lu <leonard@socialcodeinc.com> Co-authored-by: XinYang <yangxinhust@hotmail.com> Co-authored-by: Ingolf Becker <ingolf.becker@googlemail.com> Co-authored-by: Anuj Chauhan <an12ch98@gmail.com> Co-authored-by: shaoziwei <ziwei.shao@cloudchef.io>…

maxmalysh added Category: Documentation Issue Type: Bug Report labels Mar 31, 2020

auvipy added the Status: Needs Verification ✘ label Apr 1, 2020

hanchau mentioned this issue Apr 16, 2020

Fixed Issue #6017: Added Multi default logfiles and pidfiles paths #6042

Closed

auvipy added this to the 4.4.x milestone May 17, 2020

auvipy removed the Status: Needs Verification ✘ label May 17, 2020

auvipy closed this as completed May 17, 2020

mchataigner added a commit to criteo-forks/celery that referenced this issue May 17, 2020

Fix crossplatform log and pid files in multi mode

a6171e6

it relates to celery#6017

kwist-sgr mentioned this issue Jun 1, 2020

Celery 4.4.3 always trying create /var/run/celery directory, even if it's not needed. #6136

Closed

Andrew-Chen-Wang mentioned this issue Jun 30, 2020

Celery multi 4.4.6 not working due to /var/run/celery mkdir #6199

Open

18 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Celery Multi creates pid and log files in the wrong directory #6017

Celery Multi creates pid and log files in the wrong directory #6017

maxmalysh commented Mar 31, 2020 •

edited

Loading

mchataigner commented May 17, 2020

auvipy commented May 18, 2020

auvipy commented Jun 1, 2020

Celery Multi creates pid and log files in the wrong directory #6017

Celery Multi creates pid and log files in the wrong directory #6017

Comments

maxmalysh commented Mar 31, 2020 • edited Loading

mchataigner commented May 17, 2020

auvipy commented May 18, 2020

auvipy commented Jun 1, 2020

maxmalysh commented Mar 31, 2020 •

edited

Loading