Update celery to 4.1.0

[pyup-bot]

There's a new version of celery available.
You are currently using 3.1.25. I have updated it to 4.1.0

These links might come in handy: PyPI | Changelog | Homepage | Docs

Changelog

4.1.0

=====
:release-date: 2017-07-25 00:00 PM PST
:release-by: Omer Katz

• Configuration: CELERY_SEND_EVENTS instead of CELERYD_SEND_EVENTS for 3.1.x compatibility (3997)

Contributed by abhinav nilaratna.

• App: Restore behavior so Broadcast queues work. (3934)

Contributed by Patrick Cloke.

• Sphinx: Make appstr use standard format (4134) (4139)

Contributed by Preston Moore.

• App: Make id, name always accessible from logging.Formatter via extra (3994)

Contributed by Yoichi NAKAYAMA.

• Worker: Add worker_shutting_down signal (3998)

Contributed by Daniel Huang.

• PyPy: Support PyPy version 5.8.0 (4128)

Contributed by Omer Katz.

• Results: Elasticsearch: Fix serializing keys (3924)

Contributed by :github_user:staticfox.

• Canvas: Deserialize all tasks in a chain (4015)

Contributed by :github_user:fcoelho.

• Systemd: Recover loglevel for ExecStart in systemd config (4023)

Contributed by Yoichi NAKAYAMA.

• Sphinx: Use the Sphinx add_directive_to_domain API. (4037)

Contributed by Patrick Cloke.

• App: Pass properties to before_task_publish signal (4035)

Contributed by Javier Domingo Cansino.

• Results: Add SSL option for Redis backends (3831)

Contributed by Chris Kuehl.

• Beat: celery.schedule.crontab: fix reduce (3826) (3827)

Contributed by Taylor C. Richberger.

• State: Fix celery issues when using flower REST API

Contributed by Thierry RAMORASOAVINA.

• Results: Elasticsearch: Fix serializing document id.

Contributed by Acey9.

• Beat: Make shallow copy of schedules dictionary

Contributed by Brian May.

• Beat: Populate heap when periodic tasks are changed

Contributed by Wojciech Żywno.

• Task: Allow class methods to define tasks (3952)

Contributed by georgepsarakis.

• Platforms: Always return boolean value when checking if signal is supported (3962).

Contributed by Jian Yu.

• Canvas: Avoid duplicating chains in chords (3779)

Contributed by Ryan Hiebert.

• Canvas: Lookup task only if list has items (3847)

Contributed by Marc Gibbons.

• Results: Allow unicode message for exception raised in task (3903)

Contributed by George Psarakis.

• Python3: Support for Python 3.6 (3904, 3903, 3736)

Contributed by Jon Dufresne, George Psarakis, Asif Saifuddin Auvi, Omer Katz.

• App: Fix retried tasks with expirations (3790)

Contributed by Brendan MacDonell.

• • Fixes items format route in docs (3875)

Contributed by Slam.

• Utils: Fix maybe_make_aware (3850)

Contributed by Taylor C. Richberger.

• Task: Fix task ETA issues when timezone is defined in configuration (3867)

Contributed by George Psarakis.

• Concurrency: Consumer does not shutdown properly when embedded in gevent application (3746)

Contributed by Arcadiy Ivanov.

• Canvas: Fix 3725: Task replaced with group does not complete (3731)

Contributed by Morgan Doocy.

• Task: Correct order in chains with replaced tasks (3730)

Contributed by Morgan Doocy.

• Result: Enable synchronous execution of sub-tasks (3696)

Contributed by shalev67.

• Task: Fix request context for blocking task apply (added hostname) (3716)

Contributed by Marat Sharafutdinov.

• Utils: Fix task argument handling (3678) (3693)

Contributed by Roman Sichny.

• Beat: Provide a transparent method to update the Scheduler heap (3721)

Contributed by Alejandro Pernin.

• Beat: Specify default value for pidfile option of celery beat. (3722)

Contributed by Arnaud Rocher.

• Results: Elasticsearch: Stop generating a new field every time when a new result is being put (3708)

Contributed by Mike Chen.

• Requirements

• Now depends on :ref:Kombu 4.1.0 <kombu:version-4.1.0>.

• Results: Elasticsearch now reuses fields when new results are added.

Contributed by Mike Chen.

• Results: Fixed MongoDB integration when using binary encodings
  (Issue 3575).

Contributed by Andrew de Quincey.

• Worker: Making missing *args and **kwargs in Task protocol 1
  return empty value in protocol 2 (Issue 3687).

Contributed by Roman Sichny.

• App: Fixed :exc:TypeError in AMQP when using deprecated signal
  (Issue 3707).

Contributed by :github_user:michael-k.

• Beat: Added a transparent method to update the scheduler heap.

Contributed by Alejandro Pernin.

• Task: Fixed handling of tasks with keyword arguments on Python 3
  (Issue 3657).

Contributed by Roman Sichny.

• Task: Fixed request context for blocking task apply by adding missing
  hostname attribute.

Contributed by Marat Sharafutdinov.

• Task: Added option to run subtasks synchronously with
  disable_sync_subtasks argument.

Contributed by :github_user:shalev67.

• App: Fixed chaining of replaced tasks (Issue 3726).

Contributed by Morgan Doocy.

• Canvas: Fixed bug where replaced tasks with groups were not completing
  (Issue 3725).

Contributed by Morgan Doocy.

• Worker: Fixed problem where consumer does not shutdown properly when
  embedded in a gevent application (Issue 3745).

Contributed by Arcadiy Ivanov.

• Results: Added support for using AWS DynamoDB as a result backend (3736).

Contributed by George Psarakis.

• Testing: Added caching on pip installs.

Contributed by :github_user:orf.

• Worker: Prevent consuming queue before ready on startup (Issue 3620).

Contributed by Alan Hamlett.

• App: Fixed task ETA issues when timezone is defined in configuration
  (Issue 3753).

Contributed by George Psarakis.

• Utils: maybe_make_aware should not modify datetime when it is
  already timezone-aware (Issue 3849).

Contributed by Taylor C. Richberger.

• App: Fixed retrying tasks with expirations (Issue 3734).

Contributed by Brendan MacDonell.

• Results: Allow unicode message for exceptions raised in task
  (Issue 3858).

Contributed by :github_user:staticfox.

• Canvas: Fixed :exc:IndexError raised when chord has an empty header.

Contributed by Marc Gibbons.

• Canvas: Avoid duplicating chains in chords (Issue 3771).

Contributed by Ryan Hiebert and George Psarakis.

• Utils: Allow class methods to define tasks (Issue 3863).

Contributed by George Psarakis.

• Beat: Populate heap when periodic tasks are changed.

Contributed by :github_user:wzywno and Brian May.

• Results: Added support for Elasticsearch backend options settings.

Contributed by :github_user:Acey9.

• Events: Ensure Task.as_dict() works when not all information about
  task is available.

Contributed by :github_user:tramora.

• Schedules: Fixed pickled crontab schedules to restore properly (Issue 3826).

Contributed by Taylor C. Richberger.

• Results: Added SSL option for redis backends (Issue 3830).

Contributed by Chris Kuehl.

• Documentation and examples improvements by:

• Bruno Alla
• Jamie Alessio
• Vivek Anand
• Peter Bittner
• Kalle Bronsen
• Jon Dufresne
• James Michael DuPont
• Sergey Fursov
• Samuel Dion-Girardeau
• Daniel Hahler
• Mike Helmick
• Marc Hörsken
• Christopher Hoskin
• Daniel Huang
• Primož Kerin
• Michal Kuffa
• Simon Legner
• Anthony Lukach
• Ed Morley
• Jay McGrath
• Rico Moorman
• Viraj Navkal
• Ross Patterson
• Dmytro Petruk
• Luke Plant
• Eric Poelke
• Salvatore Rinchiera
• Arnaud Rocher
• Kirill Romanov
• Simon Schmidt
• Tamer Sherif
• YuLun Shih
• Ask Solem
• Tom 'Biwaa' Riat
• Arthur Vigil
• Joey Wilhelm
• Jian Yu
• YuLun Shih
• Arthur Vigil
• Joey Wilhelm
• :github_user:baixuexue123
• :github_user:bronsen
• :github_user:michael-k
• :github_user:orf
• :github_user:3lnc

4.0.2

=====
:release-date: 2016-12-15 03:31 P.M PST
:release-by: Ask Solem

• Now depends on :mod:amqp 2.1.4

This new version takes advantage of TCP Keepalive settings on Linux,
making it better at detecting closed connections, also in failover
conditions.

• Redis: Priority was reversed so, e.g. priority 0 became priority 9.

.. _version-4.0.1:

4.0.1

=====
:release-date: 2016-12-07 06:00 P.M PST
:release-by: Ask Solem

• Now depends on :mod:amqp 2.1.3

This new version takes advantage of the new TCP_USER_TIMEOUT socket option
on Linux.

• Producer: Fixed performance degradation when default exchange specified
  (Issue 651).

• QPid: Switch to using getattr in qpid.Transport.del (Issue 658)

Contributed by Patrick Creech.

• QPid: Now uses monotonic time for timeouts.

• MongoDB: Fixed compatibility with Python 3 (Issue 661).

• Consumer: __exit__ now skips cancelling consumer if connection-related
  error raised (Issue 670).

• MongoDB: Removes use of natural sort (Issue 638).

Contributed by Anton Chaporgin.

• Fixed wrong keyword argument channel error (Issue 652).

Contributed by Toomore Chiang.

• Safe argument to urllib.quote must be bytes on Python 2.x (Issue 645).

• Documentation improvments by:

• Carlos Edo
• Cemre Mengu

.. _version-4.0:

4.0

===
:release-date: 2016-10-28 16:45 P.M UTC
:release-by: Ask Solem

• Now depends on :mod:amqp 2.0.

The new py-amqp version have been refactored for better performance,
using modern Python socket conventions, and API consistency.

• No longer depends on :mod:anyjson.

Kombu will now only choose between :pypi:simplejson and the built-in
:mod:json.

Using the latest version of simplejson is recommended:

.. code-block:: console

   $ pip install -U simplejson

• Removed transports that are no longer supported in this version:

• Django ORM transport
• SQLAlchemy ORM transport
• Beanstalk transport
• ZeroMQ transport
• amqplib transport (use pyamqp).

• API Changes

• Signature of :class:kombu.Message now takes body as first argment.

   It used to be ``Message(channel, body=body, **kw)``, but now it's
   ``Message(body, channel=channel, **kw)``.

   This is unlikey to affect you, as the Kombu API does not have
   users instantiate messages manually.

• New SQS transport

Donated by NextDoor, with additional contributions from mdk.

.. note::

   ``kombu[sqs]`` now depends on :pypi:`pycurl`.

• New Consul transport.

Contributed by Wido den Hollander.

• New etcd transport.

Contributed by Stephen Milner.

• New Qpid transport.

It was introduced as an experimental transport in Kombu 3.0, but is now
mature enough to be fully supported.

Created and maintained by Brian Bouterse.

• Redis: Priority 0 is now lowest, 9 is highest.
  (backward incompatible)

This to match how priorities in AMQP works.

Fix contributed by Alex Koshelev.

• Redis: Support for Sentinel

You can point the connection to a list of sentinel URLs like:

.. code-block:: text

   sentinel://0.0.0.0:26379;sentinel://0.0.0.0:26380/...

where each sentinel is separated by a ;. Multiple sentinels are handled
by :class:kombu.Connection constructor, and placed in the alternative
list of servers to connect to in case of connection failure.

Contributed by Sergey Azovskov, and Lorenzo Mancini

• RabbitMQ Queue Extensions

New arguments have been added to :class:kombu.Queue that lets
you directly and conveniently configure the RabbitMQ queue extensions.

• Queue(expires=20.0)

   Set queue expiry time in float seconds.

   See :attr:`kombu.Queue.expires`.

• Queue(message_ttl=30.0)

   Set queue message time-to-live float seconds.

   See :attr:`kombu.Queue.message_ttl`.

• Queue(max_length=1000)

   Set queue max length (number of messages) as int.

   See :attr:`kombu.Queue.max_length`.

• Queue(max_length_bytes=1000)

   Set queue max length (message size total in bytes) as int.

   See :attr:`kombu.Queue.max_length_bytes`.

• Queue(max_priority=10)

   Declare queue to be a priority queue that routes messages
   based on the ``priority`` field of the message.

   See :attr:`kombu.Queue.max_priority`.

• RabbitMQ: Message.ack now supports the multiple argument.

If multiple is set to True, then all messages received before
the message being acked will also be acknowledged.

• amqps:// can now be specified to require SSL (Issue 610).

• Consumer.cancel_by_queue is now constant time.

• Connection.ensure* now raises :exc:kombu.exceptions.OperationalError.

Things that can be retried are now reraised as
:exc:kombu.exceptions.OperationalError.

• Redis: Fixed SSL support.

Contributed by Robert Kolba.

• New Queue.consumer_arguments can be used for the ability to
  set consumer priority via x-priority.

See https://www.rabbitmq.com/consumer-priority.html

Example:

.. code-block:: python

   Queue(
       'qname',
       exchange=Exchange('exchange'),
       routing_key='qname',
       consumer_arguments={'x-priority': 3},
   )

• Queue/Exchange: no_declare option added (also enabled for
  internal amq. exchanges) (Issue 565).

• JSON serializer now calls obj.__json__ for unsupported types.

This means you can now define a __json__ method for custom
types that can be reduced down to a built-in json type.

Example:

.. code-block:: python

   class Person:
       first_name = None
       last_name = None
       address = None

       def __json__(self):
           return {
               'first_name': self.first_name,
               'last_name': self.last_name,
               'address': self.address,
           }

• JSON serializer now handles datetimes, Django promise, UUID and Decimal.

• Beanstalk: Priority 0 is now lowest, 9 is highest.
  (backward incompatible)

This to match how priorities in AMQP works.

Fix contributed by Alex Koshelev.

• Redis: now supports SSL using the ssl argument to
  :class:~kombu.Connection.

• Redis: Fanout exchanges are no longer visible between vhosts,
  and fanout messages can be filtered by patterns.
  (backward incompatible)

It was possible to enable this mode previously using the
fanout_prefix, and fanout_patterns
transport options, but now these are enabled by default.

If you want to mix and match producers/consumers running different
versions you need to configure your kombu 3.x clients to also enable
these options:

.. code-block:: pycon

   >>> Connection(transport_options={
       'fanout_prefix': True,
       'fanout_patterns': True,
   })

• Pidbox: Mailbox new arguments: TTL and expiry.

Mailbox now supports new arguments for controlling
message TTLs and queue expiry, both for the mailbox
queue and for reply queues.

• queue_expires (float/int seconds).
• queue_ttl (float/int seconds).
• reply_queue_expires (float/int seconds).
• reply_queue_ttl (float/int seconds).

All take seconds in int/float.

Contributed by Alan Justino.

• Exchange.delivery_mode now defaults to :const:None, and the default
  is instead set by Producer.publish.

• :class:~kombu.Consumer now supports a new prefetch_count argument,
  which if provided will force the consumer to set an initial prefetch count
  just before starting.

• Virtual transports now stores priority as a property, not in
  delivery_info, to be compatible with AMQP.

• reply_to argument to Producer.publish can now be
  :class:~kombu.Queue instance.

• Connection: There's now a new method
  Connection.supports_exchange_type(type) that can be used to check if the
  current transport supports a specific exchange type.

• SQS: Consumers can now read json messages not sent by Kombu.

Contributed by Juan Carlos Ferrer.

• SQS: Will now log the access key used when authentication fails.

Contributed by Hank John.

• Added new :class:kombu.mixins.ConsumerProducerMixin for consumers that
  will also publish messages on a separate connection.

• Messages: Now have a more descriptive repr.

Contributed by Joshua Harlow.

• Async: HTTP client based on curl.

• Async: Now uses poll instead of select where available.

• MongoDB: Now supports priorities

Contributed by Alex Koshelev.

• Virtual transports now supports multiple queue bindings.

Contributed by Federico Ficarelli.

• Virtual transports now supports the anon exchange.

If when publishing a message, the exchange argument is set to '' (empty
string), the routing_key will be regarded as the destination queue.

This will bypass the routing table compeltely, and just deliver the
message to the queue name specified in the routing key.

• Zookeeper: Transport now uses the built-in suport in kazoo to handle
  failover when using a list of server names.

Contributed by Joshua Harlow.

• ConsumerMixin.run now passes keyword arguments to .consume.

Deprecations and removals

• The deprecated method Consumer.add_queue_from_dict has been removed.

Use instead:

.. code-block:: python

   consumer.add_queue(Queue.from_dict(queue_name, **options))

• The deprecated function kombu.serialization.encode has been removed.

Use :func:kombu.serialization.dumps instead.

• The deprecated function kombu.serialization.decode has been removed.

Use :func:kombu.serialization.loads instead.

• Removed module kombu.syn

detect_environment has been moved to kombu.utils.compat

.. _version-3.0.37:

3.5.0.3

---

• Adds Process._authkey alias to .authkey for 2.7 compat.
• Remove superfluous else clause from max_memory_per_child_check.
• Document and test all supported Python versions.
• Extend 'Process' to be compatible with < Py3.5.
• Use a properly initialized logger in pool.py error logging.
• _trywaitkill can now kill a whole process group if the worker process declares itself as a group leader.
• Fix cpython issue 14881 (See http://bugs.python.org/issue14881).
• Fix for a crash on windows.
• Fix messaging in case of worker exceeds max memory.

3.5.0.2

---

• max_memory_per_child was measured in kilobytes on Linux, but bytes on
  *BSD/MacOS, it's now always kilobytes.

• Windows: Adds support for max_memory_per_child, but requires the
  psutil package to be installed.

• Fixed bug in ForkingPickler.loadbuf, where it tried to pass
  a BytesIO instance directly to pickle.loads on Python 2.7.

3.5.0.1

---

• Connection: Properly handle EINTR (Issue 191).

• Fixed bug with missing CreateProcess for Windows on Python 2.7.

• Adds Process._counter for compatibility with Python <3.5.

3.5.0.0

---

• No longer supports Python 2.6

You need Python 2.7 or later to use this version of billiard.

• Merged changes from CPython 3.5

3.3.0.20

---

• Pool: Timeouts will attempt to send SIGKILL, but this signal
  does not exist on Windows. Replaced with SIGTERM.

3.3.0.19

---

• Pool: Exceptions in user timeout callbacks are now logged instead
  of crashing the pool.

Contributed by Pierre Fersing.

• Pool: Exit codes in errors were improperly being represented as signals.

• Pool: .map. and .imap now working again.

• Now builds on FreeBSD 10.

Contributed by Michael Fladischer.

3.3.0.18

---

• Now compiles on GNU/kFreeBSD

Contributed by Michael Fladischer.

• Pool: AF_PIPE address fixed so that it works on recent Windows versions
  in combination with Python 2.7.7.

Fix contributed by Joshua Tacoma.

• Pool: Fix for Supervisor object has no attribute _children error.

Fix contributed by Andres Riancho.

• Pool: Fixed bug with human_status(None).

• Pool: shrink did not work properly if asked to remove more than 1 process.

3.3.0.17

---

• Fixes SemLock on Python 3.4 (Issue 107) when using
  forking_enable(False).

• Pool: Include more useful exitcode information when processes exit.

3.3.0.16

---

• Previous release was missing the billiard.py3 package from MANIFEST
  so the installation would not work on Python 3.

3.3.0.15

---

• Pool: Fixed "cannot join process not started" error.

• Now uses billiard.py2 and billiard.py3 specific packages that are installed
  depending on the python version used.

This way the installation will not import version specific modules (and
possibly crash).

3.3.0.14

---

• Fixed problem with our backwards compatible bytes wrapper
  (Issue 103).

• No longer expects frozen applications to have a valid __file__
  attribute.

Fix contributed by George Sibble.

3.3.0.13

---

• Fixes compatability with Python < 2.7.6

• No longer attempts to handle SIGBUS

Contributed by Vishal Vatsa.

• Non-thread based pool now only handles signals:

SIGHUP, SIGQUIT, SIGTERM, SIGUSR1,
SIGUSR2.

• setup.py: Only show compilation warning for build related commands.

3.3.0.12

---

• Fixed installation for Python 3.

Contributed by Rickert Mulder.

• Pool: Fixed bug with maxtasksperchild.

Fix contributed by Ionel Cristian Maries.

• Pool: Fixed bug in maintain_pool.

3.3.0.11

---

• Fixed Unicode error when installing the distribution (Issue 89).

• Daemonic processes are now allowed to have children.

But note that it will not be possible to automatically
terminate them when the process exits.

See discussion at celery/celery#1709

• Pool: Would not always be able to detect that a process exited.

3.3.0.10

---

• Windows: Fixed problem with missing WAITABANDONED_0

Fix contributed by Matthias Wagner

• Windows: PipeConnection can now be inherited.

Fix contributed by Matthias Wagner

3.3.0.9

---

• Temporary workaround for Celery maxtasksperchild issue.

Fix contributed by Ionel Cristian Maries.

3.3.0.8

---

• Now also sets multiprocessing.current_process for compatibility
  with loggings processName field.

3.3.0.7

---

• Fixed compatibility with PyPy 2.1 + 2.2.

• Fixed problem in pypy detection.

Fix contributed by Tin Tvrtkovic.

• Now uses ctypes.find_library instead of hardcoded path to find
  the macOS CoreServices framework.

Fix contributed by Moritz Kassner.

3.3.0.6

---

• Now works without C extension again.

• New _billiard.read(fd, buffer, [len, ]) function
  implements os.read with buffer support (new buffer API)

• New pure-python implementation of Connection.send_offset.

3.3.0.5

---

• All platforms except for Windows/PyPy/Jython now requires the C extension.

3.3.0.4

---

• Fixed problem with Python3 and setblocking.

3.3.0.3

---

• Now works on Windows again.

3.3.0.2

---

• ApplyResult.terminate() may be set to signify that the job
  must not be executed. It can be used in combination with
  Pool.terminate_job.

• Pipe/_SimpleQueue: Now supports rnonblock/wnonblock arguments
  to set the read or write end of the pipe to be nonblocking.

• Pool: Log message included exception info but exception happened
  in another process so the resulting traceback was wrong.

• Pool: Worker process can now prepare results before they are sent
  back to the main process (using Worker.prepare_result).

3.3.0.1

---

• Pool: New correlation_id argument to apply_async can be
  used to set a related id for the ApplyResult object returned:

>>> r = pool.apply_async(target, args, kwargs, correlation_id='foo')
>>> r.correlation_id
'foo'

• Pool: New callback on_process_exit is called when a pool
  process exits, with signature (pid, exitcode).

Contributed by Daniel M. Taub.

• Pool: Improved the too many restarts detection.

3.3.0.0

---

• Dual code base now runs on Python 2.6+ and Python 3.

• No longer compatible with Python 2.5

• Includes many changes from multiprocessing in 3.4.

• Now uses time.monotonic when available, also including
  fallback implementations for Linux and macOS.

• No longer cleans up after receiving SIGILL, SIGSEGV or SIGFPE

Contributed by Kevin Blackham

• Finalize and register_after_fork is now aliases to multiprocessing.

It's better to import these from multiprocessing directly now
so that there aren't multiple registries.

• New billiard.queues._SimpleQueue that does not use semaphores.

• Pool: Can now be extended to support using multiple IPC queues.

• Pool: Can now use async I/O to write to pool IPC queues.

• Pool: New Worker.on_loop_stop handler can be used to add actions
  at pool worker process shutdown.

Note that, like all finalization handlers, there is no guarantee that
this will be executed.

Contributed by dmtaub.

Got merge conflicts? Close this PR and delete the branch. I'll create a new PR for you.

Happy merging! 🤖

[luzfcb]

This pull-request requires changes on https://github.com/pydanny/cookiecutter-django/blob/master/%7B%7Bcookiecutter.project_slug%7D%7D/config/settings/base.py#L268-L275
before merge.

I've had some issues with Celery 4, especially with regard to time zones and scheduled tasks.
At the moment, I do not own any projects that use celery 4.
This version apparently (by changelog) solves some of the problems I was experiencing, but it will still take a few weeks for me to have some free time to try to migrate again one of my projects .

[sfdye]

+1 for this

[sfdye]

@luzfcb Did you have the time work on this or do you need some help?

I just managed to update Celery from 3.1.25 to 4.1 for one of my projects without issues. I am willing to throw out a PR and you can have a review.

[webyneter]

@sfdye that'd be nice!

[sfdye]

@webyneter here you go #1446

[browniebroke]

Closing this as there is another PR open