Skip to content

Commit

Permalink
Added changelog for v5.4.0
Browse files Browse the repository at this point in the history
  • Loading branch information
Nusnus committed Apr 17, 2024
1 parent d0aae65 commit 2425063
Show file tree
Hide file tree
Showing 3 changed files with 322 additions and 3 deletions.
91 changes: 88 additions & 3 deletions Changelog.rst
Original file line number Diff line number Diff line change
Expand Up @@ -5,8 +5,91 @@
================

This document contains change notes for bugfix & new features
in the main branch & 5.3.x series, please see :ref:`whatsnew-5.3` for
an overview of what's new in Celery 5.3.
in the main branch & 5.4.x series, please see :ref:`whatsnew-5.4` for
an overview of what's new in Celery 5.4.

.. _version-5.4.0:

5.4.0
=====

:release-date: 2024-04-17
:release-by: Tomer Nosrati

Celery v5.4.0 and v5.3.x have consistently focused on enhancing the overall QA, both internally and externally.
This effort led to the new pytest-celery v1.0.0 release, developed concurrently with v5.3.0 & v5.4.0.

This release introduces two significant QA enhancements:

- **Smoke Tests**: A new layer of automatic tests has been added to Celery's standard CI. These tests are designed to handle production scenarios and complex conditions efficiently. While new contributions will not be halted due to the lack of smoke tests, we will request smoke tests for advanced changes where appropriate.
- `Standalone Bug Report Script <https://docs.celeryq.dev/projects/pytest-celery/en/latest/userguide/celery-bug-report.html>`_: The new pytest-celery plugin now allows for encapsulating a complete Celery dockerized setup within a single pytest script. Incorporating these into new bug reports will enable us to reproduce reported bugs deterministically, potentially speeding up the resolution process.

Contrary to the positive developments above, there have been numerous reports about issues with the Redis broker malfunctioning
upon restarts and disconnections. Our initial attempts to resolve this were not successful (#8796).
With our enhanced QA capabilities, we are now prepared to address the core issue with Redis (as a broker) again.

The rest of the changes for this release are grouped below, with the changes from the latest release candidate listed at the end.

Changes
-------
- Add a Task class specialised for Django (#8491)
- Add Google Cloud Storage (GCS) backend (#8868)
- Added documentation to the smoke tests infra (#8970)
- Added a checklist item for using pytest-celery in a bug report (#8971)
- Bugfix: Missing id on chain (#8798)
- Bugfix: Worker not consuming tasks after Redis broker restart (#8796)
- Catch UnicodeDecodeError when opening corrupt beat-schedule.db (#8806)
- chore(ci): Enhance CI with `workflow_dispatch` for targeted debugging and testing (#8826)
- Doc: Enhance "Testing with Celery" section (#8955)
- Docfix: pip install celery[sqs] -> pip install "celery[sqs]" (#8829)
- Enable efficient `chord` when using dynamicdb as backend store (#8783)
- feat(daemon): allows daemonization options to be fetched from app settings (#8553)
- Fix DeprecationWarning: datetime.datetime.utcnow() (#8726)
- Fix recursive result parents on group in middle of chain (#8903)
- Fix typos and grammar (#8915)
- Fixed version documentation tag from #8553 in configuration.rst (#8802)
- Hotfix: Smoke tests didn't allow customizing the worker's command arguments, now it does (#8937)
- Make custom remote control commands available in CLI (#8489)
- Print safe_say() to stdout for non-error flows (#8919)
- Support moto 5.0 (#8838)
- Update contributing guide to use ssh upstream url (#8881)
- Update optimizing.rst (#8945)
- Updated concurrency docs page. (#8753)

Dependencies Updates
--------------------
- Bump actions/setup-python from 4 to 5 (#8701)
- Bump codecov/codecov-action from 3 to 4 (#8831)
- Bump isort from 5.12.0 to 5.13.2 (#8772)
- Bump msgpack from 1.0.7 to 1.0.8 (#8885)
- Bump mypy from 1.8.0 to 1.9.0 (#8898)
- Bump pre-commit to 3.6.1 (#8839)
- Bump pre-commit/action from 3.0.0 to 3.0.1 (#8835)
- Bump pytest from 8.0.2 to 8.1.1 (#8901)
- Bump pytest-celery to v1.0.0 (#8962)
- Bump pytest-cov to 5.0.0 (#8924)
- Bump pytest-order from 1.2.0 to 1.2.1 (#8941)
- Bump pytest-subtests from 0.11.0 to 0.12.1 (#8896)
- Bump pytest-timeout from 2.2.0 to 2.3.1 (#8894)
- Bump python-memcached from 1.59 to 1.61 (#8776)
- Bump sphinx-click from 4.4.0 to 5.1.0 (#8774)
- Update cryptography to 42.0.5 (#8869)
- Update elastic-transport requirement from <=8.12.0 to <=8.13.0 (#8933)
- Update elasticsearch requirement from <=8.12.1 to <=8.13.0 (#8934)
- Upgraded Sphinx from v5.3.0 to v7.x.x (#8803)

Changes since 5.4.0rc2
----------------------
- Update elastic-transport requirement from <=8.12.0 to <=8.13.0 (#8933)
- Update elasticsearch requirement from <=8.12.1 to <=8.13.0 (#8934)
- Hotfix: Smoke tests didn't allow customizing the worker's command arguments, now it does (#8937)
- Bump pytest-celery to 1.0.0rc3 (#8946)
- Update optimizing.rst (#8945)
- Doc: Enhance "Testing with Celery" section (#8955)
- Bump pytest-celery to v1.0.0 (#8962)
- Bump pytest-order from 1.2.0 to 1.2.1 (#8941)
- Added documentation to the smoke tests infra (#8970)
- Added a checklist item for using pytest-celery in a bug report (#8971)

.. _version-5.4.0rc2:

Expand All @@ -33,7 +116,7 @@ an overview of what's new in Celery 5.3.
- Docfix: pip install celery[sqs] -> pip install "celery[sqs]" (#8829)
- Bump pre-commit/action from 3.0.0 to 3.0.1 (#8835)
- Support moto 5.0 (#8838)
- Another fix for `link_error` signatures being `dict`s instead of `Signature`s (#8841)
- Another fix for `link_error` signatures being `dict`s instead of `Signature` s (#8841)
- Bump codecov/codecov-action from 3 to 4 (#8831)
- Upgrade from pytest-celery v1.0.0b1 -> v1.0.0b2 (#8843)
- Bump pytest from 7.4.4 to 8.0.0 (#8823)
Expand Down Expand Up @@ -258,6 +341,8 @@ The code changes are mostly fix for regressions. More details can be found below
- Revert "Add Semgrep to CI" (#8477)
- Revert "Revert "Add Semgrep to CI"" (#8478)

.. _CELERY:

.. _version-5.3.3:

5.3.3 (Yanked)
Expand Down
1 change: 1 addition & 0 deletions docs/history/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@ version please visit :ref:`changelog`.
.. toctree::
:maxdepth: 2

whatsnew-5.4
whatsnew-5.3
whatsnew-5.1
changelog-5.1
Expand Down
233 changes: 233 additions & 0 deletions docs/history/whatsnew-5.4.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,233 @@
.. _whatsnew-5.4:

=========================================
What's new in Celery 5.4 (Opalescent)
=========================================
:Author: Tomer Nosrati (``tomer.nosrati at gmail.com``).

.. sidebar:: Change history

What's new documents describe the changes in major versions,
we also have a :ref:`changelog` that lists the changes in bugfix
releases (0.0.x), while older series are archived under the :ref:`history`
section.

Celery is a simple, flexible, and reliable distributed programming framework
to process vast amounts of messages, while providing operations with
the tools required to maintain a distributed system with python.

It's a task queue with focus on real-time processing, while also
supporting task scheduling.

Celery has a large and diverse community of users and contributors,
you should come join us :ref:`on IRC <irc-channel>`
or :ref:`our mailing-list <mailing-list>`.

.. note::

Following the problems with Freenode, we migrated our IRC channel to Libera Chat
as most projects did.
You can also join us using `Gitter <https://gitter.im/celery/celery>`_.

We're sometimes there to answer questions. We welcome you to join.

To read more about Celery you should go read the :ref:`introduction <intro>`.

While this version is **mostly** backward compatible with previous versions
it's important that you read the following section as this release
is a new major version.

This version is officially supported on CPython 3.8, 3.9 & 3.10
and is also supported on PyPy3.8+.

.. _`website`: https://docs.celeryq.dev/en/stable/

.. topic:: Table of Contents

Make sure you read the important notes before upgrading to this version.

.. contents::
:local:
:depth: 2

Preface
=======

.. note::

**This release contains fixes for many long standing bugs & stability issues.
We encourage our users to upgrade to this release as soon as possible.**

The 5.4.0 release is a new feature release for Celery.

Releases in the 5.x series are codenamed after songs of `Jon Hopkins <https://en.wikipedia.org/wiki/Jon_Hopkins>`_.
This release has been codenamed `Opalescent <https://www.youtube.com/watch?v=9ByfK25WsMM>`_.

From now on we only support Python 3.8 and above.
We will maintain compatibility with Python 3.8 until it's
EOL in 2024.

*— Tomer Nosrati*

Long Term Support Policy
------------------------

We no longer support Celery 4.x as we don't have the resources to do so.
If you'd like to help us, all contributions are welcome.

Celery 5.x **is not** an LTS release. We will support it until the release
of Celery 6.x.

We're in the process of defining our Long Term Support policy.
Watch the next "What's New" document for updates.

Wall of Contributors
--------------------

.. note::

This wall was automatically generated from git history,
so sadly it doesn't not include the people who help with more important
things like answering mailing-list questions.

Upgrading from Celery 4.x
=========================

Step 1: Adjust your command line invocation
-------------------------------------------

Celery 5.0 introduces a new CLI implementation which isn't completely backwards compatible.

The global options can no longer be positioned after the sub-command.
Instead, they must be positioned as an option for the `celery` command like so::

celery --app path.to.app worker

If you were using our :ref:`daemonizing` guide to deploy Celery in production,
you should revisit it for updates.

Step 2: Update your configuration with the new setting names
------------------------------------------------------------

If you haven't already updated your configuration when you migrated to Celery 4.0,
please do so now.

We elected to extend the deprecation period until 6.0 since
we did not loudly warn about using these deprecated settings.

Please refer to the :ref:`migration guide <conf-old-settings-map>` for instructions.

Step 3: Read the important notes in this document
-------------------------------------------------

Make sure you are not affected by any of the important upgrade notes
mentioned in the :ref:`following section <v500-important>`.

You should verify that none of the breaking changes in the CLI
do not affect you. Please refer to :ref:`New Command Line Interface <new_command_line_interface>` for details.

Step 4: Migrate your code to Python 3
-------------------------------------

Celery 5.x only supports Python 3. Therefore, you must ensure your code is
compatible with Python 3.

If you haven't ported your code to Python 3, you must do so before upgrading.

You can use tools like `2to3 <https://docs.python.org/3.8/library/2to3.html>`_
and `pyupgrade <https://github.com/asottile/pyupgrade>`_ to assist you with
this effort.

After the migration is done, run your test suite with Celery 4 to ensure
nothing has been broken.

Step 5: Upgrade to Celery 5.4
-----------------------------

At this point you can upgrade your workers and clients with the new version.

.. _v540-important:

Important Notes
===============

Supported Python Versions
-------------------------

The supported Python versions are:

- CPython 3.8
- CPython 3.9
- CPython 3.10
- PyPy3.8 7.3.11 (``pypy3``)

Experimental support
~~~~~~~~~~~~~~~~~~~~

Celery supports these Python versions provisionally as they are not production
ready yet:

- CPython 3.11

Quality Improvements and Stability Enhancements
-----------------------------------------------

Celery 5.4 focuses on elevating the overall quality and stability of the project.
We have dedicated significant efforts to address various bugs, enhance performance,
and make improvements based on valuable user feedback.

Better Compatibility and Upgrade Confidence
-------------------------------------------

Our goal with Celery 5.4 is to instill confidence in users who are currently
using Celery 4 or older versions. We want to assure you that upgrading to
Celery 5.4 will provide a more robust and reliable experience.

Dropped support for Python 3.7
------------------------------

Celery now requires Python 3.8 and above.

Python 3.7 will reach EOL in June, 2023.
In order to focus our efforts we have dropped support for Python 3.6 in
this version.

If you still require to run Celery using Python 3.7
you can still use Celery 5.2.
However we encourage you to upgrade to a supported Python version since
no further security patches will be applied for Python 3.7 after
the 23th of June, 2023.

Kombu
-----

Starting from v5.4.0, the minimum required version is Kombu 5.3.

Redis
-----

redis-py 4.5.x is the new minimum required version.


SQLAlchemy
---------------------

SQLAlchemy 1.4.x & 2.0.x is now supported in celery v5.4


Billiard
-------------------

Minimum required version is now 4.1.0


Deprecate pytz and use zoneinfo
-------------------------------

A switch have been made to zoneinfo for handling timezone data instead of pytz.

Django
------

Minimum django version is bumped to v2.2.28.
Also added --skip-checks flag to bypass django core checks.

0 comments on commit 2425063

Please sign in to comment.