Limit celery by excluding 5.3.2 and 5.3.3 #34031
Conversation
There is a new database field introduced by Celery in 5.3.2 and repeated in 5.3.3 wihch is not included in automated migrations, so users upgrading celery might have failing celery installation. The issue is already reported and acknowledged, so it is lilely to be fixed in 5.3.4 - so excluding 5.3.2 and 5.3.4 is the best approach.
|
Could you check the last sentence in the PR description? |
|
Of course :) |
|
Also - question - I understand that this is purely celery thing ? When celery uses the DB, they manage their own migrations? I looked in our migrations and I have not found any celery migrations we run :)? |
In airflow, we run a query that somehow calls the new field and fails. It was reported in our internal testing of main yesterday. |
Yes, when user configures Celery to use SQLAlchemy as results backend, Celery will take care of table migration and schema. I think the bug is in their classes which try to use a field not created because of problem in the migration script. However Celery supports other backends (Redis, RabbitMQ, custom result backend, ...), meaning users using Airflow with Celery with a different backend than SQLAlchemy will not be impacted by this bug. |
|
Yeah. It's all the "celery" internal code. And yeah - not everyone is affected. So I think the approach where we exclude only those two versions is a sound approach. I also asked if we can expect that 5.3.4 will have a fix - but I guess they have no choice but to fix it. |
|
Yep. Looks like the exclusion works as expected: 
|
BTW. This is why our |
|
Added task for it #34032 |
|
Celery v5.3.2 & v5.3.3 were yanked and the breaking changes reverted in Keep this in mind so you won't exclude a valid version with important bug fixes @potiuk |
@Nusnus This new stable patch you're talking about will be released in v5.3.4, right? |
We've completely removed the 5.3.2+ releases. We'll have a single 5.3.2 new release. Not 5.3.4. My attention right now is unfortunately going one by one on all of the changes from v5.3.1 and do my best to find out if there is anything else we missed. If we didn't miss anything else, the new release (5.3.2) should be out in the next 24-48h. |
Thank you @Nusnus for this information! However, deleting a release is completely different from yanking it, and IMHO it's a bad practice. As far as I know, to yank a python release we just need to yank it in PyPI, and update the github release by adding (YANKED) to the description. I really don't know how you manage the releases in Celery, I will check if this procedure is described somewhere. Again thank you for this comment! |
|
I think you really can't event have another 5.3.2 release. PyPi will not let you upload another 5.3.2 even if you delete the previous one |
|
@Nusnus look here pypa/packaging-problems#74 |
|
With every failure, we learn new lessons. As the most recent Owner at Celery, I am still discovering what's under the hood and I've been investigating the release flow for some time.
Unfortunately, the release flow is very lacking as I have discovered these past few days. As far as I am aware, there isn't an official updated procedure, which is why this latest release was messy. I am discussing this exact subject with the rest of the Celery Owners team to improve on this flow as part of a general effort from 5.3.0+ to increase the frequency of new releases, mainly with patch/bugfixes releases (which 5.3.2 was supposed to be, fixing important bugs), so I am totally onboard with improving this procedure.
This is good feedback, I'll pass it on to our team to avoid such practices in the future. Thank you!
Are you sure? I need to confirm that. If correct, then 5.3.4 is inevitable. |
Seems like the “solutions" there are even messier. v5.3.4 then. Thank you! |
Been there, done that :) |
There is a new database field introduced by Celery in 5.3.2 and repeated in 5.3.3 wihch is not included in automated migrations, so users upgrading celery might have failing celery installation. The issue is already reported and acknowledged, so it is lilely to be fixed in 5.3.4 - so excluding 5.3.2 and 5.3.4 is the best approach. (cherry picked from commit b6318ff)
There is a new database field introduced by Celery in 5.3.2 and repeated in 5.3.3 wihch is not included in automated migrations, so users upgrading celery might have failing celery installation. The issue is already reported and acknowledged, so it is lilely to be fixed in 5.3.4 - so excluding 5.3.2 and 5.3.4 is the best approach. (cherry picked from commit b6318ff)
There is a new database field introduced by Celery in 5.3.2 and repeated in 5.3.3 wihch is not included in automated migrations, so users upgrading celery might have failing celery installation.
The issue is already reported celery/celery#8470
and acknowledged, so it is lilely to be fixed in 5.3.4 - so excluding 5.3.2 and 5.3.3 is the best approach.
^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named
{pr_number}.significant.rstor{issue_number}.significant.rst, in newsfragments.