feat(celery): upgrade celery and its dependencies packages

[Musa10]

SUMMARY

Upgrading celery packages after a lot of bugs in using cache like memory leakes, backend keep requesting async-event api, that have been occured using basic celery configs, and fixed using celery v5.3.6

TESTING INSTRUCTIONS

Test cache and charts loading with memory usage when loading a dashboard

ADDITIONAL INFORMATION

• Has associated issue:
• Required feature flags:
  GLOBAL_ASYNC_QUERIES:TRUE
• Changes UI
• Includes DB Migration (follow approval process in SIP-59)
  • Migration is atomic, supports rollback & is backwards-compatible
  • Confirm DB migration upgrade and downgrade tested
  • Runtime estimates and downtime expectations provided
• Introduces new feature or API
• Removes existing feature or API

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (c947be3) 69.16% compared to head (2383a4f) 69.16%.

Additional details and impacted files

@@            Coverage Diff             @@
##           master   #26604      +/-   ##
==========================================
- Coverage   69.16%   69.16%   -0.01%     
==========================================
  Files        1948     1948              
  Lines       76062    76062              
  Branches     8493     8493              
==========================================
- Hits        52609    52606       -3     
- Misses      21273    21276       +3     
  Partials     2180     2180              

Flag	Coverage Δ
hive	53.68% <ø> (ø)
mysql	78.06% <ø> (ø)
postgres	78.16% <ø> (ø)
presto	53.63% <ø> (ø)
python	82.86% <ø> (ø)
sqlite	77.75% <ø> (ø)
unit	55.89% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[villebro]

Very cool, thanks for the PR @Musa10 I wasn't aware that Celery moves so quickly; here's the changelog: https://docs.celeryq.dev/en/stable/changelog.html#version-5-3-6. Lots of things have indeed changed since 5.2.2, and I get the feeling 5.3.6 is indeed more stable than 5.2.2, despite those yanked 5.3.2-5.3.4.

[villebro]

@dpgaspar @betodealmeida @john-bodley with all these nice improvements in 5.3.6 vs 5.2.2, do you think we should also set the lower bound to 5.3.6, rather than just bump the lock files?

[villebro]

Btw, I assume this issue is still unresolved.. 🙁 celery/celery#7276

[john-bodley]

@Musa10 currently your PR only impacts the frozen requirements—typically used for CI—hence if you simply pip install Apache Superset (without first installing the frozen dependencies) you’re likely not really “bumping” Celery.

In order to ensure Celery is upgraded for all situations you should increase the lower bound in the setup.py file.