chore: Bumping Celery

[john-bodley]

SUMMARY

At Airbnb we had an incident with Celery where tasks weren't being picked up and the CPU was pegged at close to 100%. We configure Celery with the gevent pool and the only workaround to remedy the problem was to restart the workers sans gevent.

We haven't been able to track down the root cause of the problem, though a Google search surfaced a similar issue (though we actually don't call AsyncResult.get() since the SQL Lab results are stored elsewhere) which I believe was fixed in 4.4.2 (which is the version Superset currently uses). The issue seems to be that there are no release note associated with said release nor does that tag exist within the Celery GitHub repo. Given there have been a number of recent releases (which mostly look like bug fixes) I felt it was prudent that we bumped the version of Celery to the latest stable version.

Note we use a MySQL database for Superset and historically have been using the mysqlclient DBAPI but we're looking into using PyMySQL which fully supports gevent. We're not sure if this was the issue, but the Celery workers poll certain async engines, i.e., Presto, every few seconds per task and thus having a fully gevent compliant DBAPI may also be benefical.

BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF

TEST PLAN

CI.

ADDITIONAL INFORMATION

• Has associated issue:
• Changes UI
• Requires DB Migration.
• Confirm DB Migration upgrade and downgrade tested.
• Introduces new feature or API
• Removes existing feature or API

[villebro]

It's a shame the otherwise well curated changelog is missing 4.4.1 and 4.4.2. But by the looks of it the more recent versions have been fixing many bugs, so I'm sure we're better off on 4.4.6 than 4.4.2.