chore(celery): Cleanup config and async query specifications #25314
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
john-bodley
added
the
review:checkpoint
john-bodley
commented
Sep 15, 2023
| 'sql_lab.get_sql_results': { | ||
| 'rate_limit': '100/s', | ||
| }, | ||
| 'email_reports.send': { |
There was a problem hiding this comment.
The email_reports.send task does not exist.
| imports = ('superset.sql_lab', "superset.tasks", "superset.tasks.thumbnails", ) | ||
| result_backend = 'redis://%s:%s/0' % (REDIS_HOST, REDIS_PORT) | ||
| broker_url = f"redis://{REDIS_HOST}:{REDIS_PORT}/0" | ||
| imports = ( |
There was a problem hiding this comment.
Imports should include importing the modules where the associated tasks are defined.
| ) | ||
| result_backend = 'redis://localhost:6379/0' | ||
| worker_log_level = 'DEBUG' |
There was a problem hiding this comment.
Setting does not exist.
| 'sql_lab.get_sql_results': { | ||
| 'rate_limit': '100/s', | ||
| }, | ||
| 'email_reports.send': { |
| }, | ||
| } | ||
| beat_schedule = { | ||
| 'email_reports.schedule_hourly': { |
There was a problem hiding this comment.
Task does not exist.
| imports = ('superset.sql_lab', "superset.tasks", "superset.tasks.thumbnails", ) | ||
| imports = ( | ||
| "superset.sql_lab", | ||
| "superset.tasks.cache", |
There was a problem hiding this comment.
Module where the caching logic is defined.
| broker_url = f"redis://{env('REDIS_HOST')}:{env('REDIS_PORT')}/0" | ||
| imports = ('superset.sql_lab', "superset.tasks", "superset.tasks.thumbnails", ) |
There was a problem hiding this comment.
No value in importing superset.tasks as the __init__.py is empty.
superset/config.py
Outdated
| @@ -1498,13 +1490,7 @@ def EMAIL_HEADER_MUTATOR( # pylint: disable=invalid-name,unused-argument | |||
|
|
|||
| # Global async query config options. | |||
| # Requires GLOBAL_ASYNC_QUERIES feature flag to be enabled. | |||
| GLOBAL_ASYNC_QUERIES_REDIS_CONFIG = { | |||
There was a problem hiding this comment.
These were the same as the defaults and thus unncessary.
| CELERY_RESULT_BACKEND = f"redis://{REDIS_HOST}:{REDIS_PORT}/{REDIS_RESULTS_DB}" | ||
| CELERY_ANNOTATIONS = {"sql_lab.add": {"rate_limit": "10/s"}} | ||
| CONCURRENCY = 1 | ||
| broker_url = f"redis://{REDIS_HOST}:{REDIS_PORT}/{REDIS_CELERY_DB}" |
There was a problem hiding this comment.
These config settings were renamed and will be removed in Celery 6.0.
| CELERY_IMPORTS = ("superset.sql_lab", "superset.tasks.thumbnails") | ||
| CELERY_ANNOTATIONS = {"sql_lab.add": {"rate_limit": "10/s"}} | ||
| CONCURRENCY = 1 | ||
| broker_url = f"redis://{REDIS_HOST}:{REDIS_PORT}/{REDIS_CELERY_DB}" |
There was a problem hiding this comment.
These config settings were renamed and will be removed in Celery 6.0.
john-bodley
force-pushed
the
john-bodley--cleanup-celery-async-configs-and-documentation
branch
from
00ece63
to
9107e1e
Compare
john-bodley
requested review from
craig-rueda,
dpgaspar and
villebro
as code owners
john-bodley
changed the title
michael-s-molina
removed
the
review:checkpoint
dpgaspar
reviewed
Sep 22, 2023
| imports = ( | ||
| 'superset.sql_lab', | ||
| 'superset.tasks', | ||
| "superset.sql_lab', |
There was a problem hiding this comment.
"superset.sql_lab', -> "superset.sql_lab",
dpgaspar
approved these changes
Oct 31, 2023
…and-documentation
cccs-rc
pushed a commit
to CybercentreCanada/superset
that referenced
this pull request
Mar 6, 2024
mistercrunch
added
🏷️ bot
sfirke
pushed a commit
to sfirke/superset
that referenced
this pull request
Mar 22, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
SUMMARY
Whilst spelunking through the code looking at global async queries and Celery I realized that there was a number of inconsistencies with how the various
CeleryConfiget al. objects were defined (referencing legacy settings or outdated tasks). This PR cleans up said logic.BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF
TESTING INSTRUCTIONS
CI.
ADDITIONAL INFORMATION