Use sane Celery defaults to prevent tasks from being delayed #3267
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Coverage remained the same at 69.301% when pulling 5896dd3726661b8c52f17d344d8997420f4b0329 on saguziel:aguziel-celery-fix-defaults into 327c052 on apache:master. |
mistercrunch
requested changes
Aug 9, 2017
superset/cli.py
Outdated
| @@ -194,7 +194,7 @@ def worker(workers): | |||
| celery_app.conf.update( | |||
| CELERYD_CONCURRENCY=config.get("SUPERSET_CELERY_WORKERS")) | |||
|
|
|||
| worker = celery_worker.worker(app=celery_app) | |||
| worker = app.Worker(optimization='fair') | |||
There was a problem hiding this comment.
app in this context is the Flask app, not celery_app
saguziel
force-pushed
the
aguziel-celery-fix-defaults
branch
from
5896dd3
to
8e90435
Compare
mistercrunch
approved these changes
Aug 10, 2017
graceguo-supercat
pushed a commit
to graceguo-supercat/superset
that referenced
this pull request
Aug 10, 2017
mistercrunch
pushed a commit
that referenced
this pull request
Aug 11, 2017
* split reducer logic for ExploreViewContainer * fix saveModal component and unit tests * revert changes in SaveModal_spec. will make another commit just to improve test coverage for SaveModal component. * improve test coverage for explore view components: - SaveModal component - URLShortLinkButton * remove comment-out code * [bugfix] wrong 'Cant have overlap between Series and Breakdowns' (#3254) * [explore] make edit datasource a basic link (#3244) * Relying on FAB for font-awesome.min.css (#3261) * Modernize SQLA pessimistic handling (#3256) Looks like SQLAlchemy has redefined the best practice around pessimistic connection handling. * [webpack] break CSS and JS files while webpackin' (#3262) * [webpack] break CSS and JS files while webpackin' * cleaning up some templates * Fix pylint issue * import logging (#3264) * [bugfix] preserve order in groupby (#3268) Recently in 4c3313b I introduced an issue where the order of groupby fields might change. This addresses this issue and will preserve ordering. * Explicitly add Flask as dependancy (#3252) * Use sane Celery defaults to prevent tasks from being delayed (#3267) * Improve the chart type of Visualize in sqllab (#3241) * Improve the chart type of Visualize in sqllab & Add some css & Fix the link address in the navbar * add vizTypes filter * Set default ports Druid (#3266) For Druid set the default port for the broker and coordinator. * [explore] Split large reducer logic in ExploreViewContainer (#3088) * split reducer logic for ExploreViewContainer * fix saveModal component and unit tests * revert changes in SaveModal_spec. will make another commit just to improve test coverage for SaveModal component. * remove comment-out code * fix merge confilicts
mistercrunch
added
🏷️ bot
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
@mistercrunch
Using these three settings (of which two are provided in the config to show it's recommended), will make Celery scheduling "fair", which reduces latency which is what is wanted when we have long running tasks (note that the defaults prioritize throughput in the case with high volumes of short-running tasks)
Still need to test this