-
-
Notifications
You must be signed in to change notification settings - Fork 3
Open
Description
Superset can use async queries to support long running queries.
This requires to add at least one new role, a worker.
This is a small spike where the rolegroups are used to provide the webserver (our standard superset), the worker and the beat which i found to not be required :
---
apiVersion: superset.stackable.tech/v1alpha1
kind: SupersetCluster
metadata:
name: superset
spec:
image:
productVersion: 4.1.4
clusterConfig:
credentialsSecret: superset-credentials
mapboxSecret: superset-mapbox-api-key
nodes:
envOverrides:
FILTER_STATE_CACHE_CONFIG: |-
{
'CACHE_TYPE': 'RedisCache',
'CACHE_DEFAULT_TIMEOUT': 86400,
'CACHE_KEY_PREFIX': 'superset_filter_cache',
'CACHE_REDIS_URL': 'redis://:{{ redisPassword }}@redis-superset-master:6379'
}
configOverrides:
superset_config.py:
EXPERIMENTAL_FILE_HEADER: |-
from flask_caching.backends.rediscache import RedisCache
EXPERIMENTAL_FILE_FOOTER: |-
RESULTS_BACKEND = RedisCache(host='redis-superset-master', port=6379, key_prefix='superset_results', password='{{ redisPassword }}')
class CeleryConfig(object):
broker_url = "redis://:{{ redisPassword }}@redis-superset-master:6379/0"
imports = (
"superset.sql_lab",
"superset.tasks.scheduler",
)
result_backend = "redis://:{{ redisPassword }}@redis-superset-master:6379/0"
worker_prefetch_multiplier = 10
task_acks_late = True
task_annotations = {
"sql_lab.get_sql_results": {
"rate_limit": "100/s",
},
}
CELERY_CONFIG = CeleryConfig
roleConfig:
listenerClass: external-stable
roleGroups:
webserver:
replicas: 1
worker:
replicas: 1
podOverrides:
spec:
containers:
- name: superset
args:
- |2
mkdir --parents /stackable/app/pythonpath
cp /stackable/config/* /stackable/app/pythonpath
cp /stackable/log_config/log_config.py /stackable/app/pythonpath
celery --app=superset.tasks.celery_app:app worker --pool=prefork -O fair -c 4
env:
- name: SUPERSET_CONFIG_PATH
value: /stackable/app/pythonpath/superset_config.py
beat:
replicas: 1
podOverrides:
spec:
containers:
- name: superset
args:
- |2
mkdir --parents /stackable/app/pythonpath
cp /stackable/config/* /stackable/app/pythonpath
cp /stackable/log_config/log_config.py /stackable/app/pythonpath
celery --app=superset.tasks.celery_app:app beat
env:
- name: SUPERSET_CONFIG_PATH
value: /stackable/app/pythonpath/superset_config.py
Metadata
Metadata
Assignees
Labels
No labels