Skip to content

Support Async Queries via Celery #698

@maltesander

Description

@maltesander

Superset can use async queries to support long running queries.

This requires to add at least one new role, a worker.

This is a small spike where the rolegroups are used to provide the webserver (our standard superset), the worker and the beat which i found to not be required :

---
apiVersion: superset.stackable.tech/v1alpha1
kind: SupersetCluster
metadata:
  name: superset
spec:
  image:
    productVersion: 4.1.4
  clusterConfig:
    credentialsSecret: superset-credentials
    mapboxSecret: superset-mapbox-api-key
  nodes:
    envOverrides:
      FILTER_STATE_CACHE_CONFIG: |-
        {
          'CACHE_TYPE': 'RedisCache',
          'CACHE_DEFAULT_TIMEOUT': 86400,
          'CACHE_KEY_PREFIX': 'superset_filter_cache',
          'CACHE_REDIS_URL': 'redis://:{{ redisPassword }}@redis-superset-master:6379'
        }
    configOverrides:
      superset_config.py:
        EXPERIMENTAL_FILE_HEADER: |-
          from flask_caching.backends.rediscache import RedisCache
        EXPERIMENTAL_FILE_FOOTER: |-
          RESULTS_BACKEND = RedisCache(host='redis-superset-master', port=6379, key_prefix='superset_results', password='{{ redisPassword }}')

          class CeleryConfig(object):
            broker_url = "redis://:{{ redisPassword }}@redis-superset-master:6379/0"
            imports = (
                "superset.sql_lab",
                "superset.tasks.scheduler",
            )
            result_backend = "redis://:{{ redisPassword }}@redis-superset-master:6379/0"
            worker_prefetch_multiplier = 10
            task_acks_late = True
            task_annotations = {
                "sql_lab.get_sql_results": {
                    "rate_limit": "100/s",
                },
            }

          CELERY_CONFIG = CeleryConfig
    roleConfig:
      listenerClass: external-stable
    roleGroups:
      webserver:
        replicas: 1
      worker:
        replicas: 1
        podOverrides:
          spec:
            containers:
              - name: superset
                args:
                  - |2
                    mkdir --parents /stackable/app/pythonpath
                    cp /stackable/config/* /stackable/app/pythonpath
                    cp /stackable/log_config/log_config.py /stackable/app/pythonpath
                    celery --app=superset.tasks.celery_app:app worker --pool=prefork -O fair -c 4
                env:
                  - name: SUPERSET_CONFIG_PATH
                    value: /stackable/app/pythonpath/superset_config.py
      beat:
        replicas: 1
        podOverrides:
          spec:
            containers:
              - name: superset
                args:
                  - |2
                    mkdir --parents /stackable/app/pythonpath
                    cp /stackable/config/* /stackable/app/pythonpath
                    cp /stackable/log_config/log_config.py /stackable/app/pythonpath
                    celery --app=superset.tasks.celery_app:app beat
                env:
                  - name: SUPERSET_CONFIG_PATH
                    value: /stackable/app/pythonpath/superset_config.py

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions