Failed to setup alerts and reports based pypi installation #29255

bionexit · 2024-06-14T14:23:49Z

Bug description

I have installed superset 4.0.1 with pypi way, seems no redis service activated while start the superset service with superset run -p 8088 --with-threads --reload --debugger -h 0.0.0.0.Currently i failed to setup alerts and reports following the document from the https://superset.apache.org/docs/configuration/alerts-reports

Please kindly help.

How to reproduce the bug

Following is my config.


FEATURE_FLAGS = {
    'ALERT_REPORTS' : True,
    'DASHBOARD_RBAC': True,
    'ENABLE_TEMPLATE_PROCESSING': True,
}


from celery.schedules import crontab

class CeleryConfig:
    beat_schedule = {
        "reports.scheduler": {
            "task": "reports.scheduler",
            "schedule": crontab(minute="*", hour="*"),
        },
        "reports.prune_log": {
            "task": "reports.prune_log",
            "schedule": crontab(minute=0, hour=0),
        },
    }
CELERY_CONFIG = CeleryConfig

SCREENSHOT_LOCATE_WAIT = 100
SCREENSHOT_LOAD_WAIT = 600

ALERT_REPORTS = True
EMAIL_NOTIFICATIONS = True
ALERT_REPORTS_NOTIFICATION_DRY_RUN = False

EMAIL_NOTIFICATIONS = False  # all the emails are sent using dryrun
SMTP_HOST = "SMTP_HOST"
SMTP_STARTTLS = True
SMTP_SSL = True
SMTP_USER = "SMTP_USER"
SMTP_PORT = 465
SMTP_PASSWORD = "SMTP_PASSWORD"
SMTP_MAIL_FROM = "SMTP_MAIL_FROM"
SMTP_SSL_SERVER_AUTH = False
#ENABLE_CHUNK_ENCODING = False


# WebDriver configuration
# If you use Firefox, you can stick with default values
# If you use Chrome, then add the following WEBDRIVER_TYPE and WEBDRIVER_OPTION_ARGS
WEBDRIVER_TYPE = "chrome"
WEBDRIVER_OPTION_ARGS = [
    "--force-device-scale-factor=2.0",
    "--high-dpi-support=2.0",
    "--headless",
    "--disable-gpu",
    "--disable-dev-shm-usage",
    "--no-sandbox",
    "--disable-setuid-sandbox",
    "--disable-extensions",
]

# This is for internal use, you can keep http
WEBDRIVER_BASEURL = "http://localhost:8088"
# This is the link sent to the recipient. Change to your domain, e.g. https://superset.mydomain.com
WEBDRIVER_BASEURL_USER_FRIENDLY = "http://IP:8088"



from superset.tasks.types import ExecutorType

THUMBNAIL_SELENIUM_USER = 'admin'
ALERT_REPORTS_EXECUTE_AS = [ExecutorType.SELENIUM]

pip list install
Package                   Version
------------------------- -----------
alembic                   1.13.1
amqp                      5.2.0
apache-superset           4.0.1
apispec                   6.6.1
apsw                      3.46.0.0
async-timeout             4.0.3
attrs                     23.2.0
Babel                     2.15.0
backoff                   2.2.1
bcrypt                    4.1.3
billiard                  4.2.0
blinker                   1.8.2
Bottleneck                1.3.8
Brotli                    1.1.0
cachelib                  0.9.0
cachetools                5.3.3
cattrs                    23.2.3
celery                    5.4.0
certifi                   2024.6.2
cffi                      1.16.0
charset-normalizer        3.3.2
click                     8.1.7
click-didyoumean          0.3.1
click-option-group        0.5.6
click-plugins             1.1.1
click-repl                0.3.0
colorama                  0.4.6
cron-descriptor           1.4.3
croniter                  2.0.5
cryptography              42.0.4
Deprecated                1.2.14
deprecation               2.1.0
dnspython                 2.6.1
email_validator           2.1.1
exceptiongroup            1.2.1
Flask                     2.3.3
Flask-AppBuilder          4.5.0
Flask-Babel               2.0.0
Flask-Caching             2.3.0
Flask-Compress            1.15
Flask-JWT-Extended        4.6.0
Flask-Limiter             3.7.0
Flask-Login               0.6.3
Flask-Migrate             3.1.0
Flask-Session             0.8.0
Flask-SQLAlchemy          2.5.1
flask-talisman            1.1.0
Flask-WTF                 1.2.1
func-timeout              4.3.5
geographiclib             2.0
geopy                     2.4.1
google-auth               2.30.0
greenlet                  3.0.3
gunicorn                  21.2.0
h11                       0.14.0
hashids                   1.3.1
holidays                  0.25
humanize                  4.9.0
idna                      3.7
importlib_metadata        7.1.0
importlib_resources       6.4.0
isodate                   0.6.1
itsdangerous              2.2.0
Jinja2                    3.1.4
jsonschema                4.22.0
jsonschema-specifications 2023.12.1
kombu                     5.3.7
korean-lunar-calendar     0.3.1
limits                    3.12.0
llvmlite                  0.42.0
Mako                      1.3.5
Markdown                  3.6
markdown-it-py            3.0.0
MarkupSafe                2.1.5
marshmallow               3.21.3
marshmallow-sqlalchemy    0.28.2
mdurl                     0.1.2
msgpack                   1.0.8
msgspec                   0.18.6
mysqlclient               2.2.4
nh3                       0.2.17
numba                     0.59.1
numexpr                   2.10.0
numpy                     1.23.5
ordered-set               4.1.0
outcome                   1.3.0.post0
packaging                 24.1
pandas                    2.0.3
paramiko                  3.4.0
parsedatetime             2.6
pgsanity                  0.2.9
pillow                    10.3.0
pip                       22.0.2
platformdirs              4.2.2
polyline                  2.0.2
prison                    0.2.1
prompt_toolkit            3.0.47
pyarrow                   14.0.2
pyasn1                    0.6.0
pyasn1_modules            0.4.0
pycparser                 2.22
pydoris                   1.0.5.3
Pygments                  2.18.0
PyJWT                     2.8.0
PyNaCl                    1.5.0
pyOpenSSL                 24.1.0
pyparsing                 3.1.2
PySocks                   1.7.1
python-dateutil           2.9.0.post0
python-dotenv             1.0.1
python-geohash            0.8.5
python-ldap               3.4.4
pytz                      2024.1
PyYAML                    6.0.1
redis                     4.6.0
referencing               0.35.1
requests                  2.32.3
requests-cache            1.2.0
rich                      13.7.1
rpds-py                   0.18.1
rsa                       4.9
selenium                  4.9.1
setuptools                59.6.0
shillelagh                1.2.19
shortid                   0.1.2
simplejson                3.19.2
six                       1.16.0
slack_sdk                 3.28.0
sniffio                   1.3.1
sortedcontainers          2.4.0
SQLAlchemy                1.4.52
SQLAlchemy-Utils          0.38.3
sqlglot                   23.17.0
sqlparse                  0.4.4
sshtunnel                 0.4.0
tabulate                  0.8.10
trio                      0.25.1
trio-websocket            0.11.1
typing_extensions         4.12.2
tzdata                    2024.1
url-normalize             1.4.3
urllib3                   2.2.1
vine                      5.1.0
wcwidth                   0.2.13
Werkzeug                  3.0.3
wrapt                     1.16.0
wsproto                   1.2.0
WTForms                   3.1.2
WTForms-JSON              0.3.5
XlsxWriter                3.0.9
zipp                      3.19.2
zstandard                 0.22.0

Screenshots/recordings

No response

Superset version

4.0.1

Python version

3.9

Node version

16

Browser

Chrome

Additional context

No response

Checklist

I have searched Superset docs and Slack and didn't find a solution to my problem.
I have searched the GitHub issue tracker and didn't find a similar bug report.
I have checked Superset's logs for errors and if I found a relevant Python stacktrace, I included it here as text in the "additional context" section.

The text was updated successfully, but these errors were encountered:

dosubot · 2024-06-14T14:29:40Z

To resolve the issue with setting up alerts and reports in Apache Superset, you need to ensure that the Redis service is properly configured and activated. Based on your provided configuration, it appears that the broker_url and result_backend for Celery are not set, which are essential for connecting to the Redis service.

Here is the updated configuration with the necessary Redis settings:

FEATURE_FLAGS = {
    'ALERT_REPORTS' : True,
    'DASHBOARD_RBAC': True,
    'ENABLE_TEMPLATE_PROCESSING': True,
}

from celery.schedules import crontab

REDIS_HOST = "localhost"  # Change this to your Redis host
REDIS_PORT = "6379"       # Change this to your Redis port if different
REDIS_CELERY_DB = "0"
REDIS_RESULTS_DB = "1"

class CeleryConfig:
    broker_url = f"redis://{REDIS_HOST}:{REDIS_PORT}/{REDIS_CELERY_DB}"
    result_backend = f"redis://{REDIS_HOST}:{REDIS_PORT}/{REDIS_RESULTS_DB}"
    beat_schedule = {
        "reports.scheduler": {
            "task": "reports.scheduler",
            "schedule": crontab(minute="*", hour="*"),
        },
        "reports.prune_log": {
            "task": "reports.prune_log",
            "schedule": crontab(minute=0, hour=0),
        },
    }
CELERY_CONFIG = CeleryConfig

SCREENSHOT_LOCATE_WAIT = 100
SCREENSHOT_LOAD_WAIT = 600

ALERT_REPORTS = True
EMAIL_NOTIFICATIONS = True
ALERT_REPORTS_NOTIFICATION_DRY_RUN = False

EMAIL_NOTIFICATIONS = False  # all the emails are sent using dryrun
SMTP_HOST = "SMTP_HOST"
SMTP_STARTTLS = True
SMTP_SSL = True
SMTP_USER = "SMTP_USER"
SMTP_PORT = 465
SMTP_PASSWORD = "SMTP_PASSWORD"
SMTP_MAIL_FROM = "SMTP_MAIL_FROM"
SMTP_SSL_SERVER_AUTH = False

WEBDRIVER_TYPE = "chrome"
WEBDRIVER_OPTION_ARGS = [
    "--force-device-scale-factor=2.0",
    "--high-dpi-support=2.0",
    "--headless",
    "--disable-gpu",
    "--disable-dev-shm-usage",
    "--no-sandbox",
    "--disable-setuid-sandbox",
    "--disable-extensions",
]

WEBDRIVER_BASEURL = "http://localhost:8088"
WEBDRIVER_BASEURL_USER_FRIENDLY = "http://IP:8088"

from superset.tasks.types import ExecutorType

THUMBNAIL_SELENIUM_USER = 'admin'
ALERT_REPORTS_EXECUTE_AS = [ExecutorType.SELENIUM]

Make sure that the Redis service is running and accessible at the specified REDIS_HOST and REDIS_PORT. This should enable the Celery workers to connect to Redis and process the tasks for alerts and reports. Additionally, ensure that the REDIS_CELERY_DB and REDIS_RESULTS_DB are correctly set to match your Redis configuration ^[1]^[2].

To continue talking to Dosu, mention @dosu.

bionexit · 2024-06-14T14:42:12Z

I installed the redis manually, and update the config as following, but NO LUCK

from celery.schedules import crontab

REDIS_HOST = "localhost"  # Change this to your Redis host
REDIS_PORT = "6379"       # Change this to your Redis port if different
REDIS_CELERY_DB = "0"
REDIS_RESULTS_DB = "1"

class CeleryConfig:
    broker_url = f"redis://{REDIS_HOST}:{REDIS_PORT}/0"
    imports = (
        "superset.sql_lab",
        "superset.tasks.scheduler",
    )
    result_backend = f"redis://{REDIS_HOST}:{REDIS_PORT}/0"
    worker_prefetch_multiplier = 10
    task_acks_late = True
    task_annotations = {
        "sql_lab.get_sql_results": {
            "rate_limit": "100/s",
        },
    }
    beat_schedule = {
        "reports.scheduler": {
            "task": "reports.scheduler",
            "schedule": crontab(minute="*", hour="*"),
        },
        "reports.prune_log": {
            "task": "reports.prune_log",
            "schedule": crontab(minute=0, hour=0),
        },
    }

In report log, seems no error message, but no email was sent or received.

dosubot · 2024-06-14T14:52:26Z

It looks like you have correctly configured the Redis settings for Celery. However, there are a few additional configurations and checks you should ensure are in place:

Enable the ALERT_REPORTS Feature Flag:
Ensure that the ALERT_REPORTS feature flag is enabled in your configuration:

FEATURE_FLAGS = {
    'ALERT_REPORTS': True,
    'DASHBOARD_RBAC': True,
    'ENABLE_TEMPLATE_PROCESSING': True,
}

Set ALERT_REPORTS_NOTIFICATION_DRY_RUN to False:
Make sure this setting is correctly set to False to enable actual notifications:
```
ALERT_REPORTS_NOTIFICATION_DRY_RUN = False
```

Configure the WebDriver:
Ensure that the WebDriver is correctly configured for taking screenshots:

WEBDRIVER_TYPE = "chrome"
WEBDRIVER_OPTION_ARGS = [
    "--force-device-scale-factor=2.0",
    "--high-dpi-support=2.0",
    "--headless",
    "--disable-gpu",
    "--disable-dev-shm-usage",
    "--no-sandbox",
    "--disable-setuid-sandbox",
    "--disable-extensions",
]
WEBDRIVER_BASEURL = "http://localhost:8088"
WEBDRIVER_BASEURL_USER_FRIENDLY = "http://IP:8088"

Set the User for Rendering Dashboards:
Ensure that the user for rendering dashboards is set:

from superset.tasks.types import ExecutorType

THUMBNAIL_SELENIUM_USER = 'admin'
ALERT_REPORTS_EXECUTE_AS = [ExecutorType.SELENIUM]

Check Celery Workers:
Ensure that Celery workers are running and can connect to Redis. You can start the Celery worker with:
```
celery -A superset.tasks.celery_app:app worker --loglevel=info
```
Check Celery Beat:
Ensure that Celery beat is running to schedule tasks:
```
celery -A superset.tasks.celery_app:app beat --loglevel=info
```
Verify Redis Connection:
Ensure that Redis is running and accessible. You can test the connection using a Redis client:
```
redis-cli -h localhost -p 6379 ping
```

SMTP Configuration:
Ensure that your SMTP settings are correctly configured for email notifications:

SMTP_HOST = "SMTP_HOST"
SMTP_STARTTLS = True
SMTP_SSL = True
SMTP_USER = "SMTP_USER"
SMTP_PORT = 465
SMTP_PASSWORD = "SMTP_PASSWORD"
SMTP_MAIL_FROM = "SMTP_MAIL_FROM"
SMTP_SSL_SERVER_AUTH = False

By ensuring these configurations and checks are in place, you should be able to set up alerts and reports in Apache Superset 4.0.1. If the issue persists, check the Superset logs for any errors or warnings that might provide more insights ^[1]^[2].

To continue talking to Dosu, mention @dosu.

bionexit · 2024-06-14T19:18:25Z

after config as above, the log report STARTTLS extension not supported by server.

sfirke · 2024-06-18T18:16:55Z

If you followed the PyPi installation, have you set up a Celery instance?

bionexit · 2024-06-19T01:49:19Z

If you followed the PyPi installation, have you set up a Celery instance?

YES, celery is running

Currently, the log shows failed as STARTTLS extension not supported by server.

And cerely is configured as

from celery.schedules import crontab

REDIS_HOST = "localhost"  # Change this to your Redis host
REDIS_PORT = "6379"       # Change this to your Redis port if different
REDIS_CELERY_DB = "0"
REDIS_RESULTS_DB = "1"

class CeleryConfig:
    broker_url = f"redis://{REDIS_HOST}:{REDIS_PORT}/0"
    imports = (
        "superset.sql_lab",
        "superset.tasks.scheduler",
    )
    result_backend = f"redis://{REDIS_HOST}:{REDIS_PORT}/0"
    worker_prefetch_multiplier = 10
    task_acks_late = True
    task_annotations = {
        "sql_lab.get_sql_results": {
            "rate_limit": "100/s",
        },
    }
    beat_schedule = {
        "reports.scheduler": {
            "task": "reports.scheduler",
            "schedule": crontab(minute="*", hour="*"),
        },
        "reports.prune_log": {
            "task": "reports.prune_log",
            "schedule": crontab(minute=0, hour=0),
        },
    }
CELERY_CONFIG = CeleryConfig


from flask_caching.backends.rediscache import RedisCache
RESULTS_BACKEND = RedisCache(
    host=REDIS_HOST, port=REDIS_PORT, key_prefix='superset_results')

dosubot bot added the alert-reports Namespace | Anything related to the Alert & Reports feature label Jun 14, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Failed to setup alerts and reports based pypi installation #29255

Failed to setup alerts and reports based pypi installation #29255

bionexit commented Jun 14, 2024 •

edited

Loading

dosubot bot commented Jun 14, 2024

bionexit commented Jun 14, 2024 •

edited

Loading

dosubot bot commented Jun 14, 2024

bionexit commented Jun 14, 2024

sfirke commented Jun 18, 2024

bionexit commented Jun 19, 2024 •

edited

Loading

Failed to setup alerts and reports based pypi installation #29255

Failed to setup alerts and reports based pypi installation #29255

Comments

bionexit commented Jun 14, 2024 • edited Loading

Bug description

How to reproduce the bug

Screenshots/recordings

Superset version

Python version

Node version

Browser

Additional context

Checklist

dosubot bot commented Jun 14, 2024

bionexit commented Jun 14, 2024 • edited Loading

dosubot bot commented Jun 14, 2024

bionexit commented Jun 14, 2024

sfirke commented Jun 18, 2024

bionexit commented Jun 19, 2024 • edited Loading

bionexit commented Jun 14, 2024 •

edited

Loading

bionexit commented Jun 14, 2024 •

edited

Loading

bionexit commented Jun 19, 2024 •

edited

Loading