[Report] Reports not working with OIDC auth #14330

dusatvoj · 2021-04-24T21:16:36Z

Can't send Report emails with error Report Schedule sellenium user not found.

Expected results

Can send Report emails

Actual results

Almost nothing in logs but Report Schedule sellenium user not found error in reports action log.

Screenshots

How to reproduce the bug

Setup smth like this

SCREENSHOT_LOCATE_WAIT = 100
SCREENSHOT_LOAD_WAIT = 600

ENABLE_ALERTS = True
FEATURE_FLAGS = {
    'ALERT_REPORTS': True
}

WEBDRIVER_TYPE = "chrome"
#WEBDRIVER_OPTION_ARGS = [
    "--force-device-scale-factor=2.0",
    "--high-dpi-support=2.0",
    "--headless",
    "--disable-gpu",
    "--disable-dev-shm-usage",
    "--no-sandbox",
    "--disable-setuid-sandbox",
    "--disable-extensions",
]

# This is for internal use, you can keep http
WEBDRIVER_BASEURL="http://localhost:8088"
# This is the link sent to the recipient, change to your domain eg. https://superset.mydomain.com
WEBDRIVER_BASEURL_USER_FRIENDLY="https://<BASE_URL>"

in config but no emails sent (or tried to send).

Environment

(please complete the following information):

superset version: Superset 1.1.0
python version: Python 3.8.7
node.js version: v12.21.0

Checklist

Make sure to follow these steps before submitting your issue - thank you!

I have checked the superset logs for python stacktraces and included it here as text if there are any.
I have reproduced the issue with at least the latest released version of superset.
I have checked the issue tracker for the same issue and I haven't found one similar.

Additional context

Apr 24 21:00:00 node6 celery[1401583]: Report state: Report Schedule sellenium user not found
Apr 24 21:00:00 node6 celery[1401583]: [2021-04-24 21:00:00,239: INFO/ForkPoolWorker-1] Report state: Report Schedule sellenium user not found

(every hour)

The text was updated successfully, but these errors were encountered:

junlincc · 2021-04-28T05:26:11Z

cc @nytai

willbarrett · 2021-04-28T17:15:05Z

cc @dpgaspar

dpgaspar · 2021-04-28T17:19:08Z

@dusatvoj

what's the value for THUMBNAIL_SELENIUM_USER? and make sure that user exists on the database

dusatvoj · 2021-04-28T20:07:30Z

What? Can't see this variable in the docs 🤔 🙃

dpgaspar · 2021-04-29T09:02:53Z

Just double checked it is there: https://superset.apache.org/docs/installation/alerts-reports

Were you able to make it work?

dusatvoj · 2021-04-29T09:25:54Z

It's written under "Old Reports feature (version 0.38 and below)" but it's still relevant? Weird

Anyway: I've tried to set it up BUT There's another error Report Schedule execution failed when generating a screenshot.

Apr 29 09:01:45 node6 celery[1496765]: Selenium timed out requesting url http://localhost:8088/superset/dashboard/7/
Apr 29 09:01:45 node6 celery[1496765]: [2021-04-29 09:01:45,886: ERROR/ForkPoolWorker-1] Selenium timed out requesting url http://localhost:8088/superset/dashboard/7/
Apr 29 09:01:46 node6 celery[1496765]: Report state: Report Schedule execution failed when generating a screenshot.
Apr 29 09:01:46 node6 celery[1496765]: [2021-04-29 09:01:46,060: INFO/ForkPoolWorker-1] Report state: Report Schedule execution failed when generating a screenshot.

Maybe It's caused by the fact we are using OIDC login (https://stackoverflow.com/questions/54010314/using-keycloakopenid-connect-with-apache-superset ... this solution bcs solution in your docs were not working too - same as this feature)

# curl http://localhost:8088/superset/dashboard/7/
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
<title>Redirecting...</title>
<h1>Redirecting...</h1>
<p>You should be redirected automatically to target URL: <a href="/login/?next=http%3A%2F%2Flocalhost%3A8088%2Fsuperset%2Fdashboard%2F7%2F">/login/?next=http%3A%2F%2Flocalhost%3A8088%2Fsuperset%2Fdashboard%2F7%2F</a>.  If not click the link.

dpgaspar · 2021-04-29T09:29:55Z

@dusatvoj good point regarding the docs, We'll fix it.

Are you using docker? have you executed the curl from inside the container?

dusatvoj · 2021-04-29T09:31:00Z

No, I have dedicated VM for superset ... and I've executed curl in the VM

dpgaspar · 2021-04-29T14:42:43Z

it should not be OIDC since your still using session cookies. that's what the worker is generating for THUMBNAIL_SELENIUM_USER. One more question/confirmation, I'm assuming that your VM has the worker and the web app executing on the same instance?

dusatvoj · 2021-04-29T15:05:09Z

yes, on the same machine

dpgaspar · 2021-04-29T15:12:15Z

ok, do you have geckodriver installed and firefox or chromedriver and chrome?
Also before the worker timeout do you have any frontend logs, if it's a login failure you should have an HTTP 302 for http://localhost:8088/superset/dashboard/7/

dusatvoj · 2021-05-11T12:45:53Z

I have installed chromedriver and firefox-esr package on Debian 10. I've tried both engines with no luck.
No, I've found a log in journal...

May 11 12:00:00 node6 celery[1817263]: [2021-05-11 12:00:00,282: INFO/ForkPoolWorker-1] Init selenium driver
May 11 12:00:00 node6 celery[1817263]: Failed at generating thumbnail [Errno 13] Permission denied: 'geckodriver.log'
May 11 12:00:00 node6 celery[1817263]: [2021-05-11 12:00:00,282: ERROR/ForkPoolWorker-1] Failed at generating thumbnail [Errno 13] Permission denied: 'geckodriver.log'
May 11 12:00:00 node6 celery[1817263]: Report state: Report Schedule execution failed when generating a screenshot.
May 11 12:00:00 node6 celery[1817263]: [2021-05-11 12:00:00,314: INFO/ForkPoolWorker-1] Report state: Report Schedule execution failed when generating a screenshot.

(I've switched experimentally to firefox-esr)

dpgaspar · 2021-05-11T13:56:15Z

Failed at generating thumbnail [Errno 13] Permission denied: 'geckodriver.log'

dusatvoj · 2021-05-11T14:25:30Z

What should be configured in superset_config.py or another config? I want to log everything to stdout / stderr (to journal).

dusatvoj · 2021-05-18T19:18:27Z

@dpgaspar I've found a solution. There were issue with bad WorkingDirectory but now there's another issue with generating screenshots 😓 .
There's timeouting selenium with firefox and chrome too. I don't know how to debug it but I think there's issue with our OAuth2 solution (https://stackoverflow.com/questions/54010314/using-keycloakopenid-connect-with-apache-superset) ... which is the only working OAuth2 SSO solution I've found. 🙃

dusatvoj · 2021-05-18T19:22:29Z

There's related log from celery ...

May 18 19:02:09 node6 celery[5345]: Selenium timed out requesting url http://localhost:8088/superset/dashboard/7/
May 18 19:02:09 node6 celery[5345]: [2021-05-18 19:02:09,007: ERROR/ForkPoolWorker-1] Selenium timed out requesting url http://localhost:8088/superset/dashboard/7/
May 18 19:02:09 node6 celery[5345]: Report state: Report Schedule execution failed when generating a screenshot.
May 18 19:02:09 node6 celery[5345]: [2021-05-18 19:02:09,596: INFO/ForkPoolWorker-1] Report state: Report Schedule execution failed when generating a screenshot.

dpgaspar · 2021-05-19T09:10:43Z

@dusatvoj if you login to your server/container are you able to launch firefox and geckodriver?

dusatvoj · 2021-05-19T11:09:07Z

superset@node6:~$ firefox
Error: no DISPLAY environment variable specified
superset@node6:~$ geckodriver
1621422469338	geckodriver	INFO	Listening on 127.0.0.1:4444
^C
superset@node6:~$

It looks that geckodriver works fine 🤔

dpgaspar · 2021-05-19T16:12:14Z

@dusatvoj

I'm troubleshooting firefox right now, it seems that everything works just fine with root user but when switching to a lower user eg: superset you get: Failed at generating thumbnail [Errno 13] Permission denied: 'geckodriver.log'

Solved that one with:

WEBDRIVER_CONFIGURATION={
    "service_log_path": "/dev/null"
}

did you do this to solve one of your first problems?

regarding the timeout to http://localhost:8088/superset/dashboard/7/ do you get to see any logs on the frontend/app?

dusatvoj · 2021-05-19T16:54:18Z

I've solved permissions issue by changing WorkingDirectory(in systemd service file) to $HOME of the user but /dev/null looks good too :D
I've changed URL to fronted (apache proxy) and it looks it's stuck on login page at keycloak ....

Headless browser is redirected to /login ...

[19/May/2021:18:46:01 +0200] "GET /login/ HTTP/1.1" 302 6863 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/90.0.4430.212 Safari/537.36"

and after that it's redirected to Keycloak instance (OIDC provider) ...

[19/May/2021:18:46:01 +0200] "GET /auth/realms/<REALM>/openid-connect/auth?...

... and again and again ... (/login -> keycloak instance -> nothing) ... but this type of authentication is the only working solution for user friendly OIDC login ... and we want to use reports too 😕 😕 😕

dpgaspar · 2021-05-19T17:40:30Z

@dusatvoj

I see so it may be because of: https://github.com/apache/superset/blob/master/superset/utils/machine_auth.py#L53
by default the test request is /login if it redirects immediately to openid then that could be your problem. You can write your own MachineAuthProvider and set it on MACHINE_AUTH_PROVIDER_CLASS config key.

Asturias-sam · 2021-06-28T13:18:00Z

@dpgaspar
I have a similar problem

Selenium timed out requesting url http://<IP>/superset/dashboard/9/
Report Schedule execution failed when generating a screenshot.

I am running it in the virtual environment without docker we have LDAP enabled , i have the property THUMBNAIL_SELENIUM_USER enables any suggestion ?

CountRedClaw · 2021-07-13T14:07:31Z

@dpgaspar
Is it a good WA to replace
driver.get(headless_url("/login/"))
with
driver.get(headless_url("/nonexistent_url"))
in order to avoid redirection to the identity provider and therefore to set cookies to proper domain?

dusatvoj · 2021-07-22T18:06:47Z

@CountRedClaw good WA but not enough 😕 I still cant generate because of the cookie 🤔 Do you have an idea how to set it properly?

heul · 2021-07-23T13:55:22Z

@dpgaspar , thanks for the hint. It might even suffice to overwrite the webdriver auth function (defined by WEBDRIVER_AUTH_FUNC). We use Azure as OAuth provider and any /login request is redirected there and then selenium times out. Works perfectly, when we added this to our superset_config.py:

from superset.utils.urls import headless_url
from superset.utils.machine_auth import MachineAuthProvider

def auth_driver(driver, user):
    # Setting cookies requires doing a request first, but /login is redirected to oauth provider, and stuck there.
    driver.get(headless_url("/doesnotexist"))

    cookies = MachineAuthProvider.get_auth_cookies(user)

    for cookie_name, cookie_val in cookies.items():
        driver.add_cookie(dict(name=cookie_name, value=cookie_val))

    return driver

WEBDRIVER_AUTH_FUNC = auth_driver

jensenity · 2021-12-26T14:34:59Z

what's the solution for this?

dusatvoj · 2021-12-26T18:31:37Z

There's just a workaround with nonexistent url

vivekpradhan · 2021-12-28T10:44:00Z

I am facing the same issue. Can someone tell me what to add to superset config to enable TRACE level logging for the geckodriver so that I can see why I am getting this error:

[2021-12-28 10:10:42,315: WARNING/ForkPoolWorker-1] Selenium timed out requesting url https://example.com/superset/dashboard/81/?standalone=3
Traceback (most recent call last):
  File "/app/superset/utils/webdriver.py", line 125, in get_screenshot
    EC.presence_of_element_located((By.CLASS_NAME, element_name))
  File "/usr/local/lib/python3.7/site-packages/selenium/webdriver/support/wait.py", line 80, in until
    raise TimeoutException(message, screen, stacktrace)
selenium.common.exceptions.TimeoutException: Message: 

Report state: Report Schedule execution failed when generating a screenshot.
[2021-12-28 10:10:46,115: INFO/ForkPoolWorker-1] Report state: Report Schedule execution failed when generating a screenshot.

vivekpradhan · 2021-12-28T15:10:39Z

I found the logs listing HTTP requests in the main superset app container. The chart is not loading with 401 status.

127.0.0.1 - - [28/Dec/2021:15:09:09 +0000] "GET /static/assets/5053927271b77cc901e5.chunk.js HTTP/1.1" 200 1258 "http://localhost:8080/superset/dashboard/81/?standalone=3" "Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Firefox/78.0"
127.0.0.1 - - [28/Dec/2021:15:09:09 +0000] "GET /api/v1/dashboard/81/datasets HTTP/1.1" 401 39 "http://localhost:8080/superset/dashboard/81/?standalone=3" "Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Firefox/78.0"
127.0.0.1 - - [28/Dec/2021:15:09:09 +0000] "GET /api/v1/dashboard/81/charts HTTP/1.1" 401 39 "http://localhost:8080/superset/dashboard/81/?standalone=3" "Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Firefox/78.0"
127.0.0.1 - - [28/Dec/2021:15:09:09 +0000] "GET /api/v1/dashboard/81 HTTP/1.1" 401 39 "http://localhost:8080/superset/dashboard/81/?standalone=3" "Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Firefox/78.0"

The default authenticate_webdriver function is not working I also tried something similar what @heul suggested. Can you tell me what the WEBDRIVER_AUTH_FUNC should be to help my user login to superset. Our superset uses google oauth.

dusatvoj · 2021-12-28T22:53:29Z

I'm using self-hosted keycloak and the workaround from @heul 's comment. It works as expected.

vivekpradhan · 2022-01-03T12:28:46Z

So @heul 's solution now works for me. The problem was I did not set THUMBNAIL_SELENIUM_USER. The default "Admin" was being used, but I had deactivated that account.

dusatvoj added the #bug Bug report label Apr 24, 2021

junlincc changed the title ~~Report Schedule sellenium user not found~~ [Report] Schedule sellenium user not found Apr 28, 2021

junlincc added validation:required A committer should validate the issue and removed #bug Bug report labels Apr 28, 2021

dpgaspar added install:config Installation - Configuration settings question & help wanted Use Github discussions instead and removed validation:required A committer should validate the issue labels May 11, 2021

dusatvoj changed the title ~~[Report] Schedule sellenium user not found~~ [Report] Reports not working with OIDC auth May 19, 2021

dpgaspar mentioned this issue May 20, 2021

fix(docker): superset permissions and firefox config #14736

Merged

8 tasks

riazarbi mentioned this issue Sep 23, 2021

[chart]"Unexpected error" in Dashboard thumbnails (Superset 1.0.1) #13490

Closed

3 tasks

OlafKocanda mentioned this issue Jan 17, 2022

Sending Email Report Selenium Driver | 3 different errors #18066

Closed

3 tasks

apache locked and limited conversation to collaborators Feb 2, 2022

geido converted this issue into discussion #18289 Feb 2, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

This issue was moved to a discussion.

[Report] Reports not working with OIDC auth #14330

[Report] Reports not working with OIDC auth #14330

dusatvoj commented Apr 24, 2021

junlincc commented Apr 28, 2021

willbarrett commented Apr 28, 2021

dpgaspar commented Apr 28, 2021

dusatvoj commented Apr 28, 2021

dpgaspar commented Apr 29, 2021

dusatvoj commented Apr 29, 2021

dpgaspar commented Apr 29, 2021

dusatvoj commented Apr 29, 2021 •

edited

dpgaspar commented Apr 29, 2021

dusatvoj commented Apr 29, 2021

dpgaspar commented Apr 29, 2021

dusatvoj commented May 11, 2021

dpgaspar commented May 11, 2021

dusatvoj commented May 11, 2021

dusatvoj commented May 18, 2021

dusatvoj commented May 18, 2021

dpgaspar commented May 19, 2021

dusatvoj commented May 19, 2021

dpgaspar commented May 19, 2021 •

edited

dusatvoj commented May 19, 2021

dpgaspar commented May 19, 2021

Asturias-sam commented Jun 28, 2021 •

edited

CountRedClaw commented Jul 13, 2021

dusatvoj commented Jul 22, 2021

heul commented Jul 23, 2021

jensenity commented Dec 26, 2021

dusatvoj commented Dec 26, 2021 •

edited

vivekpradhan commented Dec 28, 2021 •

edited

vivekpradhan commented Dec 28, 2021

dusatvoj commented Dec 28, 2021

vivekpradhan commented Jan 3, 2022

This issue was moved to a discussion.

This issue was moved to a discussion.

[Report] Reports not working with OIDC auth #14330

[Report] Reports not working with OIDC auth #14330

Comments

dusatvoj commented Apr 24, 2021

Expected results

Actual results

Screenshots

How to reproduce the bug

Environment

Checklist

Additional context

junlincc commented Apr 28, 2021

willbarrett commented Apr 28, 2021

dpgaspar commented Apr 28, 2021

dusatvoj commented Apr 28, 2021

dpgaspar commented Apr 29, 2021

dusatvoj commented Apr 29, 2021

dpgaspar commented Apr 29, 2021

dusatvoj commented Apr 29, 2021 • edited

dpgaspar commented Apr 29, 2021

dusatvoj commented Apr 29, 2021

dpgaspar commented Apr 29, 2021

dusatvoj commented May 11, 2021

dpgaspar commented May 11, 2021

dusatvoj commented May 11, 2021

dusatvoj commented May 18, 2021

dusatvoj commented May 18, 2021

dpgaspar commented May 19, 2021

dusatvoj commented May 19, 2021

dpgaspar commented May 19, 2021 • edited

dusatvoj commented May 19, 2021

dpgaspar commented May 19, 2021

Asturias-sam commented Jun 28, 2021 • edited

CountRedClaw commented Jul 13, 2021

dusatvoj commented Jul 22, 2021

heul commented Jul 23, 2021

jensenity commented Dec 26, 2021

dusatvoj commented Dec 26, 2021 • edited

vivekpradhan commented Dec 28, 2021 • edited

vivekpradhan commented Dec 28, 2021

dusatvoj commented Dec 28, 2021

vivekpradhan commented Jan 3, 2022

This issue was moved to a discussion.

dusatvoj commented Apr 29, 2021 •

edited

dpgaspar commented May 19, 2021 •

edited

Asturias-sam commented Jun 28, 2021 •

edited

dusatvoj commented Dec 26, 2021 •

edited

vivekpradhan commented Dec 28, 2021 •

edited