502 Bad Gateway

[djreducer]

Could be my problem is covered under a different issue(#10), but the fix they mention doesn't appear to be available to me.

I'm running headless Chrome from a RHEL 7.6 container. At this point, just trying to get to the login page of a device. Device has an invalid certificate.

Here's the logging output:

/opt/app-root/lib/python3.6/site-packages/urllib3/connectionpool.py:851: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings

InsecureRequestWarning)

starting browser with headless-true

DEBUG:selenium.webdriver.remote.remote_connection:POST http://127.0.0.1:53308/session/b5e0dba596d67a133b740d2fc5541579/url {"url": "http://10.10.10.1", "sessionId": "b5e0dba596d67a133b740d2fc5541579"}

INFO:seleniumwire.proxy.handler:Capturing request: http://10.10.10.1/

INFO:seleniumwire.proxy.handler:Capturing response: http://10.10.10.1/ 302 Found

DEBUG:seleniumwire.proxy.handler:http://10.10.10.1/ 302

DEBUG:seleniumwire.proxy.handler:10.10.10.1:443 200

INFO:seleniumwire.proxy.handler:Capturing request: https://10.10.10.1/

DEBUG:seleniumwire.proxy.handler:code 502, message Bad Gateway

Traceback (most recent call last):

File "/opt/app-root/lib/python3.6/site-packages/seleniumwire/proxy/proxy2.py", line 108, in do_GET

conn.request(self.command, path, req_body, dict(req.headers))

File "/opt/rh/rh-python36/root/usr/lib64/python3.6/http/client.py", line 1239, in request

self._send_request(method, url, body, headers, encode_chunked)

File "/opt/rh/rh-python36/root/usr/lib64/python3.6/http/client.py", line 1285, in _send_request

self.endheaders(body, encode_chunked=encode_chunked)

File "/opt/rh/rh-python36/root/usr/lib64/python3.6/http/client.py", line 1234, in endheaders

self._send_output(message_body, encode_chunked=encode_chunked)

File "/opt/rh/rh-python36/root/usr/lib64/python3.6/http/client.py", line 1026, in _send_output

self.send(msg)

File "/opt/rh/rh-python36/root/usr/lib64/python3.6/http/client.py", line 964, in send

self.connect()

File "/opt/rh/rh-python36/root/usr/lib64/python3.6/http/client.py", line 1400, in connect

server_hostname=server_hostname)

File "/opt/rh/rh-python36/root/usr/lib64/python3.6/ssl.py", line 407, in wrap_socket

_context=self, _session=session)

File "/opt/rh/rh-python36/root/usr/lib64/python3.6/ssl.py", line 814, in init

self.do_handshake()

File "/opt/rh/rh-python36/root/usr/lib64/python3.6/ssl.py", line 1068, in do_handshake

self._sslobj.do_handshake()

File "/opt/rh/rh-python36/root/usr/lib64/python3.6/ssl.py", line 689, in do_handshake

self._sslobj.do_handshake()

ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:777)

DEBUG:seleniumwire.proxy.handler:https://10.10.10.1/ 502

DEBUG:urllib3.connectionpool:http://127.0.0.1:53308 "POST /session/b5e0dba596d67a133b740d2fc5541579/url HTTP/1.1" 200 72

DEBUG:selenium.webdriver.remote.remote_connection:Finished Request

Terminating browser session.

INFO:seleniumwire.proxy.client:Destroying proxy

DEBUG:seleniumwire.proxy.storage:Cleaning up /opt/app-root/src/.seleniumwire/storage-55d28002-30cf-48be-a596-ed94e241aa00

DEBUG:selenium.webdriver.remote.remote_connection:DELETE http://127.0.0.1:53308/session/b5e0dba596d67a133b740d2fc5541579 {"sessionId": "b5e0dba596d67a133b740d2fc5541579"}

DEBUG:urllib3.connectionpool:http://127.0.0.1:53308 "DELETE /session/b5e0dba596d67a133b740d2fc5541579 HTTP/1.1" 200 72

DEBUG:selenium.webdriver.remote.remote_connection:Finished Request

Works fine when importing vanilla Selenium.

[wkeeling]

Could you post your the code that creates webdriver.Chrome() in your test (and all the options its passing)?

[clovis1122]

Also having this issue. Making a POST call to a Django app behind a reverse proxy. Around 7-10 seconds after the call, it seems as if the call was canceled and is reported as a 502 BAD GATEWAY in the selenium code. However, Nginx reports it as an HTTP 499: CLIENT CLOSED REQUEST and Django emits this message: Ignoring EPIPE.

Here's the code that creates the webdriver

    def initialize_driver(driver_path):
        d = DesiredCapabilities.CHROME
        d['goog:loggingPrefs'] = { 'browser':'ALL' }

        options = ChromeOption()
        options.add_argument("--headless")
        options.add_argument("--no-sandbox")
        options.add_argument("--disable-dev-shm-usage")
        options.add_argument("--disable-extensions")
        options.add_argument("--disable-gpu")
        options.add_argument("--disable-setuid-sandbox")
        options.add_argument("--ignore_ssl")
        options.add_argument("--dns-prefetch-disable")
        options.add_argument("window-size=1800,1080")
        return webdriver.Chrome(executable_path=driver_path, chrome_options=options, desired_capabilities=d)

[djreducer]

Here is my code. For some of my use cases, I've had a need to use pyvirtualdisplay. In those cases, headless is false. For the current issue, headless is true.

def initiateBrowser(headless,hostname):
    options = webdriver.ChromeOptions()
    #Options for headless use
    capabilities = DesiredCapabilities.CHROME.copy()
    options.add_argument('--no-sandbox')
    options.add_argument('--allow-running-insecure-content')
    options.add_argument('--ignore-certificate-errors')
    options.add_argument('--disable-gpu')
    options.add_argument('--window-size=1920x1080')
    capabilities['acceptSslCerts'] = True
    capabilities['acceptInsecureCerts'] = True
    capabilities['PageLoadStrategy'] = None

    if headless:
        options.add_argument('--headless')
    else:
        print("since not headless, adding more options and prefs for file download")
        download_dir = '/tmp/' + hostname
        prefs = {'download.default_directory': download_dir,
            'download.prompt_for_download': False,
            'download.directory_upgrade': True,
            'safebrowsing.enabled': False,
            'safebrowsing.disable_download_protection': True}
        options.add_experimental_option("prefs",prefs)

    driver = webdriver.Chrome('chromedriver', chrome_options=options, desired_capabilities=capabilities,service_args=['--verbose', '--log-path=/tmp/chromedriver.log'])

    return driver

[wkeeling]

Thanks for posting your code. I think I may understand what's going on here. I assume that the site you're trying to test is using a self-signed certificate for HTTPS?

[djreducer]

Correct

…

On Tue, Jul 9, 2019, 3:12 PM Will Keeling ***@***.***> wrote: Thanks for posting your code. I think I may understand what's going on here. I assume that the site you're trying to test is using a self-signed certificate for HTTPS? — You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub <#55?email_source=notifications&email_token=ALSQ7A3B3BZ7JQHXT2JR6Y3P6TWKHA5CNFSM4H4F3WR2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODZRMMNA#issuecomment-509789748>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ALSQ7A6O55IPXBO3GU24RVDP6TWKHANCNFSM4H4F3WRQ> .

[wkeeling]

The issue is that Selenium Wire's embedded proxy server isn't able to handle the self-signed certificate used by the target site. There's now a config option available which tells Selenium Wire to ignore SSL verification for self-signed certificates.

Can you try upgrading to v1.0.6 and set the verify_ssl option to False in your webdriver configuration - e.g.

...

driver = webdriver.Chrome(
    'chromedriver', 
    chrome_options=options, 
    desired_capabilities=capabilities,
    service_args=['--verbose', '--log-path=/tmp/chromedriver.log'],
    seleniumwire_options={'verify_ssl': False}   # Don't verify self-signed cert
)

@djreducer @BobDu @clovis1122

[djreducer]

Now I'm getting "signal only works on main thread"

[djreducer]

The "signal only works on main thread" message is apparently because I'm using this as part of a flask app. If I run my function outside of flask it works fine, including the selenium-wire functionality

[wkeeling]

Ok thanks. So it sounds like the 502 Bad Gateway SSL verification issue is now resolved, but you're now hitting #56 (see the issue for my proposed solution). I'll let you know once that's available - hopefully soon. Thanks!