Proxy timeout leads to 502 bad gateway

[juhoarvid]

My server under test got some delay in login. It tooks now over 5sec and I realize automation fails allways after 5sec time. Failure is:
INFO:Capturing request: http://SERVER.UNDER.TEST/login DEBUG:code 502, message Bad Gateway Traceback (most recent call last): File "/home/USER/miniconda3/lib/python3.7/site-packages/seleniumwire/proxy/proxy2.py", line 179, in do_GET res = conn.getresponse() File "/home/USER/miniconda3/lib/python3.7/http/client.py", line 1321, in getresponse response.begin() File "/home/USER/miniconda3/lib/python3.7/http/client.py", line 296, in begin version, status, reason = self._read_status() File "/home/USER/miniconda3/lib/python3.7/http/client.py", line 257, in _read_status line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1") File "/home/USER/miniconda3/lib/python3.7/socket.py", line 589, in readinto return self._sock.recv_into(b) socket.timeout: timed out DEBUG:http://SERVER.UNDER.TEST/login 502

I have seleniumwire==1.0.4 in use

After changing timeout value in https://github.com/wkeeling/selenium-wire/blob/6bb6c61/seleniumwire/proxy/proxy2.py#L38 locally, test goes PASS.

Can that timeout value be given as parameter for proxy? I don't how to do it.

options = {
'proxy': {
'http': 'http://localproxy:xxxx',
'https': http://localproxy:xxxx',
'no_proxy': 'localhost, 127.0.0.1'
}
}

[wkeeling]

Thanks for raising this. Currently the timeout value isn't configurable (hard coded to 5 seconds) but it would definitely make sense to make it configurable for situations such as this.

I'll make the change and update this issue once it's available.

[juhoarvid]

I was really needing that change and was planning to check do I have anything to give with my python skills. After taking 1.0.6 in use, I notice problem disappears even there is still that hardcoded timeout defined:
https://github.com/wkeeling/selenium-wire/blob/6bb6c61/seleniumwire/proxy/proxy2.py#L38

But that value is not used at all. Probably meaning no timeouts used for proxy bindings and therefore I did not face problem anymore with 1.0.6.

But If proxy timeout needed I still think this my ticket is valid.

[wkeeling]

Thanks @juhoarvid - I think you've found a bug. Looks as though the variable name was changed in an earlier refactoring but is no longer used, so the default of no timeout is taking effect. I guess we should probably fix that and also make the value configurable at the same time.

[juhoarvid]

One comment to this.

After using selenium-wire 1.0.6 I phase 502 in certain conditions. I am using Robot Framework for doing test automation. Tests fails due to exceptions on robot framework implementation. Tracebacks which leads to 502 are not shown. After this issue I am thinking, should traceback which leads to 502 be shown? In this issue it at least seems not to be server issue and therefore 502 is misleading debugging (https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/502)

Probably:

selenium-wire/seleniumwire/proxy/proxy2.py

Line 116 in 6bb6c61

except Exception:

        except Exception:
            if origin in self.tls.conns:
                del self.tls.conns[origin]
            self.send_error(502)
            return

Could be
(import traceback)

        except Exception:
            if origin in self.tls.conns:
                del self.tls.conns[origin]
            self.send_error(502)
            traceback.print_exc()
            return

or

        except Exception as e:
            if origin in self.tls.conns:
                del self.tls.conns[origin]
            self.send_error(502)
            raise e

Or exception splitted to smaller part where different type of situations are handled with different error code. For instance #55 (comment) mention that Nginx returns in certain condition 499 while selenium-wire do 502.

[wkeeling]

Thanks @juhoarvid

FYI version 1.0.8 is now available which contains a configurable timeout option. You can set a specific value or use None for no timeout. A default of 5 seconds is used if the option is omitted entirely.

options = {
    'connection_timeout': None  # Never timeout
}
driver = webdriver.Firefox(seleniumwire_options=options)

With regard to the 502 status, the tracebacks are in fact indirectly logged by the self.send_error(502) call (it calls log_error() in its super class). But, Selenium Wire intercepts that call and logs the message and traceback at debug level. So it's possible that you're not seeing the tracebacks if you don't have your logging set to debug - e.g.

import logging
logging.basicConfig(level=logging.DEBUG)  # Set to debug to see tracebacks

The reason for using debug level is to reduce a lot of harmless "noise" that may otherwise alarm users.