Running behind a squid proxy gives Bad Request #16

appunni-dishq · 2018-10-29T08:49:33Z

I am running a Squid proxy currently to do the Caching, so I need to do ssl termination. I am so using ssl-bump feature. But for some reason curl requests are going through but not Selenium-wire requests. I specified proxy according to the docs

wkeeling · 2018-10-29T09:13:25Z

@appunni-dishq Do you have logging turned up to DEBUG level - for example:

import logging
logging.basicConfig(level=logging.DEBUG)

from seleniumwire import webdriver

...

That may give some clues as to why Selenium Wire is not getting through the upstream proxy.

Are you able to share the configuration you're passing to the webdriver constructor?

appunni-dishq · 2018-10-29T09:55:00Z


                        settings.CHROME_WEBDRIVER,
                        desired_capabilities=options.to_capabilities(),
                        seleniumwire_options={
                            'proxy': {
                                'http': settings.ENV_HTTP_PROXY,
                                'https': settings.ENV_HTTPS_PROXY,
                                'no_proxy': 'localhost,127.0.0.1'
                            }
                        }
                    )```
And both proxies are same http proxy

appunni-dishq · 2018-10-29T09:57:33Z

@wkeeling I did turn up DEBUG logging but didn't give much information except the status code and stuff. And Selenium-wire throws error when http or https are None, which should not be the case as many implementations including in that of request module is to override proxy to empty if passed None. Also when http proxy is used in https I get 301 always.

appunni-dishq · 2018-10-29T10:02:21Z

format of Proxy is this http://x.x.x.x:port
, earlier I was using ATS but as it didn't support SSL Termination, I switched Squid.
curl -x http://x.x.x.x:port/ -iLk https://ipchicken.com works fine, without option k. It throws insecure request error in curl. Now I added a flag in chrome options "--ignore-certificate-errors" but it didn't make much difference.

appunni-dishq · 2018-10-29T12:50:38Z

@wkeeling I think that issue was because of configuration issue. As Selenium-wire send insecure proxy request, while I was expecting insecure request.

wkeeling · 2018-10-29T12:57:20Z

@appunni-dishq - OK thanks.

So just to be clear, you're using a proxy configuration such as this:

seleniumwire_options={
    'proxy': {
        'http': http://x.x.x.x:port,
        'https': http://x.x.x.x:port,
        'no_proxy': 'localhost,127.0.0.1'
     }
}

So the https option is using an http proxy URL?

Regarding the use of None for http and https, you are right, Selenium Wire should handle this and not error. I will raise a separate issue for that.

appunni-dishq · 2018-10-30T03:51:59Z

I did different implementations http proxy as well as https proxy. When http proxy is used I usually get 301 status code on all request. When I use https I am getting 400. Using curl proxy-insecure option was also giving me similar responses

wkeeling · 2018-10-30T08:37:39Z

OK thanks. I'll see if I can reproduce the issue with a Squid server.

appunni-dishq · 2018-10-31T09:18:26Z

I am using squid 4.4. This is the configuration . :

#
# Recommended minimum configuration:
#

# Example rule allowing access from your local networks.
# Adapt to list your (internal) IP networks from where browsing
# should be allowed
acl localnet src 172.12.0.0/16		# RFC 1918 local private network (LAN)
acl localnet src fc00::/7       	# RFC 4193 local private network range
acl localnet src fe80::/10      	# RFC 4291 link-local (directly plugged) machines
acl localhost src 127.0.0.1
acl localhost src localhost

acl SSL_ports port 443
acl Safe_ports port 80		# http
acl Safe_ports port 21		# ftp
acl Safe_ports port 443		# https
acl Safe_ports port 70		# gopher
acl Safe_ports port 210		# wais
acl Safe_ports port 1025-65535	# unregistered ports
acl Safe_ports port 280		# http-mgmt
acl Safe_ports port 488		# gss-http
acl Safe_ports port 591		# filemaker
acl Safe_ports port 777		# multiling http
acl CONNECT method CONNECT

#
# Recommended minimum Access Permission configuration:
#
# Deny requests to certain unsafe ports
http_access deny !Safe_ports

# Deny CONNECT to other than secure SSL ports
http_access deny CONNECT !SSL_ports

# Only allow cachemgr access from localhost
http_access allow localhost manager
http_access deny manager

# We strongly recommend the following be uncommented to protect innocent
# web applications running on the proxy server who think the only
# one who can access services on "localhost" is a local user
# http_access deny to_localhost

#
# INSERT YOUR OWN RULE(S) HERE TO ALLOW ACCESS FROM YOUR CLIENTS
#

# Example rule allowing access from your local networks.
# Adapt localnet in the ACL section to list your (internal) IP networks
# from where browsing should be allowed
http_access allow localnet
http_access allow localhost

# And finally deny all other access to this proxy
http_access deny all

sslcrtd_program /usr/lib64/squid/security_file_certgen -s /var/lib/ssl_db -M 4MB


http_port 3129 ssl-bump generate-host-certificates=on dynamic_cert_mem_cache_size=4MB  cert=/etc/squid/ssl_cert/rootCA.pem key=/etc/squid/ssl_cert/rootCA.key

https_port 3128 intercept ssl-bump generate-host-certificates=on dynamic_cert_mem_cache_size=4MB  cert=/etc/squid/ssl_cert/rootCA.pem key=/etc/squid/ssl_cert/rootCA.key options=SINGLE_DH_USE,SINGLE_ECDH_USE tls-dh=/etc/squid/ssl_cert/dhparam.pem

acl DiscoverSNIHost at_step SslBump1
# acl NoSSLIntercept ssl::server_name_regex -i "/etc/squid/server-regex.nobump"
# ssl_bump splice NoSSLIntercept
ssl_bump peek DiscoverSNIHost
#ssl_bump peek step1
ssl_bump bump all
# https_port 3129 ssl-bump generate-host-certificates=on dynamic_cert_mem_cache_size=4MB cert=/etc/squid/ssl_cert/rootCA.crt key=/etc/squid/ssl_cert/rootCA.key options=SINGLE_DH_USE,SINGLE_ECDH_USE tls-dh=/etc/squid/ssl_cert/dhparam.pem

tls_outgoing_options options=NO_SSLv3,SINGLE_DH_USE,SINGLE_ECDH_USE

tls_outgoing_options cafile=/usr/local/openssl/cabundle.file
# hardening
tls_outgoing_options cipher=HIGH:MEDIUM:!RC4:!aNULL:!eNULL:!LOW:!3DES:!MD5:!EXP:!PSK:!SRP:!DSS

# Uncomment and adjust the following to add a disk cache directory.
cache_dir ufs /var/cache/squid 100 16 256

# Leave coredumps in the first cache dir
coredump_dir /var/cache/squid


#
# Add any of your own refresh_pattern entries above these.
#
refresh_pattern ^ftp:		1440	20%	10080
refresh_pattern ^gopher:	1440	0%	1440
refresh_pattern -i (/cgi-bin/|\?) 0	0%	0
refresh_pattern .		0	20%	4320


debug_options ALL,1 9,5

wkeeling · 2018-10-31T13:01:56Z

@appunni-dishq Thanks for that. I'll see if I can reproduce and report back here.

wkeeling · 2018-11-03T15:14:28Z

@appunni-dishq Just an update on this. Running Selenium Wire through a local Squid 4.4 using Squid's default config file works without any problem for me with the following options:

options = {
    'proxy': {
        'http': 'http://localhost:3128',
        'https': 'https://localhost:3128'
    }
}
driver = webdriver.Firefox(seleniumwire_options=options)

However, using your example config file above is giving me some errors on the ssl-bump configuration. I'll do some further investigation into that, but I'm suspecting that this overall issue is probably related to ssl-bump in some way.

appunni-dishq · 2018-11-03T16:02:43Z

I am using chrome. Could be related to chrome's latest security updates. Can you please share the config file. I will try to replicate. @wkeeling Yes ssl-bump has issues. Not enough documentation regarding ssl-bump in latest tls implementations. Without ssl bump, I already have HAproxy which is more than good enough. As almost all requests are HTTPS.

wkeeling · 2018-11-03T16:45:45Z

Chrome (Version 70.0.3538.77) also seems to work fine using the default config file and same options specified above, although it does display a message about the page not being secure.

The config file is:

#
# Recommended minimum configuration:
#

# Example rule allowing access from your local networks.
# Adapt to list your (internal) IP networks from where browsing
# should be allowed
acl localnet src 0.0.0.1-0.255.255.255	# RFC 1122 "this" network (LAN)
acl localnet src 10.0.0.0/8		# RFC 1918 local private network (LAN)
acl localnet src 100.64.0.0/10		# RFC 6598 shared address space (CGN)
acl localnet src 169.254.0.0/16 	# RFC 3927 link-local (directly plugged) machines
acl localnet src 172.16.0.0/12		# RFC 1918 local private network (LAN)
acl localnet src 192.168.0.0/16		# RFC 1918 local private network (LAN)
acl localnet src fc00::/7       	# RFC 4193 local private network range
acl localnet src fe80::/10      	# RFC 4291 link-local (directly plugged) machines

acl SSL_ports port 443
acl Safe_ports port 80		# http
acl Safe_ports port 21		# ftp
acl Safe_ports port 443		# https
acl Safe_ports port 70		# gopher
acl Safe_ports port 210		# wais
acl Safe_ports port 1025-65535	# unregistered ports
acl Safe_ports port 280		# http-mgmt
acl Safe_ports port 488		# gss-http
acl Safe_ports port 591		# filemaker
acl Safe_ports port 777		# multiling http
acl CONNECT method CONNECT

#
# Recommended minimum Access Permission configuration:
#
# Deny requests to certain unsafe ports
http_access deny !Safe_ports

# Deny CONNECT to other than secure SSL ports
http_access deny CONNECT !SSL_ports

# Only allow cachemgr access from localhost
http_access allow localhost manager
http_access deny manager

# We strongly recommend the following be uncommented to protect innocent
# web applications running on the proxy server who think the only
# one who can access services on "localhost" is a local user
#http_access deny to_localhost

#
# INSERT YOUR OWN RULE(S) HERE TO ALLOW ACCESS FROM YOUR CLIENTS
#

# Example rule allowing access from your local networks.
# Adapt localnet in the ACL section to list your (internal) IP networks
# from where browsing should be allowed
http_access allow localnet
http_access allow localhost

# And finally deny all other access to this proxy
http_access deny all

# Squid normally listens to port 3128
http_port 3128

# Uncomment and adjust the following to add a disk cache directory.
#cache_dir ufs /usr/local/squid/var/cache/squid 100 16 256

# Leave coredumps in the first cache dir
coredump_dir /usr/local/squid/var/cache/squid

#
# Add any of your own refresh_pattern entries above these.
#
refresh_pattern ^ftp:		1440	20%	10080
refresh_pattern ^gopher:	1440	0%	1440
refresh_pattern -i (/cgi-bin/|\?) 0	0%	0
refresh_pattern .		0	20%	4320

appunni-dishq · 2019-05-16T10:20:38Z

#49 I think these are related

appunni-dishq closed this as completed May 16, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Running behind a squid proxy gives Bad Request #16

Running behind a squid proxy gives Bad Request #16

appunni-dishq commented Oct 29, 2018

wkeeling commented Oct 29, 2018

appunni-dishq commented Oct 29, 2018

appunni-dishq commented Oct 29, 2018

appunni-dishq commented Oct 29, 2018 •

edited

appunni-dishq commented Oct 29, 2018

wkeeling commented Oct 29, 2018

appunni-dishq commented Oct 30, 2018

wkeeling commented Oct 30, 2018

appunni-dishq commented Oct 31, 2018

wkeeling commented Oct 31, 2018

wkeeling commented Nov 3, 2018 •

edited

appunni-dishq commented Nov 3, 2018

wkeeling commented Nov 3, 2018 •

edited

appunni-dishq commented May 16, 2019

Running behind a squid proxy gives Bad Request #16

Running behind a squid proxy gives Bad Request #16

Comments

appunni-dishq commented Oct 29, 2018

wkeeling commented Oct 29, 2018

appunni-dishq commented Oct 29, 2018

appunni-dishq commented Oct 29, 2018

appunni-dishq commented Oct 29, 2018 • edited

appunni-dishq commented Oct 29, 2018

wkeeling commented Oct 29, 2018

appunni-dishq commented Oct 30, 2018

wkeeling commented Oct 30, 2018

appunni-dishq commented Oct 31, 2018

wkeeling commented Oct 31, 2018

wkeeling commented Nov 3, 2018 • edited

appunni-dishq commented Nov 3, 2018

wkeeling commented Nov 3, 2018 • edited

appunni-dishq commented May 16, 2019

appunni-dishq commented Oct 29, 2018 •

edited

wkeeling commented Nov 3, 2018 •

edited

wkeeling commented Nov 3, 2018 •

edited