Use netrc for proxy authentication

[slackhead]

This adds support for proxy authentication using usernames/passwords in ~/.netrc. If the credentials are wrong either an access denied or SSL error occurs, depending on the protocol.

The change that I made in shared.py is probably not great, but it served to test with:

try:
    host = url.host()
except AttributeError:
    host = url

There's probably a better/more secure way to do this, like casting proxy_host to a Qurl before sending it, but that's a bit beyond my knowledge.

I didn't do any documentation for this. I wanted to just fly the general idea past you first.

---

This change is 

[qutebrowser-bot]

Thanks for the pull request! Note that reviews currently can take a bit longer, as @The-Compiler is busy with exams until September.

(Powered by GitMate.io)

[slackhead]

This doesn't seem to work with socks proxies AFAICT. I need to figure out why I didn't get an auth challenge when I visited a socks proxy with authentication on.

Looking around it seems it isn't supported in Chromium. Some people had asked for it to be added, but last response seems to be WONTFIX by Chromium devs, as they consider that hardly anyone uses socks with auth anyway. These were old requests though.

I'm not sure if adding auth challenge detection for socks in qutebrowser would work, or if it needs adding to the underlying code in chromium/webengine too.

[The-Compiler]

This kind of thing definitely isn't the right way to go about this 😉

What about just making all callers pass a host (rather than a URL), and then do url.host() where netrc_authentication is called instead?

[slackhead]

I had a think about this and I'm not even sure using netrc is a good idea really.

I've been looking around to find other documentation or examples of it being used for proxy authenticaion but can't find anything. It doesn't seem to be historical linux/unix behaviour anyway.

My findings about proxy auth in chrome/chromium uncovered a few bug reports/complaints, some going back to 2010. These were mostly from windows users though. Apparently Chrome in windows takes it settings from IE, or did at one time, and something broke there at some point.

Apps like e.g. curl can use netrc for http/ftp auth, but it keeps proxy auth as separate command line flags --proxy-user etc. wget has credentials in its config file.

Historical linux behaviour is to set http(s)_proxy as http(s)://username:password@host:port however this fails with chromium and qutebrowser, also when using the --proxy-server flag in chromium. It seems to always default to prompting for the auth info whether it's set or not.

FWIW urllib seems to be able to detect proxies with auth settings:

http_proxy=user:password@192.168.1.2:35000 https_proxy=$http_proxy python3
...
>>> from urllib import request
>>> print (request.getproxies())
{'http': 'user:password@192.168.1.2:35000', 'https': 'user:password@192.168.1.2:35000'}

However, this doesn't solve switching to different proxies on the fly. Many of the bug reports/complaints about proxy auth that I found on the net were from people whose corporate networks had > 1 proxy setup with different auth on each, and the general mood was to switch to FF. Also, it seems that socks auth isn't supported at all. There aren't any prompt for credentials and this results in a ERR_SOCKS_CONNECTION_FAILED. However I seem to recall that proxy changing addons /do/ support it, so it must be in the chromium code /somewhere/.

At this point, I think a better solution would be to properly derive the auth info from the proxy URL rather than use a potentially insecure option of putting it in netrc in cleartext. I will see about doing something with that and make a new PR if I have any luck with it.

Thoughts?

[toofar]

Thoughts?

A non-serious one.

>>> request.getproxies()
{'http': 'user:password@192.168.1.2:35000'}
>>> os.environ['http_proxy'] = 'user:password@192.168.1.3:35000'
>>> request.getproxies()
{'http': 'user:password@192.168.1.3:35000'}

You can use debug-pyeval to switch proxies!!

But seriously, you have said why you are not wanting to use content.proxy before but I can't remember why. Is it because you don't want your proxy pass in your qutebrowser config file?

[slackhead]

It's because qutebrowser doesn't use the login info when it's set in content.proxy - it only looks for the host:port and ignores username:password@ - so I have to login manually every time I need to switch.

[toofar]

That seems like a bug considering the setting is parsed with this but I don't know how I can set up an authenticating proxy to test with. If QNetworkProxyFactory.setApplicationProxyFactory just doesn't work with webengine then I think the right thing to do would be to use config.val.content.proxy to configure webengine.

[The-Compiler]

One way to test is running mitmproxy --proxyauth user:pass and :set content.proxy http://user:pass@localhost:8080/ in qutebrowser. Indeed it still seems to ask for authentication.

[slackhead]

Probably the easiest way to set up a proxy to test with is with 3proxy. The config is relatively simple (usually /etc/3proxy/3proxy.cfg):

# global options
writable
daemon
pidfile /var/run/3proxy.pid
monitor /etc/3proxy/3proxy.cfg
log /var/log/3proxy.log D
logformat "L%Y-%m-%d %H:%M:%S Service: %N:%p Remote: %R:%r Request: %n:%q Client:   %C:%c - %T %E"
rotate 60

# acl rules
flush
users "testuser:CL:cleartextpassword"
auth strong
allow testuser 192.168.1.0/24 # or whatever network mask/IP address you like
deny *

# proxy servers
# http
proxy -n -p33000
# socks
socks -u2 -p33500

That will create an http(s) server on port 33000 and a socks proxy on port 33500.

[slackhead]

mitmproxy looks easier though. Does it also do socks?

[toofar]

http://doc.qt.io/qt-5/qtwebengine-overview.html#proxy-support

That is, QNetworkProxy::type(), QNetworkProxy::hostName() and QNetworkProxy::port() are taken into account. All other proxy settings such as QNetworkProxy::rawHeader(), QNetworkProxy::user(), or QNetworkProxy::password() are ignored.

And it looks like that is because chromium/net/proxy/proxy_server.h doesn't offer any API around authentication. I don't see an open bug around improving that support but I don't think Qt uses jira tickets to track features that they don't intend to implement right now so it would probably be closed right away anyway.

It looks like if we want to improve this use case we can either parse the config option in _on_proxy_authentication_required (and either just use it or set it as the default for the question) or learn to remember question answers. Or use .netrc.

[slackhead]

There's another problem here: proxy_host passed to _on_proxy_authentication_required() doesn't contain the port, only the IP. Authing by netrc will fail unless it contains a 'default' setting. If more than one proxy is on the same IP with different passes, or perhaps there's also a normal http auth on it too, then you've had it.

So it seems to me that parsing the content.proxy string for username/password, and/or providing a separate content.proxy_user containing user:password similar to curl and wget, is the only/simplest way to do this.

[slackhead]

Related - I don't see a port being looked up in netrc_authentication() either, only url.host()

       net = netrc.netrc(config.val.content.netrc_file)
       authenticators = net.authenticators(url.host())

If you are running several services on the same machine on different ports (e.g. cups and other web frontends) wouldn't this be a problem? (Someone did report a problem with netrc in IRC I seem to recall. I wonder if this was why?)

[toofar]

Yeah proxy_host not being a QUrl seems weird. I can't find anything about netrc supporting specifying ports though, just a weirdness from originally being just for ftp I suppos.

[The-Compiler]

netrc files don't seem to support a port - note that they were originally written for FTP authentication, everything else seems like a slight abuse of the file anyways.

Thus, I agree that this is kind of weird, and we should rather get the authentication information from the proxy URL. I opened #4278 for that.