Loading…
uBlock as an HTTP proxy #1475
3 participants
I'm just some dummy on the Internet, but I think:
Let's say one really parted-out uBlock and reworked it to the point that it's able to just be fed a DOM built from the HTML/HTTP response and we un-stub or implemented enough API calls that it can run some filters and render a modified DOM down to a HTML string for the proxy to send to the user. Why expect it to be great? uBlock's a wonderful implementation of an ad-blocking browsing extension compared to the other ones, and thank Zeus they can use the existing filter lists. But which killer features could even function now and what is it about uBlock that would be just so wonderful that we'd want to make a proxy out of it or port one of it's features to a proxy? You'd need an application firewall to see what stuff is really doing. Honest question.
I do think a kickass ad-blocking proxy server would be fantastic. I've tried, mostly for my iPhone.
A proxy doesn't have the same constraints that a browser extension has. It can do some unique stuff. uBlock won't reduce that initial Content-Length before it even is downloaded or prevent certain network connections from even needing to be blocked. A proxy also has some severe disadvantages as well, once the client is rendering the HTML and executing javascript it can't do much and it is really hard to predetermine the resulting behavior. Short of injecting some javascript the HTTP proxy can't do a damned thing once the page is open.
Even if someone clever puts together a little server really linked up to stuff, it's using JavaScriptCore and WebCore, maybe everything, and the extension is built right in, does it's thing, does some fuzzing to see if it can trigger any no-no's to block and then you render out HTML of the initial state-but-ad-proof, you've just wasted a lot of resources and time and you still don't know how it's going to proceed from there on.
You mentioned HTTPS a lot. This hypothetical "uBlock" proxy isn't going to be able to eavesdrop and alter encrypted SSL traffic either without hacking yourself (with your permission, but, at serious risk given the level of security of a typical user's computer, and the difficulty in checking on what was really in a cert.)
Glimmer blocker is a lot like Privoxy but with a nice Mac UI. These are HTTP proxies that snoop on and modify your web traffic.
HTTPS is supposed to ensure you are talking to who you think you are and that your traffic has not been altered. You're going to need to man-in-the-middle the SSL which is generally kind of unpleasant and will involve some work from you getting your browsers to trust either a bullshit Certificate Authority that is you or some guy who wrote the software produced. This can only work by having you force your computer to trust this meaningless certificate created out of thinner. Now every HTTPS connection you use will come from that proxy, it will decrypt the contents, make the changes, and stamp it with a that meaningless John Doe SSL cert that will keep your browser happy and "lock" icon looking secure.
I doubt uBlock wants to get into that.
uBlock or anywhateverblock I'd assume really wants to get to interact with, catch events, and generally futz with the the actual instantiation and rendering and execution of the page as it is really happening.
They'd either need to do what GhostLab is doing or figure out how to bust in to the remote debugging features that the webkit clients share but I don't know that lets you even target desktop browsers to do this effectively from the outside.
Maybe others would disagree.
FWIW you can get the same thing you have with GlimmerBlocker with off-the-shelf tools, maybe you can even figure out something to add to the proxy chain to get SSL MITM so the software can do what you want. Still, use this in conjunction with uBlock..
Do check out Privoxy. It's a multi-platform, ad-blocking, privacy protecting HTTP proxy. It has been around forever and people use it, lots of projects adapt it for various things. is not exactly busy but it isn't an abandoned project (there is some activity on their SourceForge CVS repo). You're going to find more stuff on the Internet regarding configuring that to do what you want than GlimmerBlocker.
A quick Google search will find a lot of scripts people have put together to import ABP-style lists and they often throw in some of the same HOSTS lists uBlock references.
You will want to probably chain this behind a more robust, faster proxy like Polipo. You might even be able to just use Polipo with something like this. It cannot handle a lot of rules, even if it tried it will not catch a lot of things ABP, uBlock would.
Some of these privoxy solutions out there claim to apply the cosmetic filters correctly but I can't vouch for that.
Here's a purported adblock proxy that supports MITM-SSL.
It has you trust their certificate to make it work. Probably a nice guy but he can man-in-the-middle you into thinking he is your bank website if he decided to or he were hacked.
Seriously though. Want to block ads?
- Just use uBlock everywhere.
- Keep using a HTTP proxy if you like.
Want to easily augment uBlock, do something for unprotected browsers, apps?
- Additionally install a HOSTS file. Update it semi-often, uBlock has a good reference list. There are tools to update them for you.
- If you want to do this to the "whole network" you could use dnsmasq on your router and DNS blackhole bad domains with that same list of bad hostnames. This project makes it pretty easy to do this if you've got a Linux machine or perhaps your router/gateway you can use as a DNS server. There are lots of ways to do this.
Don't rely on HOSTS and a proxy alone for basic privacy/adblocking. In certain circumstances things could become slower and lots of stuff will be missed. If an ad has an IP address in mind for a resource, some app does a DNS lookup itself on an outside server, it's no problem to get around your settings.
Overaggressive hostname blocking (hpHosts...) often just breaks websites. Occurs for uBlock users until the developers are forced to work around it in uBlock's built-in filters with an override to disable the guilty third-party rule.
- Since you use a Mac you may enjoy Little Snitch to catch apps phoning home and possibly unusual activity.
Daaaamn, @floam…thank you for the reply of unsurpassed detail!
I read everything, but I’ll limit my response to the most important points.
Do check out Privoxy. It's a multi-platform, ad-blocking, privacy protecting HTTP proxy. It has been around forever and people use it, lots of projects adapt it for various things.
Cool! I’ll check it out.
Still, use this in conjunction with uBlock..
Oh, of course!
You mentioned using other stuff in conjunction with uBlock a couple of times. I couldn’t agree more.
I hope I didn’t give the impression that I wanted to replace uBlock as a browser extension with my suggestion. Rather, I just longed for the system-wide protection offered by GlimmerBlocker, and wondered if it would be possible to abstract out a piece of uBlock that could take care of the fastest, easiest blocking on a system-wide level.
It has you trust their certificate to make it work. Probably a nice guy but he can man-in-the-middle you into thinking he is your bank website if he decided to or he were hacked.
Ew. No. I do not trust this!
Additionally install a HOSTS file. Update it semi-often, uBlock has a good reference list. There are tools to update them for you.
Oh…see, this is why I said “I’ve never been great with networking” in my original post. I think I remember hearing about a HOSTS file, but I don’t know what it is. I’ll start reading up after I finish this post. :)
If you want to do this to the "whole network" you could use dnsmasq on your router and DNS blackhole bad domains with that same list of bad hostnames.
More network ignorance here! :) I don’t really understand this. I can guess that it sounds like adding a blacklist to my router, but I don’t really know how to run custom software on a router; all I know how to do is use the shameful, frustrating, horrible web UI to poke around the router settings (many of which I also don’t understand).
This project makes it pretty easy to do this if you've got a Linux machine or perhaps your router/gateway you can use as a DNS server. There are lots of ways to do this.
I know gateways have something to do with “splitting” a single IP address into many on a subnetwork, but not much more than that. (And I wouldn’t be surprised if that description had technical inaccuracies as well.) But
Don't rely on HOSTS and a proxy alone for basic privacy/adblocking. In certain circumstances things could become slower and lots of stuff will be missed. If an ad has an IP address in mind for a resource, some app does a DNS lookup itself on an outside server, it's no problem to get around your settings.
I believe I understand this.
Use HOSTS is for basic, straightforward stuff, but it’s not foolproof. Got it.
After I learn about HOSTS files, I hope to use them soon. Maybe I could even write a script to curl the adblock feeds and keep it up-to-date automatically…(?). Is that feasible?
Since you use a Mac you may enjoy Little Snitch to catch apps phoning home and possibly unusual activity.
I do use Little Snitch! It’s pretty fantastic.
Its UI isn’t set up to make it easy to block ever-changing lists of ad servers, though. I’ve blocked a few ads with Little Snitch, but I don’t know of a way to have it do something like subscribe to a feed of blocking rules, for example.
It has you trust their certificate to make it work. Probably a nice guy but he can man-in-the-middle you into thinking he is your bank website if he decided to or he were hacked.
What ? With your personal self-signed CA.crt, only you can decrypt your HTTPS content, so no other can read it, that Man in the middle method is 100% safe, you are wrong. I'm writting a local proxy like Privoxy but support HTTPS using that technology so I'm sure that is safe.
Don't rely on HOSTS and a proxy alone for basic privacy/adblocking. In certain circumstances things could become slower and lots of stuff will be missed.
That only true with HOSTS, but with local proxy you even can have a better ad filtering performance, much faster than any kind of browser's AdBlock software by abusing multiprocessing.
Okay, to the author of this topic:
- If you want a local filtering software that can filter HTTPS, try Proxydomo (the best at the moment), or you can try GlimmerBlocker, Privoxy, Proximitron with ProxHTTPSProxy, this will help those three filter HTTPS website.
What ? With your personal self-signed CA.crt, only you can decrypt your HTTPS content, so no other can read it, that Man in the middle method is 100% safe, you are wrong. I'm writting a local proxy like Privoxy but support HTTPS using that technology so I'm sure that is safe.
A personal self-signed certificate is fine[1]. But just being given a bad cert someone made, you trust it, and then presumably go on trust the CA that made it in this context spooks me. Did I misunderstand something?
Their instructions as I understand them send you over to https://mitm.it to accept THEIR certificate. I'm confused... I didn't go so far as to try it myself and see what happens but from the way they describe it in the instructions and the fact that site does have a SSL cert my browser doesn't like I presume they want the users to go there so they can choose to trust it. Are you saying that page actually helps you generate your own cert? In that case I was wrong and I apologize.
That only true with HOSTS, but with local proxy you even can have a better ad filtering performance, much faster than any kind of browser's AdBlock software by abusing multiprocessing.
Agree. But in ... certain circumstances ... native apps that can't/won't respect your proxy settings or do things that aren't HTTP, there can be some undefined behavior. I think it's wise to keep your system-wide HOSTS a bit more conservative than the domains you pick in your adblock extension and/or adblock proxy.
[1] There are caveats and a downside or two... but nothing like what I'm talking about here so no need to complicate the issue. I'm not attacking self-signed certs where you control a CA and lock up the keys.
OK.
I was wrong, and I am sorry for not researching that project further while throwing my manic reply to @Zearin together. I had simply nearly recommended that project to him before noticed that one step in the instructions and it seemed nuts and dangerous. I regret if my advice influenced any decision making.
I had not realized that mitm.it was a part of the mitmproxy project. They're kosher and I hadn't known that the way it seems it actually works is after running the mitmproxy, by navigating there you'll not get what I saw but a tool running on your own machine to help users get certs going under their own CA in what looks like a secure, user-friendly way.
I've been playing with a number of tools recently with similar goals to the poster and started on an ad-blocking proxy as well a trying to implement a bad-hosts database like MVPS with classifications as a DNSB but not for mail servers avoiding junk. Just an itch I wanted to scratch with a particular appliance and it is seeming like a not great idea overall.
Still don't quite understand why it hits you with their their own Comodo cert intended for their hosted--really-real mitm.it page when it's only able to cause a domain mismatch. I doubt they are just lazy. What use could that have? I'll keep reading.
Several years ago, I used to use GlimmerBlocker. As a system-wide HTTP proxy, it touted benefits over traditional ad blocking extensions:
At the same time, I lamented some of its drawbacks:
When GlimmerBlocker was first released, most ads were sent over HTTP. GlimmerBlocker’s website even banks on this, stating that ads were not likely to use HTTPS because of the additional overhead. Back then, it was true. Today…not so much. :-/
Today, I use uBlock and absolutely love it, and don’t really keep up with GlimmerBlocker any more. But in spite of its large HTTPS shortcoming, there was something I miss dearly about having system-wide ad blocking.
Would it be (theoretically) possible to adapt uBlock into some kind of system-wide HTTP(S?) proxy?
Obviously, the scope would probably require a separate project. But think of it:
Rather than protecting individual browsers on each computer, I love the idea of uBlock protecting the whole computer. (Or better yet, having a dedicated machine running uBlock to protect an entire network.)
If this is a naïve request, sorry about that; I’ve been immersed in the Web since the late 90s, but I’ve never been great with networking itself. I just do my best.
Still, I love uBlock’s user-empowering mission, and its dedication to integrity and performance. I just want it to protect as much as possible.
Thanks for reading this long post!, and may The Source™ be with you.