Anti-Adblock, the future and possible solutions?

[BooBerry]

There's been multiple topics on the forums about this lately, so it I thought it might be time to post something about it here.

It doesn't look like Anti-Adblock Killer (both the filter list and userscript) is a viable solution to any new Anti-Adblock mechanisms as AAK development has slowed down considerably (e.g. 1 or 2 updates in 6+ months time - no filter updates since December). There's a fork of AAK called Adblock Protector, but it's focus seems to be using it alongside uBlock Origin. This may mean the filter list likely has rules uBO supports (e.g. different syntax).

Perhaps it's time Adguard has it's own dedicated Anti-Adblock solution(s) in place? (Okay, yes, I already know there's active Anti-Anti-Adblock happening in the filter lists as they're reported and reproduced, but this is something dedicated to just this).

At the very least, there can be an Anti-Adblock filter. While it'd make sense to merge such rules into other filter lists (English, Spyware, Annoyances) users may not know that there's Anti-Adblock rules present, hence the potential need for a specific filter just for this. Could use AAK as a starting point, or just start with something completely new.

The benefit to this approach is, the use of a userscript alongside the filter list may not be needed at all. I say may not because of the Adguard browser extensions. This might be a situation when a Adguard team-developed userscript could be an option for those with a script manager (e.g. Tampermonkey) alongside the Adguard browser extensions. Ultimately, it all depends what the Adguard browser extensions are capable of supporting. However, Adguard for Windows, Adguard for Mac, Adguard for Android, etc. should be able to use only a filter without any userscripts needed.

So here's what came to mind in a nutshell;

1. Have a Adguard team-developed Anti-Adblock filter list that covers all languages and all sites (not language-specific) that users can enable and have some peace of mind knowing there's some form of Anti-Anti-Adblock happening. This can support everything required for working rules, without the need of an additional userscript.

1a) If need be for the Adguard browser extensions, an additional userscript that can be added to the user's script manager (e.g. Greasemonkey, Tampermonkey, etc.). If the browser extensions don't require this, skip it completely!

2. New forum section under Filter Rules called Anti-Adblock issues (or something like this) where users can report sites where they encounter Anti-Adblock. Require direct URLs, Adguard they're using (Windows, Mac, Android, browser extensions, etc.), browser(s) they're using, screenshots if possible, enabled filters they're using, etc.

That's basically it. Thoughts?

[ameshkov]

You're right, we need to provide an Anti-Adblocking solution indeed.

A few points about it:

1. It must be a cross-blocker solution.
2. It might use different "strategies". For Adguard it might be compiled into a single filter. For other blockers, it could be compiled into a filter + userscript.

[BooBerry]

Sooner than later would be best, IMO.

Any updates on this, it's been over a month. Is there going to be an Adguard developed/focused Anti-Adblock filter (and userscript available for the browser extensions)?

The hard part, I assume, would be finding all the Anti-Adblock filter rules in the Adguard-developed filter lists and moving them to the dedicated Anti-Adblock script. However, this approach would bring different kinds of issues, e.g. users not enabling the new Anti-Adblock filter if/when it's available, etc.

I think the best approach would be to leave all current Anti-Adblock rules where they are now in the various filters, create and add an Anti-Adblock filter to all of Adguard's products (and push out small updates for all the products, and offer this new filter), create the Anti-Adblock section under Filter Rules on the forums and start adding new Anti-Adblock filter rules to this new filter list. Maybe it should be enabled by default in all of Adguard's products (when a warning informing users they also need a userscript if using the browser extensions in non-integration mode).

Thoughts?

[BooBerry]

I guess if you guys need to base a new Anti-Adblock filter on an existing filter, perhaps it'd be a mix of the Adblock Warning Removal List and Anti-Adblock Killer filters? At the very least, the former would be a good starting place, as the latter's development has seemingly ended (Adblock Protector probably isn't an option due to the inclusion of uBO-only syntax filters).

[ameshkov]

Any updates on this, it's been over a month. Is there going to be an Adguard developed/focused Anti-Adblock filter (and userscript available for the browser extensions)?

It's all in my head, we have not yet started the active development.

We could stick to some very simple thing, e.g. create an additional filter and forum section so that people understand that anti-adblocking issues are handled by Adguard.

It's just I'd like it to be a more "global" process involving making some improvements to the programs. Also, I don't want it to be an Adguard-specific thing.

Here's what I have in mind:

1. Separate extension or userscript, working title: "Adblock Recovery".
2. Two operation modes (you can use both):
   • Blocking access (or warn about) to the websites engaged in the anti-adblocking;
   • Trying to handle the anti-adblocking script;
3. Ideally, mark links to such websites.

So, besides actual filter, we'll need a database with the websites engaged in any form of anti-adblocking.

[seba2282]

Question. Reek Antiblock Killer isn't continuing. So when will you delete it from filters ? Gejerally copy Reek project and continuing is as Adblocker Protector.

[BooBerry]

Is it "official" yet? I haven't seen anything from Reek.

[seba2282]

https://github.com/jspenguin2017/AdBlockProtector

Here project Reek continuing. It works only with ublock origin. May all from AQ Team should look on this project.

[BooBerry]

Yes, but it's not discontinued officially by Reek, yet. It'd be foolish to rush and assume AAK is "dead" as Reek hasn't said anything on the matter. That said, I believe AAK is pretty much dead, IMO. Which is pretty disappointing. Maybe if there's no updates in 6 months to a year from now, it could be removed from the available filters?

Moving on, the Adblock Protector filter list is already an option you can add if you're using Adguard for Windows 6.2 beta. Keep in mind some rules may not work correctly since it's focused on using uBlock Origin's extended syntax.

Personally, I'd like to see a more "homegrown" solution. ;)

[seba2282]

Ok. Can you tell me, it is true, AQ will have his Anti-Adblock filter Protector integrated (ala script and list in single filter) ? When ?

[BooBerry]

You can add both the filter list and userscript as an extension in Adguard for Windows 6.2 beta. For the filter list you just add it (it's available via the Add filter button) and just select it from the list (as seen as the last option in the screenshot below);

From there you add the userscript to Adguard's extensions (you can click on the userscript link and Adguard will prompt you if you want to add it);

And that's it. You can go to https://jspenguin2017.github.io/AdBlockProtector/ and verify the filter list + script is working (you won't see the AdBlock Protector Script is not installed or not enabled! and AdBlock Protector List is not installed or not enabled! messages). Plus I can open the settings page and change the settings. ;)

[seba2282]

Ok. :)

[BooBerry]

Andrey, I've been following your discussions with the Adblock Protector developer about a dedicated extension for Anti-Adblock (instead of a userscript) and IMO, this would probably be the best (so there's no need to have Tampermonkey/Greasemonkey/Violentmonkey). This would be the best way to "back up" ad blocking browser extensions. Adguard for Windows/Mac/Android probably won't need the dedicated extension (which is a good thing).

P.S. Adblock Recovery is a good name.

[BooBerry]

AdguardTeam/AdguardBrowserExtension#701

Looks like Adblock Protector is going to be marked as incompatible in Adguard (rightfully so, most rules won't work for it anyways). Everyone using Adblock Protector shouldn't use it (or Anti-Adblock Killer since it's more-or-less dead now).

Until the "homegrown" solution can happen, anyone encountering Anti-Adblock issues on websites should report them to the Adguard forums in the Missed Ads section with [Anti-Adblock] added to the title. IMO, this is the easiest and quickest way to get these sites dealt with.

[BooBerry]

Any updates on a "homegrown" solution?

Adblock Protector (IMO, since it only supports uBlock Origin and not Adblock Plus/Adguard it should be renamed uBlock Protector) is now popping up a warning to users about Adguard not being supported, so anyone using that setup is bound to post about it on the forums/Github. I'll likely make a sticky topic about it not being compatible.

[jspenguin2017]

@BooBerry
We have nicely asked you guys to removed the filter from your settings page, we even gave you 7 days deadline when it can be easily done in one day, but you refused. More than half of the issues that are opened in our repository in the past week are invalid, we do not have the resource to handle that. We have no other choice than taking steps ourselves to motivate you to remove the listing. The redirect rule is not permanent and we will remove it later when we feel appropriate.

I agree that it should be renamed to uBlock Protector, but it's now a bit too late to rename. The project was originally aimed at Adblock Plus, which is dropped later after testing out uBlock Origin.

[BooBerry]

@jspenguin2017 I'm not a Adguard developer, and as far as I know it's been marked as incompatible on the available filter subscriptions list (hopefully it'll be completely removed from the filters repo of available known filter subscriptions list and blocked from being added manually, IMO). But sadly that doesn't stop people from trying to use it (even though me and the devs have advised users not to but they insisted upon it being available on the available known filter subscriptions list). That's why this issue exists, to move away from third-party userscripts and filters and go for something more "homegrown".

EDIT: It has been marked as incompatible.

Still, I feel it should be outright removed from the available filters list (and/or blocked from being added). However, I haven't tested it, but users can likely still manually import the filter via its direct link URL.

Also, the warning is already being noticed (and I'm pointing people to the sticky topic about removing the script/filter). I haven't seen the warning you've added, but if you haven't already you might want to adjust the warning telling the user to remove the filter and the userscript or extension as they're called in Adguard for Windows.

https://forum.adguard.com/index.php?threads/getting-message-about-adguard-all-web-pages.22016/

[jspenguin2017]

@BooBerry
Nah, I'll go the other way around.
https://github.com/jspenguin2017/AdBlockProtector/blob/master/AdBlockProtectorList.txt#L14
Or you guys won't be motivated to update the list.
I have not tested if the rule actually works, it is treated as comment in uBO. Looks like it actually does work, that's good to know.
There is still no officials from Adguard who responded to either GitHub threads, looks like you guys don't care, didn't you guys claimed "24x7 support"?

[seba2282]

@jspenguin2017 Adblock Protector should as filter for Adblockers for different browsers (not only UBO for Chrome). And list and script should be in it. AQ/UBO/Adblock (Plus)/etc browsers Chrome/firefox/Edge/etc and Windows versions or extensions.

Can you tell me, do you know, reek continuing project killer or not ?

[BooBerry]

@seba2282 Reek's been MIA (I haven't seen anything officially stating that development of AAK has ended) and there hasn't been an AAK update in over 6 months. Personally, I think it's dead.

I think @ameshkov and his idea with the "Adblock Recovery" mentioned above is probably the best way forward. :)

Otherwise, remove the Adblock Protector userscript and filter from Adguard and if you encounter any websites employing Anti-Adblock scripts, report them on the Adguard forums in the Missed Ads section for the filter devs to look at.

[jspenguin2017]

I'm just tired of explaining the same thing over and over to people like @seba2282 . That's why I'm asking you guys to remove the filter entry.

[seba2282]

@ameshkov if reek killer dead, so why is it in AQ products all time ?

[BooBerry]

@seba2282 because the Adguard team hasn't decided the fate of AAK in Adguard yet (e.g. removing it) - it's still being discussed.

[jspenguin2017]

@BooBerry I'd say Reek's Userscript is beyond "outdated", it can be abused as malware dropper, to bypass cross origin security policy, the script callbacks to reeksite.com/public/xhr.php to load some resources, the problem is the domain is taken over by some unknown people, and it may serve malware through the Userscript.
https://github.com/reek/anti-adblock-killer/blob/master/anti-adblock-killer.user.js#L478

[BooBerry]

Yikes!

Yeah, AAK should be removed sooner than later, IMO.

[jspenguin2017]

On newer browsers, it should^tm be fine, since GM_xmlHttpRequest is pretty much always present. Another reference of reeksite.com is here:
https://github.com/reek/anti-adblock-killer/blob/master/anti-adblock-killer.user.js#L1398
This function is only referenced once, it may try to load resource from reeksite.com, even on newer browsers, on tv3play.tv3.ee.
IMO This is a bigger problem and is a real threat, as we have no control over what reeksite.com serves anymore, not even Reek himself.

[BooBerry]

Trying to visit reeksite.com tries to load up some sort of tracker (and some check credit site lulz) it looks like. Adguard's browsing security and my ESET both blocked it.

Double yikes.

[jspenguin2017]

@ameshkov Ah, I see. I have browser protection turned off since it is a pain when I'm using my local server. I guess that patches the vulnerability in AAK Script...
Can you hash the script and enforce a secure checksum until Reek is back?

[ameshkov]

Can you hash the script and enforce a secure checksum until Reek is back?

Nope, but this thing makes me think we need to introduce some kind of a killswitch in AG. I'll consider possible options.

Anyway, it won't help people who use other script hosts.

[jspenguin2017]

I'm going to try to initiate a community effort to be the alternative to AdBlock Protector.
I will make reasonable effort to kick start this if enough people signed up to be reporter / developer.
Let me know if anyone wish to help:
https://gitlab.com/xuhaiyang1234/AAK-Cont

[seba2282]

Can you add this rule : ||atendesoftware.pl and ||redcdn.pl , @ameshkov ? I saw, this domains show ads type view seconds mp4 video (popups on Firefox, first video ads in player before normal video example in Chrome).

[BooBerry]

@seba2282 you should report that on the forums.

[mikhoul]

Just want to follow THIS discussion.

[jspenguin2017]

@mikhoul There won't be anything interesting going on in this thread, Adguard will have their own solution, I will continue to maintain uBlock Protector (now strictly for the setup I use), and AAK-Cont is going to be mostly made of automation scripts that port uBlock Protector, compatibility is not that much better, as it takes forever to resolve an issue due to lack of interest and testers.
I'm implementing a proxy based Userscript/adblocker host engine. The idea is to intercept and patch pages on the network level, which free me from restrictions of uBlock Origin and Violentmonkey. It will also completely remove race condition, and browser cache would cache the patched version.

[mikhoul]

Thanks @jspenguin2017 for the answer,

I use mainly Firefox and I favor projects that run on most browsers to contribute, do you know if AAK-Cont could be used with Greasemonkey ?

If yes I will switch to AAK-cont and open an account on Gitlab to contribute to the project in my spare time 😉

What is the main reason to only support Chrome and Violent monkey instead of being more compatible with all options available ?

IMO it's always good to have the choice of browsers, Userscript host since if one fail or disappear you can rely on the others, also it push for more innovations.

Best Regards

[jspenguin2017]

@mikhoul Yes, AAK-Cont does run on Greasemonkey. Giwayume rewrote 3516 lines of code just to make it working on Greasemonkey, you really should thank him.
I strictly only support a single setup in uBlock Protector because I don't have time to test on everything. And I really doubt how many skilled contributors I can attract by offering support for another setup. The increase in setup-specific issues opening would heavily outweigh the increase in popularity.
Also, I want my code to be fast, and offering support for multiple setup works against performance. With only one setup, I can make assumptions that are just not safe for a platform-independent solution. I can also write my code the way I want, ECMAScript 2017? No problem because Chrome supports it.
Chromium, uBlock Origin, and Violentmonkey are not going to fail, if they fail, I'll fix them. They are open source, there is no failing in open source software.

If you contribute to AAK-Cont, then soon you'll understand why I only support one setup.
Managing uBlock Protector takes more than just spare time.

[jspenguin2017]

@mikhoul You know that Greasemonkey, along with other legacy extensions, will be forcefully uninstalled by November this year, right? If something is going to fail, Greasemonkey is the first in line.

[mikhoul]

@jspenguin2017 Not really @arantius is working on Greasemonkey 4.0 and should be ready for FF57 😄

More Information here if you want to contribute or have some Ideas for the new API: https://groups.google.com/forum/#!topic/greasemonkey-users/Va5kqhkgVG8

Regards

[jspenguin2017]

@mikhoul The new Greasemonkey won't be the old Greasemonkey anymore, it would behave similar to everything else, which would probably remove the problem which is causing Greasemonkey to have compatibility issues.

Whatever happens, I would not support it in uBlock Protector. You said I should support more setups, I agree, but that doesn't come without cost, if you want to pay that cost, come over to AAK-Cont and start coding.

[BooBerry]

Anyways, to get this back on topic.

@ameshkov any updates? Is the "Adblock Recovery" idea going to progress?

[seba2282]

@ameshkov Can you add special filter with list and script (all in one) to protect Adguard ? And remove reek old filter. 😊

[BooBerry]

@seba2282 well, that's what @ameshkov posted above in his first couple posts, a conceptual idea (not just for Adguard either).

If you're using Adguard for Windows, Adguard for Mac, etc. you probably won't need a userscript companion since the desktop apps are much more powerful than the browser extensions. That said, browser extension users will likely need the companion userscript (or better yet, a dedicated companion browser extension if possible).

[arantius]

I'm working on making something called Greasemonkey that runs user scripts manage to exist in a FF57+ world. But given that this is unpaid volunteer labor, there's a limit to what I can accomplish. Other user script engines also exist.

[jspenguin2017]

@BooBerry From what I read, AdFender and Adguard for Windows are transparent proxies, they are not magic, they have different limitations than extensions. The best would be a transparent proxy working together with an extension, then you'll have full control over what is happening in the browser.

Transparent proxies are local servers that filter everything going in and out of your system, they have more power over the content that is delivered to the browser, but have less control over what the browser do. For example, they can't forcefully close a tab, and it can't forcefully open many tabs (it probably can open 1 or 2, but the rest would get blocked). Servers also tend to be heavier, but since they can be in native code, it might offset the performance back.

The idea is simple, but the execution is hard, because Microsoft won't let everyone to make transparent proxy, there probably won't ever have open source transparent proxy for Windows. But explicit proxy is possible, that is what I was doing, but I'm coding it in JavaScript, which can have some performance issues.

[ameshkov]

@ameshkov any updates? Is the "Adblock Recovery" idea going to progress?

Yeah, it is, I'll post a link here once the first alpha is out.

they have different limitations than extensions. The best would be a transparent proxy working together with an extension

That's absolutely true.

[ameshkov]

@arantius

I'm working on making something called Greasemonkey that runs user scripts manage to exist in a FF57+ world.

Cheers!

How it goes? I reckon this is the WebExt branch: https://github.com/arantius/greasemonkey/tree/webbymonkey

[seba2282]

@jspenguin2017 In last time I can't mount your script on Edge + Tampermonkey. What did you make with scripts. If I uninstall Tampermonkey, I see new tab with txt your script in Edge. If I have Tampermonkey, script isn't loading and install... Earlier worked medium....

[BooBerry]

Yeah, it is, I'll post a link here once the first alpha is out.

Oh? Is it listed on Git yet?

[jspenguin2017]

@seba2282 I wrote it in ECMAScript 8, Edge probably haven't caught up yet.

[ameshkov]

Oh? Is it listed on Git yet?

Not yet, I don't like to publish anything until it's at least partly ready.

[arantius]

How it goes? I reckon this is the WebExt branch: https://github.com/arantius/greasemonkey/tree/webbymonkey

Exactly. It's functional, minimally. You can install and use it for some real use cases. Most recent change has been adding API support, but new async incompatible versions of them.

[seba2282]

@jspenguin2017 Ok. Sorry, I write it, but I see future Adguard with dedicated filter with list and scripts Protector Team Adguard. Don't Angry on my opinion. 😊