Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ABP's $rewrite filter option #46

Closed
mapx- opened this issue May 22, 2018 · 43 comments
Closed

ABP's $rewrite filter option #46

mapx- opened this issue May 22, 2018 · 43 comments
Labels
declined declined discussion weighing in community's input on a specific topic

Comments

@mapx-
Copy link

mapx- commented May 22, 2018

see
https://issues.adblockplus.org/ticket/6592
https://issues.adblockplus.org/ticket/6622

related:
https://issues.adblockplus.org/ticket/6242

@mapx- mapx- changed the title ABP's $redirect filter option ABP's $redirect filter option - discussion May 22, 2018
@mapx- mapx- changed the title ABP's $redirect filter option - discussion ABP's $rewrite filter option - discussion May 22, 2018
@uBlock-user uBlock-user added the discussion weighing in community's input on a specific topic label May 22, 2018
@gwarser
Copy link
Member

gwarser commented May 22, 2018

Why not:

||server.com/content/*.m3u8?userId=*$rewrite=s/userId=\d+//

@gwarser gwarser added discussion weighing in community's input on a specific topic and removed discussion weighing in community's input on a specific topic labels May 22, 2018
@jspenguin2017
Copy link

jspenguin2017 commented May 23, 2018

So /(^https?:\/\/[^/])/$script,rewrite=$1.evil.com basically let you inject arbitrary script to every page?
This can go so wrong, I'd say just hard code the few special websites that need this option in uBO-Extra.

@gwarser
Copy link
Member

gwarser commented May 23, 2018

Allow only to delete parts of the url?

@gorhill
Copy link
Member

gorhill commented May 23, 2018

Copy/pasting here a comment I made yesterday in Teams after I pondered this for a while:


gorhill May 22, 2018, 10:19 AM EDT

I won't implementing this filter option, I see too many issues with it. I am however open to implement a different filter option with similar purpose, but which would not suffer the issues I see with how rewrite has been designed. However, I need to see more cases being solved by such filter option than the just one case mentioned. So far I don't see this.

My concerns:

Security: testing same origin for redirect URL is not enough: both github.com/gorhill/ and github.com/toto/ have the same origin, however the content of both URLs is not controlled by the same person. My hunch tells me this is not good.

Security: even with strictly same origin, a malicious filter list author could add bad stuff to a network request.

Performance: the rewrite= option requires that the filter is a regex. This is bad, as this prevents such filter from being tokenizable, and as a consequence the filter must be tested against every single network request. With uBO this can be somewhat mitigated (not with ABP) by using type option (ex. xmlhttprequest) and party-ness option (ex. third-party), but this is still be a concern for me.

Given these concerns, I see a better way to implement similar option but with a more focused purpose: to remove specific query parameters from a URL:

||content.uplynk.com/ext/*.m3u8?$querystrip=*

Where the querystrip option would mean: "remove all query parameters matching the given lists of tokens or pattern".

Sticking to remove query parameters takes care of the ownership and malicious filter list author issue for the most part -- the filter removes query parameters, it can never rewrite them into something else.

The performance concern no longer exist with such filter, since it does not have to be a regex. The value of the querystrip option dictates which query parameter must be removed. It could be * to remove all of them, or a |-separated list of tokens which tells which specific query parameters should be removed.

Now this does not remove some other concerns I see with rewrite=.

One is that it is designed as a block filter.

What if I really want to block using ||content.uplynk.com/ext/? The way uBO/ABP and all similar-purpose blockers is to stop trying to find a match when a hit is found. If the filter with the rewrite option is found first, the really blocking filter won't be found and the network request will go through. This can be addressed in uBO using the important filter option, but still, my gut tells me there is something wrong about a blocking filter which actually does not block -- I feel more thinking is needed there.

My current thinking is that a querystrip filter applies if and only if the network request was neither blocked or excepted. Not blocked because block filters must result in the network request not being made by the browser, and not excepted because the purpose of exception filters is strictly to counter block filters. So if a network request goes through both block/exception filters unscathed, than it would be fair game for further handling with a querystrip option.

Anyway, as said I still need more than just one case to be an argument for such filter -- the last thing I want is to add technical debt to uBO for little tangible benefits overall. Note that a site could simply convert their GET request into a POST one and this would bypass both rewrite= or querystrip= filters -- so far the case for such filter options is thin.

@gwarser
Copy link
Member

gwarser commented May 23, 2018

remove specific query parameters

utm_* stuff (more like privacy and tracking)

querystrip may be not enough.
Apart from the POST requests - some pages use path to encode query parameters, for example amazon1:
amazon
And some other pages use hash 2, 3 (not even handled by network code).

@gwarser
Copy link
Member

gwarser commented May 23, 2018

Also ABP code can handle https://server.com/content/foo-with-ads.m3u8 -> .../foo.m3u8

This can go so wrong, I'd say just hard code the few special websites that need this option in uBO-Extra.

👍

and other cases (utm_*) can be handled by specialized URL redirecting extensions.

@jspenguin2017
Copy link

jspenguin2017 commented May 23, 2018

@gorhill

Yea, querystrip sounds way better.

Also I don't see how the new ABP build can get pass AMO review since it allow injection of arbitrary script. (Or I'm missed something?)

@gwarser other cases (utm_*) can be handled by specialized URL redirecting extensions.

The background script of uBO-Extra can be made configurable (unlike the content script), so we can include a switch to enable / disable part of its URL rewriting rules.

@gwarser
Copy link
Member

gwarser commented May 23, 2018

One is that it is designed as a block filter.

How to make this exception-only option? Request can be blocked by one filter, and then repaired by @@...$querystrip=*.

@gorhill
Copy link
Member

gorhill commented May 23, 2018

then repaired by @@...$querystrip=*

Exception filters also obey the first-match rule. You could end up with another exception filter such that the querystrip filter is not seen, and you also would be unable to except such filter if needed if it is found to cause issue on one site. Hence the need to lookup such filters after having both gone through block/exception filters and finding that none applied before caring about querystrip filters. This means added overhead overall since in basic settings most network requests are neither blocked or excepted.

A scriptlet for that specific purpose seems to be the right approach (an XMLHttpRequest wrapper to remove query parameters), it's injected only on sites where it's needed, can be excepted, and adds no global overhead to network request handling.

@hfiguiere
Copy link

Also I don't see how the new ABP build can get pass AMO review since it allow injection of arbitrary script. (Or I'm missed something?)

No it doesn't inject arbitrary scripts.

@jspenguin2017
Copy link

@hfiguiere What mechanism it has to prevent #46 (comment) from happening?

@hfiguiere
Copy link

The example in #46 (comment) change the origin, so the rewrite doesn't happen in that case, and the original query is let through.

@jspenguin2017
Copy link

jspenguin2017 commented May 23, 2018

@hfiguiere OK, so what about rewriting an URL from RawGit? You can control the content served from RawGit pretty easily, and all files served from RawGit are under cdn.rawgit.com origin. All apps using RawGit CDN can be attacked.

@jspenguin2017
Copy link

Nope, it ended poorly: https://www.bleepingcomputer.com/news/security/adblock-plus-filters-can-be-exploited-to-run-malicious-code/

@uBlock-user uBlock-user changed the title ABP's $rewrite filter option - discussion ABP's $rewrite filter option Apr 16, 2019
@rain-1

This comment has been minimized.

@Rudde

This comment has been minimized.

@uBlockOrigin uBlockOrigin locked and limited conversation to collaborators May 3, 2019
@gwarser
Copy link
Member

gwarser commented Sep 16, 2021

@KiARC
Copy link

KiARC commented Oct 4, 2023

Since this was marked as duplicate elsewhere, I'll mention it here: What if the syntax to rewrite a URL only functioned in My Filters, and was not parsed anywhere else? This would protect users from malicious list maintainers while still allowing people to do this. An example I have for a use case is redirecting requests to ajax.googleapis.com/ajax/libs/jquery to cdnjs.cloudflare.com/ajax/libs/jquery to avoid the Google API call.

@gorhill
Copy link
Member

gorhill commented Oct 4, 2023

only functioned in My Filters

uBO is mature software at this point, and my focus is to do work on code which help filter list maintainers. What you ask can already be accomplish by specialized extensions out there.

@KiARC
Copy link

KiARC commented Oct 4, 2023

That's fair, really Request Control is the right extension for this, but unfortunately uBO has a habit of breaking literally any other network filtering extension I install no matter what order they're installed in. In this case, if I block the Google API request and allow the Cloudflare one, uBO blocks the whole request before Request Control can rewrite it, and if I allow both calls uBO inexplicably forwards the request to Google without letting Request Control interact with it, at least as far as the logger says.

@gorhill
Copy link
Member

gorhill commented Oct 4, 2023

I block the Google API request and allow the Cloudflare one, uBO blocks the whole request before Request Control can rewrite it

Blocking has precedence over redirecting, regardless of extension order of processing. Just don't block network requests you want to be redirected to a different server.


uBO inexplicably forwards the request to Google

uBO doesn't "forward" requests, it merely do not block them. Requests which are not blocked by one extension can be freely redirected by another one. If it's not working, you need to investigate your other extension as to why it's not working.

@KiARC
Copy link

KiARC commented Oct 4, 2023

That's new information to me, thank you, but when I allow it in uBO for Request Control to redirect my log seems to show a request to Google and then a request to Cloudflare, which suggests to me that the request is not being redirected but repeated. This behavior is not caused by Request Control, as when uBO is off it does not happen, as far as I am able to tell. Is there something I'm overlooking?

EDIT: I'll try logging the network requests at the OS level to see if maybe it's just an issue with how the browser is reporting them.

@gorhill
Copy link
Member

gorhill commented Oct 4, 2023

my log

What is that "log"? If it's uBO logger, then it's normal it's showing you a network request to Google not being blocked (if that is the one you want to be redirected), because everything shown in the uBO's own logger is from the point of view of uBO, and if the request is not blocked by uBO, then the logger will show it as not being blocked. Use the browser dev tools to see the final outcome of which network requests are blocked/redirected/allowed.

@KiARC
Copy link

KiARC commented Oct 4, 2023

I initially used the uBO logger and saw that, and then checked the network tab in DevTools and was getting inconsistent results. Like I said, I'll try to examine it better.

@YoshiTabletopGamer
Copy link

YoshiTabletopGamer commented Oct 16, 2023

only functioned in My Filters

uBO is mature software at this point, and my focus is to do work on code which help filter list maintainers. What you ask can already be accomplish by specialized extensions out there.

Halfway there (after slash) gorhill/uBlock@2e4525fe3c 🤷
See #2229 (comment)

@stephenhawk8054
Copy link
Member

work on code which help filter list maintainers.

@YoshiTabletopGamer
Copy link

YoshiTabletopGamer commented Oct 16, 2023

work on code which help filter list maintainers.

I'm just saying a lot of features have been added without that much community input. Ublock Origin is not mainly and should not be mainly something like a user script extension. Sure, userResourcesLocation exists, but Ublock Origin had the benefit of getting filterlists from absolutely any source (with the exception of userResourcesLocation) without worrying about some potentially dangerous script execution (Don't understand this wrong, I'm not arguing against adding actual useful features which are necessary)

@stephenhawk8054
Copy link
Member

stephenhawk8054 commented Oct 16, 2023

So what should filter list maintainers do when a large website with large users target uBO specifically and change scripts everyday and quick fix list cannot keep up?

Then hundreds of users come to uBO reddit and github to complain every day?

Can you help us managing those users? Can you help us update the filter lists when the website updates while it's 1AM-3AM and filter lists maintainers cannot stay up that late?

@YoshiTabletopGamer
Copy link

YoshiTabletopGamer commented Oct 16, 2023

Like Yuki said,

One reason I love and trust uBO is that he always takes serious consideration of security and privacy #46 (comment). Those who don't and take every nice or convenient feature demanded suffer vulnerability. Prevention is better than cure.

Which is better, feature creep and increased attack surface, or having to update filter lists more frequently

Can you help us update the filter lists when the website updates while it's 1AM-3AM and filter lists maintainers cannot stay up that late?

Are they all in the same timezone?

If we must do this to defeat anti blockers, we need to think carefully, mitigating the attack surface. The adding of trusted scriptlets to UbO must be done carefully. I'm talking about the attack exposure.

@YoshiTabletopGamer
Copy link

YoshiTabletopGamer commented Oct 16, 2023

The community is pressuring the maintainers for quickly defeating anti blockers, but I still believe we need to justify new trusted filters and as I previously said, we must not trade security for functionality. See #2229 (comment)

@gorhill
Copy link
Member

gorhill commented Oct 16, 2023

Halfway there (after slash)

As opposed to rewrite, only uBO's own lists can use it. The trusted framework didn't exist back then in uBO.

@YoshiTabletopGamer
Copy link

YoshiTabletopGamer commented Oct 16, 2023

Hi, gorhill. Sorry for the rant, but I never expected anti blockers to force us to add fully custom filters Edit: (I mean trusted scriptlets). See #2229 (comment)

@stephenhawk8054
Copy link
Member

stephenhawk8054 commented Oct 16, 2023

Then the best way for gorhill is to abandon the project and let whoever can take those ideas do whatever they want. I'll support that decision with both hands and I could be able not to waste my time dealing with filter lists and users any more.

If you have any ideas to both develop the way you think it's better and to counter with the codes website changes everyday, just develop and PR it. Talking sarcastically and literaturely without any actions to contribute just wastes time on both sides.

If that's difficult, you can voluntarily help us update the filter lists and talk with users out there. Is it ok?

@uBlock-user
Copy link
Contributor

I never expected anti blockers to force us to add fully custom filters.

You must be new here then. As new tricks and ways appear, uBO continues to develop new features too. This has been ongoing for last 5 yrs and beyond...

@gorhill
Copy link
Member

gorhill commented Oct 16, 2023

I never expected anti blockers to force us to add fully custom filters

I don't understand what that means. Can you clarify?

@stephenhawk8054
Copy link
Member

In case you don't know, in the last uBO stable release, uBO survived that large website anti blockers solely due to 1 of the trusted scriptlets for weeks before new release could come out while there was litetally no other ways to counter with other scriptlets.

@YoshiTabletopGamer
Copy link

YoshiTabletopGamer commented Oct 16, 2023

I never expected anti blockers to force us to add fully custom filters.

You must be new here then. As new tricks and ways appear, uBO continues to develop new features too. This has been ongoing for last 5 yrs and beyond...

Do not misunderstand! I have known for years scriptlets have been essential to uBO. I'm talking about the trusted scriptlets. By "fully custom filters" I mean filters that can now do pretty much the same things a user script can do. It's now clear that the evidence for usage of trusted filters has been growing. See also #2229 (comment)

@uBlock-user
Copy link
Contributor

I'm talking about the trusted scriptlets.

I mean filters that can now do pretty much the same things a user script can do

Necessity is the mother of invention. There came a time when we cannot limit ourselves, hence the concept of trusted sources came. Also this concept arrived in AdGuard first fyi.

As for "filters that can now do pretty much the same things a user script can do", it's been like this even before trusted sources came with the help of advanced setting called userResourcesLocation where you can link multiple user-scripts and make filters out of them.

@YoshiTabletopGamer
Copy link

YoshiTabletopGamer commented Oct 16, 2023

I know, I myself use medium mode and am an "Advanced User". I just don't use a userResourcesLocation (not an argument)

@uBlock-user
Copy link
Contributor

uBlock-user commented Oct 16, 2023

The userScript point of yours has nothing to do trusted sources concept which you were attributing the reason. That's my point. Once userResourcesLocation came into existence, many things became possible. trustedSources is now one more step in opening many more possibilities which were still not available to filter lists maintainers.

@YoshiTabletopGamer
Copy link

The userScript point of yours has nothing to do trusted sources concept which you were attributing the reason. That's my point. Once userResourcesLocation came into existence, many things became possible. trustedSources is now one more step in opening many more possibilities which were still not available to filter lists maintainers.

Yes, I still remember when the subreddit was once filled with advice on how to bypass Twitch ads.

@gorhill
Copy link
Member

gorhill commented Oct 16, 2023

feature creep

This is not "feature creep", what is being done is necessary to fulfilled user expectations from their content blockers given how websites evolved. "Feature creep" would be warranted if uBO become large inefficient piece of software, but that is not the case, I am confident that it's still the most efficient content blocker by a good margin given it's capabilities.

image

The added features are for the most part not visible to users, i.e. we are not adding buttons and widgets in the user interface, we are adding filtering capabilities in order to improve content blocking in way that is seamless to users, and a testament to this is that vast majority of people out there (before Youtube crackdown anyways) have no idea that the "seamless" aspect is due to filter list maintainers constantly working behind the scene, with the capabilities offered by the filtering engines in uBO.

It seems at this point you are arguing against evolving uBO to keep up with websites, arguing for stagnation which will lead to irrelevancy. If you have better solutions to the issues we are facing, best to suggest something actionable and concrete, so far you are not proposing anything actionable beside what is essentially "stop working on uBO".

@YoshiTabletopGamer
Copy link

I'm not arguing against Ublock Origin's performance, I just had this (unlikely, but potentially (in the very, very long term, actionable) worry here). I'm not arguing for stagnation in any way, shape or form.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
declined declined discussion weighing in community's input on a specific topic
Projects
None yet
Development

No branches or pull requests