Update benchmarks to include whitelisting checks

[mjethani]

From here: #2165 (comment)

The question is why neither adblockplus nor ublock apply genericblock here?

I have always wondered if the benchmarks were entirely fair, because of all the other things that content blockers do before they finally block or allow a URL request.

Adblock Plus makes two calls to the engine before it evaluates the URL itself:

1. Check for a $document whitelisting filter for the origin URL
2. Check for a $genericblock whitelisting filter for the origin URL

@gorhill it would be nice if you could share the exact steps in uBlock Origin as well.

Now it looks like tsurlfilter already does all of this internally.

For what it's worth, I had proposed that Adblock Plus do all of these checks in a single call, but like so many things it seems to have ended up in a black hole.

Should we update the benchmarks to reflect these additional calls that content blockers must make to check for whitelisting?

[ameshkov]

Yeah, I suppose we should do that.

Also, it seems there are already some pretty serious differences that I wasn't aware of (i.e. the absence of $genericblock support in ubo), these differences won't go away.

[remusao]

So should we use the same mechanism we used to preprocess the lists and remove some filters that are not supported by all blockers? As far as I can see there is only one $genericblock filter in the lists used in the benchmark, maybe we can just remove it. Are you aware of any other discrepancy in support for options?

[mjethani]

Is it correct to assume that uBlock Origin's measurements already take exceptions into account?

The two checks that Adblock Plus does are for $document and $genericblock and both are not supported by uBlock Origin.

@remusao do Cliqz and Brave support these options?

[remusao]

@remusao do Cliqz and Brave support these options?

No.

From what I can see Brave does not support $genericblock and has some support for $document. Overall I think the goal was to stick to uBlock Origin's behavior whenever possible.

[gorhill]

The two checks that Adblock Plus does are for $document and $genericblock and both are not supported by uBlock Origin.

On the other end uBO does mind filter options not supported by ABP, they may have a cost in uBO even if when a filter list does not use them. For instance, consider the important option which requires an extra step in uBO for every network request.

In any case, the value of the comparative benchmark here for me is more about whether uBO would seriously diverge in performance from other blockers that I would have to look at what can be improved, and to make the case that webRequest-based blockers can be fast enough that performance is an invalid argument for declarativeNetRequest.

[mjethani]

… to make the case that webRequest-based blockers can be fast enough that performance is an invalid argument for declarativeNetRequest.

Not to go off topic, but I'll just say that Google and eyeo have not been doing this in good faith.

The performance improvements in Adblock Plus were phenomenal. As noted above, whitelisting logic must go over every frame in the hierarchy. This means several additional calls must be made to the matcher. The entire whitelisting logic became incredibly faster as part of the performance work. I did this so Google would not have any excuse. If I remember correctly, request blocking overall became about twenty times faster as part of this work.

Most companies would brag about such remarkable improvements to their software, but eyeo seemed to shove it all under the carpet.

It all adds up.

But these benchmarks can still be used to convince others who may still be making arguments in good faith.

[mjethani]

But these benchmarks can still be used to convince others who may still be making arguments in good faith.

To my knowledge, Google has never presented any data about performance.

But the proof is in the pudding. I stand ready to benchmark any declarativeNetRequest-based content blocker on Chrome with uBlock Origin on Firefox on the same system. We can look at the overall CPU and memory usage. We can look at various other measures. I'd be willing to bet that uBlock Origin will win hands down.

[mjethani]

Unfortunately the way the whitelisting logic works is that we need to check not only the frame URL but also the URLs of all of the ancestor frames. This means we would need the entire frame hierarchy in the data set. We do not have this at the moment.

@ameshkov why does tsurlfilter only care about the current frame? Is there a way to disable the $document and $genericblock checks in tsurlfilter for now?

We could of course remove all such filters.

[mjethani]

The parent can be a data: or about:blank frame.

Also the filter @@||example.com^$document should whitelist any example.com document, including any subdocuments. If example.com loads any frames from example.net and those frames make requests that might otherwise be blocked, they should be whitelisted too.

[remusao]

Ok, my insights are not the freshest at this point but I would assume that to allow a request from example.com loaded as an iframe the type should be $subdocument or $frame and not $document.

[mjethani]

In the above example example.com is the top frame whereas example.net is a subframe. The filter @@||example.com^$document should still whitelist requests from example.net — if it's a subframe of example.com. The target of the filter here is example.com, which is the top frame.

[mjethani]

@ameshkov why does tsurlfilter only care about the current frame?

Alright, I take it that tsurlfilter assumes that the current frame is the top frame. In this case, we can also assume the same for Adblock Plus and do a one-level check.

This does not affect any of the other content blockers based on the comments above.

Note that adding the whitelisting logic will make the results for Adblock Plus similar to those for tsurlfilter and then they'll be different from uBlock Origin, Cliqz, and Brave.

We could still remove the $document and $genericblock filters but that would be orthogonal to the whitelisting logic.

[ameshkov]

@ameshkov why does tsurlfilter only care about the current frame? Is there a way to disable the $document and $genericblock checks in tsurlfilter for now?

Please don't do that just yet. The API is not final and will probably change by the time we integrate it into the extension. I'll see to us taking care of this problem as well.

[mjethani]

This is now approved and merged here: #2172