Inconsistent blocking numbers when adding custom lists

[Mikey1993]

Got this when looking into #295 :
STR (Steps To Reproduce):
Step 1: Click "Purge all caches".
Step 2: Update the lists (with NO no regional lists) - Click "Update now" button.
Step 3: Add https://raw.githubusercontent.com/r4vi/block-the-eu-cookie-shit-list/master/filterlist.txt to the custom filter list.
Step 4: Hit "Apply" button.
Step 5: Tick the "Block-EU-Cookie-Shit-List".
Step 6: Click "Apply Changes" button.
Step 7: The list is said to be block "366 used out of 814".
Step 8: Repeat Steps 1-2.
Step 9: The "Block-EU-Cookie-Shit-List" reports to block "814 used out of 814"
Step 10: Get confused.
Step 11: Pick a regional list (ANY regional list I believe will reproduce it) (I chose ISR: EasyList Hebrew)
Step 12: Click "Apply Changes" button.
Step 13: The "Block-EU-Cookie-Shit-List" reports now that it's blocking 366 used out of 814.
Step 14: Untick the just selected regional list.
Step 15: Click "Apply Changes" button.
Step 16: The "Block-EU-Cookie-Shit-List" reports now that it's blocking 366 used out of 814.
Step 17: Get confused again. (Because the "Block-EU-Cookie-Shit-List" should be back to "814 used out of 814".... or is it?)

[gorhill]

There must be one or more filter list which content intersects with the content of "Block-EU-Cookie-Shit-List".

The lists are all loaded asynchronously, so the order in which they are processed is unpredictable. This means when there are duplicates, sometimes it's the filters from one list which will be loaded, another time it may be the filters from another list which are loaded.

[gorhill]

"Fanboy's Annoyance" intersects with "Block-EU-Cookie-Shit-List", and I get similar counts for "Block-EU-Cookie-Shit-List", sometimes 367, sometimes 814. It just depends of which list is processed first by uBlock, and this depends on which one will be fetched first from the net or locally.

[Mikey1993]

So there is no way to reliably know the amount of the unique rules that is present in a particular list to the whole loaded lists?
How this will settle down with the "search rules inside cached lists" feature that might be implemented (or other variant of it?)? Will the lists be changing on every purge + update?

How about after applying or turning off a lists, asynchronously, to check the lists for unique rules from top to bottom, and the same scenario then for the custom lists?
This will produce 100% reliable numbers, and the user won't get confused.

[gorhill]

So there is no way to reliably know the amount of the unique rules that is present in a particular list to the whole loaded lists?

That would cost quite a lot CPU- and memory-wise: this would require a two-pass load of the filters, with keeping reverse look-up filter information for each list. Loading fast has priority over information about unique filters in a list relative to the other currently selected.

[Mikey1993]

Well, hope this one will be implemented after all in the future, in spite of the CPU/Memory cost.
Thanks for the in depth explanation.