Custom filter lists in Shields don't respect Expires: field

[Furax-31]

Description

Custom filters don't update automatically

Actual result:

Custom filters don't update automatically. You have to force the update manually.

Expected result:

Filter lists should update automatically.

Reproduces how often:

Add a list manually, after 3 days it is still not updated

Desktop Brave version:

All nightly build since 1.31.16

[antonok-edm]

@Furax-31 thanks for reporting! Given how recently the subscriptions feature landed, I imagine you're looking at a list with a shorter update interval specified. The lists should update automatically after 7 days. Unfortunately, at the moment, the browser doesn't check any of the list metadata, including the Expires field where a list maintainer can customize this interval - that's definitely planned as a followup improvement. There are some clever load-balancing algorithms in other adblockers which we'd probably want to respect as well (e.g. https://adblockplus.org/blog/downloading-a-file-regularly-how-hard-can-it-be).

I'll leave this issue open to track support for the Expires field, since I hadn't already opened one yet.

On the other hand, if the lists aren't updating soon after 7 days have passed, please let us know as well.

[stephendonner]

Verification PASSED using

Brave | 1.57.36 Chromium: 115.0.5790.171 (Official Build) beta (x86_64)
-- | --
Revision | a4bb683f14dc402f5da13707e13559dc4d5ef08b
OS | macOS Version 11.7.9 (Build 20G1426)

Case 1 - PASSED

Steps:

1. installed 1.57.36
2. launched Brave
3. opened brave://settings/shields/filters
4. pasted https://raw.githubusercontent.com/miyurusankalpa/adblock-list-sri-lanka/master/lkfilter.txt into the Enter filter list URL textbox, and clicked Add
5. closed Brave
6. opened Local State from /Users/sdonner/Library/Application Support/BraveSoftware/Brave-Browser-Beta and confirmed expires is 120
7. examined the values for last_successful_update_attempt and last_update_attempt, which were both 13335905915841497
8. subtracted 417600000000 (from the example in the test plan) from the above 13335905915841497 and got 13335488315841497
9. relaunched Brave
10. confirmed 5 days ago timestamp in brave://settings/shields/filters
11. waited 5 minutes
12. confirmed the timestamp didn't update
13. closed Brave
14. subtracted 14,400,000,000 microseconds from 13335488315831497 and replaced the values for last_successful_update_attempt and last_update_attempt with 13335473915841497
15. relaunched Brave
16. confirmed timestamp still read 5 days ago
17. waited 5 minutes
18. confirmed timestamp refreshed with 0 seconds ago

step 4	step 6	step 7	step 8	steps 10-12	step 14	step 16	step 18

Case 2 - PASSED

Steps:

1. installed 1.57.36
2. launched Brave
3. opened brave://settings/shields/filters
4. pasted https://raw.githubusercontent.com/miyurusankalpa/adblock-list-sri-lanka/master/lkfilter.txt into the Enter filter list URL textbox, and clicked Add
5. closed Brave
6. opened Local State from /Users/sdonner/Library/Application Support/BraveSoftware/Brave-Browser-Beta and confirmed expires is 120
7. examined the values for last_successful_update_attempt and last_update_attempt, which were both 13335911840504428
8. subtracted 431520000000 microseconds from 13335911840504428 to get 13335480320504428
9. replaced the values in step 7 with 13335480320504428
10. relaunched Brave
11. waited, and noted 5 days ago last-updated timestamp
12. after about a minute, confirmed the list auto-refreshed, with 0 seconds ago

step 4	step 6	step 7	step 9	step 11	step 12

[Uni-verse]

Verified on Samsung Galaxy S21 5G using version(s):

Brave	1.57.45 Chromium: 116.0.5845.82 (Official Build) (64-bit) 
Revision	c43241b101a636729252df90d1a1a7b72f2a9f94
OS	Android 13; Build/TP1A.220624.014; 33; REL

Note: QA to spot check custom filters/adblock list to ensure no regressions have occurred since the test plan in brave/brave-core#18836 (comment) cannot be tested on Android due to restricted access to file system.

• Ensured using ||brave.com custom filter will disable js for brave.com
• Ensured disabling shields will disable the custom filter list and re-enable js on brave.com
• Ensured enabling Bypass Paywalls Clean Filters in adblock filters is removing paywall on nytimes.com`

Allow brave	Brave.com	blocking brave.com	JS disabled	Disabling shields restore site

Default	Bypass paywall enabled	With filter