Syntax improvement project

[DandelionSprout]

So in the past few months I had begun to notice that there were many issue reports on GitHub about substantial problems with list version syntaxes, so I decided to begin volunteering to try to fix it, which is currently going pretty poorly even after I got pull/merge privileges on GitHub. So as far as I am personally concerned, it's time to make a solid checklist so I won't have to check in on 20 different issue reports to remember what I wanted to fix, and to easier show what I need help from Ador and badmojr with.

Stage 1 (Most important)

• Become able to see the [Travis / Azure Pipeline / whatever else] logs for the list generator script(s).
• Make the list generator recognise changes made to build.sh and filter.sh scripts on GitHub.

Stage 2

• Remove entries with double-dots. (Domain format for mutiple lists contain invalid entries (including HTML) #547, Bad entries in some hosts files #531)
• Remove entries with backslashes in them. (Invalid hosts in list? #561, Bad entries in some hosts files #531, Unbound failed to load with Ultimate #326, plausibly dnsmasq throws up an error about a line in the block list #219)
• Remove entries that end with dots (Domain format for mutiple lists contain invalid entries (including HTML) #547, Bad entries in some hosts files #531, [xtreme | rpz] wrong record format #529)
• Remove entries containing "<", ">", " ", and other HTML elements (Domain format for mutiple lists contain invalid entries (including HTML) #547, Bad entries in some hosts files #531)
• Add an [Adblock Plus n.n] header to the adblocker list versions. (Please add an Adblock Plus version with [AdBlock Plus 1.1] version TAG #358, Errors. #172)
• A way to edit the GitHub Readme without having it be reverted 6 hours later.

Stage 3

• An easier and more direct way to submit whitelistings. (Question regarding blocklist filtering #537, graph.qq.com #558, Remove cloudfront.net from Xtreme extension #533 www.abebooks.co.uk #527, etc.)
• Implement a max-length of 240~247 characters per entry for RPZ list versions. ([ultimate - rpz format] Domainname length overflow #526)
• Implement a max-length of 63 characters per entry for Unbound list versions. (Some listed domains exceed IDN's maximum length #477)
• Combine the IPv4 and IPv6 versions of dnsmasq into one version. (dnsmasq IPv4 and IPv6 format #246)

Website-related bonus stuff

• Possibly create separate Telegram servers for issue reports and Rose-bot button clicks.
• https://app.energized.pro/ is down. (app.energized.pro seems to be down #559)
• All watchdog-(...).energized.pro image files are down, causing https://energized.pro/check/ to display false positives for all lists.

[ghost]

Maybe adding to stage 2:

• evaluate source reliability / quality , e.g. Airelle's as pointed out here Domains misclassified as porn #552 (comment) | us.dyn.indeed.com #538 (comment) | www.e-junkie.com #460 (comment) | h2.shared.global.fastly.net  #556 (comment) | au.download.windowsupdate.com.c.footprint.net #352 (comment) | False-positive host lxer.com #563 (comment)

[DandelionSprout]

Okay, so it turns out that the extent to which the GitHub files are ignored by Energized, is far bigger than I initially believed (My initial beliefs was that I had somehow broke something when writing #560).

In particular, #530 still hasn't gone live after more than 3 weeks, which not only means that Energized is still blocking Falun Gong domains due to the political beliefs of neoHosts Full; but also that the past 48 hours of my life that I spent on this repo was a complete waste.

Either Ador needs to set a lot of things straight here within the next few days and explain how (and/or make it easier) to contribute to Energized, or I'm leaving this project.

[kah0922]

I think this is probably the best place to mention this, but the adblock lists need to be simplified. For example, in the blu list, there is the rule ||pushwoosh.com^, yet there are 66 other entries in that list that all end with pushwoosh.com, even though the adblock syntax would block all of it due to the ||pushwoosh.com^ rule. An even worse example is ||siteintercept.qualtrics.com^, where subdomains of that site account for 771 rules, and that's not accounting for the 4327 hits for instances of qualtrics.com domains, which at that point you might as well just add a rule for ||qualtrics.com^.

Edit: Also there's a whole bunch of instances where there's ||badsite.com and then ||www.badsite.com. Once again, adblock syntax takes care of this. There are 11,517 instances of ||www. in the blu list, and they're all completely redundant rules.

[DandelionSprout]

Okay, so with Ador having been AFK for a month and a half now without doing even cursory checks of reported issues (Many of them very critical), my patience is done for. There is no way I can salvage a project whose salvation methods can only be accessed by one man who is AFK for weeks or months at a time.

My involvement as an Energized Protection team member is hereby over.

[ghost]

Too bad, really, but cannot be helped then. It seems that even just reporting false positives a/o syntax issues is a waste of time under the circumstance.

[AvinashReddy3108]

Okay, so with Ador having been AFK for a month and a half now without doing even cursory checks of reported issues (Many of them very critical), my patience is done for. There is no way I can salvage a project whose salvation methods can only be accessed by one man who is AFK for weeks or months at a time.

My involvement as an Energized Protection team member is hereby over.

We'll be sad to see you go away like that. How about you and I work together while Ador's AFK?
Feel free to ping me on Telegram: @AvinashReddy3108 is my username there.