Skynet blocking GitHub access by "forgetting" existing whitelist entry

[TheStork239]

Brief Description Of Issue

Every few days Skynet blocks access to Github.com (e.g. IP 140.82.121.3) due to Github IPs erroneously ending on an IP block list, but somehow the existing whitelisting of that IP does not get checked or executed by Skynet.

Logs contain blocks of e.g. 140.82.121.3, but when trying to add that IP to the whitelist Skynet notifies that it has already been added.

Therefore it seems like Skynet is somehow missing putting the whitelist entries for GitHub into action.

(I am not the only one with the issue; there seems to be a number of people with this issue on discussion forums)
I am using an RT-AC88U router with Merlin 386.3_2 firmware, with Diversion running in addition to Skynet.

Steps To Reproduce Issue

Seems to occur randomly every few days. Issue seems to be temporarily fixed by either 1) disabling Skynet off, 2) manually refreshing whitelist entries, or 3) rebooting router. However, that is only a temporary fix and the issue seems to recur.

Expected Behaviour

Github.com and related IPs are already on the built-in Skynet whitelist, so access to GitHub.com should not be blocked by Skynet.

Output of ( sh /jffs/scripts/firewall debug info )

[Adamm00]

I had some ideas for whitelist improvements but it got put on the back-burner. This is nothing more then an (annoying) false positive from firehol lvl3. You can temporarily remove this from your filter list. I’ll see about other whitelist improvements when I get a chance.

[p3yot3]

Thanks for reporting this issue - I thought it was just me !

[Adamm00]

I've pushed v7.2.6 f0cea07

This will add GitHubs asn to Skynet's whitelist to avoid any IP changes. I have plans to make the whitelist more accurate with domains, but it will take some time to implement as my free time is limited.

As for some of those commenting on SNB, I find it distasteful. I've provided free support and the groundwork for countless addons for 7+ years. Rather then take shots at someone whose not around to defend themselves, create a pull request and remember this is an "open source" project.

[TheStork239]

Thank you @Adamm00 ; really appreciate your help and support. I’ll give this fix a try later on.

[dave14305]

As for some of those commenting on SNB, I find it distasteful. I've provided free support and the groundwork for countless addons for 7+ years. Rather then take shots at someone whose not around to defend themselves, create a pull request and remember this is an "open source" project.

Will do!

[p3yot3]

Updated & so far so good. Thanks for all your hard work - it is very much appreciated by most of us!

[TheStork239]

Updated and seems to do the trick. Thanks! I’ll close the issue.

[entrop1x]

I've updated Skynet to v7.2.6 and the 140.82.121.3 ip still ends up getting blocked.

I had to add it to Whitelist, but even I don't know how long that'll work.
I think this issue should remain open.

[TheStork239]

Ok. I’ll re-open the issue as it still seems to be causing trouble for some.

[TheStork239]

I can confirm that even with v7.2.6 GitHub.com does from time to time still get blocked. It seems to be fixed (temporarily) with a manual refresh of entries in the whitelist, through Skynet command line. It appears as if somehow Skynet “forgets” the whitelist until the entries are refreshed.

[dave14305]

I believe the problem is some kind of rate limiting by bgpview.io. If you remove the & at the end of line 879, it works more reliably, albeit slower.

IPSet_ASUS/firewall.sh

Line 879 in f0cea07

curl -fsL --retry 3 --connect-timeout 3 "https://api.bgpview.io/asn/$asn/prefixes" | grep -oE '.{20}([0-9]{1,3}\.){3}[0-9]{1,3}\\/[0-9]{1,2}' | grep -vF "parent" | grep -oE '([0-9]{1,3}\.){3}[0-9]{1,3}\\/[0-9]{1,2}' | tr -d "\\" | awk -v asn="$asn" '{printf "add Skynet-Whitelist %s comment \"CDN-Whitelist: %s\"\n", $1, asn }' &

[Adamm00]

I believe the problem is some kind of rate limiting by bgpview.io. If you remove the & at the end of line 879, it works more reliably, albeit slower.

IPSet_ASUS/firewall.sh

Line 879 in f0cea07

curl -fsL --retry 3 --connect-timeout 3 "https://api.bgpview.io/asn/$asn/prefixes" | grep -oE '.{20}([0-9]{1,3}\.){3}[0-9]{1,3}\\/[0-9]{1,2}' | grep -vF "parent" | grep -oE '([0-9]{1,3}\.){3}[0-9]{1,3}\\/[0-9]{1,2}' | tr -d "\\" | awk -v asn="$asn" '{printf "add Skynet-Whitelist %s comment \"CDN-Whitelist: %s\"\n", $1, asn }' &

Thanks, thought we got past this months ago but I guess bpgview increased their rate limiting again. Will look into it when I get some free time this week.

[macexx]

Though even if bgpview has a harsh rate limit there will still be two issues.

1, BGPview only pulls the registered CIDR´s from IANA for given AS.

So in githubs case there is quite a lot of addresses used that is not covered(different Data centers, subsidiaries, ISPs, sub contractors, remote APIs etc,.)

For an example using an vpn to US endpoint virgina(cloudflare dns) and trying to build a docker container(to not get IP of a CDN)
a lookup gave me "185.199.108.0/22 AS54113" which is not covered by bgpview but by githubs own list.

So should probably be pulled from github itself with something similar to what is done for amazon,
Will add a lot of duplicates but ipset will sort that out anyways.

curl -fsL --retry 3 --connect-timeout 3 https://api.github.com/meta | grep -oE '([0-9]{1,3}\.){3}[0-9]{1,3}/[0-9]{1,2}' | awk '{printf "add Skynet-Whitelist %s comment \"CDN-Whitelist: Github\"\n", $1 }'

2, Why is Github blacklisted in the first place?
Firehol lvl3 was suppose to be a 30 day list but firehol as a hole is not maintained any moore couple that with some sources it pulls from that is not maintained and more importantly it pulls from vxvault, a superbad combo.

So if if some code is uploaded to vxvault for testing from a source github, discord etc., and it can check that the file is "hosted"., and it contains malicious code etc., it will blacklist that IP. (and this list is not actively checked at all so a false positive wont get removed until it falls from the list which seem to be along long time).

This is probably why github is on the list and a huge amount of discord IPs.

BAD BAD VERY BAD
Vxvault uses "X-Forwarded-For" from the upload, to determinate the IP to block.

As long as your packet doesn't travel through a load-balancer/proxy that changes the value of "X-Forwarded-For" header, it will remain.

"X-Forwarded-For" can easily be changed by any client, so you can easily set "8.8.8.8(google dns)" upload some malicious code and get google dns banned from the lvl3 list.

Well you need a login on vxvault to do all this which probably explains why there is quite a low number of banned IPs on that list.(though found an addon for discord that has this pre entered which probably explains all the discord IPs on that list)

So this list should probably be removed and add ciarmy http://cinsscore.com/list/ci-badguys.txt since its what the list pulls its majority from any ways.

[Adamm00]

Though even if bgpview has a harsh rate limit there will still be two issues.

1, BGPview only pulls the registered CIDR´s from IANA for given AS.

So in githubs case there is quite a lot of addresses used that is not covered(different Data centers, subsidiaries, ISPs, sub contractors, remote APIs etc,.)

For an example using an vpn to US endpoint virgina(cloudflare dns) and trying to build a docker container(to not get IP of a CDN) a lookup gave me "185.199.108.0/22 AS54113" which is not covered by bgpview but by githubs own list.

So should probably be pulled from github itself with something similar to what is done for amazon, Will add a lot of duplicates but ipset will sort that out anyways.

curl -fsL --retry 3 --connect-timeout 3 https://api.github.com/meta | grep -oE '([0-9]{1,3}\.){3}[0-9]{1,3}/[0-9]{1,2}' | awk '{printf "add Skynet-Whitelist %s comment \"CDN-Whitelist: Github\"\n", $1 }'

2, Why is Github blacklisted in the first place? Firehol lvl3 was suppose to be a 30 day list but firehol as a hole is not maintained any moore couple that with some sources it pulls from that is not maintained and more importantly it pulls from vxvault, a superbad combo.

So if if some code is uploaded to vxvault for testing from a source github, discord etc., and it can check that the file is "hosted"., and it contains malicious code etc., it will blacklist that IP. (and this list is not actively checked at all so a false positive wont get removed until it falls from the list which seem to be along long time).

This is probably why github is on the list and a huge amount of discord IPs.

BAD BAD VERY BAD Vxvault uses "X-Forwarded-For" from the upload, to determinate the IP to block.

As long as your packet doesn't travel through a load-balancer/proxy that changes the value of "X-Forwarded-For" header, it will remain.

"X-Forwarded-For" can easily be changed by any client, so you can easily set "8.8.8.8(google dns)" upload some malicious code and get google dns banned from the lvl3 list.

Well you need a login on vxvault to do all this which probably explains why there is quite a low number of banned IPs on that list.(though found an addon for discord that has this pre entered which probably explains all the discord IPs on that list)

So this list should probably be removed and add ciarmy http://cinsscore.com/list/ci-badguys.txt since its what the list pulls its majority from any ways.

Thanks ce999d8

Will keep the rest in mind.

[dave14305]

add Skynet-Whitelist 140.82.112.0/20 comment "CDN-Whitelist: Github"
This should be capturing the 140.82.121.3 address.

[dave14305]

There are overlapping ranges (many smaller CIDRs within the /20 I mentioned above). Does that matter?

[Adamm00]

I can confirm that with the latest v7.2.7, Github is still getting blocked. As is DuckDuckGo. @Adamm00 I hope you have the time and persistence to make this go away once and for all. Best wishes.
thelonelycoder

Mind giving me examples of blocked addresses so we can identify what’s going on?

[decoderman]

I can confirm that with the latest v7.2.7, Github is still getting blocked. As is DuckDuckGo. @Adamm00 I hope you have the time and persistence to make this go away once and for all. Best wishes.
thelonelycoder

Mind giving me examples of blocked addresses so we can identify what’s going on?

nslookup only gives me one IP. And since I restarted Skynet it will not be blocked until it happens next time:
Non-authoritative answer:
Name: duckduckgo.com
Address: 40.114.177.156

Updated to your attempted fix and it immediately blocked DDG. Now I get:
Non-authoritative answer:
Name: duckduckgo.com
Address: 191.235.123.80

[Adamm00]

So...

https://i.imgur.com/QaugUKJ.png
https://i.imgur.com/wUnkZla.png

But this will take some time to get right. It is linked to many moving pieces and depends on how reliable dnsmasq is.

[macexx]

I can confirm that with the latest v7.2.7, Github is still getting blocked. As is DuckDuckGo. @Adamm00 I hope you have the time and persistence to make this go away once and for all. Best wishes.
thelonelycoder

Mind giving me examples of blocked addresses so we can identify what’s going on?

nslookup only gives me one IP. And since I restarted Skynet it will not be blocked until it happens next time: Non-authoritative answer: Name: duckduckgo.com Address: 40.114.177.156

Updated to your attempted fix and it immediately blocked DDG. Now I get: Non-authoritative answer: Name: duckduckgo.com Address: 191.235.123.80

Well this is expected behavior, you will get a different IP, kind of how the internet works with CDN´s / Load balancers.
(Thou if you use something like stubby with an DNS list this will exponentially give you more IP lookups since the DNS servers are always called from different locations)

So if you white list a domain it will whitelist the "DNS look up IP" when added, so if this changes on your next DNS lookup(haven't checked the code for when it updates) it "might" get blocked if the new IP is in a block list.

This is why whitelisting of domains get super tricky with an "block list" which Skynet pretty much is.
(Sure you could have some code hammering DNS lookups to a list but would be resource heavy and plain stupid since there is nothing to verify against)
And this is how it should work. (So it will work on and off)

And in your case with Duckduck.
They don't have their own dedicated IP´s, but are hosted in this case on Azure(Microsofts server farm).

So you cant go about to whitelist the entire Azure since there definitely is malicious stuff on a big server cluster as that.
(Guess that ship has kinda sailed thou since AWS is white listed)

So some options:

Check what list is blocking the IP/Range and remove it (Do your own custom list, option exists in Skynet).
Thou try to find out why, might be a good reason why its blocked in the first place!!!!

White list CIDRs (or the built in AS option "check bgpview.io for AS-number" )

Stop using Skynet, (or any custom rules)
Firewalls are maintenance that's why network engineers exists(if it was setup and forget Asus would have implemented it along time ago)

,

[dave14305]

v7.3.0 breaks dnsmasq, presumably because the ipset line in dnsmasq.conf.add is too long to be handled correctly, depending on the length of the shared whitelists.

[alexeyza]

Not sure if it is related to what @dave14305 said, but as soon as I upgraded to v7.3.0 (from v7.2.8), none of my devices were able to connect to the router. I couldn't even ssh to it or stop Skynet (had no way to see any errors or logs). I ended up factory resetting the router, and reinstalling everything. Later, a couple minutes after re-installing Skynet v7.3.0, same thing happened again. (I'm using RT-AC86U)

[TheStork239]

Same here @alexeyza; shortly after upgrading Skynet to 7.3.0 my router (AC88U) went totally unresponsive, so had to factory reset and restore to a set-up prior to 7.3.0. Now works again, but don't dare to apply 7.3.0.

[Adamm00]

So apparently people have huge shared whitelists? Beyond what diversion and Skynet add by default. I’ll revert this one when I get access to a computer in a few hours unless someone wants to make a pull request in the mean time… 🤦‍♂️

[alexeyza]

Actually, my whitelists have about 20 items in them (13 items in diversion + a few in Skynet), and whatever is the default Diversion + Skynet lists. So, not sure if that's the issue.

Thanks for looking into this.

Edit:

# wc -l /jffs/addons/shared-whitelists/*
52 /jffs/addons/shared-whitelists/shared-Diversion-whitelist

[Adamm00]

Anyone with the issue what’s the output of;

wc -l /jffs/addons/shared-whitelists/*

You can then follow @dave14305 instructions to temporarily fix dnsmasq until I can revert the commit (GitHub web somehow doesn’t allow reverting commits and I’m stuck on a train for another hour or two).

https://www.snbforums.com/threads/whats-appending-this-line-in-dnsmasq-conf-add.75457/post-721350

My apologies to anyone it affected, this has been working locally for weeks without issue.

[TheStork239]

Anyone with the issue what’s the output of;

wc -l /jffs/addons/shared-whitelists/*

I have only added 5 domains manually to the whitelist, in addition to default Skynet+Diversion. The output I get from the above command (having restored from a backup to 7.2.5) is:

45 /jffs/addons/shared-whitelists/shared-Diversion-whitelist
       13 /jffs/addons/shared-whitelists/shared-Skynet-whitelist
       22 /jffs/addons/shared-whitelists/shared-Skynet2-whitelist
       80 total

[Adamm00]

Going to try tackle this again soon. From a quick test it seems a line around 1000 characters (all domains from /jffs/addons/shared-whitelists/* combined) causes dnsmasq to fail.

Jan 13 21:34:40 Skynet: [i] Restarting Firewall Service
Jan 13 21:34:40 rc_service: service 21463:notify_rc restart_firewall
Jan 13 21:34:40 rc_service: waitting "start_dnsmasq" via watchdog ...
Jan 13 21:34:51 Diversion: auto-added IPv6 entries to blocking list(s) and blacklist, this might have taken some time
Jan 13 21:34:51 Diversion: restarted Dnsmasq to apply settings
Jan 13 21:34:51 stubby[21596]: Stubby version: Stubby 0.4.0
Jan 13 21:34:51 dnsmasq[21598]: bad option at line 56 of /etc/dnsmasq.conf
Jan 13 21:34:51 dnsmasq[21598]: FAILED to start up
Jan 13 21:34:51 custom_script: Running /jffs/scripts/service-event-end (args: start dnsmasq)

I welcome anyone else to take a look as my free time is very limited recently.

[dave14305]

OpenWrt looks like it will go with the equivalent of one domain per line.
openwrt/openwrt#4667

[Adamm00]

7e374be

For those of you wanting to try this out, there's a new branch to avoid a situation like last time. I'd appreciate any feedback.

[dave14305]

7e374be

For those of you wanting to try this out, there's a new branch to avoid a situation like last time. I'd appreciate any feedback.

Depending on the number of domains people whitelist, you might consider a timeout on the Skynet-WhitelistDomains set (e.g. 86400), to prevent unnecessary growth as IPs change. And if you go that route, it’s usually a good idea to complement that with max-cache-ttl=86400 in dnsmasq.conf.add. I’m just imagining the worst-case scenario ahead of time.

I like the clever use of xargs! I learn something new every time there’s a new feature added!

[Adamm00]

7e374be
For those of you wanting to try this out, there's a new branch to avoid a situation like last time. I'd appreciate any feedback.

Depending on the number of domains people whitelist, you might consider a timeout on the Skynet-WhitelistDomains set (e.g. 86400), to prevent unnecessary growth as IPs change. And if you go that route, it’s usually a good idea to complement that with max-cache-ttl=86400 in dnsmasq.conf.add. I’m just imagining the worst-case scenario ahead of time.

I like the clever use of xargs! I learn something new every time there’s a new feature added!

Yeah I still need to work out a few details in the implementation on how to avoid stale entries and so that entries are removed correctly when the domain is no longer whitelisted. So expect to see that soon once I can confirm the implementation doesn’t break dnsmasq or anything like last time and proves reliable.

As for xargs, thanks, I still try stay creative… admittedly had to dust off a few bash cobwebs when taking a look.

[Adamm00]

@decoderman

So while stress testing this solution with 150+ domains in a new testing shard list I kept running into an issue with Diversion. It seems something doesn't sit right and causes diversion to get hung up on processing the blacklist file that swells up to 100MB+ of nonsense preventing DNS from ever starting up until the USB device is removed (or blacklist file deleted). Perhaps something worth looking into on your end?

https://i.imgur.com/FiMxpu5.png
https://i.imgur.com/PkPJNGT.png

[decoderman]

Definitely something to look into. I’ve heard of exponential growth of the blacklist before from a user or two.
Are the entries gibberish or just duplicates?

[Adamm00]

Definitely something to look into. I’ve heard of exponential growth of the blacklist before from a user or two. Are the entries gibberish or just duplicates?

https://drive.google.com/file/d/1RxIUsXVwLgfrQEHSnG71BVttBjq3cOqD/view?usp=sharing

Can't open the file in any editor without it crashing.

EDIT: It finally opened and contains 102,883,329 blank lines lol. I see how the router might crash when trying to process that.

[decoderman]

Can't open the file in any editor without it crashing.

EDIT: It finally opened and contains 102,883,329 blank lines lol. I see how the router might crash when trying to process that.

I'll get to it ASAP. Thanks for the evidence.

[decoderman]

@Adamm00 Thanks again for finding the blacklist bug in Diversion. That is a nasty one indeed. Whenever one switches on IPv6 on the router, then each empty line gets duplicated. If one switches IPv6 on and off repeatedly for some reason, all empty lines get duplicated again. Do this a number of times and one ends up with a large blacklist with lots of empty lines.

This even happens if the blacklist is initially empty...
That also explains why some Diversion users reported that it takes forever for Diversion SSH UI to load. I count the lines for the blacklists, whitelist and blocking file to display in the UI. It takes a while for wc -l to count large files. Especially if the file is 98MB empty lines as in your case.

Will release an update as soon as testing is over.

[dave14305]

7e374be

For those of you wanting to try this out, there's a new branch to avoid a situation like last time. I'd appreciate any feedback.

I've got v7.3.1 running (mostly due to the latest Cyclops threats). I don't have many Diversion domains whitelisted, so my Skynet-WhitelistDomains ipset entry in dnsmasq.conf.add has mostly the built-in hard-coded domains (e.g. entware.net, githubusercontent.com, etc.).

This made me realize that these hard-coded domains are not likely to get populated in the Whitelist ipset since most of them are resolved on the router and won't ever be queried through dnsmasq, but directly by WAN DNS servers (with default firmware settings for local resolver).

Only domains requested from a LAN client will reach dnsmasq and populate the ipset, which is good, but doesn't help the router's own whitelisted domains. Unless you nslookup bin.entware.net 127.0.0.1 to prime the ipset, for example.

[Adamm00]

Fair point, I do hate the fact we have to go back to cycling domains with nslookup and wish we had something faster like dig built-in, but I think the best middle ground here is just todo hard coded domains and not others.

Thanks for pointing this out, I do still get notifications when people post on issues etc and read them all so I apologize if I'm slow to reply as I usually wait until I have free time to work on things. Waiting on some legal hurdles on getting an AXE11000 which should be some inspiration to dig back into things. Will try push an update for this soon

[ToniToneLo]

Interesting.