Join GitHub today
GitHub is home to over 31 million developers working together to host and review code, manage projects, and build software together.Sign up
Some thoughts about Symbiosis firewall management #136
The Symbiosis firewall seems to work well, but I have issues with it, which may be caused by my lack of understanding of the nuances of ruby, I guess. Mostly I want to stop the bots that relentlessly look for logins and passwords, while at present I think that Exim is doing a reasonable job of keeping spam out.
This attempts to explain my understanding of what is happening, so please bear with me - and maybe you wlll see if I have it wrong.
I am interested in improving the action of /usr/sbin/symbiosis-firewall-blacklist which runs every 15 minutes (or so) and is responsible for creating and deleting files in /etc/symbiosis/firewall/blacklist.d. The files are named for IP addresses (and have .auto) appended to them when created by the script. IP files can be created here to put into the firewall, and are ignored by the system. The files contain port numbers, 'all' or be empty (which implies 'all') - the port numbers are loaded into iptables and block attempts to connect from the named IP to the appropriate port.
The symbiosis-firewall-blacklist script has two distinct phases:
So let's look at phase 1:
Each file in the patterns.d directory contains the name of a log file. This file is scanned looking for a regular expression match to find likely candidates for blocking. The script uses an sqlite3 database (in /var/lib/symbiosis/firewall-blacklist-logtail.db) to remember where it was in the file the last time it looked at that file. (I suspect that this is the cause of some people's uncertainty about this system. If you add a pattern which you've found in a log file, then it won't be activated until the problem happens again and re-appears in the log file so it can be seen.) However doing this makes sense for efficiency reasons.
The script uses pattern.rb to scan the log file. When matches are found it creates a two dimensional hash
So it's found some lines in a logfile matching the regular expression. Now it wants to decide if any of these entries are worthy of being blacklisted.
This happens if any of the hit counts are greater than 20 (by default), or the value supplied by the -a (--block-after). While it's doing this, it's summing all the hits for the ip and writes them to another sqlite3 database (/var/lib/symbiosis/firewall-blacklist-count.db) along with the IP and a timestamp.
It now looks to see if this IP has been really bad, and is worth blocking completely. It creates a sum of the hits recorded for this IP in the last 24 hours and if this is over 100 (default) or the value determined from -b (--block-all-after) then this IP is set up to be blocked completely for all ports.
So now if it has some candidates, it will create files with the ip address as the names and attach .auto.
Phase 2 is reasonably simple. It looks for the modification time of the file, and if it's expired it's removed.
The Phase 1 algorithm really needs improving and I would say that this is really an urgently needed change. In many cases the system is not reactive enough - and I see that people are using other systems because it's not good enough.
When looking for candidates, the system only 'knows' about the matches it has found in the current slice of the log file that it's inspecting for that pass. For my site, this is typically 3 or 6. My system is not busy so not a lot happens every 15 minutes. The window for selecting candidates is much too small - and is essentially avoidable by delaying attacks.
Second, the system completely ignores the history that it has carefully stored away in the sqlite3 file. Mostly I find that these robots come back from the same IP address often several days later. The system should make much better use of the history that it has gathered.
Third, the current system will theoretically generate port specific matches in iptables. But it's not clear how often this will happen, and frankly I don't care, I'd be happy to completely block every bad site from everything. As an aside it would be good if the code making the firewall understood about 'multiport dports' to make less lines in the filters.
The current selection code is:
I am no ruby programmer - so this is a syntactic guess but maybe this will be an improvement - without changing patterns.rb.
This may cause problems if this hits ip addresses that should be in use. There are no tools for removing good guys from the history file. Maybe this can be supplied as a small tool. Perhaps when an address is placed in the whitelist directory, the address should be removed from the blacklist database if it's there.
Here's a package that contains the changes I've made to the standard system to implement these ideas. Things altered a little from the speculation above. There's also a posting on the Bytemark forum https://forum.bytemark.co.uk/t/improving-symbiosis-firewall/2971.