Nftfw - Nftables firewall builder for Debian

The nftfw package builds firewalls for nftables. The system creates a simple and easy-to-use configuration model for firewall management. The model was created for the iptables based firewall package supplied as part of Bytemark's Symbiosis hosting package and also for Sympl, a fork of Symbiosis. The firewall is controlled using files in a directory structure that maps onto the parts of the firewall. To add a rule, you just add a file. To block an IP address with a specific set of ports, you just add a file.

nftfw doesn't need Sympl or Symbiosis, it's stand-alone and will run on any Debian Buster system or later. It should also work on other Linux distributions derived from Debian. The package is written in Python 3 (version 3.6). These days it's being maintained using Python 3.11, and it now requires at least Python 3.9.

nftfw can be installed from a Debian binary package, there is a zip file called nftfw_current.zip in the package directory containing the most recent version. For safety, nftfw needs some configuration after installation. See the installation document Install nftfw from Debian package for a how-to guide.

New in current release

For full update information see the Changelog. The current release moves from v0.9.16 to v0.9.20.

The v0.9.20 release depends on the Debian version of the Python nftables module. Before running dpkg -i on the package, do execute:

apt install python3-nftables

to make the update of nftfw painless. If you don't, dpkg will stop, and you will need to take remedial action to install things. See above (and below) for a link to the Installation document that explains how to extract yourself from this problem.

In addition, there is a change to the default settings for the nftfw file in cron.d. When installing dpkg will query whether you want to keep your value or take the released one - use the 'D' option to see the differences. You probably want to keep your version, but make the suggested change by hand.

Main changes:

v0.9.17: The systemd path and service have been told not to worry about ratelimiting starts and restarts. This can happen if nftfwedit is used in a loop to remove or add several ips. nftfw will manage and delay multiple starts.

v0.9.18: Add a flag (-g) to nftfwls to remove GeoIP output. Previously GeoIP information was always shown if the geoip package was installed.

The packaging system for building the debian package has been changed to use the more up-to-date pyproject.toml configuration file.

v0.9.19: Add additional technique for cleaning firewall database of old values. The existing system expires IP addresses after some longish period if they haven't been back to annoy the system. The idea here is to continue to block active sites, but slowly delete ones that have stopped. This system is retained.

However, we now have the rise of botnets, cloud computing and the presence of many hosting companies who simply don't care about what their customers do. The firewall database can get filled with IP addresses that have triggered a pattern match a few times but have never been back to do it again. nftfw counts 'incidents' and 'matchcounts'. Incidents are the number of times an ipaddress has triggered a match in one scan of the log files, and matchcounts record how many abuse patterns have appeared during that scan. It seems sensible to delete these addresses rather sooner than the normal long timeout period.

The new feature is run before the extant removal system. By default, it's not activated and can be turned on from config.ini. The new default settings can be found in the default config.ini file under the [Blacklist] section.

v0.9.19: I've removed my version of nftables.py from the distribution. When I started nftfw the standard library was unable to access the full capabilities of nftables so I added some code and included it in the distribution. The version supplied with python3-nftables now does what was missing and more. I've included this as a requirement for the package, but you may need to use apt install python3-nftables.

v0.9.19: Finally, I've updated documentation and done a pylint sweep using pylint for python 3.11.2, which has changed some minor bits of coding.

v0.9.20: The default time to run the database tidy code in /etc/cron.d/nftfw was incorrectly specified. It was always the intention to run it overnight. The 'doh' moment was to put the time into cron as hh mm, when it should be mm hh. This is now changed.

Add a new tool nftnetchk that compares the current firewall database IPs with the networks found in blacknets.d. If any of the IPs are part of a network that's blocked in blacknets.d , then they are not needed in the database. The manual page for nftnetchk supplies a recipe for deleting the IPs.

Features

• Easy-to-use firewall admin. Five directories control the firewall. Placing files in the directories create firewall rules configured from the file names. Two directories, incoming.d and outgoing.d, supply rules allowing access to ports for incoming and outgoing connections. These files are usually empty, but can contain IP addresses to make the rule more specific. Two more directories, blacklist.d and whitelist.d, contain IP addresses, blocking or allowing access for named addresses. These files can contain ports, modifying the action of the rule. The final directory, blacknets.d can contain files with lists of IP address ranges and makes rules that block access to all the addresses. Changing the firewall is simply a matter of making or removing a file in one of these directories. The directory contents are described in detail in the User's Guide, while the How do I... or Quick Users' Guide gives a more task oriented description.

• Automatic blacklisting. The system contains a log file scanner that uses regular expressions to detect unwanted access and then creates files in the blacklist.d directory to block access to any matched IP address. Files to scan, the relevant ports to block for the file and the regular expressions for matching are all contained in a set of files in patterns.d. Pattern files are small text files, easy to add and edit, and the system contains a method of testing them. The nftfw configuration file controls the number of matched lines needed for blocking and how long to wait before removing the IP address from the blacklist.

• Blacklisting by address range. The system may be supplied with lists of IP address ranges used to block all the addresses in the ranges. This can be used to block access to specific countries, or unwanted access from organisations.

• Firewall feedback. The blacklist scanner can be told how to scan the syslog file looking for log entries from nftables and updates the blacklist database when a blocked IP address returns, keeping it in the firewall until it stops being active.

• Automatic whitelisting. The whitelist scanner looks in the system's wtmp file for logins from users and automatically whitelists their IP address.

• Full use of nftables sets. Blacklist and whitelist rules use nftables sets, and nftfw tries not to perform a full firewall reload until it's needed. If just the blacklist or whitelist sets alter, then only the changed sets are reloaded.

• Configurable nftables template. A user editable template provides the framework for nftables. nftfw uses the template on every firewall build, using 'includes' to pull in its own rules. The use of a template allows for local changes, perhaps to support internal LAN interfaces on a gateway machine. A sample version of the template file used on my gateway machine is supplied in the source distribution.

• Editable nftables commands. Rules in incoming.d and outgoing.d use small action files that are shell scripts to create commands for nftables rules. The scripts are called with a defined set of environment variables and generate output using echo. The idea is that local tailoring should be possible and easy.

• Blacklist monitoring. The system provides a tool listing the current blacklist status. For each live entry it shows: the IP address and optionally the country of origin, the blocked ports, the date and time of the first and last access and the difference between the two times. HTML output can be generated so the data can be seen from the web. A sample PHP webpage is provided.

• Admin editing. A database editor allows admins to add and delete entries from the blacklist database.

• Initial configuration. The system comes with a fully configured set of firewall rules and is supplied with some working pattern files that are in use now keeping the bad guys out.

Other documents

All documents can be found on the web from the nftfw website.

See documents in the docs directory:

• Install nftfw from Debian package
  • Installation from the Debian package found in the package directory.
• Installing nftfw manually
  • Full installation of the system for Debian Buster or later.
• Manual Installation Instructions
  • For those who want a bare bones list of tasks.
• Installing Geolocation
  • Installing Geolocation, adding country detection to nftfwls, which is optional but desirable.
• Getting CIDR lists
  • How to get CIDR files for use with the blacknet feature.
• Using fail2ban with nftfw
  • How to configure fail2ban to use nftfw as its firewall.
• sympl-email-changes - changes to Sympl bullseye email installation
  • I've added a repository that steps through the changes I make to the standard exim4/dovecot systems on Sympl to improve feedback and detection of bad IPs.
• Updating nftfw
  • How to update a manual installation of nftfw.
• User's Guide
  • The full User guide, the first section explains how the system is controlled.
• How do I.. or Quick User's Guide
  • Answers a bunch of questions about the system.
• Manual Page index
  • Manual Page index

Request for help

I wanted to do this because I like the simplicity and ease of controlling the firewall. The control system lacks danger, messing with complex tables isn't needed to add or remove a rule, you just create or delete a file. Controlling things using the file system is very much part of the UNIX ethos that I embraced willingly many years ago. The user interface to this system is entirely down to the efforts of Patrick Cherry who ran Bytemark, a hosting company in the UK who I used for many years.

Most of what I understand about firewalls has been picked up over the years, largely from folklore. Mine seem to work. However, there may be glaring errors in what this system delivers, helpful suggestions are always welcomed.