ChangeLog

v0.9.16 Fix sample pattern in patterns.d for exim4
 	Change to GeoIP CSV files

v0.9.15 Sort out DNSBL lookups to use correct return values

v0.9.14 Change Python packaging to use pyproject.toml
	Minor documentation changes to bring things up-to-date for
	bullseye and bookworm.
	Changes to the Makefile creating Html files,
	This version built on bookworm.

v0.9.13 Change default mask for IPv6 addresses from /64 to /112
	This now only masks the last block of 4 characters in an
	IPv6 address. This follows a change in the Sympl firewall
	distribution, done because it was considered that the /64
	setting is too aggressive.

v0.9.12 Reorder failure tests in nft_python.py is if there is an error
	it will be logged properly.

v0.9.11 Files in incoming.d and outgoing.d can contain IP addresses
	applying the action rule only to those addresses. Fix to
	ensure that a rule contains addresses it is only applied
	for a network protocol (IPv4 or IPv6) when an address of
	the particular type appears in the list.

v0.9.10	Missed json.schema which is part of the original python3-nftables
	module and which isn't used by nftfw. However, it seems prudent
	to include it here.

v0.9.9	Change interface to nftables to use the python3 bindings to
	libnftables. The distributed version of the python module
	doesn't have a full mapping of the library interfaces so I've
	created an edited version to allow for rule testing. I did
	first created an add-on that contained the bits I needed, but
	that felt wrong.
	The system now allows choosing between the original code
	using /usr/sbin/nft to load information, and the new
	python version.
	Add nft_select to config.ini to allow use of old code
	in case of need.

v0.9.8	Update fail2ban document.
	Other documentation changes and edits. Make am_i_root() use the
	logging interface if running in the background. Some commands are
	now running as part of system tasks. Change warning message when
	nftfw backup file exists. It now tells users what to do to allow
	backups to be taken.

v0.9.7  Change nftfwedit to allow service names in port lists (-p argument).
	Changes to nftfwedit interface to assist external automatic editing:
	  -b adds IP into database and creates file in blacklist.
        	The count & incident count are initially set to
		10/1. Subsequent use with the same ip address increments the
		counts. The 10 is the default value taken from block_after
		in the config.
	  -a adds ip into the database, but doesn't create the file.
	        Both commands now ignore any whitelisted address.
	  -r is a new option, just to remove any file in blacklist.d
                and leave the database alone.
	   -m is a new option allowing the matchcount to be supplied.
	IP address output from nftfwedit now shows ports as service names
	where they can be decoded.
	Change to cidrlists/getgeocountry to reflect changes in the
	zip file that's downloaded.
	Add fail2ban integration files

v0.9.6	Reorder ignore and established rules in nftfw_init
	The original ordering tended to output unwanted packets
	on connection close.
	Rework all the import statements to avoid use of implicit
	imports.
	Use pylint to find and clean some minor problems.
	Ensure that action scripts are run as a mortal user
	and not root.

v0.9.5	The getgeocountry script in cidrlists was broken because
	evidently MaxMind changed their packaging of the zip file.
	The script tries to get the name of their distribution
	which includes the date, and this was failing. This bug
	is fixed, and also some more error reporting was added
	to the script to report other failures.

v0.9.4	Changes in Debian packaging suggested by comments
	when version 0.9.3 was uploaded to mentors.debian.net

v0.9.3	Eventually, I'll get the links for github correct.

v0.9.2	Eventually, I'll get the links for github correct.

v0.9.1  More documentation changes to get new github links
	to work properly.

v0.9.0	Add Debian package to release
	Install debian and package directories
	Documentation changes to prefer use of /etc/nftfw

v0.8.3	Some readability changes in the rule.d directory.
	Added documentation for the Debian package

v0.8.2	Minor text changes, remove inadvertent pdf file in docs
	Recoded rule.d/icmp.sh and rule.d/icmpv6.sh
	Fix Uninstall.sh nftfw script removal

v0.8.1	Documentation update
	Create pdf's from the main documentation files
	Fix typo in Uninstall.sh

v0.8.0	Summary of changes from 0.7 requiring some reconfiguration
	It's highly likely that you will need to update your config.ini
	and nftfw_init.nft files from the distributed versions.

	a) Edit config.ini to remove:
	   [Owner] section - ownership of files created in etc/nftfw
	   now taken from owner of that directory
	   nftfw_base - nftfw now uses its own control files
	b) etc/nftfw/original renamed etc/nftfw/etc_nftfw
	c) Change to nftfw_init.nft to include essential ipv6 icmp
	   coding. Change to rule.d/essential-icmpv6.sh. Can remove
	   reference to this rule in incoming.d and outgoing.d.
	d) Updated regular expressions in exim4.patterns - now
	   finds IP addresses correctly

	Other changes:
	New import_tool to import Symbiosis/Sympl configs
	New Uninstall.sh to remove manual installation
	Many documentation changes

v0.7.14	Repair the tests directory that hadn't been updated correctly.
	Tests now create reference data if the srcdata file
	doesn't exist in the data directory.

	Change default for nftables_conf in config.ini. It now defaults to
	/etc/nftables.conf and is overriden by the distributed config.ini
	file that places it in etc/nftfw. This is safe for installing on
	a new system, while being considerably easier to explain.

	Add etc/nftfw/local.d directory to allow for locally modified rules
	matching Sympl's usage. Names in local.d will supersede files in
	rule.d. This means that rule.d can be updated when new
	installations happen. Add some missing rules to match Sympl.
	Change Install.sh to match this change.

	Remove support for using a Symbiosis/Sympl firewall configuration
	as part of the nftfw setup. nftfw can now only use its own
	files. Provide import_tool to import Symbiosis/Sympl setup into nftfw.

	Add essential ipv6 icmp rules to nftfw_init.nft before any
	blacklist whitelist checking. Otherwise if an ipv6 neighbour is
	blacklisted, everything for that address is blocked.

	Add Uninstall.sh which will remove the manually installed nftfw
	installation from the system.

v0.7.13	Want to be able to write files in etc/nftfw owned by someone other
	than root making it easier for a mortal to control. Originally
	this was done by putting the owner/group into the config.ini. It's
	going to be easier to distribute if the owner of new files
	is taken from the owner of etc/nftfw. Most of the changes for this
	are in config.py, with some changes in Install.sh.

	Insert checks at program startup to verify that the
	directories and config files exist in the right places.
	In the event of non-configuration or mis-configuration
	the scripts will now complain meaningfully.

	Decided that the use of 'etc/nftfw/original' to retain the distributed
	config files was perhaps obvious what the files were. Renaming
	this to etc_nftfw as it is in the source. For the Debian package, this
	will be a symlink to /usr/share/doc/nftfw/etc_nftfw. Changes to do
	this are in Install.sh, and will also need to put some code into
	the Debian install to recognise that 'original' might have an old
	installation set and will need deleting.

v0.7.12 Debian packaging wants to run tests from its
	working directory. Add an __init__.py to ./tests
	to allow this. Further changes to the test package
	makes it happier to run when being used by debian packaging.
	Debian will copy data file fairly easily, so the tests now
	treat this as read-only and create and destroy working
	directories.
	Add fake local logs to pattern files so tests don't look at the
	system, even when they are just looking for logs.
	Fix config problems with manual pages
           Fix titles in section 5
	   Add \- in front of the title line to keep makewhatis happy
	Change structure of etc_nftfw to make Debian package generation
	easier. etc_nftfw is now a clone of what will appear in the
	just installed /etc/nftfw or /usr/local/etc/nftfw. For current
	Install.sh this is copied to /usr/local/etc/nftfw/original,
	for Debian it will be in /usr/share/doc/original.

v0.7.11 Check and update released patterns for detecting exim4
	problems - original rules were not parsing IP addresses
	correctly. Look in newly installed
	.../etc/nftfw/original/patterns/exim4.patterns
	and update your working copy in
	.../etc/nftfw/patterns.d/exim4.patterns

v0.7.10 Ignore errors when reading UTF-8 files.
	Attempted in 0.7.7 but incorrectly implemented. RTFM.
	Replacing by backslashes only works on write.

v0.7.9 	Some minor cosmetic changes
	Make blacklist tidy be aware of timezones
	Make 'drop' the default rule for incoming table,
	may be better to slow down port scans
	Add pyproject.toml
	Add better ipaddress selection to cidrlist/getcountrynet
	Small changes to the Installation documents, some text tidying and
	some notes about installation on the Raspberry Pi OS
	Fix default regular expressions in patterns.d/exim4.patterns
	it needed to be make more specific so that HELO's with IP
	addresses are ignored in the log file and the sender address
	is used.

v0.7.8  Fix Nftfwls which didn't cope with IP addresses not found in the
        MaxMind database.

v0.7.7  Ignore UTF-8 errors in logfiles. Replace offending text
	with backslashed values.

v0.7.6  Ignore broken error in nftfwls. Happened when using
	nftfwls -a into the head command
	Fix incorrectly formatted help messages for
	nftfwedit.

v0.7.5  Expand documentation on nftfw_init.nft changing the
	How Do I document and the nftfw_files man page.

v0.7.4	Fix shell coding error in Install.sh. POSIX shells need
	explicit path names when using '.' commands unless . is in
	PATH.

	Test and supply a zsh 'workaround' to cope with the
	problem it supplies when dealing with spaces in variables.

	Make the ftp-passive more robust. It's really intended to
	be used with pure-ftp, and will need hand editing if pure
	is not installed. Now it will just generate nothing if it's not
	set up properly. Not sure if this is a good thing, will people
	look to see if it's working?

v0.7.3	Automate Install.sh as suggested by mrbluecoat on github.
	The install script now looks for Autoinstall.conf and takes
	its answers from there. This file is ignored by git so can be
	retained and used for later updates.

v0.7.2  Add -a (altered) flag to nftfw to print changes in
	config from the compiled in values. Can replace config.ini
	by the values.

v0.7.1  Add necessary change to the systemd setup to report on
	the blacknets.d directory.
	Documentation update.

v0.7.0	Add blacknets.d to etc/nftfw - containing files with lists of
	network CIDR ranges that can block countries and other
	unwanted groups based on ip range. The code understands that
	ipv4 addresses can be expressed as ipv6 addresses.
	The code checks that overlapping ranges are suppressed.

	Provide sample infrastructure to allow automatic downloads
	of GeoIP country CIDR ranges.
	Updated documentation
	Update Install.sh to flag if config.ini and nftfw_init.nft
	needs updating.

v0.6.0  Revised some comments that were a little obscure.
	Add new entry to the How Do I Document
	Replace incron use by systemd path service
	Revise scheduler to remove incron flag which isn't needed.
	Replace use of incron by a systemd path service
	which seems to do the job.
	Code changed to remove incron flag and change to
	scheduler which makes it work without either incron
	or systemd.
	Incron references removed from documentation.
	Installation instructions updated

v0.5.1  Pure FTP needs to allow access to a port range on input
	for it to work. Add new rule - ftp-passive.sh to determine
	the range from /etc/pure-ftpd/conf/PassivePortRange and
	then add it to the allowed inbound ports. Put 21-ftp-passive in
	incoming.d to include the range.

v0.5.0  Tag a new release

v0.4.5  Don't use interval sets by default for blacklist and whitelist
	sets. This is a change in distributed config rather than
	functionality. Use of the system has found that certain
	national actors frequently use adjacent IP addresses and
	this triggers the nft bug causing table reloads to fail.

v0.4.4  v0.4.2 removed intervals from sets as well as merge
	intervals are needed for ipv6

v0.4.3	Make the pattern reader regex compilation more robust.
	Ignore case when compiling regexes for log scanning (should
	have done this before).
	Only allow one regex group in the match, so that brackets ()
	in patterns now must be prefixed by a back slash.

	Upgraded test suite initialisation. Prevents the nftfw code
	from timing out any blacklist entries by touching the files
	in the reference 'data' directory.
	The initialise script is now called from the Makefile if the
	'data' directory is empty.

v0.4.2  Added blacklist_set_auto_merge and whitelist_set_auto_merge to
	config.ini.

        These two variables control the type of sets automatically
	generated for blacklist and whitelist tables. When true, nftfw
	uses auto_merged, interval sets for the blacklist or whitelist
	sets it makes. This set type automatically create single entries
	containing an address range for adjacent IP addresses. The feature
	is desirable because it reduces the number of matches.

        However, at present, the auto-merged, interval sets can cause the
	nft program to fail, flagging an error. There is a bug causing nft
	to succeed in loading the set when a full install is performed but
	failing when attempting a reload.

        The bug has been reported to the nftables development team, but no
	fix has been generated as of the current releases. nftfw will work
	around this bug, automatically generating a full install when an
	attempt at a set reload fails. However, it seems a good idea to
	provide a way of turning this feature off.

	Added hostname printing into the print function of nftfwedit, made
	optional using a command line option, because it can be slow
	when printing many addresses with no hostname.



v0.4.1  Removed reject-www-data from default outgoing setting
	advice says that this causes more problems than it solves

v0.4.0  Change the README.md file, lots of what was there is
	now in other files.

v0.3.4  More documentation changes, make the geoiplocation code more
	robust.

v0.3.3  Change internal database interface slightly
	Add reset of missing files in blacklist.d, this has never happened
	but might.
	Update documentation

v0.3.2 	New document 'Installation Instructions'
	Augment logic in fwmanage. When adding a new IP into an set that
	creates a range, the partial load fails. Now test the partial load
	and do a full install when needed

v0.3.1  Put back the code that allows nftfwls to run
	without the python geoip2 package.
	Typo fixes in documents suggested by hairydog
	Changes to the Installation document
	New document 'Installing GeoLocation'
	New document 'Updating nftfw'

v0.2.3	Small change to the Installation document
	Added extra tests to test_04

v0.2.2  Added a set of pytest tests, which pulled out some omissions
	in checking, rather than actual bugs. See README files in
	the test directory. Added a Makefile to make linting the main
	source simpler.
	Tests needed a standalone bootstrap system, added to makefile
	Changed the way that ftp-helper is implemented

v0.2.1  Just when it looked fine, noticed an omission in the ipv6
	setup.

v0.2.0  Make a release - it seems to be functioning as expected


v0.1.9  Add some frequency stats to nftfwedit and log values from
	blacklist - puts some of the number into context

v0.1.8	Redo the management of Debian logger and how config.py interacts
	with other modules. New module loggermanager.py does this, and
	provides better control of what is happening.

	Large name changes to bring the code up to pylint standard, except
	I allow 1 and 2 character variable names, which are extremely
	local.
	argument-rgx=[a-z_][a-z0-9_]{0,30}$
	attr-rgx=[a-z_][a-z0-9_]{0,30}$
	also add 'log' to good-names otherwise it complains a lot
	about the log calls

	Revise commenting strategy to make is somewhat more formal


v0.1.7  Redo set naming algorithm.
	Change default blacklist action to drop which
	causes less traffic.
	Remove .patterns from argument to nftfw -p
	Fix a misconception in the nftfw_init.nft
	file.

v0.1.6	Blacklist won't blacklist addresses or networks
	in whitelist.d
	Reworked IP address checking into one place

v0.1.5  Fix typos in code highlighted by the installation
	on a vanilla Sympl system
	Update Installation document to incorporate things
 	learnt from the installation exercise

v0.1.4  Ensure glob in pattern match resolves to
	unique set of canonical files.
	Some amount of pythonification
	Add to installation documentation

v0.1.3 	Add nftfwedit to print info on IPs, and provide
	command line access to add, delete and blacklist
	ips.
	Documentation update

v0.1.2  Amend logging for blacklist detection to give an indication
	of what pattern triggered it.
	Add sample web page for nftfwls -w
	Add user control: files automatically created in etc/nftfw
	will be owned by user nominated in config.ini, this follows
	Symbiosis/Sympl practice of making files in etc/nftfw
	accessible to a non-root admin user
	Install.sh now asks for user name

v0.1.1 	Tidy links in documents
	Flatten distribution - removing original file tree for system.
	Replace 'dist' in etc_nftfw by 'original' to avoid
	problems with python package names.
	Seems like a good idea to have a ChangeLog.
	Change blacklist algorithm somewhat, realising
	that the update option needs to update times on
	files in blacklist.d, otherwise they expire because
	expiry uses file mtimes.
	Change nftfw_init.nft to add related accepts
	to the output chain, needed to be able to make
	sensible decisions about blocking connections
	later. Also tidy some of the extra packet output,
	and ensure that the lookback interface is handled
	properly.
	Documentation fix to How_do_i.
	Make Install delete files in etc/nftfw/original
	Add dropcounter to drop rules
	Add HTML output for nftfwls

v0.1.0  Move to github