Native blocklist loader for large IP lists

[ul0gic]

This came up in #117: what happens if a ban list holds millions of addresses? It deserves its own design discussion, so here it is. @mwriter, it started with your question, so your input would be especially valuable.

Why not through bans

Every ban in fail2zig is tracked individually: a durable record, who owns it, and a check against the kernel every few seconds, so fail2zig can always tell you exactly what's enforced and why. That's what makes bans trustworthy, and it's also why they have a ceiling (65,536 across the daemon). Pushing millions of addresses through that machinery would cost gigabytes of state and constant verification work, and could bring a small box to its knees.

What we're researching: a native blocklist loader

The kernel itself handles very large lists well, as long as they're loaded as a block rather than one ban at a time. The idea is to build that into fail2zig:

• a list (from a file or a URL) loads as its own set, separate from bans
• overlapping addresses and ranges are merged first, so the kernel holds as few entries as possible
• updates replace the whole set at once, so there's never a half-loaded list
• you set a size limit, and fail2zig accounts for the memory up front
• status shows each list with its source, entry count and last refresh

We think this would be a great fit for fail2zig: one tool for both detection-driven bans and large curated lists.

What would help us shape it

• Which lists you'd use (sources), and roughly how big they are
• How often they update, and whether you'd want fail2zig to fetch them or read a file you manage
• Whether some lists should apply to every jail or only to specific ports or services
• Anything you've found painful with blocklists elsewhere

This is research, not a release commitment yet. We'll post here as the design firms up.

[mwriter]

@ul0gic

Yep, that's a topic for a long discussion ;D
What's more, there is also a virtually infinite address space IPv6.
We need to think...

[mwriter]

@ul0gic

I've had first thought on this matter.
What if we implemented a mode for permanent banning and ignoring?
In other words, an user sets this mode (for example, via bantime = "inf" and ignoretime = "inf"), and after that, these jails no longer require any additional reads or checks.

[ul0gic]

@mwriter The cost you're pointing at is real. The expensive part isn't holding an entry... it's re-checking it. With tens of thousands of bans the daemon spends most of its idle time re-reading the kernel set and comparing it to what it expects. So a list that's loaded once and left alone is the right shape, and bantime = "permanent" already covers the permanent half for detection bans.

The part I'd push back on is "no further checks" meaning never looking again. If fail2zig stops looking it can't tell you the list is still enforced. An nft flush, a reboot with the wrong unit ordering, another tool rewriting the table... and the list is gone while status still says it's loaded. "What fail2zig says is enforced is enforced" is the promise worth keeping.

I was thinking we check cheaply instead of not at all. The kernel can answer for a handful of specific entries in one message without walking the set, and it broadcasts an event when a table or set gets flushed or deleted. So fail2zig keeps a hash of what it loaded, spot-checks a few entries on a slow schedule, listens for deletions, and if either says something changed it reloads the whole block in one shot and says so. At idle that's close to free even at millions of entries.

Two things we should probably keep separate. A permanent ban from detection still needs its record so unban, list and history work on it. A curated list needs none of that... a source, a count, a timestamp. Giving lists their own home seems cleaner than making a jail behave differently when bantime is infinite. Does that split match your use? Or do you want permanent detection bans to skip the checks too?

On IPv6... a list there is ranges, not addresses, and kernel sets hold intervals natively, so a list of /64s or /48s stays small. Merging overlapping ranges before loading would be on us.

Questions for you:

• Which lists would you use (files you maintain, URLs like a DROP list), roughly how large, and how often do they change?
• Global, or tied to a jail or port? My guess is mostly global.
• If a URL can't be fetched... keep the last good copy and flag it, or drop the list? I'd keep it.
• Should list hits be visible anywhere, or is "blocked, not logged" fine?

No timing promise yet, but this is high on the list.

[mwriter]

An nft flush, a reboot with the wrong unit ordering, another tool rewriting the table... and the list is gone while status still says it's loaded.

I hadn't thought about those factors. However, a reboot still means that the rules are applied again, and therefore cannot be considered such a factor.

Or do you want permanent detection bans to skip the checks too?

It seemed to me that in certain modes, it would make sense to minimize the load on system resources.
However, the question really is: which is more important - workload or security?
It seems to me that we could create a jail mode where bans are permanent, and the user can be absolutely certain that nothing will overwrite the rules that have already been added, and eliminate any re-checks for such jails.

[mwriter]

• Which lists would you use (files you maintain, URLs like a DROP list), roughly how large, and how often do they change?

After all, for now, only offline sources are supported, and that's all we're talking about. But it seems to me that this topic isn't specifically about my personal preferences, but rather about versatility ;D

• Global, or tied to a jail or port?

I still think that the entire logic is currently based on grouping by jails, and that's how it should continue to be.

• Should list hits be visible anywhere, or is "blocked, not logged" fine?

You mean logging to the internal database, right?
This seems like a separate logical point.
It seems to me that there should be a separate setting for each of these points, and every logical action related to a jail would be governed by the final value of the corresponding setting for that jail.

[ul0gic]

@mwriter Fair point on reboot... the rules come back with the daemon, so that one's off the list. Flush and other tools rewriting the table stay on it, and those are the two that happen in the field.

On workload vs security, I don't think we have to pick. The check is only expensive today because it's a full read of the set every few seconds. We cut that in half this week and the next step is to stop polling altogether and let the kernel tell us when something changes... it sends an event on every flush and delete, which costs nothing while nothing happens. Once that's in, a permanent jail gets what you're after, no periodic work, and we still know the moment something touches the rules. So I'd rather make the check free for every jail than add a mode that turns it off.

On your other three: agreed, lists stay grouped by jail like everything else. Files you manage come first, and URLs can come later. And yes, whether hits get recorded is its own per-jail setting, same as the rest.

Let's keep this post open as the design record, and I'll post here when there's something to try.