High %CPU

[mwriter]

@ul0gic

# fail2zig status --details
+-------------------------------------------------------------------------------+
| fail2zig 0.4.5 — status                                                       |
+-------------------------------------------------------------------------------+
| Active bans: 2101                                                             |
| Total bans:  2101                                                             |
| Knowledge:   fresh (readback 4319 ms ago)                                     |
| Protection:  active                                                           |
| Backend:     nftables                                                         |
| Jails:       2                                                                |
| Storage:     healthy                                                          |
| Worker:      busy for 9 ms                                                    |
+-------------------------------------------------------------------------------+

# fail2zig jails --details
JAIL       STATE   ACTIVE MAX RETRY FIND TIME BAN TIME ACTION   ENFORCING SOURCE  HEALTH
-----------------------------------------------------------------------------------------
sshd       enabled 59     5         1h           1w    nftables true      journal ok
portsentry enabled 2042   1         1h           1w    nftables true      file    ok
Total: 2 jails

file2zig %CPU:

25.3
23.6
27.0
28.3
40.6
23.6
23.3
26.0
24.3
24.0
23.0
25.3
35.3
24.0
24.0
43.0
23.6
23.3
36.6
20.6
24.3
26.0
32.6
44.6
23.0
24.0
24.3
25.6
24.3
24.6
33.3
44.6
42.6
24.3
24.0
23.0

It seems to me that this is an abnormal activity, something needs to be changed. Maybe add manual control of check timeouts?

[mwriter]

@ul0gic

fail2zig 0.4.6
I continue to observe high every-second %CPU loads with a large number of added rules (primarily from portsentry). All the rules have already been applied, there are almost no new addresses in the list, but the load is consistently high. If these are not repeated recordings, then some kind of constant checks. Maybe we should add other check timeouts after all?
It seems to me that something is still wrong with the architecture.

[ul0gic]

@mwriter Apologies for the delay on this one. It has been tricky and we have been deep in the weeds on it, but we found where the CPU goes.

We reproduced your situation in the lab: a single-CPU host with about 2,000 PortSentry bans applied and nothing new arriving. The daemon sat at roughly 50 % of that CPU, so your samples are real and not repeated recordings.

To pin it down we built a temporary instrumented version of the daemon that times every piece of its work on each 100 ms tick, ran short measured sessions at 0, 500 and 2,000 bans on that host, and then attached the kernel profiler to the running daemon to see exactly where the time went.

The periodic firewall checks are not the cost. Reading the full set back from nftables and comparing it is under 1 % at 2,000 bans. The cost is in the daemon's own status bookkeeping. On every tick it rebuilds the published jail view, and for each active ban it rescans the entire ban list. That grows with the square of the ban count, so it is invisible with a few dozen bans and dominant with a couple of thousand. The profiler put about two thirds of the daemon's CPU in that one path.

A check-interval setting would not help because the interval is not what costs. The fix is a direct lookup instead of a rescan, and refreshing the view once a second instead of after every poll. Our target for the same 2,000-ban idle state is about 2 % of one CPU. The same work also fixes the slow stop with many rules, which took 44 s at 2,000 bans in the lab.

This goes into v0.4.7, a stabilization release. We will post the measured before and after numbers here when the candidate is ready. Thanks for the samples; they are what made this reproducible.

[ul0gic]

v0.4.7 addresses this. Measured on a one-vCPU Debian 13 host with 2,000 nftables bans over five minutes: mean idle CPU 2.61 %, maximum 6.0 %, against about 56 % before. The causes were a full set readback per verification cycle and the journal source spawning journalctl about seven times a second when idle; both are gone. The healthy stop at that size takes about one second instead of being killed at the 10 s timeout. Tracked as #108 and #109.

[ul0gic]

@mwriter Closing this one, since v0.4.7 fixed the idle CPU you reported. If you see it climb again, please open a new discussion or issue with a status --details snapshot, and we'll pick it straight back up. Thanks again for pushing on it until we found the cause.