Watchdog and Resource Profiles

Watchdog & Resource Profiles

Type: Architecture Scope: Runtime pressure monitoring, operating modes, server profiles, adaptive behavior Since: v1.78.0 (watchdog), v1.80.0 (profiles) Authority: Go watchdog (internal/watchdog/), safety subsystem (internal/safety/) Terminology: Glossary & Vocabulary

---

Purpose

See also: Metrics Architecture | Health Model | Architecture Overview

This page describes how NFTBan monitors its own resource consumption and adapts behavior under pressure. The watchdog and safety subsystems are distinct from the validator — they measure operational pressure, not protection health.

System	What it answers	Authority
Validator	Is the firewall protecting correctly?	Health states (PROTECTED/DEGRADED/DOWN/IDLE)
Watchdog	Is the daemon running within resource bounds?	Pressure scores + operating modes
Safety	Should we skip expensive operations?	Memory pressure levels + CIDR limits

These are complementary, not overlapping. The validator can report PROTECTED while the watchdog is in DEGRADED mode (system is protecting but under CPU pressure).

---

Watchdog Architecture

The watchdog runs as a goroutine inside the daemon. It continuously monitors four pressure dimensions, computes scores, and adjusts daemon behavior through RuntimeControls.

Source: internal/watchdog/doc.go

┌─────────────────────────────────────────────────────────┐
│                     Watchdog Core                        │
│  ┌─────────────────────────────────────────────────────┐│
│  │                   State Machine                      ││
│  │  NORMAL <-> DEGRADED <-> SURVIVAL                    ││
│  └──────────────────────┬──────────────────────────────┘│
│                         |                                │
│     ┌───────────────────┴──────────────────┐             │
│     v                                      v             │
│  ┌─────────────────┐           ┌─────────────────────┐  │
│  │   Collectors    │           │      Actions         │  │
│  │  - Process      │           │  - Throttle workers  │  │
│  │  - Runtime      │           │  - Profile (pprof)   │  │
│  │  - System       │           │  - Memory valve      │  │
│  │  - Kernel       │           │  - Degrade mode      │  │
│  │  - nftables     │           └─────────────────────┘  │
│  └─────────────────┘                                     │
│                         |                                │
│                         v                                │
│              ┌─────────────────────────┐                 │
│              │    Flight Recorder      │                 │
│              │  - Event log (JSON)     │                 │
│              │  - Periodic snapshots   │                 │
│              │  - pprof profiles       │                 │
│              └─────────────────────────┘                 │
└─────────────────────────────────────────────────────────┘

Pressure dimensions

Dimension	What it measures	Sources
CPU	Process CPU%, system load, softnet drops	/proc/stat, procfs, /proc/net/softnet_stat
MEM	RSS vs budget, heap vs GOMEMLIMIT, GC fraction	/proc/meminfo, Go runtime
IO	iowait%, disk usage, log partition fullness	/proc/stat, syscall.Statfs()
NET	conntrack utilization, softnet drops rate, NIC drops	/proc/sys/net/netfilter/, /sys/class/net/

Each dimension produces a score (0-100). The state machine uses hysteresis to prevent flapping between levels.

Pressure levels

Level	Score	Meaning
OK	Below warn threshold	Normal operation
WARN	Enter at 60, exit at 50	Reduced operation, backpressure enabled
CRITICAL	Enter at 80, exit at 70	Essential operations only

Thresholds are configurable in /etc/nftban/conf.d/watchdog.conf. Exit thresholds are lower than entry thresholds (hysteresis prevents flapping).

---

Operating Modes

The watchdog drives three daemon operating modes. Mode transitions happen automatically based on pressure scores.

Source: internal/watchdog/types.go:270-297

Mode	Trigger	Workers	Expensive Collectors	NFT Scan	Telemetry
NORMAL	All dimensions OK	10	ON	ON	100%
DEGRADED	Any dimension WARN	5	OFF	OFF	50%
SURVIVAL	Any dimension CRITICAL	2	OFF	OFF	25%

RuntimeControls use Go atomic operations for lock-free thread-safe access. The watchdog updates controls; daemon subsystems read them without locking.

What changes per mode

NORMAL: Full functionality. All collectors run. Full telemetry sampling.

DEGRADED: Worker pool reduced to 5. Expensive collectors (full nft ruleset scan, top process tracking) disabled. Telemetry sampled at 50%. Collection interval multiplied by 2.0x.

SURVIVAL: Worker pool reduced to 2. Only essential operations. Telemetry at 25%.

Prometheus metrics

• nftban_pressure_score{dim} — score per dimension (gauge, 0-100)
• nftban_pressure_state{dim,level} — state per dimension (gauge)
• nftban_operating_mode{mode} — current mode (gauge)
• nftban_watchdog_action_total{action} — actions taken (counter)

---

Server Profiles

NFTBan detects server resources at startup and adjusts behavior based on hardware class. This is not a configuration setting — it is automatic.

Source: internal/safety/mem.go:133-286

Profile detection

DetectServerProfile() reads:

• CPU cores from /proc/cpuinfo
• Total and available RAM from /proc/meminfo and cgroup limits
• Control panel presence (cPanel, DirectAdmin, Plesk, CyberPanel, CloudPanel, VestaCP, HestiaCP)

Hardware classes

Profile	RAM	Memory Budget	Max CIDRs	Workers
Small	<=4 GB	440 MB max	75,000	1/core (panel) or 2/core, max 8
Medium	4-8 GB	590 MB max	100,000	1/core (panel) or 2/core, max 8
Large	>8 GB	1.15 GB max	150,000	2/core, max 8

Control panel adjustment

Servers with a control panel (cPanel, DirectAdmin, Plesk, etc.) receive more conservative budgets because the panel itself consumes significant memory.

Server type	Budget calculation
Panel server	20% of available RAM (capped by tier)
Bare server	35% of available RAM (capped by tier)

Minimum budget floor: 64 MB regardless of server size.

Where profiles are wired

• daemon_init.go:104-106 — CIDR limit set at boot based on server tier
• watchdog/config.go:131 — memory budget passed to watchdog
• daemon_handlers_sync.go:214-350 — feeds and geoban skipping under pressure

---

Memory Pressure Levels

The safety subsystem tracks memory consumption relative to the budget and triggers progressive protection.

Source: internal/safety/limits.go:449-517

Level	% of Budget	Behavior
Normal	0-70%	Full operations
Warning	70-85%	Log warning, continue all operations
High	85-95%	Skip geoban refresh (ShouldSkipGeoban())
Critical	>95%	Skip feeds + trim cold bans (ShouldSkipFeeds())

Adaptive behavior under pressure

Feeds loading (daemon_handlers_sync.go:214-218): When ShouldSkipFeeds() returns true (Critical pressure), the daemon skips loading all feed data. This prevents memory exhaustion from large feed sets (some feeds contain 50,000+ CIDRs).

Geoban loading (daemon_handlers_sync.go:325-328): When ShouldSkipGeoban() returns true (High or Critical pressure), the daemon skips loading geoban country data. Geoban can require 20,000+ CIDRs per country (China, Russia, etc.).

Memory allocation check (daemon_handlers_sync.go:263, 350): Before loading feeds or geoban, the daemon calls safety.CanAllocate() with an estimate of bytes needed (300 bytes per CIDR). If allocation would exceed the budget, loading is skipped even if pressure level alone would allow it.

CIDR hard limits (safety/limits.go:175-189): Regardless of pressure, CIDRs are capped per tier:

• Small: 75,000 max
• Medium: 100,000 max
• Large: 150,000 max

Protection state tracking

When protection triggers, the daemon writes state to /var/lib/nftban/state/protection.json recording what was skipped and why.

Prometheus metrics:

• nftban_protection_active — 1 if any skip occurred
• nftban_protection_feeds_skipped — 1 if feeds skipped
• nftban_protection_geoban_skipped — 1 if geoban skipped
• nftban_memory_pressure_level — current level (0-3)

---

Configuration

All watchdog settings are in /etc/nftban/conf.d/watchdog.conf (333 lines). Override with /etc/nftban/conf.d/watchdog.conf.local.

Key settings

Setting	Default	Purpose
NFTBAN_WATCHDOG_ENABLED	true	Enable watchdog
NFTBAN_WATCHDOG_INTERVAL	90	Check interval (seconds)
NFTBAN_MEM_BUDGET_BYTES	1073741824 (1 GB)	Memory budget (auto-adjusted by profile)
NFTBAN_PRESSURE_WARN_ENTER	60	Score to enter WARN
NFTBAN_PRESSURE_WARN_EXIT	50	Score to exit WARN
NFTBAN_PRESSURE_CRIT_ENTER	80	Score to enter CRITICAL
NFTBAN_PRESSURE_CRIT_EXIT	70	Score to exit CRITICAL
NFTBAN_RSS_CRIT_PERCENT	95	RSS critical threshold (% of budget)
NFTBAN_CPU_CRIT_PERCENT	80	CPU critical threshold
NFTBAN_DEGRADE_REDUCE_WORKERS_TO	2	Workers in DEGRADED mode
NFTBAN_SURVIVAL_REDUCE_WORKERS_TO	1	Workers in SURVIVAL mode
NFTBAN_DEGRADED_INTERVAL_MULTIPLIER	2.0	Collection interval multiplier in DEGRADED

Auto-profiling (incident capture)

When thresholds are breached, the watchdog captures pprof profiles for post-mortem analysis:

• CPU profile: 30s capture, 15-minute cooldown
• Heap profile: on demand, 30-minute cooldown
• Goroutine dump: on demand, 5-minute cooldown

Memory valve

debug.FreeOSMemory() is triggered when memory pressure is high and CPU is below 40%. Cooldown: 10 minutes. This releases unused memory back to the OS without disrupting daemon operation.

---

Verification

# Check watchdog status
curl -s http://127.0.0.1:9580/metrics | grep nftban_pressure_score
# Shows per-dimension pressure (CPU, MEM, IO, NET)

# Check operating mode
curl -s http://127.0.0.1:9580/metrics | grep nftban_operating_mode
# mode="normal" = 1 when in NORMAL mode

# Check server profile at startup
journalctl -u nftband --no-pager | grep "Server tier"
# Example: "Server tier: medium (max CIDRs: 100000)"

# Check protection state
cat /var/lib/nftban/state/protection.json 2>/dev/null | jq .
# Shows feeds_skipped, geoban_skipped, reason

# Check memory pressure
curl -s http://127.0.0.1:9580/metrics | grep nftban_memory_pressure_level
# 0=normal, 1=warning, 2=high, 3=critical

---

Limitations

• Shell exporter is not profile-aware. The shell unified exporter runs on a fixed 60-second timer regardless of server size or pressure mode. Only the Go daemon adapts behavior based on profiles.
• Memory pressure metrics gap. The functions SetMemoryPressureLevel() and SetProtectionActive() are defined but not continuously updated from the watchdog tick. The watchdog updates its own pressure_score/pressure_state metrics but does not bridge to the safety-layer gauges.
• Ban eviction not wired. GetEvictableBans() logic exists for trimming old permanent bans under critical pressure, but is not invoked from the watchdog.