BotScan

BotScan (Web Access-Log Abuse Scanner)

BotScan is a log-driven detector that reads web server access logs and bans source IPs that match web-abuse patterns or cross volume thresholds. It is the owner of web-endpoint abuse detection.

Last verified: v1.190.1.

---

Role and boundary

BotScan is a scanner, not a runtime traffic guard. It reacts to what is already recorded in the access log, on a timer cadence — it is not in the packet path.

Module	Owns	Does not own
BotScan	Web access-log abuse — pattern matches and endpoint-volume floods	Packet-rate enforcement; credential-failure scoring
BotGuard	Runtime HTTP burst/concurrency behavior (the http_bot_guard nft chain)	Access-log content; endpoint-volume detection
Login Monitoring	Credential-failure / auth_failure events from auth logs	The 200-body result of a web request

---

How it works

• Trigger model: a timer-driven batch scanner, not realtime. A flood is detected on the next scan cycle, not at request time.
• Source: web server access logs, read by the BotScan collector (nftban-botscan-collector.service, running as the nftban user with CAP_DAC_READ_SEARCH) into the spool directory /run/nftban/botscan. The scanner (nftban-botscan.service) reads the spool — it does not need elevated read capability itself.
• Latency: detection lag equals the collector/scanner timer cadence. BotScan is for sustained abuse, not single-packet response.
• Health: when no web access log is present or readable, BotScan reports WARN_NO_LOGS / NO_LOGS rather than appearing healthy with no input.

---

Detection classes

404 flood

Per-IP count of 404 responses within a window. Default threshold BOTSCAN_404_THRESHOLD=50 within BOTSCAN_404_WINDOW=300 seconds.

Exploit / webshell / scanner patterns

Configurable URL/User-Agent patterns with per-pattern threshold, window, and ban duration. Managed with nftban botscan patterns ….

Bad-bot / AI-bot categories

User-Agent category lists (badbots, aibots, scanner, custom). Enable or disable banning per bot with nftban botscan blockbot <name> / allowbot <name>.

Verified-crawler handling

A crawler claiming to be Googlebot/Bingbot by User-Agent is verified by forward-confirmed reverse DNS (FCrDNS) before being exempted, because the User-Agent string alone is spoofable. Verification is fail-closed and controlled by BOTSCAN_VERIFY_CRAWLERS (default true). A spoofed crawler is treated as a bad bot.

Endpoint-flood protection (XML-RPC / wp-login)

As of v1.190.1, BotScan counts requests to a watched endpoint per source IP, per endpoint, per window, independent of response status, and bans an IP that exceeds the threshold.

• Why status-independent: POST /xmlrpc.php and POST /wp-login.php floods are frequently answered with 200 OK — the request is well-formed, the payload is the attack. A detector keyed on error status alone never sees them.
• Volume-based: a single request never bans, so legitimate XML-RPC traffic (Jetpack, pingbacks, mobile apps) below the threshold is unaffected. It is not a blanket URL-pattern ban.
• Defaults (read from main.conf, override in main.conf.local):

Setting	Default	Meaning
BOTSCAN_ENDPOINT_FLOOD_ENABLED	true	Master switch.
BOTSCAN_ENDPOINT_FLOOD_METHOD	POST	HTTP method counted.
BOTSCAN_ENDPOINT_FLOOD_THRESHOLD	30	Requests per window, per IP, per endpoint, before a ban.
BOTSCAN_ENDPOINT_FLOOD_WINDOW	60	Window length, in seconds.
BOTSCAN_ENDPOINT_FLOOD_BAN	3600	Ban duration, in seconds.
BOTSCAN_ENDPOINT_FLOOD_ENDPOINTS	xmlrpc.php wp-login.php	Space-separated watched endpoints.

---

Configuration

BotScan defaults live in /etc/nftban/main.conf; override in /etc/nftban/main.conf.local. Pattern definitions ship under /etc/nftban/patterns.d/botscan/ with a three-tier override.local no-clobber model.

Key defaults: BOTSCAN_404_THRESHOLD=50, BOTSCAN_404_WINDOW=300, BOTSCAN_VERIFY_CRAWLERS=true, BOTSCAN_LOG_FILE=/var/log/nftban/botscan.log, plus the endpoint-flood knobs above.

---

Systemd units

Unit	Purpose
nftban-botscan-collector.service	Collector — reads access logs as nftban + CAP_DAC_READ_SEARCH, writes the spool /run/nftban/botscan.
nftban-botscan-collector.timer	Schedules the collector.
nftban-botscan.service	Scanner — classifies spooled log lines and emits ban signals.
nftban-botscan.timer	Schedules the scanner.

---

Enforcement path

BotScan does not write nftables directly. The scanner emits a batch signal (/var/lib/nftban/botguard/batch_signals.jsonl) that the daemon (nftband) applies to the nftables ban set. Both IPv4 and IPv6 sources are handled. No new nftables rules or schema fields are introduced by the access-log detectors — they feed the existing ban/enforcement path.

---

Verification

# 1. What BotScan decided (report layer)
grep -E 'endpoint_flood|404 flood|pattern' /var/log/nftban/botscan.log

# 2. Kernel evidence — the banned IP is in the active ban set
nftban status <ip>
nft list set ip nftban blacklist_manual_ipv4 2>/dev/null | grep <ip>

# 3. Scanner ran and is healthy
systemctl status nftban-botscan.service
nftban botscan status

A log line is a report, not proof of enforcement — confirm the ban in kernel state. When the ban duration elapses, the entry is removed on the normal expiry path.

---

Limitations

• Not realtime. Detection lag equals the scanner timer cadence; BotScan addresses sustained abuse, not single-request response.
• Requires readable access logs. No web log, or an unreadable one, means no BotScan coverage (surfaced in health as WARN_NO_LOGS / NO_LOGS).
• Low-and-slow L7 floods that stay under both the endpoint-flood volume threshold and the BotGuard burst rate are not detected by either module. This gap is acknowledged, not hidden.

---

References

• BotGuard (runtime HTTP burst/concurrency): BotGuard
• Login Monitoring (credential-failure owner): Login-Monitoring
• Source: cli/lib/nftban/core/nftban_botscan.sh, cli/lib/nftban/cli/cmd_botscan.sh