Troubleshooting

Troubleshooting

For the 1.x line. Running 0.5.5 or earlier? See Troubleshooting 0.x.

Start here:

systemctl status ddos-stage1 ddos-stage2
journalctl -fu ddos-stage1
journalctl -fu ddos-stage2

The Sensor Captures Nothing

The capture status line shows raw_captured=0 and it never climbs.

Traffic is not routed through the gateway. This is the usual cause. If the source and the protected host are on the same bridge and the same subnet, they talk directly at Layer 2 and the gateway never sees the packets, even when it is configured as their default gateway. Confirm independently:

sudo tcpdump -i ens19 -c 20

If tcpdump sees nothing either, the problem is the network layout, not FLOD. See the network placement section in Installation.

The wrong interface. Check the ExecStart line in /etc/systemd/system/ddos-stage1.service against ip link.

The filter excludes your traffic. The sensor only captures traffic destined for the addresses given by --victim-ips or --victim-subnet. Traffic to anything else is dropped in the kernel before it reaches the sensor. To confirm this is the cause, restart briefly with --no-filter and see whether counts start climbing. Do not leave it that way.

Missing capture rights. The journal will say so at startup. Either run as root or grant the capability:

sudo setcap cap_net_raw+ep /usr/local/bin/ddos_stage1

Packets Are Captured but Not Analysed

Compare raw_captured against forwarded in the status line.

parse_failed counts frames that were not parseable at all. A small number on a noisy link is normal; a large proportion is not.

non_ip counts non IP frames, which should be near zero given the filter.

truncated counts frames whose payload was cut short by the snapshot length. This is expected and harmless. The sensor reads only headers, and a truncated frame is still fully analysed. A large truncated count alongside parse_failed=0 is the normal, healthy state.

Nothing Is Ever Flagged

Still warming up. The sensor learns your baseline before it will flag anything. Until warm up finishes there is deliberately no enforcement. The journal shows progress.

Sensitivity too low. Lower --k in steps of half a point. Every step widens what counts as an anomaly.

Traffic genuinely is not anomalous. If an attack is distributed enough to resemble normal traffic in both rate and source diversity, this detector will not separate it. That is a real limitation of a header only feature set, not a misconfiguration.

Legitimate Traffic Is Being Flagged

Check the flagged windows in the journal:

ANOMALY window 34 | flags=0x02 | r=24.5 (boundary=68.2) |
h=0.9533 (boundary=0.9565) | dom_ratio=0.200

Read the margins. If the measured value is barely past the boundary, and dom_ratio is low, traffic is well distributed and this is a false positive. flags=0x01 is a rate anomaly, flags=0x02 is an entropy anomaly.

This happens when your normal traffic is very consistent. A small variance puts the boundary almost on top of the average, so ordinary fluctuation crosses it.

First, check which host it is. If the flagging is concentrated on one host, and that host is the gateway rather than a service behind it, the fix is to stop protecting it rather than to retune anything. A gateway carries its own management traffic at a volume and variability unlike anything behind it, and --victim-subnet sweeps it into the protected set when its address is in the same range. Its spread then sets the floors for every other target as well.

# Which hosts are flagging, and how often
journalctl -u ddos-stage1 --since -1h --no-pager -o cat |
    grep ANOMALY | grep -o 'victim=[^]]*' | sort | uniq -c | sort -rn

Swap --victim-subnet for --victim-ips naming only the services, then recalibrate. See Configuration for the details and the check that confirms what the sensor resolved.

Remedies, in order of preference:

1. Measure the right boundaries instead of guessing at them. Once traffic is back to normal:

   sudo python3 scripts/calibrate.py --auto-debug --apply

   It reads your own traffic and sets the sigma floors so the boundary sits past ordinary variation. Run it without --apply first if you want to see the numbers before committing to them.

2. Raise --k so the boundary sits further from the average.

3. Raise block_rate_floor_pps and ratelimit_rate_floor_pps so a low absolute rate cannot trigger enforcement whatever the baseline says.

4. Whitelist the sources if they are known and fixed.

Rate limited addresses recover on their own when their entry expires. To clear one immediately, use the firewall page or:

sudo ipset del ddos_ratelimit <address>

Everything Is Flagged and It Never Stops

A specific and self sustaining version of the case above. Almost every window fires, the flags are rate anomalies, and the measured rate is far above the boundary rather than barely past it:

ANOMALY window 1  | flags=0x01 | r=264.6 (boundary=204.2) | dom_ratio=0.115
ANOMALY window 2  | flags=0x01 | r=240.9 (boundary=201.7) | dom_ratio=0.156
ANOMALY window 14 | flags=0x01 | r=209.1 (boundary=201.7) | dom_ratio=0.159

Note that the boundary barely moves while the traffic sits well above it. That is the signature: your traffic has genuinely grown, the learned average is still at the old level, and it cannot catch up. A flagged window does not update the baseline, which is what stops a slow attacker from teaching the sensor that a flood is normal. When every window is flagged, that same protection means the average is stuck.

From 1.2.0, this resolves on its own, just not quickly: past --max-baseline-freeze-windows (default 400) consecutive frozen windows, the current traffic is accepted as the new baseline regardless, logged as baseline was frozen for N consecutive windows... accepting current traffic as the new baseline. Detection is not weakened while it waits; every anomalous window is still flagged and reported the whole time, only the baseline's own learning pauses. If you would rather not wait, or you are on an earlier version, wipe the learned baseline so it relearns at the current level immediately:

sudo systemctl stop ddos-stage1
sudo rm /var/lib/ddos_stage1/baselines.json
sudo systemctl start ddos-stage1

Warm up takes 200 windows before detection resumes.

Remove the file, not the directory. Without the directory the sensor still runs, but it cannot save anything, so every restart costs a full warm up with no enforcement during it. It says so once per attempt:

Persistence: failed to open '/var/lib/ddos_stage1/baselines.json.tmp' for
writing: No such file or directory. Does the parent directory exist and is it
writable?

Put it back with the ownership the service account needs:

sudo install -d -m 700 -o ddos-stage1 -g ddos-stage1 /var/lib/ddos_stage1

Removing the directory deliberately is occasionally useful, when you want two runs to learn independently rather than one inheriting the other's baseline. Restore it afterwards.

scripts/calibrate.py reports this condition when it sees it, and --clear-baseline does the wipe as part of applying new values:

sudo python3 scripts/calibrate.py --auto-debug --apply --clear-baseline

If it comes back, the traffic growth was not the whole story. Check whether something genuinely is sending more than it used to before dismissing it.

The Kernel Backend Will Not Start

Only applies with --capture-mode kernel. In every case the sensor exits with the reason rather than starting degraded, and --capture-mode pcap gets you running again while you sort it out.

No object at that path. The installer builds and installs it, but a manual build writes into the source tree, not the runtime path. Re-run the build as root so it installs both places:

sudo ./scripts/build-ebpf.sh

The kernel rejected the programs. The message includes the verifier's own log. This is the kernel refusing to run the bytecode, not a compile failure, so rebuilding without changing anything will not help. Include that log if you report it.

Needs a target list. Matching happens in the kernel, so --victim-ips or --victim-subnet is required. There is no equivalent of running unfiltered.

Works by hand but not under systemd. Almost always capabilities. Running as root by hand grants everything; the unit grants a specific set. Check it has all of these:

AmbientCapabilities=CAP_NET_RAW CAP_BPF CAP_NET_ADMIN CAP_PERFMON
LimitMEMLOCK=infinity

CAP_NET_RAW alone is enough for libpcap but not for loading programs or attaching them.

The Build Cannot Find a Toolchain

The eBPF programs need a nightly Rust toolchain with rust-src, plus bpf-linker and the LLVM it links against. scripts/install.sh sets all of this up and picks a bpf-linker matching the LLVM your distribution ships.

rustup not found, but you installed it. If the installer ran under sudo, rustup went to root's home. sudo also rebuilds PATH, so a later plain sudo call cannot see it. The scripts search your home, root's home, and the invoking user's home, so run the build the same way you ran the installer.

A link error naming an LLVM symbol. bpf-linker and LLVM move together and the wrong pairing fails at link time. Re-running the installer picks a version against the LLVM you actually have. Do not pin a version by hand.

None of this affects the pcap backend, which needs no toolchain beyond a normal Rust install.

I Locked Myself Out

Blocking your own management address is the most common self inflicted problem. From console access:

sudo ipset del ddos_blocklist <your-address>
sudo ipset del ddos_ratelimit <your-address>

Then add yourself to the whitelist before doing anything else.

To clear everything:

sudo ipset flush ddos_blocklist
sudo ipset flush ddos_ratelimit

Enforcement Does Not Take Effect

Confirm the sets exist and the rules are attached:

sudo ipset list ddos_blocklist
sudo iptables -L INPUT -n --line-numbers | grep ddos
sudo iptables -L FORWARD -n --line-numbers | grep ddos

Both chains should have a rule for each set. Stage 2 creates them at startup and logs a warning if it cannot.

The rate limit needs the kernel hashlimit module. If it is missing, blocks will work and rate limits will not.

Note that both sets match on source address. Traffic is stopped based on where it came from, not where it was going.

The Dashboard Will Not Load

Certificate warning. Expected with the self signed certificate the installer generates. Click through, or install a certificate issued for the host.

Connection refused. Check Stage 2 is running and nothing else holds port 8000.

Redirected to login repeatedly. Sessions last ten minutes of inactivity. If it happens immediately, the browser is refusing the session cookie. Over plain HTTP this can happen if the browser was previously served HTTPS from the same host.

The Dashboard Loads but Renders Nothing

Almost always a stale cached script from before an update. Force a full reload with Ctrl+Shift+R.

Recent versions tell the browser to revalidate its cached copies, so this should no longer occur. If it does after an update, the browser is holding files from before that change.

Data Looks Stale

The dashboard polls every two seconds and the sensor reports on a matching heartbeat.

If one host's figures update and another's do not, the quiet one is simply reporting on its heartbeat while the busy one reports on every flagged window. That is expected.

If nothing updates at all, Stage 2 has probably lost the socket. Check the journal and restart Stage 2 first, then Stage 1.

Stage 2 Falls Behind

Verdicts and rate limits arrive seconds late, the sensor logs IPC: write failed (its socket to Stage 2 is full), or the dashboard lags an attack. Stage 2 handles windows one after another, so a single slow window delays every window behind it.

Since 1.6.1 Stage 2 says where the time went. Search its log:

journalctl -u ddos-stage2 | grep -E "Handling one window took|Latency: summary" | tail

A Handling one window took line lists the seconds spent in inference, the database write, the flow snapshot and enforcement. The Latency: summary line adds a busy figure, the time Stage 2 spends handling each window after it arrives. A large handoff with a small busy means windows arrived late and Stage 2 was not the slow part.

Two causes are known and fixed in 1.6.1: the auto-label job used to hold a capture file's lock through its whole scoring pass while Stage 2 waited on it, and enforcement used to act on every flow in the sensor's snapshot instead of the flows to the host under attack. If you are on an older release, update. One IPC: write failed per Stage 2 restart is normal, since the sensor cannot write while Stage 2 is down. If a slow window warning keeps appearing, keep the lines and see Filing a Bug.

A Benchmark Run Refuses to Start or Looks Wrong

scripts/benchmark_live.sh stops before it starts in four cases, each with a message that says why:

• Another benchmark is running. The driver holds a lock in its output directory, and it also refuses while a benchmark sampler runs on the gateway. Two drivers at once feed traffic into each other's phases and the first to finish deletes the other's sampler files. If a run died without cleaning up, stop the leftover sampler on the gateway with pkill -f '[s]ystem_sampler.sh'.
• The ingress interface is not idle. With every generator stopped it must stay under IDLE_INGRESS_MAX_PPS (300 packets a second). Something is still sending, often a generator left by an interrupted session. Find it before retrying, because calibration would learn that traffic as Normal.
• The egress interface has no address. The gateway cannot forward to the targets. Fix that first.
• The attack source count is outside its range.

A run that started can still be spoiled. Read the calibration section of its report before trusting the rest: floors far above your Normal traffic, or most windows flagged during calibration, mean something else was sending. See Benchmarking.

Auto Labeling Always Reports Zero Rows

journalctl -u ddos-stage2-auto-label showing "Auto-labeled 0 row(s)" on every run, even with a full anomalous_capture.csv, is usually not a bug. Two independent gates both have to clear, and either one alone explains a zero:

Row age. AUTO_LABEL_DELAY_HOURS defaults to 24. A row captured this morning stays ineligible until this time tomorrow, regardless of anything else.

Model freshness. Every model scoring a row must have been trained after that row was captured. If you have never run a retrain, or your last one predates your current traffic, nothing captured since then can clear this check until the next retrain. See Automatic Labeling for --training-csv and --retrain-interval. Confirm with:

ls -la /var/lib/flod/*.joblib

Confidence. Both models must reach AUTO_LABEL_CONFIDENCE_THRESHOLD (default 0.90) on the same class. A Random Forest limited to a shallow depth often tops out below that on generated attack traffic even when it names the right class every time, so a run can stage a small share of a file whose rows are all correctly detected. Compare how many rows the two models call the same class with how many clear the threshold before treating a low count as a fault. Running once with AUTO_LABEL_CONFIDENCE_THRESHOLD=0.80 shows the difference.

If a row is old enough and every model postdates it but you still see zero, check for degenerate rows: a window with entropy, proto_ratio, dominant_ip_ratio, source_port_entropy, ttl_variance, and fingerprint_diversity all reading exactly 0.0 is refused regardless of confidence or agreement. That is by design: a zero-traffic window carries no evidence for any class. Before 1.6.1 these windows filled most of anomalous_capture.csv (36,000 of 50,000 rows in one lab gateway), which is why its queue looked full while nothing staged. They are no longer written.

Every staged row can also be DDoS. ddos_capture.csv only receives windows the Random Forest called DDoS and anomalous_capture.csv only windows the Isolation Forest flagged, so ordinary traffic never reaches the labeling job in bulk. That is the design, not a fault. If you need Normal or Flash Crowd rows, capture them yourself following the clean rule in Configuration.

Merge Is Unavailable on the Auto Label Page

The page says "No training CSV is configured (TRAINING_CSV_PATH), so Merge is unavailable." The setting lives in the ddos-stage2 service unit, and update.sh rebuilds that unit every time it runs, writing the path only when --training-csv is given. A plain update.sh removes it. Rerun with the flag to turn Merge back on:

sudo bash scripts/update.sh --training-csv /path/to/training_data.csv

The same file is where Merge appends and what the retrain timer trains on. Discard works either way.

The Isolation Forest Flags Almost Every Window

Nearly every window reads Anomalous, including plainly ordinary traffic, even right after a retrain. Check whether the training set you trained on was captured under the same sigma floors the sensor runs now. Compare sigma_h and sigma_r in the training CSV with the values in /var/lib/flod/anomalous_capture.csv: a training set with sigma_h near 0.05 and sigma_r pinned at one value, against captures with sigma_h at your current entropy floor, explains it. The Random Forest ignores those two columns and keeps classifying correctly, and the Anomalous label never drives enforcement, so nothing is being blocked wrongly. Recapture under the current floors, or train the Isolation Forest on rows from this deployment, and see Configuration. Once the training set and the floors agree the flag rate falls: in the lab it was 0.3% to 1.2% of Normal windows and 0.0% to 1.7% of Flash Crowd windows under the deployed tuning.

Records Are Missing from the Event Log

If the journal shows enforcement happening but the log page does not list it, check for database errors:

journalctl -u ddos-stage2 | grep -i sqlite

Older versions could lose incident records under load while enforcement itself succeeded. Updating fixes it.

A Record Shows n/a, or an Entropy of Zero

n/a means the value was never measured, and is correct rather than a fault. A release has no rate, because releasing an address is not an observation of traffic. An address you blocked by hand has none either. A window verdict with no single dominant sender has no source rate to attribute.

An entropy of exactly 0.0000 on a row written before 1.0.1 means the same thing, unknown, but was stored as a number. Zero entropy otherwise means traffic concentrated on a single source, so on those older rows it reads as the opposite of what may have happened. From 1.0.1 onward an unmeasured entropy is recorded as unmeasured, and the value that is recorded comes from the window that drove the action rather than from whichever host reported most recently.

Existing rows were not rewritten, because the correct value for them cannot be recovered.

Services Will Not Start

Stage 1 exits immediately. Usually the interface does not exist or capture rights are missing. The journal names the reason and lists available interfaces.

Stage 2 exits immediately. Usually a missing Python dependency after a partial update. Re run scripts/update.sh.

Stage 1 runs but Stage 2 sees nothing. Stage 2 owns the socket, so start it first:

sudo systemctl restart ddos-stage2
sudo systemctl restart ddos-stage1

Cannot Log In

There is no default account. If one was never created:

cd stage2
sudo venv/bin/python3 setup_admin.py

Repeated failed logins from one address lock that address out temporarily. Wait it out.

If the journal mentions a stored hash not being in the expected format, the account predates the current password scheme. Re run setup_admin.py to reset it.

Filing a Bug

Include the output of systemctl status for both services, the relevant journal extract, a capture status line, your ExecStart line, and your distribution and kernel version.

Redact real addresses if you would rather not publish them, but say what you replaced them with so the report still makes sense.

Security vulnerabilities go through the process in SECURITY.md, not the issue tracker.