Skip to content

Suricata: 'truncated packet' alerts flooding logs due to virtio-net rx-gro-hw #342

Description

@Xore

Symptom

The VPS Suricata instance logs a huge volume of decoder-event alerts:

  • SID 2200003 — "SURICATA IPv4 truncated packet"
  • SID 2200122 — "SURICATA AF-PACKET truncated packet"

Sampling one hour of eve-*.json: ~39,700 occurrences in that single hour, and in a fresh 5000-line sample these two SIDs alone were ~70% of all alert volume — drowning out real signal.

Root cause

Not a portbridge/MTU misconfiguration. Decoded a sample alert's raw packet bytes:

  • Ethernet + IPv4 header parse cleanly, IP header declares total_length = 1535 bytes
  • The captured frame Suricata actually received was only ~1510 bytes of IP payload — a ~25 byte shortfall vs. what the header claims

ethtool -k on the VPS's public interface showed all the usual offload suspects already off (tcp-segmentation-offload, generic-segmentation-offload, generic-receive-offload, large-receive-offload). But rx-gro-hw: on — a separate hardware/hypervisor-accelerated GRO feature, not controlled by the standard gro off toggle, and common on virtio-net (KVM/QEMU) interfaces. Confirmed the interface is virtio-net (ID_NET_DRIVER=virtio_net). This offload coalesces multiple physical frames into one oversized "packet" with an inflated IP total-length; Suricata's AF_PACKET capture then reports the coalesced/truncated result as a decoder error.

Live-verified the fix before committing to it: toggled ethtool -K <iface> rx-gro-hw off on the running VPS and watched fresh eve.json output. In a subsequent clean 2m15s window (traffic after the toggle had fully taken effect), truncation-alert count was 0, down from ~70% of alert volume beforehand.

Fix

  • New vps/disable-nic-hw-gro.sh: installs a systemd-networkd .link file matching Driver=virtio_net with GenericReceiveOffloadHardware=false, so the setting survives reboots/interface recreation without needing a custom ethtool-invoking service (native systemd 259 directive, exactly for this).
  • Documented as a VPS deployment step alongside the existing honeypot-firewall.sh manual-apply convention.
  • Applying the live ethtool -K rx-gro-hw off toggle to the running VPS directly (already done for verification) so alert volume drops immediately, ahead of the persistent fix landing.

Not a portbridge packet-size issue in the end, but the original hunch ("adjust packet size because of the port bridge") was right in spirit — it's a network-capture framing issue, just one layer further down (hypervisor NIC offload, not portbridge's own socket handling).

Metadata

Metadata

Assignees

Labels

bugSomething isn't workingin-progressActively being worked on

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions