Operations and Troubleshooting

Operations and Troubleshooting

First-response checklist

Run these before changing anything:

sudo axdp status
sudo axdp backend
sudo axdp ports
sudo axdp stats
sudo cat /etc/auto_xdp/runtime-state.json
sudo cat /etc/auto_xdp/machine-state.json

Then collect service logs:

sudo journalctl -u xdp-port-sync -u auto-xdp-relay -n 200 --no-pager

For OpenRC, inspect rc-service ... status and the distribution's service logs.

A service is listening but unreachable

1. Confirm the socket and bind address:

   sudo ss -lntup
   sudo axdp ports

2. Check discovery.exclude_loopback, exclude_bind_cidrs, and exclude_ports.

3. Confirm the ingress interface appears in axdp backend and machine-state.json.

4. Check ACL/trust intent. A TCP ACL may be the only path to an otherwise closed port.

5. Inspect drop reasons in axdp tui or counters in axdp stats.

6. Test from a new remote connection; existing conntrack state can produce a different result.

7. Check cloud security groups, provider firewalls, routing, and the application itself.

For an intentional temporary opening:

sudo axdp permanent add tcp 8080 emergency-test

Remove it after diagnosis.

A private listener is being exposed

Exclude its port or local bind network:

sudo axdp exclude port add 3306
sudo axdp exclude src add 10.0.0.0/8
sudo axdp ports

exclude src means the local bind source/address. If a process binds 0.0.0.0, a private-CIDR exclusion will not classify it as private-only; exclude the port or fix the bind address.

XDP fell back to nftables

sudo axdp backend
uname -r
sudo mount | grep ' /sys/fs/bpf '
command -v bpftool clang tc nft

Common causes:

• kernel lacks the required ARRAY_OF_MAPS behavior (normally need 5.10+);
• BPF creation denied by privileges, lockdown, container policy, or LSM;
• missing BPF object or required pinned map;
• verifier rejection on the current kernel/toolchain;
• NIC/driver cannot attach native XDP and generic mode is disallowed;
• tc egress tracker cannot be installed or verified.

Use bash setup_xdp.sh --dry-run and reinstall with logs. Do not manually detach a healthy fallback until the XDP cause is understood.

XDP program disappeared after a link bounce

The launcher verifies every configured interface and reattaches the pinned program when possible:

sudo axdp restart
sudo axdp backend
ip -d link show dev eth0
tc filter show dev eth0 egress

If tc restoration fails while XDP remains attached, backend selection aborts rather than creating an unsafe mixed state.

Configuration change did not apply

Direct file saves are not watched automatically:

sudo axdp restart
sudo journalctl -u xdp-port-sync -n 100 --no-pager

An invalid TOML reload retains the previous configuration. Validate syntax with Python 3.11+:

python3 -c 'import tomllib; tomllib.load(open("/etc/auto_xdp/config.toml", "rb"))'

On Python 3.10, use the installed tomli module instead.

Unexpected rate-limit drops

Match the counter to its layer:

• SYN_RATE_DROP: per source.
• SYN_AGG_RATE_DROP: source prefix.
• TCP_CONN_LIMIT_DROP: established per source.
• TCP_CONN_PREFIX_LIMIT_DROP: established per prefix.
• TCP_CONN_PORT_LIMIT_DROP: total on port.
• UDP_RATE_DROP, UDP_AGG_RATE_DROP, UDP_GLOBAL_RATE_DROP: corresponding UDP layers.

Check the owning process and /etc/services name shown by axdp ports. Adjust the narrow process/service override instead of globally disabling protection. NAT gateways may require less aggressive prefix grouping or a specific trusted/ACL rule.

TUI has no live events

sudo axdp under-attack
sudo systemctl status auto-xdp-relay
sudo ls -ld /var/run/auto_xdp
sudo ls -l /var/run/auto_xdp/pkt_events.sock
sudo journalctl -u auto-xdp-relay -n 100 --no-pager

Under-attack mode intentionally suppresses drop events. If the relay is missing after an upgrade, rerun bash setup_xdp.sh --force to install its service.

AbuseIPDB is not blocking

• Confirm XDP is active; the integration is XDP-only.
• Confirm [abuseipdb].enabled = true.
• Wait for the first fetch or inspect debug logs.
• Confirm outbound HTTPS/DNS access to the configured base URL.
• Remember that trusted CIDRs bypass the blocklist.
• Only IPv4 entries are supplied.

An empty or failed refresh intentionally leaves the previous map and never activates an empty new policy.

Inspect pins and rules safely

sudo find /sys/fs/bpf/xdp_fw -maxdepth 1 -type f -print
sudo bpftool prog show pinned /sys/fs/bpf/xdp_fw/prog
sudo bpftool map show
sudo nft list table inet auto_xdp

Do not delete pins, detach XDP, or flush the nft table during ordinary diagnosis. Those actions can remove protection and complicate rollback.

Collect a useful bug report

Include:

• Auto XDP release/commit and readlink -f /usr/local/lib/auto_xdp/current;
• distribution, kernel, architecture, init system, and NIC/driver;
• axdp backend, redacted config, and relevant service logs;
• exact source/destination/protocol and expected versus actual verdict;
• verifier log or packet capture when applicable;
• whether nftables reproduces the behavior.

Never post secrets or private infrastructure details. Report firewall bypasses and privilege/security issues through GitHub Private Vulnerability Reporting, not a public issue.