Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

level=error msg="unable to commit delete decisions conn.Receive: netlink receive: no such file or directory" #230

Closed
ne20002 opened this issue Jan 16, 2023 · 19 comments
Closed

level=error msg="unable to commit delete decisions conn.Receive: netlink receive: no such file or directory" #230

ne20002 opened this issue Jan 16, 2023 · 19 comments

Comments

@ne20002
Copy link

ne20002 commented Jan 16, 2023

I just upgraded to newest version 0.0.25 and found this in the logfile

level=error msg="unable to commit delete decisions conn.Receive: netlink receive: no such file or directory"

What may be the cause of this message?
@LaurenceJJones
Copy link
Contributor

LaurenceJJones commented Jan 19, 2023 •
edited

Guess you use NFTABLES?

Error line:

cs-firewall-bouncer/main.go

Line 222 in bb8370a

log.Errorf("unable to commit delete decisions %v", err)

Commit func:

cs-firewall-bouncer/nftables.go

Line 526 in c0a9f1d

func (n *nft) Commit() error {

Searching for message returns stackexchange post https://unix.stackexchange.com/questions/537912/nftables-rule-no-such-file-or-directory-error

Do your chains exist?

EDIT:
Also to note we released a a packet counting feature so it would be

I was getting Error: Could not process rule: No such file or directory for nftables counter rules on an embedded Linux system built with Yocto.

In my case, the issue was due to kernel configuration. I had to enable the following:

    CONFIG_NFT_COUNTER=m so that I could create counters.
    CONFIG_NFT_OBJREF=m so that I could refer to the counters by name in rules.

This mailing list post helped me:

[[OpenWrt-Devel] nftables: named counters broken on 18.06.4](https://desiato.infradead.org/pipermail/openwrt-devel/2019-September/024588.html) — September 2019

For us to know if this is the issue you could open your configuration and change the prometheus enabled -> false. To see if its the issue.

@ne20002
Copy link
Author

ne20002 commented Jan 24, 2023 •
edited

As far as I see, the bouncer is creating table, chains and rules (for nftables in my case) only or input hook. As I want to use it on a (OpenWrt) router I need to create my rules manually to cover forward hook.
I would like to try a new configuration with prometheus disabled but havent found info for this on documentation (https://docs.crowdsec.net/docs/bouncers/firewall/).
OpenWrt supports counter in nftables so I don't think this is an issue.

@alimdi
Copy link

alimdi commented Apr 12, 2023 •
edited

Same problem here without any way to determine what's causing the error.
Thus the bouncer doesn't add IPs to the blacklist even if the table and chain are present.

time="12-04-2023 13:16:48" level=debug msg="adding 162.0.209.211 to buffer "
time="12-04-2023 13:16:48" level=error msg="unable to commit add decisions conn.Receive: netlink receive: invalid argument"
time="12-04-2023 13:16:48" level=debug msg="committed added decisions"
time="12-04-2023 13:16:48" level=info msg="49614 decisions added"

configuration :

mode: nftables
nftables:
  ipv4:
    enabled: true
    set-only: true
    table: crowdsec
    chain: crowdsec-chain

The set is empty :

root@ddx4:/# nft list table crowdsec
table ip crowdsec {
        set crowdsec-blacklists {
                type ipv4_addr
                flags timeout
        }

        chain crowdsec-chain {
                type filter hook input priority filter; policy accept;
                ip saddr @crowdsec-blacklists drop
        }
}

@LaurenceJJones
Copy link
Contributor

LaurenceJJones commented Apr 12, 2023 •
edited

We figured out the issue is most likely linked to running nftables in ipv4 or ipv6 only. We have a current RC pending if you wish to download and replace the binary with to test if it fixes for you.

Instructions are for amd64 if you are not using it then please go to releases and update link to relevant platform 😄

sudo systemctl stop crowdsec-firewall-bouncer
cd /tmp
sudo cp $(which crowdsec-firewall-bouncer) crowdsec-firewall-bouncer.bak
wget https://github.com/crowdsecurity/cs-firewall-bouncer/releases/download/v0.0.26-rc3/crowdsec-firewall-bouncer-linux-amd64.tgz
tar -xvf crowdsec-firewall-bouncer-linux-amd64.tgz
sudo cp crowdsec-firewall-bouncer-v0.0.26-rc3/crowdsec-firewall-bouncer $(which crowdsec-firewall-bouncer)
sudo systemctl start crowdsec-firewall-bouncer
rm -rf crowdsec-firewall-bouncer-v0.0.26-rc3

@alimdi
Copy link

alimdi commented Apr 12, 2023

It's definitely working the RC version.
Thank you for your help, much appreciated.

@LaurenceJJones LaurenceJJones mentioned this issue Apr 12, 2023
Closed
@ne20002
Copy link
Author

ne20002 commented Apr 17, 2023 •
edited

I just updated to 0.0.26-rc4 on OpenWrt. The error message still appears. :(

time="17-04-2023 14:50:38" level=info msg="backend type : nftables"
time="17-04-2023 14:50:38" level=info msg="nftables initiated"
time="17-04-2023 14:50:38" level=info msg="Using API key auth"
time="17-04-2023 14:50:38" level=info msg="Processing new and deleted decisions . . ."
time="17-04-2023 14:50:43" level=info msg="715 decisions deleted"
time="17-04-2023 14:58:45" level=info msg="flushing 'crowdsec-blacklists' set in 'crowdsec' table"
time="17-04-2023 14:58:45" level=info msg="flushing 'crowdsec6-blacklists' set in 'crowdsec6' table"
time="17-04-2023 14:58:45" level=error msg="unable to commit add decisions Receive: netlink receive: no such file or directory"
time="17-04-2023 14:58:45" level=info msg="Shutting down firewall-bouncer service"
time="17-04-2023 14:58:47" level=info msg="backend type : nftables"
time="17-04-2023 14:58:47" level=info msg="nftables initiated"
time="17-04-2023 14:58:47" level=info msg="Using API key auth"
time="17-04-2023 14:58:47" level=info msg="Processing new and deleted decisions . . ."
time="17-04-2023 14:58:52" level=info msg="929 decisions deleted"
time="17-04-2023 15:07:25" level=info msg="49191 decisions added"
time="17-04-2023 15:07:30" level=info msg="1 decision deleted"
time="17-04-2023 15:18:03" level=info msg="1 decision added"
time="17-04-2023 15:18:13" level=error msg="unable to commit expired decisions conn.Receive: netlink receive: no such file or directory"

My bouncer config is

root@FB4040:/tmp/log# cat /var/etc/crowdsec/bouncers/crowdsec-firewall-bouncer.yaml
mode: nftables
pid_dir: /var/run/
update_frequency: 10s
daemonize: true
log_mode: file
log_dir: /var/log/
log_level: info
log_compression: true
log_max_size: 100
log_max_backups: 3
log_max_age: 30
api_url: http://cs-lapi:8014/
api_key: *************************
insecure_skip_verify: true
disable_ipv6: false
deny_action: drop
deny_log: false
supported_decisions_type:
  - ban
#to change log prefix
deny_log_prefix: "crowdsec: "
#to change the blacklists name
blacklists_ipv4: crowdsec-blacklists
blacklists_ipv6: crowdsec6-blacklists
#type of ipset to use
ipset_type: nethash
#if present, insert rule in those chains
iptables_chains:
  - INPUT
#  - FORWARD
#  - DOCKER-USER
## nftables
nftables:
  ipv4:
    enabled: true
    set-only: true
    table: crowdsec
    chain: input
  ipv6:
    enabled: true
    set-only: true
    table: crowdsec6
    chain: input
# packet filter
pf:
  # an empty disables the anchor
  anchor_name: ""
prometheus:
  enabled: false
  listen_addr: 127.0.0.1
  listen_port: 60601

This is the nftables tables/chains:

table ip crowdsec {
	set crowdsec-blacklists {
		type ipv4_addr
		flags timeout
	}

	chain input {
		type filter hook input priority filter + 4; policy accept;
		iifname { "wg1", "eth1" } ct state new ip saddr @crowdsec-blacklists counter packets 0 bytes 0 drop
	}

	chain forward {
		type filter hook forward priority filter + 4; policy accept;
		iifname { "wg1", "eth1" } ct state new ip saddr @crowdsec-blacklists counter packets 0 bytes 0 drop
	}
}


table ip6 crowdsec6 {
	set crowdsec6-blacklists {
		type ipv6_addr
		flags timeout
	}

	chain input {
		type filter hook input priority filter + 4; policy accept;
		iifname { "wg1", "eth1" } ct state new ip6 saddr @crowdsec6-blacklists counter packets 0 bytes 0 drop
	}

	chain forward {
		type filter hook forward priority filter + 4; policy accept;
		iifname { "wg1", "eth1" } ct state new ip6 saddr @crowdsec6-blacklists counter packets 0 bytes 0 drop
	}
}

@ne20002
Copy link
Author

ne20002 commented Apr 18, 2023

This is a log as of today:

time="17-04-2023 18:02:03" level=error msg="unable to commit expired decisions conn.Receive: netlink receive: no such file or directory"
time="17-04-2023 18:04:13" level=info msg="1 decision added"
time="17-04-2023 18:22:03" level=info msg="1 decision added"
time="17-04-2023 18:40:23" level=info msg="1 decision added"
time="17-04-2023 18:51:53" level=error msg="unable to commit expired decisions conn.Receive: netlink receive: no such file or directory"
time="17-04-2023 18:52:03" level=error msg="unable to commit expired decisions conn.Receive: netlink receive: no such file or directory"
time="17-04-2023 19:00:23" level=info msg="1 decision added"
time="17-04-2023 19:04:13" level=info msg="1 decision deleted"
time="17-04-2023 19:06:23" level=info msg="1 decision added"
time="17-04-2023 19:22:03" level=info msg="1 decision deleted"
time="17-04-2023 19:22:33" level=info msg="1 decision added"
time="17-04-2023 19:40:23" level=info msg="1 decision deleted"
time="17-04-2023 19:51:54" level=info msg="15 decisions deleted"
time="17-04-2023 19:52:00" level=error msg="unable to commit add decisions conn.Receive: netlink receive: no such file or directory"
time="17-04-2023 19:52:05" level=info msg="15 decisions deleted"
time="17-04-2023 19:57:23" level=info msg="1 decision added"
time="17-04-2023 20:06:23" level=info msg="1 decision deleted"
time="17-04-2023 20:15:23" level=info msg="1 decision added"
time="17-04-2023 20:51:53" level=error msg="unable to commit expired decisions conn.Receive: netlink receive: no such file or directory"
time="17-04-2023 20:52:03" level=error msg="unable to commit expired decisions conn.Receive: netlink receive: no such file or directory"
time="17-04-2023 20:57:23" level=info msg="1 decision deleted"
time="17-04-2023 21:03:33" level=info msg="1 decision added"
time="17-04-2023 21:04:02" level=info msg="1 decision added"
time="17-04-2023 21:04:43" level=info msg="2 decisions added"
time="17-04-2023 21:15:23" level=info msg="1 decision deleted"
time="17-04-2023 21:16:03" level=info msg="1 decision added"
time="17-04-2023 21:16:22" level=info msg="1 decision added"
time="17-04-2023 21:33:03" level=info msg="1 decision added"
time="17-04-2023 21:42:53" level=error msg="unable to commit expired decisions conn.Receive: netlink receive: no such file or directory"
time="17-04-2023 21:45:03" level=info msg="1 decision deleted"
time="17-04-2023 21:51:53" level=error msg="unable to commit expired decisions conn.Receive: netlink receive: no such file or directory"
time="17-04-2023 21:51:58" level=info msg="795 decisions added"
time="17-04-2023 21:52:03" level=error msg="unable to commit expired decisions conn.Receive: netlink receive: no such file or directory"
time="17-04-2023 21:52:09" level=info msg="1078 decisions added"
time="17-04-2023 21:57:52" level=info msg="1 decision added"
time="17-04-2023 22:03:33" level=info msg="1 decision deleted"
time="17-04-2023 22:03:42" level=info msg="1 decision added"
time="17-04-2023 22:04:03" level=error msg="unable to commit expired decisions conn.Receive: netlink receive: no such file or directory"
time="17-04-2023 22:04:12" level=info msg="1 decision deleted"
time="17-04-2023 22:04:42" level=error msg="unable to commit expired decisions conn.Receive: netlink receive: no such file or directory"
time="17-04-2023 22:13:32" level=info msg="1 decision deleted"
time="17-04-2023 22:16:03" level=info msg="1 decision deleted"
time="17-04-2023 22:16:22" level=error msg="unable to commit expired decisions conn.Receive: netlink receive: no such file or directory"
time="17-04-2023 22:51:53" level=error msg="unable to commit expired decisions conn.Receive: netlink receive: no such file or directory"
time="17-04-2023 22:52:03" level=error msg="unable to commit expired decisions conn.Receive: netlink receive: no such file or directory"
time="17-04-2023 22:57:52" level=error msg="unable to commit expired decisions conn.Receive: netlink receive: no such file or directory"
time="17-04-2023 23:03:43" level=info msg="1 decision deleted"
time="17-04-2023 23:11:53" level=info msg="1 decision added"
time="17-04-2023 23:19:53" level=info msg="1 decision added"
time="17-04-2023 23:31:32" level=info msg="1 decision added"
time="17-04-2023 23:33:32" level=info msg="1 decision added"
time="17-04-2023 23:51:52" level=error msg="unable to commit expired decisions conn.Receive: netlink receive: no such file or directory"
time="17-04-2023 23:51:57" level=info msg="794 decisions added"
time="17-04-2023 23:52:03" level=error msg="unable to commit expired decisions conn.Receive: netlink receive: no such file or directory"
time="17-04-2023 23:52:34" level=info msg="11126 decisions added"
time="17-04-2023 23:53:23" level=info msg="7 decisions added"
time="18-04-2023 00:11:53" level=info msg="1 decision deleted"
time="18-04-2023 00:19:43" level=error msg="unable to commit expired decisions conn.Receive: netlink receive: no such file or directory"
time="18-04-2023 00:19:53" level=info msg="1 decision deleted"
time="18-04-2023 00:23:03" level=info msg="1 decision added"
time="18-04-2023 00:25:03" level=info msg="1 decision added"
time="18-04-2023 00:31:33" level=info msg="1 decision deleted"
time="18-04-2023 00:35:33" level=info msg="1 decision added"
time="18-04-2023 00:39:33" level=info msg="1 decision added"
time="18-04-2023 00:45:23" level=info msg="1 decision added"
time="18-04-2023 00:51:53" level=error msg="unable to commit expired decisions conn.Receive: netlink receive: no such file or directory"
time="18-04-2023 00:52:03" level=error msg="unable to commit expired decisions conn.Receive: netlink receive: no such file or directory"
time="18-04-2023 00:52:32" level=info msg="1 decision added"
time="18-04-2023 00:53:23" level=info msg="7 decisions deleted"
time="18-04-2023 01:23:02" level=info msg="1 decision deleted"
time="18-04-2023 01:25:03" level=error msg="unable to commit expired decisions conn.Receive: netlink receive: no such file or directory"
time="18-04-2023 01:29:33" level=info msg="1 decision added"
time="18-04-2023 01:39:33" level=info msg="1 decision deleted"
time="18-04-2023 01:43:03" level=info msg="1 decision added"
time="18-04-2023 01:45:23" level=info msg="1 decision deleted"
time="18-04-2023 01:51:54" level=error msg="unable to commit expired decisions conn.Receive: netlink receive: no such file or directory"
time="18-04-2023 01:51:59" level=error msg="unable to commit add decisions conn.Receive: netlink receive: no such file or directory"
time="18-04-2023 01:52:05" level=info msg="18 decisions deleted"
time="18-04-2023 01:52:23" level=error msg="unable to commit expired decisions conn.Receive: netlink receive: no such file or directory"
time="18-04-2023 01:52:33" level=info msg="1 decision deleted"
time="18-04-2023 02:05:43" level=info msg="1 decision added"
time="18-04-2023 02:29:33" level=info msg="1 decision deleted"
time="18-04-2023 02:43:03" level=info msg="1 decision deleted"
time="18-04-2023 02:48:53" level=info msg="1 decision added"
time="18-04-2023 02:51:53" level=error msg="unable to commit expired decisions conn.Receive: netlink receive: no such file or directory"
time="18-04-2023 02:52:03" level=error msg="unable to commit expired decisions conn.Receive: netlink receive: no such file or directory"
time="18-04-2023 02:59:03" level=info msg="1 decision deleted"
time="18-04-2023 02:59:13" level=info msg="1 decision deleted"
time="18-04-2023 03:03:13" level=info msg="1 decision deleted"
time="18-04-2023 03:05:43" level=error msg="unable to commit expired decisions conn.Receive: netlink receive: no such file or directory"
time="18-04-2023 03:48:53" level=info msg="1 decision deleted"
time="18-04-2023 03:51:54" level=error msg="unable to commit expired decisions conn.Receive: netlink receive: no such file or directory"
time="18-04-2023 03:52:00" level=error msg="unable to commit add decisions conn.Receive: netlink receive: no such file or directory"
time="18-04-2023 03:52:05" level=info msg="16 decisions deleted"
time="18-04-2023 04:14:53" level=info msg="1 decision added"
time="18-04-2023 04:51:53" level=error msg="unable to commit expired decisions conn.Receive: netlink receive: no such file or directory"
time="18-04-2023 04:52:03" level=error msg="unable to commit expired decisions conn.Receive: netlink receive: no such file or directory"
time="18-04-2023 05:51:54" level=error msg="unable to commit expired decisions conn.Receive: netlink receive: no such file or directory"
time="18-04-2023 05:51:59" level=error msg="unable to commit add decisions conn.Receive: netlink receive: no such file or directory"
time="18-04-2023 05:52:05" level=info msg="13 decisions deleted"
time="18-04-2023 06:18:03" level=info msg="2 decisions added"
time="18-04-2023 06:20:12" level=info msg="1 decision deleted"
time="18-04-2023 06:29:53" level=info msg="1 decision added"
time="18-04-2023 06:51:53" level=error msg="unable to commit expired decisions conn.Receive: netlink receive: no such file or directory"
time="18-04-2023 06:52:03" level=error msg="unable to commit expired decisions conn.Receive: netlink receive: no such file or directory"
time="18-04-2023 07:18:02" level=info msg="2 decisions deleted"
time="18-04-2023 07:29:52" level=info msg="1 decision deleted"
time="18-04-2023 07:44:52" level=info msg="1 decision added"
time="18-04-2023 07:51:53" level=error msg="unable to commit expired decisions conn.Receive: netlink receive: no such file or directory"
time="18-04-2023 07:51:58" level=info msg="831 decisions added"
time="18-04-2023 07:52:03" level=info msg="16 decisions deleted"
time="18-04-2023 07:52:08" level=info msg="825 decisions added"
time="18-04-2023 07:57:22" level=info msg="1 decision added"
time="18-04-2023 08:12:33" level=info msg="1 decision added"
time="18-04-2023 08:14:32" level=info msg="1 decision added"
time="18-04-2023 08:21:43" level=info msg="1 decision added"
time="18-04-2023 08:38:53" level=info msg="1 decision added"
time="18-04-2023 08:44:53" level=error msg="unable to commit expired decisions conn.Receive: netlink receive: no such file or directory"
time="18-04-2023 08:51:52" level=error msg="unable to commit expired decisions conn.Receive: netlink receive: no such file or directory"
time="18-04-2023 08:52:03" level=error msg="unable to commit expired decisions conn.Receive: netlink receive: no such file or directory"
time="18-04-2023 08:57:22" level=info msg="1 decision deleted"
time="18-04-2023 09:12:33" level=error msg="unable to commit expired decisions conn.Receive: netlink receive: no such file or directory"
time="18-04-2023 09:14:32" level=info msg="1 decision deleted"
time="18-04-2023 09:21:42" level=info msg="1 decision deleted"
time="18-04-2023 09:31:13" level=info msg="1 decision deleted"
time="18-04-2023 09:31:22" level=info msg="1 decision deleted"
time="18-04-2023 09:32:02" level=info msg="1 decision added"
time="18-04-2023 09:38:52" level=error msg="unable to commit expired decisions conn.Receive: netlink receive: no such file or directory"
time="18-04-2023 09:51:54" level=info msg="4 decisions deleted"
time="18-04-2023 09:51:59" level=error msg="unable to commit add decisions conn.Receive: netlink receive: no such file or directory"
time="18-04-2023 09:52:04" level=info msg="4 decisions deleted"
time="18-04-2023 10:32:02" level=error msg="unable to commit expired decisions conn.Receive: netlink receive: no such file or directory"
time="18-04-2023 10:32:12" level=info msg="1 decision deleted"
time="18-04-2023 10:35:43" level=info msg="1 decision added"
time="18-04-2023 10:35:52" level=info msg="1 decision added"
time="18-04-2023 10:51:53" level=error msg="unable to commit expired decisions conn.Receive: netlink receive: no such file or directory"
time="18-04-2023 10:52:02" level=error msg="unable to commit expired decisions conn.Receive: netlink receive: no such file or directory"
time="18-04-2023 11:38:03" level=info msg="1 decision added"
time="18-04-2023 11:51:54" level=error msg="unable to commit expired decisions conn.Receive: netlink receive: no such file or directory"
time="18-04-2023 11:51:59" level=error msg="unable to commit add decisions conn.Receive: netlink receive: no such file or directory"
time="18-04-2023 11:52:05" level=info msg="19 decisions deleted"
time="18-04-2023 11:53:02" level=info msg="1 decision added"
time="18-04-2023 12:01:32" level=info msg="1 decision added"
time="18-04-2023 12:38:03" level=info msg="1 decision deleted"
time="18-04-2023 12:39:43" level=info msg="1 decision added"
time="18-04-2023 12:42:23" level=info msg="1 decision added"
time="18-04-2023 12:51:53" level=error msg="unable to commit expired decisions conn.Receive: netlink receive: no such file or directory"
time="18-04-2023 12:52:03" level=error msg="unable to commit expired decisions conn.Receive: netlink receive: no such file or directory"
time="18-04-2023 12:53:02" level=info msg="1 decision deleted"
time="18-04-2023 12:56:02" level=info msg="1 decision added"
time="18-04-2023 13:01:32" level=info msg="1 decision deleted"
time="18-04-2023 13:03:53" level=error msg="unable to commit add decisions conn.Receive: netlink receive: no such file or directory"
time="18-04-2023 13:09:53" level=info msg="1 decision added"
time="18-04-2023 13:10:02" level=info msg="1 decision added"
time="18-04-2023 13:10:22" level=info msg="1 decision added"
time="18-04-2023 13:39:42" level=info msg="1 decision deleted"
time="18-04-2023 13:44:43" level=info msg="1 decision added"
time="18-04-2023 13:45:33" level=info msg="1 decision deleted"
time="18-04-2023 13:45:42" level=info msg="1 decision deleted"
time="18-04-2023 13:51:54" level=error msg="unable to commit expired decisions conn.Receive: netlink receive: no such file or directory"
time="18-04-2023 13:52:49" level=info msg="15810 decisions added"
time="18-04-2023 13:52:54" level=info msg="26 decisions deleted"
time="18-04-2023 13:55:53" level=error msg="unable to commit expired decisions conn.Receive: netlink receive: no such file or directory"
time="18-04-2023 13:56:03" level=info msg="1 decision deleted"
time="18-04-2023 14:03:52" level=info msg="1 decision deleted"
time="18-04-2023 14:09:53" level=info msg="1 decision deleted"
time="18-04-2023 14:10:03" level=info msg="1 decision deleted"
time="18-04-2023 14:10:23" level=info msg="1 decision deleted"
time="18-04-2023 14:35:43" level=info msg="1 decision added"
time="18-04-2023 14:44:43" level=info msg="1 decision deleted"
time="18-04-2023 14:49:52" level=info msg="1 decision added"
time="18-04-2023 14:51:53" level=error msg="unable to commit expired decisions conn.Receive: netlink receive: no such file or directory"
time="18-04-2023 14:52:03" level=error msg="unable to commit expired decisions conn.Receive: netlink receive: no such file or directory"
time="18-04-2023 15:30:43" level=info msg="1 decision added"
time="18-04-2023 15:35:43" level=info msg="1 decision deleted"
time="18-04-2023 15:49:53" level=error msg="unable to commit expired decisions conn.Receive: netlink receive: no such file or directory"
time="18-04-2023 15:50:03" level=info msg="1 decision deleted"
time="18-04-2023 15:51:53" level=error msg="unable to commit expired decisions conn.Receive: netlink receive: no such file or directory"
time="18-04-2023 15:52:04" level=error msg="unable to commit expired decisions conn.Receive: netlink receive: no such file or directory"
time="18-04-2023 15:52:20" level=info msg="15805 decisions added"
time="18-04-2023 15:53:03" level=info msg="1 decision added"
time="18-04-2023 16:23:52" level=info msg="1 decision added"
time="18-04-2023 16:30:43" level=info msg="1 decision deleted"
time="18-04-2023 16:51:53" level=error msg="unable to commit expired decisions conn.Receive: netlink receive: no such file or directory"
time="18-04-2023 16:52:03" level=error msg="unable to commit expired decisions conn.Receive: netlink receive: no such file or directory"
time="18-04-2023 16:53:03" level=info msg="1 decision deleted"
time="18-04-2023 16:59:03" level=info msg="1 decision added"
time="18-04-2023 17:04:14" level=info msg="1 decision added"
time="18-04-2023 17:23:52" level=info msg="1 decision deleted"
time="18-04-2023 17:51:54" level=error msg="unable to commit expired decisions conn.Receive: netlink receive: no such file or directory"
time="18-04-2023 17:52:19" level=info msg="15805 decisions added"
time="18-04-2023 17:52:24" level=error msg="unable to commit expired decisions conn.Receive: netlink receive: no such file or directory"

@ne20002
Copy link
Author

ne20002 commented Apr 19, 2023

Logging with debug level does not really give more information :(

time="19-04-2023 14:41:06" level=debug msg="req-api: GET http://cs-lapi:8014/v1/decisions/stream?"
time="19-04-2023 14:41:06" level=debug msg="resp-api: http 200"
time="19-04-2023 14:41:06" level=debug msg="[headers] Date : [Wed, 19 Apr 2023 14:41:06 GMT]"
time="19-04-2023 14:41:06" level=debug msg="[headers] Content-Length : [27]"
time="19-04-2023 14:41:06" level=debug msg="[headers] Content-Type : [application/json; charset=utf-8]"
time="19-04-2023 14:41:06" level=debug msg="Response: HTTP/1.1 200 OK\r\nContent-Length: 27\r\nContent-Type: application/json; charset=utf-8\r\nDate: Wed, 19 Apr 2023 14:41:06 GMT\r\n\r\n{\"deleted\":null,\"new\":null}"
time="19-04-2023 14:41:16" level=debug msg="req-api: GET http://cs-lapi:8014/v1/decisions/stream?"
time="19-04-2023 14:41:16" level=debug msg="resp-api: http 200"
time="19-04-2023 14:41:16" level=debug msg="[headers] Content-Type : [application/json; charset=utf-8]"
time="19-04-2023 14:41:16" level=debug msg="[headers] Date : [Wed, 19 Apr 2023 14:41:16 GMT]"
time="19-04-2023 14:41:16" level=debug msg="[headers] Content-Length : [27]"
time="19-04-2023 14:41:16" level=debug msg="Response: HTTP/1.1 200 OK\r\nContent-Length: 27\r\nContent-Type: application/json; charset=utf-8\r\nDate: Wed, 19 Apr 2023 14:41:16 GMT\r\n\r\n{\"deleted\":null,\"new\":null}"
time="19-04-2023 14:41:26" level=debug msg="req-api: GET http://cs-lapi:8014/v1/decisions/stream?"
time="19-04-2023 14:41:26" level=debug msg="resp-api: http 200"
time="19-04-2023 14:41:26" level=debug msg="[headers] Content-Type : [application/json; charset=utf-8]"
time="19-04-2023 14:41:26" level=debug msg="[headers] Date : [Wed, 19 Apr 2023 14:41:26 GMT]"
time="19-04-2023 14:41:26" level=debug msg="[headers] Content-Length : [167]"
time="19-04-2023 14:41:26" level=debug msg="Response: HTTP/1.1 200 OK\r\nContent-Length: 167\r\nContent-Type: application/json; charset=utf-8\r\nDate: Wed, 19 Apr 2023 14:41:26 GMT\r\n\r\n{\"deleted\":[{\"duration\":\"-5.257460673s\",\"id\":68124674,\"origin\":\"fail2ban\",\"scenario\":\"fail2ban/nginx-444\",\"scope\":\"Ip\",\"type\":\"ban\",\"value\":\"221.11.5.55\"}],\"new\":null}"
time="19-04-2023 14:41:26" level=debug msg="deleted '221.11.5.55'"
time="19-04-2023 14:41:26" level=debug msg="committing expired decisions"
time="19-04-2023 14:41:31" level=debug msg="will delete 221.11.5.55"
time="19-04-2023 14:41:31" level=error msg="unable to commit expired decisions conn.Receive: netlink receive: no such file or directory"
time="19-04-2023 14:41:36" level=debug msg="req-api: GET http://cs-lapi:8014/v1/decisions/stream?"
time="19-04-2023 14:41:36" level=debug msg="resp-api: http 200"
time="19-04-2023 14:41:36" level=debug msg="[headers] Date : [Wed, 19 Apr 2023 14:41:36 GMT]"
time="19-04-2023 14:41:36" level=debug msg="[headers] Content-Length : [27]"
time="19-04-2023 14:41:36" level=debug msg="[headers] Content-Type : [application/json; charset=utf-8]"
time="19-04-2023 14:41:36" level=debug msg="Response: HTTP/1.1 200 OK\r\nContent-Length: 27\r\nContent-Type: application/json; charset=utf-8\r\nDate: Wed, 19 Apr 2023 14:41:36 GMT\r\n\r\n{\"deleted\":null,\"new\":null}"
time="19-04-2023 14:41:46" level=debug msg="req-api: GET http://cs-lapi:8014/v1/decisions/stream?"
time="19-04-2023 14:41:46" level=debug msg="resp-api: http 200"
time="19-04-2023 14:41:46" level=debug msg="[headers] Content-Type : [application/json; charset=utf-8]"
time="19-04-2023 14:41:46" level=debug msg="[headers] Date : [Wed, 19 Apr 2023 14:41:46 GMT]"
time="19-04-2023 14:41:46" level=debug msg="[headers] Content-Length : [27]"
time="19-04-2023 14:41:46" level=debug msg="Response: HTTP/1.1 200 OK\r\nContent-Length: 27\r\nContent-Type: application/json; charset=utf-8\r\nDate: Wed, 19 Apr 2023 14:41:46 GMT\r\n\r\n{\"deleted\":null,\"new\":null}"
time="19-04-2023 14:41:56" level=debug msg="req-api: GET http://cs-lapi:8014/v1/decisions/stream?"
time="19-04-2023 14:41:56" level=debug msg="resp-api: http 200"
time="19-04-2023 14:41:56" level=debug msg="[headers] Content-Type : [application/json; charset=utf-8]"
time="19-04-2023 14:41:56" level=debug msg="[headers] Date : [Wed, 19 Apr 2023 14:41:56 GMT]"
time="19-04-2023 14:41:56" level=debug msg="[headers] Content-Length : [168]"
time="19-04-2023 14:41:56" level=debug msg="Response: HTTP/1.1 200 OK\r\nContent-Length: 168\r\nContent-Type: application/json; charset=utf-8\r\nDate: Wed, 19 Apr 2023 14:41:56 GMT\r\n\r\n{\"deleted\":[{\"duration\":\"-9.257480717s\",\"id\":68124675,\"origin\":\"fail2ban\",\"scenario\":\"fail2ban/nginx-444\",\"scope\":\"Ip\",\"type\":\"ban\",\"value\":\"111.33.20.65\"}],\"new\":null}"
time="19-04-2023 14:41:56" level=debug msg="deleted '111.33.20.65'"
time="19-04-2023 14:41:56" level=debug msg="committing expired decisions"

@LaurenceJJones
Copy link
Contributor

So I been playing around with nftables abit, and nomally that error happens when the underlying table / chain does not exist. Do you have any other program that interfaces with nftables that could be flushing the set and then our bouncer does not know it has been removed?

@LaurenceJJones
Copy link
Contributor

You can confirm if they are missing if you run

nft list tables

It will not show the crowdsec ones

@ne20002
Copy link
Author

ne20002 commented Apr 21, 2023

Hi @LaurenceJJones

actually it shows the crowdsec tables.

With the OpenWrt package the tables, chains and sets are created with the init script. As in the config file, the bouncer uses set-only mode (which I beleive is for exactly this situation). There is no other process manipulating the sets. But they are created with
'flags timeout' whereas ips can be removed by nftables itself.
Also there is a lot of calls successful, which makes me think the table, chain, set isn't the problem.

@LaurenceJJones
Copy link
Contributor

Interesting, if we set it to timeout, we shouldn't care to remove deleted decisions as nftables will self remove.

I think you're on the right track, I will test this once I am able to do so.

@trick77
Copy link

trick77 commented Apr 25, 2023

Same error message here on a Debian 11 Linux server (using nftables/firewalld). The tables do exist though:

$ nft list tables
table inet filter
table ip crowdsec
table ip6 crowdsec6

iptables is present as well, it gets installed with firewalld. Can't remove it.

@ne20002
Copy link
Author

ne20002 commented May 5, 2023

The messages still appear with 0.0.26-rc6

@mmetc
Copy link
Contributor

mmetc commented May 11, 2023

Should be fixed in #279

0.0.27-rc1 soon

@mmetc mmetc closed this as completed in b811ee5 May 11, 2023
@mmetc
Copy link
Contributor

mmetc commented May 11, 2023

0.0.27-rc1 is out, should fix this issue

https://packagecloud.io/crowdsec/crowdsec-testing

@ne20002
Copy link
Author

ne20002 commented May 13, 2023 •
edited

Hi @mmtec

Please reopen: this is with 0.0.27-rc1:

time="13-05-2023 17:00:15" level=info msg="backend type : nftables"
time="13-05-2023 17:00:15" level=info msg="nftables initiated"
time="13-05-2023 17:00:15" level=info msg="Using API key auth"
time="13-05-2023 17:00:15" level=info msg="Processing new and deleted decisions . . ."
time="13-05-2023 17:00:18" level=info msg="1075 decisions deleted"
time="13-05-2023 17:00:21" level=info msg="28814 decisions added"
time="13-05-2023 17:05:08" level=info msg="failed to flush chunk of 25 elements, will retry each one: conn.Receive: netlink receive: no such file or directory"
time="13-05-2023 17:05:12" level=info msg="25 decisions deleted"
time="13-05-2023 17:05:20" level=info msg="25 decisions deleted"

Maybe a mitigation may be to increase (by 25%?) the timeout/expire values when adding elements to the sets by the bouncer? So that they are not gone when trying to remove them?

@mmetc
Copy link
Contributor

mmetc commented May 14, 2023 •
edited

Hi @ne20002

Actually, it works as intended, and the message down to "info" level. The logic is now

  • delete (up to) 200 ips
  • if flush returns an error, repeat the deletions without batching them

The message can be improved but it's not meant to alarm, you may perceive it as that because it was associated with a fatal error before

@ne20002
Copy link
Author

ne20002 commented May 14, 2023

Hi @mmetc
The message appeared only once until now, so not really an issue. A more clear maessage would be helpful. ;)

As stated: wouldn't increasing the timeout/expire values when adding the ips to set mitigate the problem? I understand it as now

  • an ip is added with a timeout which is the ban time for the ip until it is automatically
  • removed from the set by the bouncer.
    If there is a dealy, the ip may have gone from the set which causes the error. So adding extra 15min to the timeout when adding to the set may ensure that the ip is still there when the bouncer tries to remove it (at least, most the times).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@trick77 @LaurenceJJones @ne20002 @alimdi @mmetc