libvirt: Failed to initialize a valid firewall backend

[TimKlampe]

Issue description

Using the latest packages from the unstable channel I am unable to enable the default virsh network. The error indicates an issue with the firewall backend, but I disabled the firewall systemwide through:
networking.firewall.enable = false;

What am I missing here?

Steps to reproduce

sudo virsh net-list --all
 Name      State      Autostart   Persistent
----------------------------------------------
 default   inactive   yes         yes

sudo virsh net-start default
error: Failed to start network default
error: internal error: Failed to initialize a valid firewall backend

Technical details

• system: "x86_64-linux"
• host os: Linux 5.4.3, NixOS, 20.03pre205710.352f030b715 (Markhor)
• multi-user?: yes
• sandbox: yes
• version: nix-env (Nix) 2.3.1
• channels(root): "nixos-20.03pre205710.352f030b715"
• channels(tim): ""
• nixpkgs: /nix/var/nix/profiles/per-user/root/channels/nixos

[teto]

Have you looked into libvirtd logs for a better characterization of the issue ?

[TimKlampe]

Yes but I could not find anything meaninful, but I'll continue trying to find something

[izuk]

I'm seeing this the libvirtd.log:

2019-12-19 04:34:37.251+0000: 4078: error : virFileReadAll:1431 : Failed to open file '/sys/class/net/virbr0-nic/operstate': No such file or directory
2019-12-19 04:34:37.251+0000: 4078: error : virNetDevGetLinkInfo:2460 : unable to read: /sys/class/net/virbr0-nic/operstate: No such file or directory

Also, I noticed that ebtables was updated recently.

[rembo10]

I'm facing the same issue. I have the nixos firewall enabled, and it complains it can't find ebtables. If I install it separately that error goes away but I still see the errors posted by @izuk above

[izuk]

I confirmed that reverting ebtables to 2.0.10-4 fixes this.

Not sure what the next step is.

[izuk]

Idea: set EBTABLES_PATH=/sbin/ebtables-legacy in libvirt config.

[andersk]

When #75026 bumped ebtables from 2.0.10-4 to 2.0.11, the ebtables binary was renamed to ebtables-legacy, so the libvirtd configure script fails to find it at build time.

$ nix log `which libvirtd` | grep ebtables
[0.0 MiB DL]
checking for ebtables... /sbin/ebtables
  CC       util/libvirt_util_la-virebtables.lo
libtool: finish: PATH="/nix/store/66l2b2byvrlnx8s9g557ms876n6s3adl-dnsmasq-2.80/bin:/nix/store/r1pnay5aw9cfi833n0rczfm9vq0dgiwm-iproute2-5.4.0/bin:/nix/store/hyavnr4a090q4q6hvkgrd4a5gk90617h-iptables-1.8.4/bin:/nix/store/0yakr80cfsqvc1mfpcs87v0ib6p7g359-ebtables-2.0.11/bin:/nix/store/6rn9l7b4xqx29b3a1hkvpvs0jgfn0s83-lvm2-2.03.01/bin:/nix/store/xqc2dzks2v9ddf7q3i1xmw2560ys97l7-systemd-243.3/bin:/nix/store/s59i6kddn8z76gvn45zc0gq8n9z939sc-numad-0.5/bin:/nix/store/dx9m1whplkkhlch81z2k655sqpy53lij-pkg-config-0.29.2/bin:/nix/store/941fc45yy1vx1yhn9ph88bq80gzbpb0p-patchelf-0.9/bin:/nix/store/47hi526v96skkwgdj7d9c31ccd1xivrx-gcc-wrapper-8.3.0/bin:/nix/store/c6jvs50xrcy2m1nbrmpjwad51f62fcrx-gcc-8.3.0/bin:/nix/store/41asifinxaxn2jjjrlfq9yk57fqp7yyj-glibc-2.27-bin/bin:/nix/store/d7hlkykjjqs3f200jnmjm1y2hzgvbqa8-coreutils-8.31/bin:/nix/store/b31bs3028964clajg0aalqdyh3im00fd-binutils-wrapper-2.31.1/bin:/nix/store/m8ih8gw7qnd2c7p0qgv3cn4b6abkbanb-binutils-2.31.1/bin:/nix/store/41asifinxaxn2jjjrlfq9yk57fqp7yyj-glibc-2.27-bin/bin:/nix/store/d7hlkykjjqs3f200jnmjm1y2hzgvbqa8-coreutils-8.31/bin:/nix/store/8bvz3q83035x61dvgr2ybdjk05sadk70-libxml2-2.9.10-dev/bin:/nix/store/qbdnzlq4qan57rgzps1rh7zh3vpkmyh2-libxml2-2.9.10-bin/bin:/nix/store/9gh2s7a8g3y3d0drd9g9c0a5yykdhwl4-nettle-3.5.1-dev/bin:/nix/store/cxx645avlqq66f0n9qizhlffjlm7h65i-gnutls-3.6.11.1-bin/bin:/nix/store/ihgwxaqh6c80kh581vhw8lkgljd4mdnv-perl-5.30.1/bin:/nix/store/gk0lrrvqz491xn0mgrshzgvazx4rghl2-python-2.7.17/bin:/nix/store/qz58s29a7zfjwbhs882zbyrd45wpp49f-ncurses-6.1-20190112-dev/bin:/nix/store/3cz3xgnghkpy41ab9hcj7011sqbrpg4n-ncurses-6.1-20190112/bin:/nix/store/j1cp9hbbk7lkimm8v5q20rd870qli11x-readline-6.3p08/bin:/nix/store/q94jgppkjmqs1djawx4c28ywdars5qcn-gettext-0.20.1/bin:/nix/store/wkvgpjv5n6w7zy79cpj7alvjnp702xix-libtasn1-4.14-dev/bin:/nix/store/lghdrqnhnh1dblkfkcmnnwg78fc4vmfz-libgcrypt-1.8.5-dev/bin:/nix/store/sdz4bsn2hx4w7n7vla0bd759xi7cwmbd-yajl-2.1.0/bin:/nix/store/3pg1f05s5aylkyih3k9hgdbi3fcp1f6i-libxslt-1.1.34-dev/bin:/nix/store/lnf79xfv8hjz7kmh7y9gsam29vj6anlv-libxslt-1.1.34-bin/bin:/nix/store/z2kncdwxwzzwq2d6w2bir8hsqiq1l72j-perl5.30.1-XML-XPath-1.44/bin:/nix/store/0da7jlxvwg9gpx0j1b2571gfdgq85sz4-perl5.30.1-libwww-perl-6.42/bin:/nix/store/00zf0sh2yxsmi083my3daaxc8iy5ynga-curl-7.67.0-dev/bin:/nix/store/jqylvcycv8iyryic8ndz5dapgna1mav6-nghttp2-1.40.0-bin/bin:/nix/store/czcsr5z4xx8ib1r442n4irvrvxycj463-libkrb5-1.17-dev/bin:/nix/store/dzga394c4rdg0si8jh0rkczb8k7v3p37-libkrb5-1.17/bin:/nix/store/vxrhhpfpvljqpi4yl54xq5hdm5rvnqsw-openssl-1.1.1d-bin/bin:/nix/store/zkx1azl49fjmxk4nn0q89dm46ir16lqi-curl-7.67.0-bin/bin:/nix/store/7b0xvwvhrp4f0sgpdvkviwasf94k52cz-libpcap-1.9.1/bin:/nix/store/2hgyczlndp1kc092kh2jv46scmzvj6ig-libtool-2.4.6/bin:/nix/store/86izrw1r1jz7nsymbf2jk554f6iw0927-gnum4-1.4.18/bin:/nix/store/89a2q6ldkkb7fdalfb2ykqya4xqsv8vv-autoconf-2.69/bin:/nix/store/s9h15viqp818w1g1gqpsrpxvxavkr7d1-automake-1.16.1/bin:/nix/store/6rn9l7b4xqx29b3a1hkvpvs0jgfn0s83-lvm2-2.03.01/bin:/nix/store/295sx902z8n4icq429q3p75srpxdx0wa-util-linux-2.33.2-bin/bin:/nix/store/xqc2dzks2v9ddf7q3i1xmw2560ys97l7-systemd-243.3/bin:/nix/store/2ca2m1w2b8ywjbyhzlwmgg6pn0sh5m3d-libnl-3.5.0-bin/bin:/nix/store/s59i6kddn8z76gvn45zc0gq8n9z939sc-numad-0.5/bin:/nix/store/l8r8r8i3g7zdlxb0sx1hr4h5kxvadkrq-zfs-user-0.8.2/bin:/nix/store/ymq7j40f74p088n57039cqy94dccirxz-libcap-ng-0.7.9/bin:/nix/store/kyv09y33s761v2ld2gs8klvfbrp2f6rr-numactl-2.0.13/bin:/nix/store/0f88fqzjphdvipikv52nhbf73k8rbss9-attr-2.4.48-bin/bin:/nix/store/a6jsvy1qjqix9xfypdmc86h0ixws6c6j-parted-3.3/bin:/nix/store/d7hlkykjjqs3f200jnmjm1y2hzgvbqa8-coreutils-8.31/bin:/nix/store/i72wars5xl7yqgfbx0g5lfkhr78m48m6-findutils-4.7.0/bin:/nix/store/13n1kg7j0248q2n069vnzbwiy8nh82yg-diffutils-3.7/bin:/nix/store/gv56q6g8c7s6amyh72ahyh5i1s4bdnbf-gnused-4.7/bin:/nix/store/qp2s597ignw8m5ywzwx98mjv9waskqyj-gnugrep-3.3/bin:/nix/store/791r84jgd68r4cg2v7a8hjfa0ycx3kdw-gawk-5.0.1/bin:/nix/store/gvmn6xqpghbza0v8szhs22pg7qacn4l9-gnutar-1.32/bin:/nix/store/87j3arwf856dp46l1hfd1bzb2b3ii0w6-gzip-1.10/bin:/nix/store/v6m9vzg7zqv6bqm6vkdznaxxx8r70rmv-bzip2-1.0.6.0.1-bin/bin:/nix/store/8k67igas5d2i9mvdkvf56n17gflyv5yl-gnumake-4.2.1/bin:/nix/store/xb062l4b76zyhq6grqf4iyfdikkpg8fl-bash-4.4-p23/bin:/nix/store/0apklljlfn37szygbwl984vlnkj8hw9w-patch-2.7.6/bin:/nix/store/x4l3gg14lvapvhkvz6bbafpl7zqlfa3p-xz-5.2.4-bin/bin:/sbin" ldconfig -n /nix/store/nanyvn8hjwhh97sg4avslapyx9spi90a-libvirt-5.4.0/lib

$ ls /nix/store/0yakr80cfsqvc1mfpcs87v0ib6p7g359-ebtables-2.0.11/bin
ebtablesd        ebtables-legacy-restore  ebtablesu
ebtables-legacy  ebtables-legacy-save

[andersk]

Impure workaround for the desperate: add the ebtables package to environment.systemPackages, and symlink /sbin/ebtables to /run/current-system/sw/bin/ebtables-legacy.

(I also tried ebtables from the iptables-nftables-compat package, which does not seem to work.)

[balsoft]

Another workaround for less desperate: inherit (nixpkgs-19.03) ebtables in your overlay.

[kamidon]

I verified that providing an EBTABLES_PATH to the right ebtables-legacy binary in the derivation allows my VMs to work again. I tested in a overlay I maintain, from which a minimal (untested) extract to work around the issue until someone has time to come up with a more correct patch to submit upstream is:

self: super:
{
  # Patch libvirt to use ebtables-legacy
  libvirt = if super.libvirt.version <= "5.4.0" && super.ebtables.version > "2.0.10-4"
    then
      super.libvirt.overrideAttrs (oldAttrs: rec {
        EBTABLES_PATH="${self.ebtables}/bin/ebtables-legacy";
      })
    else super.libvirt;
}

[izuk]

I've confirmed that the overlay in #75878 (comment) works.

[d4g]

I also have the issue. This is actually a major issue, if libvirt does not work anymore.

[costrouc]

Had the same issue with unstable. The overlay above fixed it. Do we just need a PR to staging-next that adds that environment variable to the build? I'll submit that PR if that is all that is necessary -- didn't see any submitted yet.