Every LXD container fails to start

[wucke13]

Describe the bug
I'm not able to start any (freshly created) container in lxd.

To Reproduce
Steps to reproduce the behavior:

1. virtualisation.lxd.enable = true
2. sudo lxd --init
3. lxc launch images:alpine/3.10

Expected behavior
Container starts without issues.

Additional context
Log gathered via lxc info --show-log enabled-sponge

lxc enabled-sponge 20190928102355.329 WARN     initutils - initutils.c:setproctitle:341 - Invalid argument - Failed to set cmdline
lxc enabled-sponge 20190928102355.370 ERROR    cgfsng - cgroups/cgfsng.c:mkdir_eexist_on_last:1279 - File exists - Failed to create directory "/sys/fs/cgroup/cpuset//lxc.monitor/enabled-sponge"
lxc enabled-sponge 20190928102355.370 ERROR    cgfsng - cgroups/cgfsng.c:monitor_create_path_for_hierarchy:1300 - Failed to create cgroup "/sys/fs/cgroup/cpuset//lxc.monitor/enabled-sponge"
lxc enabled-sponge 20190928102355.370 ERROR    cgfsng - cgroups/cgfsng.c:cgfsng_monitor_create:1390 - Failed to create cgroup "/sys/fs/cgroup/cpuset//lxc.monitor/enabled-sponge"
lxc enabled-sponge 20190928102355.370 ERROR    cgfsng - cgroups/cgfsng.c:mkdir_eexist_on_last:1279 - File exists - Failed to create directory "/sys/fs/cgroup/cpuset//lxc.monitor/enabled-sponge-1"
lxc enabled-sponge 20190928102355.370 ERROR    cgfsng - cgroups/cgfsng.c:monitor_create_path_for_hierarchy:1300 - Failed to create cgroup "/sys/fs/cgroup/cpuset//lxc.monitor/enabled-sponge-1"
lxc enabled-sponge 20190928102355.370 ERROR    cgfsng - cgroups/cgfsng.c:cgfsng_monitor_create:1390 - Failed to create cgroup "/sys/fs/cgroup/cpuset//lxc.monitor/enabled-sponge-1"
lxc enabled-sponge 20190928102355.371 ERROR    cgfsng - cgroups/cgfsng.c:mkdir_eexist_on_last:1279 - File exists - Failed to create directory "/sys/fs/cgroup/cpuset//lxc.monitor/enabled-sponge-2"
lxc enabled-sponge 20190928102355.371 ERROR    cgfsng - cgroups/cgfsng.c:monitor_create_path_for_hierarchy:1300 - Failed to create cgroup "/sys/fs/cgroup/cpuset//lxc.monitor/enabled-sponge-2"
lxc enabled-sponge 20190928102355.371 ERROR    cgfsng - cgroups/cgfsng.c:cgfsng_monitor_create:1390 - Failed to create cgroup "/sys/fs/cgroup/cpuset//lxc.monitor/enabled-sponge-2"
lxc enabled-sponge 20190928102355.372 ERROR    cgfsng - cgroups/cgfsng.c:__do_cgroup_enter:1500 - No space left on device - Failed to enter cgroup "/sys/fs/cgroup/cpuset//lxc.monitor/enabled-sponge-3/cgroup.procs"
lxc enabled-sponge 20190928102355.372 ERROR    start - start.c:__lxc_start:2009 - Failed to enter monitor cgroup
lxc enabled-sponge 20190928102355.372 ERROR    lxccontainer - lxccontainer.c:wait_on_daemonized_start:873 - Received container state "STOPPING" instead of "RUNNING"
lxc enabled-sponge 20190928102355.374 WARN     cgfsng - cgroups/cgfsng.c:cgfsng_monitor_destroy:1180 - No space left on device - Failed to move monitor 20481 to "/sys/fs/cgroup/cpuset//lxc.pivot/cgroup.procs"

lxc 20190928102355.374 WARN     commands - commands.c:lxc_cmd_rsp_recv:135 - Connection reset by peer - Failed to receive response for command "get_state"

Regarding the not enough space, I'm quiet sure that the report is wrong:

$ df -x tmpfs -x devtmpfs -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda2       466G  367G   97G  80% /
/dev/sda2       466G  367G   97G  80% /nix
/dev/sda2       466G  367G   97G  80% /var
/dev/sda1       500M   31M  470M   7% /boot
/dev/sda2       466G  367G   97G  80% /home

Metadata

• system: "x86_64-linux"
• host os: Linux 5.3.0, NixOS, 19.09beta439.b30f86ffc6a (Loris)
• multi-user?: yes
• sandbox: yes
• version: nix-env (Nix) 2.3
• channels(root): "nixos-19.09beta439.b30f86ffc6a"
• channels(wucke13): "nixos-unstable-19.09pre192418.e19054ab3cd"
• nixpkgs: /nix/var/nix/profiles/per-user/root/channels/nixos

Maintainer information:

# a list of nixpkgs attributes affected by the problem
attribute:
# a list of nixos modules affected by the problem
module:
  lxd

[wucke13]

@fpletz Maybe updating lxd helps?
I started working on a PR, though I depend on a haskellPackage which fails to build.
https://github.com/wucke13/nixpkgs/tree/lxd

[Lassulus]

tested with #70209 merged on top inside a nogui VM

[root@nixos:~]# lxd init
Would you like to use LXD clustering? (yes/no) [default=no]:  
Do you want to configure a new storage pool? (yes/no) [default=yes]: 
Name of the new storage pool [default=default]: 
Name of the storage backend to use (btrfs, dir, lvm) [default=btrfs]: dir
Would you like to connect to a MAAS server? (yes/no) [default=no]: 
Would you like to create a new local network bridge? (yes/no) [default=yes]: 
What should the new bridge be called? [default=lxdbr0]: 
What IPv4 address should be used? (CIDR subnet notation, “auto” or “none”) [default=auto]: 
What IPv6 address should be used? (CIDR subnet notation, “auto” or “none”) [default=auto]: 
Would you like LXD to be available over the network? (yes/no) [default=no]: 
Would you like stale cached images to be updated automatically? (yes/no) [default=yes] 
Would you like a YAML "lxd init" preseed to be printed? (yes/no) [default=no]: 

[root@nixos:~]# lxc launch images:alpine/3.10
Creating the container
Container name is: superb-kingfish          
Starting superb-kingfish

[root@nixos:~]# lxc list
+-----------------+---------+----------------------+-----------------------------------------------+------------+-----------+
|      NAME       |  STATE  |         IPV4         |                     IPV6                      |    TYPE    | SNAPSHOTS |
+-----------------+---------+----------------------+-----------------------------------------------+------------+-----------+
| superb-kingfish | RUNNING | 10.202.182.82 (eth0) | fd42:c87d:5a30:1d5c:216:3eff:fe5f:6ecd (eth0) | PERSISTENT | 0         |
+-----------------+---------+----------------------+-----------------------------------------------+------------+-----------+

[root@nixos:~]# lxc exec superb-kingfish -- ls -la /bin
total 892
drwxr-xr-x    2 root     root          4096 Oct 14 13:00 .
drwxr-xr-x   19 root     root          4096 Oct 14 13:00 ..
lrwxrwxrwx    1 root     root            12 Jun 19 17:14 arch -> /bin/busybox
lrwxrwxrwx    1 root     root            12 Jun 19 17:14 ash -> /bin/busybox
lrwxrwxrwx    1 root     root            12 Jun 19 17:14 base64 -> /bin/busybox
lrwxrwxrwx    1 root     root            12 Jun 19 17:14 bbconfig -> /bin/busybox
---s--x--x    1 root     root         14040 Jun 12 17:52 bbsuid
-rwxr-xr-x    1 root     root        833104 Jun 12 17:52 busybox
lrwxrwxrwx    1 root     root            12 Jun 19 17:14 cat -> /bin/busybox
lrwxrwxrwx    1 root     root            12 Jun 19 17:14 chgrp -> /bin/busybox
lrwxrwxrwx    1 root     root            12 Jun 19 17:14 chmod -> /bin/busybox
lrwxrwxrwx    1 root     root            12 Jun 19 17:14 chown -> /bin/busybox
lrwxrwxrwx    1 root     root            12 Jun 19 17:14 conspy -> /bin/busybox
lrwxrwxrwx    1 root     root            12 Jun 19 17:14 cp -> /bin/busybox
lrwxrwxrwx    1 root     root            12 Jun 19 17:14 date -> /bin/busybox
lrwxrwxrwx    1 root     root            12 Jun 19 17:14 dd -> /bin/busybox
lrwxrwxrwx    1 root     root            12 Jun 19 17:14 df -> /bin/busybox
lrwxrwxrwx    1 root     root            12 Jun 19 17:14 dmesg -> /bin/busybox
lrwxrwxrwx    1 root     root            12 Jun 19 17:14 dnsdomainname -> /bin/busybox
...

[wucke13]

Wrong commit?

[Lassulus]

yip