Error when pushing file into new container

[fredluetkemeier]

Required information

• Distribution: Debian
• Distribution version: 11 (bullseye)
• The output of "incus info" or if that fails:
  • Kernel version: 5.10.0-28-amd64
  • LXC version: couldn't find this
  • Incus version: 0.5.1
  • Storage backend in use: dir

Issue description

An error occurs when trying to push a new file into a new container. This worked up until very recently.

Steps to reproduce

1. Launch a new incus container: incus launch images:ubuntu/22.04 test
2. Push a file into new container: incus file push ./script.sh test/home/ubuntu/script.sh

Information to attach

The following error occurs: Error: error receiving version packet from server: read unix @->/proc/self/fd/26/forkfile.sock: read: connection reset by peer

• Any relevant kernel output (dmesg)

• Container log (incus info NAME --show-log)

  • Log:

    lxc test 20240216060550.724 ERROR cgroup2_devices - ../src/lxc/cgroups/cgroup2_devices.c:bpf_program_load_kernel:332 - Operation not permitted - Failed to load bpf program: (null)
    lxc test 20240216060550.725 WARN conf - ../src/lxc/conf.c:lxc_map_ids:3621 - newuidmap binary is missing
    lxc test 20240216060550.725 WARN conf - ../src/lxc/conf.c:lxc_map_ids:3627 - newgidmap binary is missing
    lxc test 20240216060550.725 WARN conf - ../src/lxc/conf.c:lxc_map_ids:3621 - newuidmap binary is missing
    lxc test 20240216060550.725 WARN conf - ../src/lxc/conf.c:lxc_map_ids:3627 - newgidmap binary is missing

• Container configuration (incus config show NAME --expanded)

  • architecture: x86_64
    config:
    image.architecture: amd64
    image.description: Ubuntu jammy amd64 (20240214_20:43)
    image.os: Ubuntu
    image.release: jammy
    image.serial: "20240214_20:43"
    image.type: squashfs
    image.variant: default
    volatile.base_image: 6ab6ffeb00fa7af7a586c76c7601d50a84601c020e1fddd14c18acb1eb1befd5
    volatile.cloud-init.instance-id: c861e59e-4fea-4f11-b9b9-4c0d52b0d601
    volatile.eth0.host_name: veth07654474
    volatile.eth0.hwaddr: 00:16:3e:e0:8a:6b
    volatile.idmap.base: "0"
    volatile.idmap.current: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
    volatile.idmap.next: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
    volatile.last_state.idmap: '[{"Isuid":true,"Isgid":false,"Hostid":1000000,"Nsid":0,"Maprange":1000000000},{"Isuid":false,"Isgid":true,"Hostid":1000000,"Nsid":0,"Maprange":1000000000}]'
    volatile.last_state.power: RUNNING
    volatile.uuid: d9f73071-dc0b-49f5-9f8d-5b2fef931b1f
    volatile.uuid.generation: d9f73071-dc0b-49f5-9f8d-5b2fef931b1f
    devices:
    eth0:
    name: eth0
    network: incusbr0
    type: nic
    root:
    path: /
    pool: default
    type: disk
    ephemeral: false
    profiles:
    • default
      stateful: false
      description: ""

• Main daemon log (at /var/log/incus/incusd.log)

  • time="2024-02-16T00:05:11-06:00" level=warning msg="Failed to initialize fanotify, falling back on inotify" err="Failed to watch directory "/dev": no such device"
    time="2024-02-16T00:05:14-06:00" level=error msg="Failed initialising instance" err="Failed getting root disk: No root device could be found" instance=test project=default type=container
    time="2024-02-16T00:06:18-06:00" level=error msg="SFTP server stopped with error" err="signal: aborted" instance=test instanceType=container project=default stderr="runtime/cgo: bad stack bounds: lo=0xffff80030d9ccf60 hi=0x7ffcf26330f0"
    time="2024-02-16T00:16:01-06:00" level=error msg="SFTP server stopped with error" err="signal: aborted" instance=test instanceType=container project=default stderr="runtime/cgo: bad stack bounds: lo=0xffff800383b9a9c0 hi=0x7ffc7c465690"

• Output of the client with --debug

  • Not sure how to do this

• Output of the daemon with --debug (alternatively output of incus monitor --pretty while reproducing the issue)

  • DEBUG [2024-02-16T00:23:08-06:00] Event listener server handler started id=54c93a15-0e43-460c-a29e-d5dc3810b94e local=/var/lib/incus/unix.socket remote=@
    DEBUG [2024-02-16T00:23:27-06:00] Handling API request ip=@ method=GET protocol=unix url=/1.0 username=fred
    DEBUG [2024-02-16T00:23:27-06:00] Handling API request ip=@ method=POST protocol=unix url="/1.0/instances/test/files?path=%2Fhome%2Fubuntu%2Fscript.sh" username=fred
    ERROR [2024-02-16T00:23:27-06:00] SFTP server stopped with error err="signal: aborted" instance=test instanceType=container project=default stderr="runtime/cgo: bad stack bounds: lo=0xffff80036249d520 hi=0x7ffc9db62b30"

[fredluetkemeier]

So upon further investigation, it appears this may be related to a recent repository change (https://github.com/zabbly/incus), and not incus itself.

Downgrading all packages (incus, incus-base, incus-client) to version 0.5.1-202401301901-debian11 resolves the issue.

I think the problematic version is 0.5.1-202402142309-debian11.

[stgraber]

Looks like a Go crash in forkfile.

A recent package update has updated Go to version 1.22.

I'll have to try to reproduce your issue here to see if it's specific to running on Debian 11.

[stgraber]

I've got the issue reproduced on Debian 11, now to figure out what to do to sort it out :)

[stgraber]

Tracked it down to a Go issue. This seems to affect everyone doing containers stuff from Go and it hits older glibc systems (ubuntu 20.04, debian 11).

I've reverted the package to Go 1.21 for the time being, the repository should get updated with new builds in the next 3-4 hours.

[Salamandar]

Hi, the issue is still present in incus 0.6-202402281848-debian11, also installed from the zabbly repositories.

[stgraber]

@Salamandar

root@d11:~# dpkg -l | grep incus
ii  incus                                0.6-202402281848-debian11      amd64        Incus -  Container and virtualization daemon (container-only)
ii  incus-base                           0.6-202402281848-debian11      amd64        Incus -  Container and virtualization daemon (container-only)
ii  incus-client                         0.6-202402281848-debian11      amd64        Incus -  Command line client
root@d11:~# cat /etc/debian_version 
11.9
root@d11:~# incus launch images:ubuntu/22.04 test
Launching test
root@d11:~# incus file push /etc/hosts test/root/foo
root@d11:~# incus exec test -- cat /root/foo       
127.0.1.1	d11
127.0.0.1	localhost
::1		localhost ip6-localhost ip6-loopback
ff02::1		ip6-allnodes
ff02::2		ip6-allrouters

root@d11:~# 

[Salamandar]

@stgraber hah, sorry, i did some crap. I had multiple "things" installed on the server (lxd and incus), and i mixed up my scripts. It's indeed working in Incus (but not in lxd :D ). Sorry for the noise.