Failure to start LXD VM due to long UNIX Socket Path

[cbartz]

Required information

• Distribution: Ubuntu
• Distribution version: 22.04
• The output of "lxc info":

config: {}
api_extensions:
- storage_zfs_remove_snapshots
- container_host_shutdown_timeout
- container_stop_priority
- container_syscall_filtering
- auth_pki
- container_last_used_at
- etag
- patch
- usb_devices
- https_allowed_credentials
- image_compression_algorithm
- directory_manipulation
- container_cpu_time
- storage_zfs_use_refquota
- storage_lvm_mount_options
- network
- profile_usedby
- container_push
- container_exec_recording
- certificate_update
- container_exec_signal_handling
- gpu_devices
- container_image_properties
- migration_progress
- id_map
- network_firewall_filtering
- network_routes
- storage
- file_delete
- file_append
- network_dhcp_expiry
- storage_lvm_vg_rename
- storage_lvm_thinpool_rename
- network_vlan
- image_create_aliases
- container_stateless_copy
- container_only_migration
- storage_zfs_clone_copy
- unix_device_rename
- storage_lvm_use_thinpool
- storage_rsync_bwlimit
- network_vxlan_interface
- storage_btrfs_mount_options
- entity_description
- image_force_refresh
- storage_lvm_lv_resizing
- id_map_base
- file_symlinks
- container_push_target
- network_vlan_physical
- storage_images_delete
- container_edit_metadata
- container_snapshot_stateful_migration
- storage_driver_ceph
- storage_ceph_user_name
- resource_limits
- storage_volatile_initial_source
- storage_ceph_force_osd_reuse
- storage_block_filesystem_btrfs
- resources
- kernel_limits
- storage_api_volume_rename
- macaroon_authentication
- network_sriov
- console
- restrict_devlxd
- migration_pre_copy
- infiniband
- maas_network
- devlxd_events
- proxy
- network_dhcp_gateway
- file_get_symlink
- network_leases
- unix_device_hotplug
- storage_api_local_volume_handling
- operation_description
- clustering
- event_lifecycle
- storage_api_remote_volume_handling
- nvidia_runtime
- container_mount_propagation
- container_backup
- devlxd_images
- container_local_cross_pool_handling
- proxy_unix
- proxy_udp
- clustering_join
- proxy_tcp_udp_multi_port_handling
- network_state
- proxy_unix_dac_properties
- container_protection_delete
- unix_priv_drop
- pprof_http
- proxy_haproxy_protocol
- network_hwaddr
- proxy_nat
- network_nat_order
- container_full
- candid_authentication
- backup_compression
- candid_config
- nvidia_runtime_config
- storage_api_volume_snapshots
- storage_unmapped
- projects
- candid_config_key
- network_vxlan_ttl
- container_incremental_copy
- usb_optional_vendorid
- snapshot_scheduling
- snapshot_schedule_aliases
- container_copy_project
- clustering_server_address
- clustering_image_replication
- container_protection_shift
- snapshot_expiry
- container_backup_override_pool
- snapshot_expiry_creation
- network_leases_location
- resources_cpu_socket
- resources_gpu
- resources_numa
- kernel_features
- id_map_current
- event_location
- storage_api_remote_volume_snapshots
- network_nat_address
- container_nic_routes
- rbac
- cluster_internal_copy
- seccomp_notify
- lxc_features
- container_nic_ipvlan
- network_vlan_sriov
- storage_cephfs
- container_nic_ipfilter
- resources_v2
- container_exec_user_group_cwd
- container_syscall_intercept
- container_disk_shift
- storage_shifted
- resources_infiniband
- daemon_storage
- instances
- image_types
- resources_disk_sata
- clustering_roles
- images_expiry
- resources_network_firmware
- backup_compression_algorithm
- ceph_data_pool_name
- container_syscall_intercept_mount
- compression_squashfs
- container_raw_mount
- container_nic_routed
- container_syscall_intercept_mount_fuse
- container_disk_ceph
- virtual-machines
- image_profiles
- clustering_architecture
- resources_disk_id
- storage_lvm_stripes
- vm_boot_priority
- unix_hotplug_devices
- api_filtering
- instance_nic_network
- clustering_sizing
- firewall_driver
- projects_limits
- container_syscall_intercept_hugetlbfs
- limits_hugepages
- container_nic_routed_gateway
- projects_restrictions
- custom_volume_snapshot_expiry
- volume_snapshot_scheduling
- trust_ca_certificates
- snapshot_disk_usage
- clustering_edit_roles
- container_nic_routed_host_address
- container_nic_ipvlan_gateway
- resources_usb_pci
- resources_cpu_threads_numa
- resources_cpu_core_die
- api_os
- container_nic_routed_host_table
- container_nic_ipvlan_host_table
- container_nic_ipvlan_mode
- resources_system
- images_push_relay
- network_dns_search
- container_nic_routed_limits
- instance_nic_bridged_vlan
- network_state_bond_bridge
- usedby_consistency
- custom_block_volumes
- clustering_failure_domains
- resources_gpu_mdev
- console_vga_type
- projects_limits_disk
- network_type_macvlan
- network_type_sriov
- container_syscall_intercept_bpf_devices
- network_type_ovn
- projects_networks
- projects_networks_restricted_uplinks
- custom_volume_backup
- backup_override_name
- storage_rsync_compression
- network_type_physical
- network_ovn_external_subnets
- network_ovn_nat
- network_ovn_external_routes_remove
- tpm_device_type
- storage_zfs_clone_copy_rebase
- gpu_mdev
- resources_pci_iommu
- resources_network_usb
- resources_disk_address
- network_physical_ovn_ingress_mode
- network_ovn_dhcp
- network_physical_routes_anycast
- projects_limits_instances
- network_state_vlan
- instance_nic_bridged_port_isolation
- instance_bulk_state_change
- network_gvrp
- instance_pool_move
- gpu_sriov
- pci_device_type
- storage_volume_state
- network_acl
- migration_stateful
- disk_state_quota
- storage_ceph_features
- projects_compression
- projects_images_remote_cache_expiry
- certificate_project
- network_ovn_acl
- projects_images_auto_update
- projects_restricted_cluster_target
- images_default_architecture
- network_ovn_acl_defaults
- gpu_mig
- project_usage
- network_bridge_acl
- warnings
- projects_restricted_backups_and_snapshots
- clustering_join_token
- clustering_description
- server_trusted_proxy
- clustering_update_cert
- storage_api_project
- server_instance_driver_operational
- server_supported_storage_drivers
- event_lifecycle_requestor_address
- resources_gpu_usb
- clustering_evacuation
- network_ovn_nat_address
- network_bgp
- network_forward
- custom_volume_refresh
- network_counters_errors_dropped
- metrics
- image_source_project
- clustering_config
- network_peer
- linux_sysctl
- network_dns
- ovn_nic_acceleration
- certificate_self_renewal
- instance_project_move
- storage_volume_project_move
- cloud_init
- network_dns_nat
- database_leader
- instance_all_projects
- clustering_groups
- ceph_rbd_du
- instance_get_full
- qemu_metrics
- gpu_mig_uuid
- event_project
- clustering_evacuation_live
- instance_allow_inconsistent_copy
- network_state_ovn
- storage_volume_api_filtering
- image_restrictions
- storage_zfs_export
- network_dns_records
- storage_zfs_reserve_space
- network_acl_log
- storage_zfs_blocksize
- metrics_cpu_seconds
- instance_snapshot_never
- certificate_token
- instance_nic_routed_neighbor_probe
- event_hub
- agent_nic_config
- projects_restricted_intercept
- metrics_authentication
- images_target_project
- cluster_migration_inconsistent_copy
- cluster_ovn_chassis
- container_syscall_intercept_sched_setscheduler
- storage_lvm_thinpool_metadata_size
- storage_volume_state_total
- instance_file_head
- resources_pci_vpd
- qemu_raw_conf
- storage_cephfs_fscache
- vsock_api
- storage_volumes_all_projects
- projects_networks_restricted_access
- cluster_join_token_expiry
- remote_token_expiry
- init_preseed
- cpu_hotplug
api_status: stable
api_version: "1.0"
auth: trusted
public: false
auth_methods:
- tls
environment:
  addresses: []
  architectures:
  - x86_64
  - i686
  certificate: |
    -----BEGIN CERTIFICATE-----
    MIICAjCCAYegAwIBAgIRAJrMJKkDW+Ad9tsMyXY1mAowCgYIKoZIzj0EAwMwMzEc
    MBoGA1UEChMTbGludXhjb250YWluZXJzLm9yZzETMBEGA1UEAwwKcm9vdEBzcXVp
    ZDAeFw0yMzExMjAwNzIyNDdaFw0zMzExMTcwNzIyNDdaMDMxHDAaBgNVBAoTE2xp
    bnV4Y29udGFpbmVycy5vcmcxEzARBgNVBAMMCnJvb3RAc3F1aWQwdjAQBgcqhkjO
    PQIBBgUrgQQAIgNiAAQcYTk+Me71KxfYD72/l7BxN6o5wSay5BZIt2g49zK5iGHM
    v1b1hh+O4lzAmXV4xEwSoT+B1VQ1/XYw+9M3k/W/YYS4MrjH5gBBooiRHFVuFVnt
    vSpdCHiKQERqiXF2QlmjXzBdMA4GA1UdDwEB/wQEAwIFoDATBgNVHSUEDDAKBggr
    BgEFBQcDATAMBgNVHRMBAf8EAjAAMCgGA1UdEQQhMB+CBXNxdWlkhwR/AAABhxAA
    AAAAAAAAAAAAAAAAAAABMAoGCCqGSM49BAMDA2kAMGYCMQC+hnOZZ4MEpbgEhyS6
    MQqI72Px8NW+o2LdIWU78gJ/8THkvhFzU4U99b4WBYnTroMCMQC0EUqFzP77bOCp
    WJVQtF6HK146l/639ED5Gp8p10q8Up9uV0/bw6kUYXYSIERLIJs=
    -----END CERTIFICATE-----
  certificate_fingerprint: f336157139686162d891253af88a84f732ae2d6c029651e081306b2a563d27aa
  driver: lxc | qemu
  driver_version: 5.0.2 | 7.1.0
  firewall: nftables
  kernel: Linux
  kernel_architecture: x86_64
  kernel_features:
    idmapped_mounts: "true"
    netnsid_getifaddrs: "true"
    seccomp_listener: "true"
    seccomp_listener_continue: "true"
    shiftfs: "false"
    uevent_injection: "true"
    unpriv_fscaps: "true"
  kernel_version: 5.15.0-87-generic
  lxc_features:
    cgroup2: "true"
    core_scheduling: "true"
    devpts_fd: "true"
    idmapped_mounts_v2: "true"
    mount_injection_file: "true"
    network_gateway_device_route: "true"
    network_ipvlan: "true"
    network_l2proxy: "true"
    network_phys_macvlan_mtu: "true"
    network_veth_router: "true"
    pidfd: "true"
    seccomp_allow_deny_syntax: "true"
    seccomp_notify: "true"
    seccomp_proxy_send_notify_fd: "true"
  os_name: Ubuntu
  os_version: "22.04"
  project: default
  server: lxd
  server_clustered: false
  server_event_mode: full-mesh
  server_name: squid
  server_pid: 12638
  server_version: 5.0.2
  storage: dir
  storage_version: "1"
  storage_supported_drivers:
  - name: lvm
    version: 2.03.07(2) (2019-11-30) / 1.02.167 (2019-11-30) / 4.45.0
    remote: false
  - name: zfs
    version: 2.1.5-1ubuntu6~22.04.1
    remote: false
  - name: btrfs
    version: 5.4.1
    remote: false
  - name: ceph
    version: 15.2.17
    remote: true
  - name: cephfs
    version: 15.2.17
    remote: true
  - name: cephobject
    version: 15.2.17
    remote: true
  - name: dir
    version: "1"
    remote: false

Issue description

When adding a disk to an LXD VM instance using a directory on the host with a rather long name, the instance fails to start because the UNIX socket path is too long:

ubuntu@ubuntu:~$ sudo cat /var/snap/lxd/common/lxd/logs/stg-ps6-small2-3-166d3e5f-ba4f-4cd0-817e-1ba1994e8c8d/qemu.log
qemu-system-x86_64:/var/snap/lxd/common/lxd/logs/stg-ps6-small2-3-166d3e5f-ba4f-4cd0-817e-1ba1994e8c8d/qemu.conf:254: UNIX socket path '/var/snap/lxd/common/lxd/devices/stg-ps6-small2-3-166d3e5f-ba4f-4cd0-817e-1ba1994e8c8d/virtio-fs.metrics.sock' is too long
Path must be less than 108 bytes

The problem occurred in an OpenStack VM (instance of a GitHub self-hosted runner charm), but could be reproduced locally inside a multipass vm.

Steps to reproduce

1. lxc launch ubuntu:22.04 stg-ps6-small2-3-166d3e5f-ba4f-4cd0-817e-1ba1994e8c8d --vm
2. lxc stop stg-ps6-small2-3-166d3e5f-ba4f-4cd0-817e-1ba1994e8c8d
3. mkdir /home/ubuntu/shared_fs
4. lxc config device add stg-ps6-small2-3-166d3e5f-ba4f-4cd0-817e-1ba1994e8c8d metrics disk source=/home/ubuntu/shared_fs/ path=/metrics-exchange
5. lxc start stg-ps6-small2-3-166d3e5f-ba4f-4cd0-817e-1ba1994e8c8d
6. sudo cat /var/snap/lxd/common/lxd/logs/stg-ps6-small2-3-166d3e5f-ba4f-4cd0-817e-1ba1994e8c8d/qemu.log

Information to attach

• Any relevant kernel output (dmesg)
• Container log (lxc info NAME --show-log)
• Container configuration (lxc config show NAME --expanded)
• Main daemon log (at /var/log/lxd/lxd.log or /var/snap/lxd/common/lxd/logs/lxd.log)
• Output of the client with --debug
• Output of the daemon with --debug (alternatively output of lxc monitor while reproducing the issue)

[tomponline]

Hi @cbartz please can you try this again on LXD 5.0.3, as I believe its already been fixed by:

https://github.com/canonical/lxd/blob/stable-5.0/lxd/device/device_utils_disk.go#L464-L468

[tomponline]

Ah no, it wont actually because of https://github.com/canonical/lxd/blob/stable-5.0/lxd/device/disk.go#L936 I think.
We need to see about using the same technique and passing a file handle to QEMU.

[hamistao]

So far I managed to get the file from the socket listener object, and pass it over to d.addFileDescriptor to get a valid file descriptor for the socket file and passing it over to qemu via the fd field under chardev and fsdev in the qemu.config file. I oppened a draft PR on this approach. Unfortunately this always results in the following error on qemu.log:

qemu-system-x86_64:/var/snap/lxd/common/lxd/logs/stg-ps6-small2-3-166d3e5f-ba4f-4cd0-817e-1ba1994e8c8d/qemu.conf:273: warning: '-fsdev proxy' and '-virtfs proxy' are deprecated, use 'local' instead of 'proxy, or consider deploying virtiofsd as alternative to 9p
qemu-system-x86_64:/var/snap/lxd/common/lxd/logs/stg-ps6-small2-3-166d3e5f-ba4f-4cd0-817e-1ba1994e8c8d/qemu.conf:267: Failed to write msg. Wrote -1 instead of 12.
qemu-system-x86_64:/var/snap/lxd/common/lxd/logs/stg-ps6-small2-3-166d3e5f-ba4f-4cd0-817e-1ba1994e8c8d/qemu.conf:267: vhost_backend_init failed: Protocol error

This is possibly a result of the qemu version being used in the snap, I am currently running some tests using other versions to see if this alters the behavior. An alternative approach could be to use symbolic links to the socket files, but I haven't figured an apropriate place to create these links on without resulting in a path longer than 107 bytes.

[tomponline]

@hamistao Don't worry about this line:

qemu-system-x86_64:/var/snap/lxd/common/lxd/logs/stg-ps6-small2-3-166d3e5f-ba4f-4cd0-817e-1ba1994e8c8d/qemu.conf:273: warning: '-fsdev proxy' and '-virtfs proxy' are deprecated, use 'local' instead of 'proxy, or consider deploying virtiofsd as alternative to 9p

That is known, see #12828

[tomponline]

One approach could be to open a file handle and then pass it as a file path using /proc/self/fd/<fdnum>

[hamistao]

If I am understanding correctly, I have tried this but it doesn't work because qemu doesn't recognize /proc/self as the same directory as LXD. Using /proc/{pid} instead also doesn't work because qemu doesn't have permission to acces this directory.

[roosterfish]

Might it be that the protocol error in #13320 is coming from having both path and fd config setting in the qemu config file for the chardev device? I have added a comment here #13320 (comment).

[hamistao]

Might it be that the protocol error in #13320 is coming from having both path and fd config setting in the qemu config file for the chardev device? I have added a comment here #13320 (comment).

this isn't the case, although it seemed like the path was still being included, the chardev section in the qemu.config generated looks like this:

# metrics drive (virtio-fs)
[chardev "lxd_metrics"]
backend = "socket"
fd = "6"

[cbartz]

This (or a similar error) is even easier to reproduce (without adding a disk) by simply specifying a long instance name::

╭─ubuntu@isd1045-cos-integration ~ [lxd:gh-runner]
╰─$ lxd --version                                                                                                                     
5.0.3
╭─ubuntu@isd1045-cos-integration ~ [lxd:gh-runner]
╰─$ lxc launch ubuntu:22.04 runner-charm-os-service-checks-1-400033f2168e6db04cc9b6c9  --vm
Creating runner-charm-os-service-checks-1-400033f2168e6db04cc9b6c9
Starting runner-charm-os-service-checks-1-400033f2168e6db04cc9b6c9
Error: Failed to run: forklimits limit=memlock:unlimited:unlimited fd=3 fd=4 -- /snap/lxd/28373/bin/qemu-system-x86_64 -S -name runner-charm-os-service-checks-1-400033f2168e6db04cc9b6c9 -uuid f83633e9-8787-41a6-adb3-5d3b37497229 -daemonize -cpu host,hv_passthrough -nographic -serial chardev:console -nodefaults -no-user-config -sandbox on,obsolete=deny,elevateprivileges=allow,spawn=allow,resourcecontrol=deny -readconfig /var/snap/lxd/common/lxd/logs/runner-charm-os-service-checks-1-400033f2168e6db04cc9b6c9/qemu.conf -spice unix=on,disable-ticketing=on,addr=/var/snap/lxd/common/lxd/logs/runner-charm-os-service-checks-1-400033f2168e6db04cc9b6c9/qemu.spice -pidfile /var/snap/lxd/common/lxd/logs/runner-charm-os-service-checks-1-400033f2168e6db04cc9b6c9/qemu.pid -D /var/snap/lxd/common/lxd/logs/runner-charm-os-service-checks-1-400033f2168e6db04cc9b6c9/qemu.log -smbios type=2,manufacturer=Canonical Ltd.,product=LXD -runas lxd: : exit status 1
Try `lxc info --show-log local:runner-charm-os-service-checks-1-400033f2168e6db04cc9b6c9` for more info
╭─ubuntu@isd1045-cos-integration ~ [lxd:gh-runner]
╰─$ lxc info --show-log local:runner-charm-os-service-checks-1-400033f2168e6db04cc9b6c9                                                                                                                                                                                        1 ↵
Name: runner-charm-os-service-checks-1-400033f2168e6db04cc9b6c9
Status: STOPPED
Type: virtual-machine
Architecture: x86_64
Created: 2024/05/02 08:48 CEST

Log:

qemu-system-x86_64:/var/snap/lxd/common/lxd/logs/runner-charm-os-service-checks-1-400033f2168e6db04cc9b6c9/qemu.conf:225: UNIX socket path '/var/snap/lxd/common/lxd/logs/runner-charm-os-service-checks-1-400033f2168e6db04cc9b6c9/virtio-fs.config.sock' is too long
Path must be less than 108 bytes

[hamistao]

@cbartz It is indeed practically the same error, with the difference that the socket whose path is too large in this case is the qemu config socket and not the device socket.