Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Failure to start LXD VM due to long UNIX Socket Path #12539

Closed
6 tasks
cbartz opened this issue Nov 21, 2023 · 10 comments · Fixed by #13320
Closed
6 tasks

Failure to start LXD VM due to long UNIX Socket Path #12539

cbartz opened this issue Nov 21, 2023 · 10 comments · Fixed by #13320
Assignees
Labels
Bug Confirmed to be a bug
Milestone

Comments

@cbartz
Copy link
Contributor

cbartz commented Nov 21, 2023

Required information

  • Distribution: Ubuntu
  • Distribution version: 22.04
  • The output of "lxc info":
config: {}
api_extensions:
- storage_zfs_remove_snapshots
- container_host_shutdown_timeout
- container_stop_priority
- container_syscall_filtering
- auth_pki
- container_last_used_at
- etag
- patch
- usb_devices
- https_allowed_credentials
- image_compression_algorithm
- directory_manipulation
- container_cpu_time
- storage_zfs_use_refquota
- storage_lvm_mount_options
- network
- profile_usedby
- container_push
- container_exec_recording
- certificate_update
- container_exec_signal_handling
- gpu_devices
- container_image_properties
- migration_progress
- id_map
- network_firewall_filtering
- network_routes
- storage
- file_delete
- file_append
- network_dhcp_expiry
- storage_lvm_vg_rename
- storage_lvm_thinpool_rename
- network_vlan
- image_create_aliases
- container_stateless_copy
- container_only_migration
- storage_zfs_clone_copy
- unix_device_rename
- storage_lvm_use_thinpool
- storage_rsync_bwlimit
- network_vxlan_interface
- storage_btrfs_mount_options
- entity_description
- image_force_refresh
- storage_lvm_lv_resizing
- id_map_base
- file_symlinks
- container_push_target
- network_vlan_physical
- storage_images_delete
- container_edit_metadata
- container_snapshot_stateful_migration
- storage_driver_ceph
- storage_ceph_user_name
- resource_limits
- storage_volatile_initial_source
- storage_ceph_force_osd_reuse
- storage_block_filesystem_btrfs
- resources
- kernel_limits
- storage_api_volume_rename
- macaroon_authentication
- network_sriov
- console
- restrict_devlxd
- migration_pre_copy
- infiniband
- maas_network
- devlxd_events
- proxy
- network_dhcp_gateway
- file_get_symlink
- network_leases
- unix_device_hotplug
- storage_api_local_volume_handling
- operation_description
- clustering
- event_lifecycle
- storage_api_remote_volume_handling
- nvidia_runtime
- container_mount_propagation
- container_backup
- devlxd_images
- container_local_cross_pool_handling
- proxy_unix
- proxy_udp
- clustering_join
- proxy_tcp_udp_multi_port_handling
- network_state
- proxy_unix_dac_properties
- container_protection_delete
- unix_priv_drop
- pprof_http
- proxy_haproxy_protocol
- network_hwaddr
- proxy_nat
- network_nat_order
- container_full
- candid_authentication
- backup_compression
- candid_config
- nvidia_runtime_config
- storage_api_volume_snapshots
- storage_unmapped
- projects
- candid_config_key
- network_vxlan_ttl
- container_incremental_copy
- usb_optional_vendorid
- snapshot_scheduling
- snapshot_schedule_aliases
- container_copy_project
- clustering_server_address
- clustering_image_replication
- container_protection_shift
- snapshot_expiry
- container_backup_override_pool
- snapshot_expiry_creation
- network_leases_location
- resources_cpu_socket
- resources_gpu
- resources_numa
- kernel_features
- id_map_current
- event_location
- storage_api_remote_volume_snapshots
- network_nat_address
- container_nic_routes
- rbac
- cluster_internal_copy
- seccomp_notify
- lxc_features
- container_nic_ipvlan
- network_vlan_sriov
- storage_cephfs
- container_nic_ipfilter
- resources_v2
- container_exec_user_group_cwd
- container_syscall_intercept
- container_disk_shift
- storage_shifted
- resources_infiniband
- daemon_storage
- instances
- image_types
- resources_disk_sata
- clustering_roles
- images_expiry
- resources_network_firmware
- backup_compression_algorithm
- ceph_data_pool_name
- container_syscall_intercept_mount
- compression_squashfs
- container_raw_mount
- container_nic_routed
- container_syscall_intercept_mount_fuse
- container_disk_ceph
- virtual-machines
- image_profiles
- clustering_architecture
- resources_disk_id
- storage_lvm_stripes
- vm_boot_priority
- unix_hotplug_devices
- api_filtering
- instance_nic_network
- clustering_sizing
- firewall_driver
- projects_limits
- container_syscall_intercept_hugetlbfs
- limits_hugepages
- container_nic_routed_gateway
- projects_restrictions
- custom_volume_snapshot_expiry
- volume_snapshot_scheduling
- trust_ca_certificates
- snapshot_disk_usage
- clustering_edit_roles
- container_nic_routed_host_address
- container_nic_ipvlan_gateway
- resources_usb_pci
- resources_cpu_threads_numa
- resources_cpu_core_die
- api_os
- container_nic_routed_host_table
- container_nic_ipvlan_host_table
- container_nic_ipvlan_mode
- resources_system
- images_push_relay
- network_dns_search
- container_nic_routed_limits
- instance_nic_bridged_vlan
- network_state_bond_bridge
- usedby_consistency
- custom_block_volumes
- clustering_failure_domains
- resources_gpu_mdev
- console_vga_type
- projects_limits_disk
- network_type_macvlan
- network_type_sriov
- container_syscall_intercept_bpf_devices
- network_type_ovn
- projects_networks
- projects_networks_restricted_uplinks
- custom_volume_backup
- backup_override_name
- storage_rsync_compression
- network_type_physical
- network_ovn_external_subnets
- network_ovn_nat
- network_ovn_external_routes_remove
- tpm_device_type
- storage_zfs_clone_copy_rebase
- gpu_mdev
- resources_pci_iommu
- resources_network_usb
- resources_disk_address
- network_physical_ovn_ingress_mode
- network_ovn_dhcp
- network_physical_routes_anycast
- projects_limits_instances
- network_state_vlan
- instance_nic_bridged_port_isolation
- instance_bulk_state_change
- network_gvrp
- instance_pool_move
- gpu_sriov
- pci_device_type
- storage_volume_state
- network_acl
- migration_stateful
- disk_state_quota
- storage_ceph_features
- projects_compression
- projects_images_remote_cache_expiry
- certificate_project
- network_ovn_acl
- projects_images_auto_update
- projects_restricted_cluster_target
- images_default_architecture
- network_ovn_acl_defaults
- gpu_mig
- project_usage
- network_bridge_acl
- warnings
- projects_restricted_backups_and_snapshots
- clustering_join_token
- clustering_description
- server_trusted_proxy
- clustering_update_cert
- storage_api_project
- server_instance_driver_operational
- server_supported_storage_drivers
- event_lifecycle_requestor_address
- resources_gpu_usb
- clustering_evacuation
- network_ovn_nat_address
- network_bgp
- network_forward
- custom_volume_refresh
- network_counters_errors_dropped
- metrics
- image_source_project
- clustering_config
- network_peer
- linux_sysctl
- network_dns
- ovn_nic_acceleration
- certificate_self_renewal
- instance_project_move
- storage_volume_project_move
- cloud_init
- network_dns_nat
- database_leader
- instance_all_projects
- clustering_groups
- ceph_rbd_du
- instance_get_full
- qemu_metrics
- gpu_mig_uuid
- event_project
- clustering_evacuation_live
- instance_allow_inconsistent_copy
- network_state_ovn
- storage_volume_api_filtering
- image_restrictions
- storage_zfs_export
- network_dns_records
- storage_zfs_reserve_space
- network_acl_log
- storage_zfs_blocksize
- metrics_cpu_seconds
- instance_snapshot_never
- certificate_token
- instance_nic_routed_neighbor_probe
- event_hub
- agent_nic_config
- projects_restricted_intercept
- metrics_authentication
- images_target_project
- cluster_migration_inconsistent_copy
- cluster_ovn_chassis
- container_syscall_intercept_sched_setscheduler
- storage_lvm_thinpool_metadata_size
- storage_volume_state_total
- instance_file_head
- resources_pci_vpd
- qemu_raw_conf
- storage_cephfs_fscache
- vsock_api
- storage_volumes_all_projects
- projects_networks_restricted_access
- cluster_join_token_expiry
- remote_token_expiry
- init_preseed
- cpu_hotplug
api_status: stable
api_version: "1.0"
auth: trusted
public: false
auth_methods:
- tls
environment:
  addresses: []
  architectures:
  - x86_64
  - i686
  certificate: |
    -----BEGIN CERTIFICATE-----
    MIICAjCCAYegAwIBAgIRAJrMJKkDW+Ad9tsMyXY1mAowCgYIKoZIzj0EAwMwMzEc
    MBoGA1UEChMTbGludXhjb250YWluZXJzLm9yZzETMBEGA1UEAwwKcm9vdEBzcXVp
    ZDAeFw0yMzExMjAwNzIyNDdaFw0zMzExMTcwNzIyNDdaMDMxHDAaBgNVBAoTE2xp
    bnV4Y29udGFpbmVycy5vcmcxEzARBgNVBAMMCnJvb3RAc3F1aWQwdjAQBgcqhkjO
    PQIBBgUrgQQAIgNiAAQcYTk+Me71KxfYD72/l7BxN6o5wSay5BZIt2g49zK5iGHM
    v1b1hh+O4lzAmXV4xEwSoT+B1VQ1/XYw+9M3k/W/YYS4MrjH5gBBooiRHFVuFVnt
    vSpdCHiKQERqiXF2QlmjXzBdMA4GA1UdDwEB/wQEAwIFoDATBgNVHSUEDDAKBggr
    BgEFBQcDATAMBgNVHRMBAf8EAjAAMCgGA1UdEQQhMB+CBXNxdWlkhwR/AAABhxAA
    AAAAAAAAAAAAAAAAAAABMAoGCCqGSM49BAMDA2kAMGYCMQC+hnOZZ4MEpbgEhyS6
    MQqI72Px8NW+o2LdIWU78gJ/8THkvhFzU4U99b4WBYnTroMCMQC0EUqFzP77bOCp
    WJVQtF6HK146l/639ED5Gp8p10q8Up9uV0/bw6kUYXYSIERLIJs=
    -----END CERTIFICATE-----
  certificate_fingerprint: f336157139686162d891253af88a84f732ae2d6c029651e081306b2a563d27aa
  driver: lxc | qemu
  driver_version: 5.0.2 | 7.1.0
  firewall: nftables
  kernel: Linux
  kernel_architecture: x86_64
  kernel_features:
    idmapped_mounts: "true"
    netnsid_getifaddrs: "true"
    seccomp_listener: "true"
    seccomp_listener_continue: "true"
    shiftfs: "false"
    uevent_injection: "true"
    unpriv_fscaps: "true"
  kernel_version: 5.15.0-87-generic
  lxc_features:
    cgroup2: "true"
    core_scheduling: "true"
    devpts_fd: "true"
    idmapped_mounts_v2: "true"
    mount_injection_file: "true"
    network_gateway_device_route: "true"
    network_ipvlan: "true"
    network_l2proxy: "true"
    network_phys_macvlan_mtu: "true"
    network_veth_router: "true"
    pidfd: "true"
    seccomp_allow_deny_syntax: "true"
    seccomp_notify: "true"
    seccomp_proxy_send_notify_fd: "true"
  os_name: Ubuntu
  os_version: "22.04"
  project: default
  server: lxd
  server_clustered: false
  server_event_mode: full-mesh
  server_name: squid
  server_pid: 12638
  server_version: 5.0.2
  storage: dir
  storage_version: "1"
  storage_supported_drivers:
  - name: lvm
    version: 2.03.07(2) (2019-11-30) / 1.02.167 (2019-11-30) / 4.45.0
    remote: false
  - name: zfs
    version: 2.1.5-1ubuntu6~22.04.1
    remote: false
  - name: btrfs
    version: 5.4.1
    remote: false
  - name: ceph
    version: 15.2.17
    remote: true
  - name: cephfs
    version: 15.2.17
    remote: true
  - name: cephobject
    version: 15.2.17
    remote: true
  - name: dir
    version: "1"
    remote: false

Issue description

When adding a disk to an LXD VM instance using a directory on the host with a rather long name, the instance fails to start because the UNIX socket path is too long:

ubuntu@ubuntu:~$ sudo cat /var/snap/lxd/common/lxd/logs/stg-ps6-small2-3-166d3e5f-ba4f-4cd0-817e-1ba1994e8c8d/qemu.log
qemu-system-x86_64:/var/snap/lxd/common/lxd/logs/stg-ps6-small2-3-166d3e5f-ba4f-4cd0-817e-1ba1994e8c8d/qemu.conf:254: UNIX socket path '/var/snap/lxd/common/lxd/devices/stg-ps6-small2-3-166d3e5f-ba4f-4cd0-817e-1ba1994e8c8d/virtio-fs.metrics.sock' is too long
Path must be less than 108 bytes

The problem occurred in an OpenStack VM (instance of a GitHub self-hosted runner charm), but could be reproduced locally inside a multipass vm.

Steps to reproduce

  1. lxc launch ubuntu:22.04 stg-ps6-small2-3-166d3e5f-ba4f-4cd0-817e-1ba1994e8c8d --vm
  2. lxc stop stg-ps6-small2-3-166d3e5f-ba4f-4cd0-817e-1ba1994e8c8d
  3. mkdir /home/ubuntu/shared_fs
  4. lxc config device add stg-ps6-small2-3-166d3e5f-ba4f-4cd0-817e-1ba1994e8c8d metrics disk source=/home/ubuntu/shared_fs/ path=/metrics-exchange
  5. lxc start stg-ps6-small2-3-166d3e5f-ba4f-4cd0-817e-1ba1994e8c8d
  6. sudo cat /var/snap/lxd/common/lxd/logs/stg-ps6-small2-3-166d3e5f-ba4f-4cd0-817e-1ba1994e8c8d/qemu.log

Information to attach

  • Any relevant kernel output (dmesg)
  • Container log (lxc info NAME --show-log)
  • Container configuration (lxc config show NAME --expanded)
  • Main daemon log (at /var/log/lxd/lxd.log or /var/snap/lxd/common/lxd/logs/lxd.log)
  • Output of the client with --debug
  • Output of the daemon with --debug (alternatively output of lxc monitor while reproducing the issue)
@tomponline tomponline added the Bug Confirmed to be a bug label Nov 27, 2023
@tomponline tomponline added this to the lxd-6.1 milestone Mar 18, 2024
@hamistao hamistao self-assigned this Mar 25, 2024
@tomponline
Copy link
Member

Hi @cbartz please can you try this again on LXD 5.0.3, as I believe its already been fixed by:

https://github.com/canonical/lxd/blob/stable-5.0/lxd/device/device_utils_disk.go#L464-L468

@tomponline
Copy link
Member

Ah no, it wont actually because of https://github.com/canonical/lxd/blob/stable-5.0/lxd/device/disk.go#L936 I think.
We need to see about using the same technique and passing a file handle to QEMU.

@hamistao
Copy link
Contributor

So far I managed to get the file from the socket listener object, and pass it over to d.addFileDescriptor to get a valid file descriptor for the socket file and passing it over to qemu via the fd field under chardev and fsdev in the qemu.config file. I oppened a draft PR on this approach. Unfortunately this always results in the following error on qemu.log:

qemu-system-x86_64:/var/snap/lxd/common/lxd/logs/stg-ps6-small2-3-166d3e5f-ba4f-4cd0-817e-1ba1994e8c8d/qemu.conf:273: warning: '-fsdev proxy' and '-virtfs proxy' are deprecated, use 'local' instead of 'proxy, or consider deploying virtiofsd as alternative to 9p
qemu-system-x86_64:/var/snap/lxd/common/lxd/logs/stg-ps6-small2-3-166d3e5f-ba4f-4cd0-817e-1ba1994e8c8d/qemu.conf:267: Failed to write msg. Wrote -1 instead of 12.
qemu-system-x86_64:/var/snap/lxd/common/lxd/logs/stg-ps6-small2-3-166d3e5f-ba4f-4cd0-817e-1ba1994e8c8d/qemu.conf:267: vhost_backend_init failed: Protocol error

This is possibly a result of the qemu version being used in the snap, I am currently running some tests using other versions to see if this alters the behavior. An alternative approach could be to use symbolic links to the socket files, but I haven't figured an apropriate place to create these links on without resulting in a path longer than 107 bytes.

@tomponline
Copy link
Member

@hamistao Don't worry about this line:

qemu-system-x86_64:/var/snap/lxd/common/lxd/logs/stg-ps6-small2-3-166d3e5f-ba4f-4cd0-817e-1ba1994e8c8d/qemu.conf:273: warning: '-fsdev proxy' and '-virtfs proxy' are deprecated, use 'local' instead of 'proxy, or consider deploying virtiofsd as alternative to 9p

That is known, see #12828

@tomponline
Copy link
Member

One approach could be to open a file handle and then pass it as a file path using /proc/self/fd/<fdnum>

@hamistao
Copy link
Contributor

hamistao commented Apr 12, 2024

If I am understanding correctly, I have tried this but it doesn't work because qemu doesn't recognize /proc/self as the same directory as LXD. Using /proc/{pid} instead also doesn't work because qemu doesn't have permission to acces this directory.

@roosterfish
Copy link
Contributor

Might it be that the protocol error in #13320 is coming from having both path and fd config setting in the qemu config file for the chardev device? I have added a comment here #13320 (comment).

@hamistao
Copy link
Contributor

hamistao commented Apr 12, 2024

Might it be that the protocol error in #13320 is coming from having both path and fd config setting in the qemu config file for the chardev device? I have added a comment here #13320 (comment).

this isn't the case, although it seemed like the path was still being included, the chardev section in the qemu.config generated looks like this:

# metrics drive (virtio-fs)
[chardev "lxd_metrics"]
backend = "socket"
fd = "6"

@cbartz
Copy link
Contributor Author

cbartz commented May 2, 2024

This (or a similar error) is even easier to reproduce (without adding a disk) by simply specifying a long instance name::

╭─ubuntu@isd1045-cos-integration ~ [lxd:gh-runner]
╰─$ lxd --version                                                                                                                     
5.0.3
╭─ubuntu@isd1045-cos-integration ~ [lxd:gh-runner]
╰─$ lxc launch ubuntu:22.04 runner-charm-os-service-checks-1-400033f2168e6db04cc9b6c9  --vm
Creating runner-charm-os-service-checks-1-400033f2168e6db04cc9b6c9
Starting runner-charm-os-service-checks-1-400033f2168e6db04cc9b6c9
Error: Failed to run: forklimits limit=memlock:unlimited:unlimited fd=3 fd=4 -- /snap/lxd/28373/bin/qemu-system-x86_64 -S -name runner-charm-os-service-checks-1-400033f2168e6db04cc9b6c9 -uuid f83633e9-8787-41a6-adb3-5d3b37497229 -daemonize -cpu host,hv_passthrough -nographic -serial chardev:console -nodefaults -no-user-config -sandbox on,obsolete=deny,elevateprivileges=allow,spawn=allow,resourcecontrol=deny -readconfig /var/snap/lxd/common/lxd/logs/runner-charm-os-service-checks-1-400033f2168e6db04cc9b6c9/qemu.conf -spice unix=on,disable-ticketing=on,addr=/var/snap/lxd/common/lxd/logs/runner-charm-os-service-checks-1-400033f2168e6db04cc9b6c9/qemu.spice -pidfile /var/snap/lxd/common/lxd/logs/runner-charm-os-service-checks-1-400033f2168e6db04cc9b6c9/qemu.pid -D /var/snap/lxd/common/lxd/logs/runner-charm-os-service-checks-1-400033f2168e6db04cc9b6c9/qemu.log -smbios type=2,manufacturer=Canonical Ltd.,product=LXD -runas lxd: : exit status 1
Try `lxc info --show-log local:runner-charm-os-service-checks-1-400033f2168e6db04cc9b6c9` for more info
╭─ubuntu@isd1045-cos-integration ~ [lxd:gh-runner]
╰─$ lxc info --show-log local:runner-charm-os-service-checks-1-400033f2168e6db04cc9b6c9                                                                                                                                                                                        1 ↵
Name: runner-charm-os-service-checks-1-400033f2168e6db04cc9b6c9
Status: STOPPED
Type: virtual-machine
Architecture: x86_64
Created: 2024/05/02 08:48 CEST

Log:

qemu-system-x86_64:/var/snap/lxd/common/lxd/logs/runner-charm-os-service-checks-1-400033f2168e6db04cc9b6c9/qemu.conf:225: UNIX socket path '/var/snap/lxd/common/lxd/logs/runner-charm-os-service-checks-1-400033f2168e6db04cc9b6c9/virtio-fs.config.sock' is too long
Path must be less than 108 bytes

@hamistao
Copy link
Contributor

hamistao commented May 2, 2024

@cbartz It is indeed practically the same error, with the difference that the socket whose path is too large in this case is the qemu config socket and not the device socket.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Bug Confirmed to be a bug
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants