Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Error: Failed container creation: Create container from image: Image create: Unpack failed, Failed to run: unsquashfs #5449

Closed
falstaff1288 opened this issue Feb 1, 2019 · 42 comments

Comments

6 participants
@falstaff1288
Copy link

commented Feb 1, 2019

Required information

  • Distribution: Arch Linux
  • Distribution version: System updated on 2019-01-31 with the command pacman -Syu
  • The output of "lxc info" or if that fails:
lxc info
config: {}
api_extensions:
- storage_zfs_remove_snapshots
- container_host_shutdown_timeout
- container_stop_priority
- container_syscall_filtering
- auth_pki
- container_last_used_at
- etag
- patch
- usb_devices
- https_allowed_credentials
- image_compression_algorithm
- directory_manipulation
- container_cpu_time
- storage_zfs_use_refquota
- storage_lvm_mount_options
- network
- profile_usedby
- container_push
- container_exec_recording
- certificate_update
- container_exec_signal_handling
- gpu_devices
- container_image_properties
- migration_progress
- id_map
- network_firewall_filtering
- network_routes
- storage
- file_delete
- file_append
- network_dhcp_expiry
- storage_lvm_vg_rename
- storage_lvm_thinpool_rename
- network_vlan
- image_create_aliases
- container_stateless_copy
- container_only_migration
- storage_zfs_clone_copy
- unix_device_rename
- storage_lvm_use_thinpool
- storage_rsync_bwlimit
- network_vxlan_interface
- storage_btrfs_mount_options
- entity_description
- image_force_refresh
- storage_lvm_lv_resizing
- id_map_base
- file_symlinks
- container_push_target
- network_vlan_physical
- storage_images_delete
- container_edit_metadata
- container_snapshot_stateful_migration
- storage_driver_ceph
- storage_ceph_user_name
- resource_limits
- storage_volatile_initial_source
- storage_ceph_force_osd_reuse
- storage_block_filesystem_btrfs
- resources
- kernel_limits
- storage_api_volume_rename
- macaroon_authentication
- network_sriov
- console
- restrict_devlxd
- migration_pre_copy
- infiniband
- maas_network
- devlxd_events
- proxy
- network_dhcp_gateway
- file_get_symlink
- network_leases
- unix_device_hotplug
- storage_api_local_volume_handling
- operation_description
- clustering
- event_lifecycle
- storage_api_remote_volume_handling
- nvidia_runtime
- container_mount_propagation
- container_backup
- devlxd_images
- container_local_cross_pool_handling
- proxy_unix
- proxy_udp
- clustering_join
- proxy_tcp_udp_multi_port_handling
- network_state
- proxy_unix_dac_properties
- container_protection_delete
- unix_priv_drop
- pprof_http
- proxy_haproxy_protocol
- network_hwaddr
- proxy_nat
- network_nat_order
- container_full
- candid_authentication
- backup_compression
- candid_config
- nvidia_runtime_config
- storage_api_volume_snapshots
- storage_unmapped
- projects
- candid_config_key
- network_vxlan_ttl
- container_incremental_copy
- usb_optional_vendorid
- snapshot_scheduling
- container_copy_project
- clustering_server_address
- clustering_image_replication
- container_protection_shift
api_status: stable
api_version: "1.0"
auth: trusted
public: false
auth_methods:
- tls
environment:
 addresses: []
 architectures:
 - x86_64
 - i686
 certificate: |
   -----BEGIN CERTIFICATE-----
...
   -----END CERTIFICATE-----
 certificate_fingerprint: ...
 driver: lxc
 driver_version: 3.1.0
 kernel: Linux
 kernel_architecture: x86_64
 kernel_version: 4.20.5-arch1-1-ARCH
 server: lxd
 server_pid: 1422
 server_version: "3.8"
 storage: lvm
 storage_version: 2.02.183(2) (2018-12-07) / 1.02.154 (2018-12-07) / 4.39.0 / ./configure
   --prefix=/usr --sbindir=/usr/bin --sysconfdir=/etc --localstatedir=/var --enable-applib
   --enable-cmdlib --enable-dmeventd --enable-lvmetad --enable-lvmpolld --enable-pkgconfig
   --enable-readline --enable-udev_rules --enable-udev_sync --enable-use-lvmetad
   --with-cache=internal --with-default-dm-run-dir=/run --with-default-locking-dir=/run/lock/lvm
   --with-default-pid-dir=/run --with-default-run-dir=/run/lvm --with-systemdsystemunitdir=/usr/lib/systemd/system
   --with-thin=internal --with-udev-prefix=/usr --enable-udev-systemd-background-jobs
 server_clustered: false
 server_name: ...
 project: default 

Issue description

After updating my ArchLinux system, whenever I attempt to create a new container with an image that wasn't downloaded/wasn't already in my list of download images, I get the following error :

$ lxc launch ubuntu:xenial test1 -c security.nesting=true
Creating test1
Error: Failed container creation: Create container from image: Image create: Unpack failed, Failed to run: unsquashfs -f -d /var/lib/lxd/storage-pools/LVM/images/b579861d72c87f006a28ba6031e3fbee0688c1f26302d86ae8db08cf2d0b63f7/rootfs -n /var/lib/lxd/images/b579861d72c87f006a28ba6031e3fbee0688c1f26302d86ae8db08cf2d0b63f7.rootfs: FATAL ERROR:Data queue size is too large.  FATAL ERROR:Data queue size is too large
  • I have no issue creating containers with images that were present before rebooting my system. I can create snapshots and restore them with my curent containers without any issues.
  • unsquashfs can be ran with any issues in the command line :
# unsquashfs -d test /var/lib/lxd/images/df7155e205ba18e619d72ec7e8bff9d558d693a94d512cbb6ad56b09955d92ad.rootfs
Parallel unsquashfs: Using 4 processors
9676 inodes (10928 blocks) to write

[=========================================================================================================================================================================-] 10928/10928 100%

created 8186 files
created 944 directories
created 1474 symlinks
created 7 devices
created 0 fifos
# ls -l test/
total 80
drwxr-xr-x  2 root root 4096 27 jun  2018 bin
drwxr-xr-x  2 root root 4096 30 mai  2016 boot
drwxr-xr-x  3 root root 4096 27 jun  2018 dev
drwxr-xr-x 35 root root 4096 27 jun  2018 etc
drwxr-xr-x  2 root root 4096 30 mai  2016 home
drwxr-xr-x  7 root root 4096 22 jun  2012 lib
drwxr-xr-x  2 root root 4096 27 jun  2018 lib64
drwxr-xr-x  2 root root 4096 27 jun  2018 media
drwxr-xr-x  2 root root 4096 30 mai  2016 mnt
drwxr-xr-x  2 root root 4096 27 jun  2018 opt
drwxr-xr-x  2 root root 4096 30 mai  2016 proc
drwx------  2 root root 4096 27 jun  2018 root
drwxr-xr-x  2 root root 4096 27 jun  2018 run
drwxr-xr-x  2 root root 4096 27 jun  2018 sbin
drwxr-xr-x  2 root root 4096 10 jun  2012 selinux
drwxr-xr-x  2 root root 4096 27 jun  2018 srv
drwxr-xr-x  2 root root 4096 14 jui  2013 sys
drwxrwxrwt  2 root root 4096 27 jun  2018 tmp
drwxr-xr-x 10 root root 4096 27 jun  2018 usr
drwxr-xr-x 11 root root 4096 27 jun  2018 var

Information to attach

  • Any relevant kernel output (dmesg)
    See below
  • Container log (lxc info NAME --show-log)
    Can't create containers
  • Container configuration (lxc config show NAME --expanded)
    Same as above
  • Main daemon log (at /var/log/lxd/lxd.log or /var/snap/lxd/common/lxd/logs/lxd.log)
    Output from the command journalctl -u lxd
-- Logs begin at Mon 2018-09-10 05:42:33 EDT. --
jan 31 22:36:06 nas.maison.lan lxd[1422]: t=2019-01-31T22:36:06-0500 lvl=eror msg="Failed to create LVM storage volume for container \"iscsi-target\" on storage pool \"LVM\": Image create: Unpack failed, Failed to run: unsquashfs -f -d /var/lib/lxd/storage-pools/LVM/images/df7155e205ba18e619d72ec7e8bff9d558d693a94d512cbb6ad56b09955d92ad/rootfs -n /var/lib/lxd/images/df7155e205ba18e619d72ec7e8bff9d558d693a94d512cbb6ad56b09955d92ad.rootfs: FATAL ERROR:Data queue size is too large.  FATAL ERROR:Data queue size is too large"
jan 31 22:36:35 nas.maison.lan lxd[1422]: t=2019-01-31T22:36:35-0500 lvl=eror msg="Failed to create LVM storage volume for container \"iscsi-target\" on storage pool \"LVM\": Image create: Unpack failed, Failed to run: unsquashfs -f -d /var/lib/lxd/storage-pools/LVM/images/df7155e205ba18e619d72ec7e8bff9d558d693a94d512cbb6ad56b09955d92ad/rootfs -n /var/lib/lxd/images/df7155e205ba18e619d72ec7e8bff9d558d693a94d512cbb6ad56b09955d92ad.rootfs: FATAL ERROR:Data queue size is too large.  FATAL ERROR:Data queue size is too large"
jan 31 22:36:51 nas.maison.lan lxd[1422]: t=2019-01-31T22:36:51-0500 lvl=eror msg="Failed to create LVM storage volume for container \"iscsi-target\" on storage pool \"LVM\": Image create: Unpack failed, Failed to run: unsquashfs -f -d /var/lib/lxd/storage-pools/LVM/images/df7155e205ba18e619d72ec7e8bff9d558d693a94d512cbb6ad56b09955d92ad/rootfs -n /var/lib/lxd/images/df7155e205ba18e619d72ec7e8bff9d558d693a94d512cbb6ad56b09955d92ad.rootfs: FATAL ERROR:Data queue size is too large.  FATAL ERROR:Data queue size is too large"
jan 31 22:44:40 nas.maison.lan lxd[1422]: t=2019-01-31T22:44:40-0500 lvl=eror msg="Failed to create LVM storage volume for container \"iscsi-target\" on storage pool \"LVM\": Image create: Unpack failed, Failed to run: unsquashfs -f -d /var/lib/lxd/storage-pools/LVM/images/df7155e205ba18e619d72ec7e8bff9d558d693a94d512cbb6ad56b09955d92ad/rootfs -n /var/lib/lxd/images/df7155e205ba18e619d72ec7e8bff9d558d693a94d512cbb6ad56b09955d92ad.rootfs: FATAL ERROR:Data queue size is too large.  FATAL ERROR:Data queue size is too large"
jan 31 22:49:43 nas.maison.lan lxd[1422]: t=2019-01-31T22:49:43-0500 lvl=eror msg="Failed to create LVM storage volume for container \"iscsi-target\" on storage pool \"LVM\": Image create: Unpack failed, Failed to run: unsquashfs -f -d /var/lib/lxd/storage-pools/LVM/images/df7155e205ba18e619d72ec7e8bff9d558d693a94d512cbb6ad56b09955d92ad/rootfs -n /var/lib/lxd/images/df7155e205ba18e619d72ec7e8bff9d558d693a94d512cbb6ad56b09955d92ad.rootfs: FATAL ERROR:Data queue size is too large.  FATAL ERROR:Data queue size is too large"
jan 31 22:56:30 nas.maison.lan lxd[1422]: t=2019-01-31T22:56:30-0500 lvl=eror msg="Failed to create LVM storage volume for container \"iscsi-target\" on storage pool \"LVM\": Image create: Unpack failed, Failed to run: unsquashfs -f -d /var/lib/lxd/storage-pools/LVM/images/df7155e205ba18e619d72ec7e8bff9d558d693a94d512cbb6ad56b09955d92ad/rootfs -n /var/lib/lxd/images/df7155e205ba18e619d72ec7e8bff9d558d693a94d512cbb6ad56b09955d92ad.rootfs: FATAL ERROR:Data queue size is too large.  FATAL ERROR:Data queue size is too large"
jan 31 22:56:45 nas.maison.lan lxd[1422]: t=2019-01-31T22:56:45-0500 lvl=warn msg="Unable to update backup.yaml at this time" name=fitting-chigger rootfs=/var/lib/lxd/containers/fitting-chigger/rootfs
jan 31 22:57:16 nas.maison.lan lxd[1422]: No such file or directory - Failed to receive file descriptor
jan 31 22:59:49 nas.maison.lan lxd[1422]: t=2019-01-31T22:59:49-0500 lvl=eror msg="Failed to create LVM storage volume for container \"test1\" on storage pool \"LVM\": Image create: Unpack failed, Failed to run: unsquashfs -f -d /var/lib/lxd/storage-pools/LVM/images/b579861d72c87f006a28ba6031e3fbee0688c1f26302d86ae8db08cf2d0b63f7/rootfs -n /var/lib/lxd/images/b579861d72c87f006a28ba6031e3fbee0688c1f26302d86ae8db08cf2d0b63f7.rootfs: FATAL ERROR:Data queue size is too large.  FATAL ERROR:Data queue size is too large"
jan 31 23:00:21 nas.maison.lan lxd[1422]: t=2019-01-31T23:00:21-0500 lvl=eror msg="Failed to create LVM storage volume for container \"test1\" on storage pool \"LVM\": Image create: Unpack failed, Failed to run: unsquashfs -f -d /var/lib/lxd/storage-pools/LVM/images/b579861d72c87f006a28ba6031e3fbee0688c1f26302d86ae8db08cf2d0b63f7/rootfs -n /var/lib/lxd/images/b579861d72c87f006a28ba6031e3fbee0688c1f26302d86ae8db08cf2d0b63f7.rootfs: FATAL ERROR:Data queue size is too large.  FATAL ERROR:Data queue size is too large"
  • Output of the client with --debug
lxc launch -v --debug ubuntu:xenial test1 -c security.nesting=true
DBUG[01-31|23:15:14] Connecting to a local LXD over a Unix socket 
DBUG[01-31|23:15:14] Sending request to LXD                   method=GET url=http://unix.socket/1.0 etag=
DBUG[01-31|23:15:14] Got response struct from LXD 
DBUG[01-31|23:15:14] 
	{
		"config": {},
		"api_extensions": [
			"storage_zfs_remove_snapshots",
			"container_host_shutdown_timeout",
			"container_stop_priority",
			"container_syscall_filtering",
			"auth_pki",
			"container_last_used_at",
			"etag",
			"patch",
			"usb_devices",
			"https_allowed_credentials",
			"image_compression_algorithm",
			"directory_manipulation",
			"container_cpu_time",
			"storage_zfs_use_refquota",
			"storage_lvm_mount_options",
			"network",
			"profile_usedby",
			"container_push",
			"container_exec_recording",
			"certificate_update",
			"container_exec_signal_handling",
			"gpu_devices",
			"container_image_properties",
			"migration_progress",
			"id_map",
			"network_firewall_filtering",
			"network_routes",
			"storage",
			"file_delete",
			"file_append",
			"network_dhcp_expiry",
			"storage_lvm_vg_rename",
			"storage_lvm_thinpool_rename",
			"network_vlan",
			"image_create_aliases",
			"container_stateless_copy",
			"container_only_migration",
			"storage_zfs_clone_copy",
			"unix_device_rename",
			"storage_lvm_use_thinpool",
			"storage_rsync_bwlimit",
			"network_vxlan_interface",
			"storage_btrfs_mount_options",
			"entity_description",
			"image_force_refresh",
			"storage_lvm_lv_resizing",
			"id_map_base",
			"file_symlinks",
			"container_push_target",
			"network_vlan_physical",
			"storage_images_delete",
			"container_edit_metadata",
			"container_snapshot_stateful_migration",
			"storage_driver_ceph",
			"storage_ceph_user_name",
			"resource_limits",
			"storage_volatile_initial_source",
			"storage_ceph_force_osd_reuse",
			"storage_block_filesystem_btrfs",
			"resources",
			"kernel_limits",
			"storage_api_volume_rename",
			"macaroon_authentication",
			"network_sriov",
			"console",
			"restrict_devlxd",
			"migration_pre_copy",
			"infiniband",
			"maas_network",
			"devlxd_events",
			"proxy",
			"network_dhcp_gateway",
			"file_get_symlink",
			"network_leases",
			"unix_device_hotplug",
			"storage_api_local_volume_handling",
			"operation_description",
			"clustering",
			"event_lifecycle",
			"storage_api_remote_volume_handling",
			"nvidia_runtime",
			"container_mount_propagation",
			"container_backup",
			"devlxd_images",
			"container_local_cross_pool_handling",
			"proxy_unix",
			"proxy_udp",
			"clustering_join",
			"proxy_tcp_udp_multi_port_handling",
			"network_state",
			"proxy_unix_dac_properties",
			"container_protection_delete",
			"unix_priv_drop",
			"pprof_http",
			"proxy_haproxy_protocol",
			"network_hwaddr",
			"proxy_nat",
			"network_nat_order",
			"container_full",
			"candid_authentication",
			"backup_compression",
			"candid_config",
			"nvidia_runtime_config",
			"storage_api_volume_snapshots",
			"storage_unmapped",
			"projects",
			"candid_config_key",
			"network_vxlan_ttl",
			"container_incremental_copy",
			"usb_optional_vendorid",
			"snapshot_scheduling",
			"container_copy_project",
			"clustering_server_address",
			"clustering_image_replication",
			"container_protection_shift"
		],
		"api_status": "stable",
		"api_version": "1.0",
		"auth": "trusted",
		"public": false,
		"auth_methods": [
			"tls"
		],
		"environment": {
			"addresses": [],
			"architectures": [
				"x86_64",
				"i686"
			],
			"certificate": "-----BEGIN CERTIFICATE-----\n.../A=\n-----END CERTIFICATE-----\n",
			"certificate_fingerprint": "",
			"driver": "lxc",
			"driver_version": "3.1.0",
			"kernel": "Linux",
			"kernel_architecture": "x86_64",
			"kernel_version": "4.20.5-arch1-1-ARCH",
			"server": "lxd",
			"server_pid": 1422,
			"server_version": "3.8",
			"storage": "lvm",
			"storage_version": "2.02.183(2) (2018-12-07) / 1.02.154 (2018-12-07) / 4.39.0 / ./configure --prefix=/usr --sbindir=/usr/bin --sysconfdir=/etc --localstatedir=/var --enable-applib --enable-cmdlib --enable-dmeventd --enable-lvmetad --enable-lvmpolld --enable-pkgconfig --enable-readline --enable-udev_rules --enable-udev_sync --enable-use-lvmetad --with-cache=internal --with-default-dm-run-dir=/run --with-default-locking-dir=/run/lock/lvm --with-default-pid-dir=/run --with-default-run-dir=/run/lvm --with-systemdsystemunitdir=/usr/lib/systemd/system --with-thin=internal --with-udev-prefix=/usr --enable-udev-systemd-background-jobs",
			"server_clustered": false,
			"server_name": "nas.maison.lan",
			"project": "default"
		}
	} 
Creating test1
DBUG[01-31|23:15:14] Connecting to a remote simplestreams server 
DBUG[01-31|23:15:14] Connected to the websocket 
DBUG[01-31|23:15:14] Sending request to LXD                   method=POST url=http://unix.socket/1.0/containers etag=
DBUG[01-31|23:15:14] 
	{
		"architecture": "",
		"config": {
			"security.nesting": "true"
		},
		"devices": {},
		"ephemeral": false,
		"profiles": null,
		"stateful": false,
		"description": "",
		"name": "test1",
		"source": {
			"type": "image",
			"certificate": "",
			"alias": "xenial",
			"server": "https://cloud-images.ubuntu.com/releases",
			"protocol": "simplestreams",
			"mode": "pull"
		},
		"instance_type": ""
	} 
DBUG[01-31|23:15:14] Got operation from LXD 
DBUG[01-31|23:15:14] 
	{
		"id": "26181d2c-9f67-43c5-9e1f-0b2cfc4a2c78",
		"class": "task",
		"description": "Creating container",
		"created_at": "2019-01-31T23:15:14.375383514-05:00",
		"updated_at": "2019-01-31T23:15:14.375383514-05:00",
		"status": "Running",
		"status_code": 103,
		"resources": {
			"containers": [
				"/1.0/containers/test1"
			]
		},
		"metadata": null,
		"may_cancel": false,
		"err": ""
	} 
DBUG[01-31|23:15:14] Sending request to LXD                   method=GET url=http://unix.socket/1.0/operations/26181d2c-9f67-43c5-9e1f-0b2cfc4a2c78 etag=
DBUG[01-31|23:15:14] Got response struct from LXD 
DBUG[01-31|23:15:14] 
	{
		"id": "26181d2c-9f67-43c5-9e1f-0b2cfc4a2c78",
		"class": "task",
		"description": "Creating container",
		"created_at": "2019-01-31T23:15:14.375383514-05:00",
		"updated_at": "2019-01-31T23:15:14.375383514-05:00",
		"status": "Running",
		"status_code": 103,
		"resources": {
			"containers": [
				"/1.0/containers/test1"
			]
		},
		"metadata": null,
		"may_cancel": false,
		"err": ""
	} 
Error: Failed container creation: Create container from image: Image create: Unpack failed, Failed to run: unsquashfs -f -d /var/lib/lxd/storage-pools/LVM/images/b579861d72c87f006a28ba6031e3fbee0688c1f26302d86ae8db08cf2d0b63f7/rootfs -n /var/lib/lxd/images/b579861d72c87f006a28ba6031e3fbee0688c1f26302d86ae8db08cf2d0b63f7.rootfs: FATAL ERROR:Data queue size is too large.  FATAL ERROR:Data queue size is too large
  • Output of the daemon with --debug (alternatively output of lxc monitor while reproducing the issue)
    The output above from the command journalctl -u lxd should be sufficent. If not, please let me know.

Please advise and thanks in advance for your help.

@falstaff1288 falstaff1288 changed the title Error: Failed container creation: Create container from image: Image create: Unpack failed, Failed to run: unsquashfs [LVM backend] Error: Failed container creation: Create container from image: Image create: Unpack failed, Failed to run: unsquashfs Feb 1, 2019

@falstaff1288 falstaff1288 changed the title [LVM backend] Error: Failed container creation: Create container from image: Image create: Unpack failed, Failed to run: unsquashfs [LVM storage] Error: Failed container creation: Create container from image: Image create: Unpack failed, Failed to run: unsquashfs Feb 1, 2019

@falstaff1288

This comment has been minimized.

Copy link
Author

commented Feb 1, 2019

Had no issue manually creating a thin LV in LXDThinPool although I get some warnings but the thin data pool usage is below 8% and around 2% for the metadata pool

$ sudo lvcreate -n ThinLVTest -V 1G --thinpool vgssd1/LXDThinPool
  WARNING: Sum of all thin volume sizes (181,00 GiB) exceeds the size of thin pool vgssd1/LXDThinPool and the size of whole volume group (<100,00 GiB).
  WARNING: You have not turned on protection against thin pools running out of space.
  WARNING: Set activation/thin_pool_autoextend_threshold below 100 to trigger automatic extension of thin pools before they get full.
  Logical volume "ThinLVTest" created.

$ lsblk /dev/sdb2
NAME                                                                                 MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sdb2                                                                                   8:18   0  100G  0 part 
├─vgssd1-LXDThinPool_tmeta                                                           254:4    0    1G  0 lvm  
│ └─vgssd1-LXDThinPool-tpool                                                         254:6    0   98G  0 lvm  
│   ├─vgssd1-LXDThinPool                                                             254:7    0   98G  0 lvm  
│   ├─vgssd1-containers_gitea                                                        254:8    0   10G  0 lvm  /var/lib/lxd/storage-pools/LVM/containers/gitea
│   ├─vgssd1-containers_bind--snap1                                                  254:9    0   10G  1 lvm  
│   ├─vgssd1-containers_gitea--snap0                                                 254:10   0   10G  1 lvm  
│   ├─vgssd1-containers_gitea--snap1                                                 254:11   0   10G  1 lvm  
│   ├─vgssd1-containers_bind                                                         254:12   0   10G  0 lvm  /var/lib/lxd/storage-pools/LVM/containers/bind
│   ├─vgssd1-containers_gitea--snap2                                                 254:13   0   10G  1 lvm  
│   ├─vgssd1-containers_bind--snap0                                                  254:14   0   10G  1 lvm  
│   ├─vgssd1-containers_docker                                                       254:15   0   10G  0 lvm  /var/lib/lxd/storage-pools/LVM/containers/docker
│   ├─vgssd1-containers_gitea--snap3                                                 254:16   0   10G  1 lvm  
│   ├─vgssd1-containers_docker--snap0                                                254:17   0   10G  1 lvm  
│   ├─vgssd1-containers_docker--snap1                                                254:18   0   10G  1 lvm  
│   ├─vgssd1-containers_ansible----test                                              254:19   0   10G  0 lvm  /var/lib/lxd/storage-pools/LVM/containers/ansible-test
│   ├─vgssd1-images_49fbcd263660ebe9026fb3cc9ae6dba461814bc257401efe87adb19a6d37ffeb 254:20   0   10G  0 lvm  
│   ├─vgssd1-containers_prometheus                                                   254:21   0   10G  0 lvm  
│   ├─vgssd1-images_3862bc84db95e069c6938d879b6ab0d00854265bf231fc85ef220f27d847c6b6 254:22   0   10G  0 lvm  
│   ├─vgssd1-images_dd2dcba03b23a9ceac220e49c972fe4270cabe1a30eefec5d8387d7ee2e87c12 254:23   0   10G  0 lvm  
│   ├─vgssd1-images_091ea27f573bf334d4e65305e6a5acdeeccfbbc38ce77afb439d5a160fd9d5fc 254:24   0   10G  0 lvm  
│   ├─vgssd1-images_bfb2c59415c520100db4b6231fe3386ee84f2c98c5d1bda00f734226084e834a 254:25   0   10G  0 lvm  
│   └─vgssd1-ThinLVTest                                                              254:31   0    1G  0 lvm  
└─vgssd1-LXDThinPool_tdata                                                           254:5    0   98G  0 lvm  
  └─vgssd1-LXDThinPool-tpool                                                         254:6    0   98G  0 lvm  
    ├─vgssd1-LXDThinPool                                                             254:7    0   98G  0 lvm  
    ├─vgssd1-containers_gitea                                                        254:8    0   10G  0 lvm  /var/lib/lxd/storage-pools/LVM/containers/gitea
    ├─vgssd1-containers_bind--snap1                                                  254:9    0   10G  1 lvm  
    ├─vgssd1-containers_gitea--snap0                                                 254:10   0   10G  1 lvm  
    ├─vgssd1-containers_gitea--snap1                                                 254:11   0   10G  1 lvm  
    ├─vgssd1-containers_bind                                                         254:12   0   10G  0 lvm  /var/lib/lxd/storage-pools/LVM/containers/bind
    ├─vgssd1-containers_gitea--snap2                                                 254:13   0   10G  1 lvm  
    ├─vgssd1-containers_bind--snap0                                                  254:14   0   10G  1 lvm  
    ├─vgssd1-containers_docker                                                       254:15   0   10G  0 lvm  /var/lib/lxd/storage-pools/LVM/containers/docker
    ├─vgssd1-containers_gitea--snap3                                                 254:16   0   10G  1 lvm  
    ├─vgssd1-containers_docker--snap0                                                254:17   0   10G  1 lvm  
    ├─vgssd1-containers_docker--snap1                                                254:18   0   10G  1 lvm  
    ├─vgssd1-containers_ansible----test                                              254:19   0   10G  0 lvm  /var/lib/lxd/storage-pools/LVM/containers/ansible-test
    ├─vgssd1-images_49fbcd263660ebe9026fb3cc9ae6dba461814bc257401efe87adb19a6d37ffeb 254:20   0   10G  0 lvm  
    ├─vgssd1-containers_prometheus                                                   254:21   0   10G  0 lvm  
    ├─vgssd1-images_3862bc84db95e069c6938d879b6ab0d00854265bf231fc85ef220f27d847c6b6 254:22   0   10G  0 lvm  
    ├─vgssd1-images_dd2dcba03b23a9ceac220e49c972fe4270cabe1a30eefec5d8387d7ee2e87c12 254:23   0   10G  0 lvm  
    ├─vgssd1-images_091ea27f573bf334d4e65305e6a5acdeeccfbbc38ce77afb439d5a160fd9d5fc 254:24   0   10G  0 lvm  
    ├─vgssd1-images_bfb2c59415c520100db4b6231fe3386ee84f2c98c5d1bda00f734226084e834a 254:25   0   10G  0 lvm  
    └─vgssd1-ThinLVTest                                                              254:31   0    1G  0 lvm  

$ sudo lvs
  LV                                                                      VG        Attr       LSize   Pool        Origin                                                                  Data%  Meta%  Move Log Cpy%Sync Convert
[...]
  LXDThinPool                                                             vgssd1    twi-aotz-- <98,00g                                                                                     7,52   2,04                            
  ThinLVTest                                                              vgssd1    Vwi-a-tz--   1,00g LXDThinPool                                                                         0,00                                   
  containers_ansible--test                                                vgssd1    Vwi-aotz--  10,00g LXDThinPool                                                                         11,13                                  
  containers_bind                                                         vgssd1    Vwi-aotz--  10,00g LXDThinPool containers_bind-snap1                                                   2,72                                   
  containers_bind-snap0                                                   vgssd1    Vri-a-tz--  10,00g LXDThinPool                                                                         2,72                                   
  containers_bind-snap1                                                   vgssd1    Vri-a-tz--  10,00g LXDThinPool                                                                         2,72                                   
  containers_docker                                                       vgssd1    Vwi-aotz--  10,00g LXDThinPool                                                                         13,85                                  
  containers_docker-snap0                                                 vgssd1    Vri-a-tz--  10,00g LXDThinPool containers_docker                                                       13,59                                  
  containers_docker-snap1                                                 vgssd1    Vri-a-tz--  10,00g LXDThinPool containers_docker                                                       13,63                                  
  containers_gitea                                                        vgssd1    Vwi-aotz--  10,00g LXDThinPool                                                                         9,45                                   
  containers_gitea-snap0                                                  vgssd1    Vri-a-tz--  10,00g LXDThinPool containers_gitea                                                        9,38                                   
  containers_gitea-snap1                                                  vgssd1    Vri-a-tz--  10,00g LXDThinPool containers_gitea                                                        9,62                                   
  containers_gitea-snap2                                                  vgssd1    Vri-a-tz--  10,00g LXDThinPool containers_gitea                                                        9,43                                   
  containers_gitea-snap3                                                  vgssd1    Vri-a-tz--  10,00g LXDThinPool containers_gitea                                                        9,44                                   
  containers_prometheus                                                   vgssd1    Vwi-a-tz--  10,00g LXDThinPool images_49fbcd263660ebe9026fb3cc9ae6dba461814bc257401efe87adb19a6d37ffeb 16,00                                  
  images_091ea27f573bf334d4e65305e6a5acdeeccfbbc38ce77afb439d5a160fd9d5fc vgssd1    Vwi-a-tz--  10,00g LXDThinPool                                                                         2,31                                   
  images_3862bc84db95e069c6938d879b6ab0d00854265bf231fc85ef220f27d847c6b6 vgssd1    Vwi-a-tz--  10,00g LXDThinPool                                                                         6,43                                   
  images_49fbcd263660ebe9026fb3cc9ae6dba461814bc257401efe87adb19a6d37ffeb vgssd1    Vwi-a-tz--  10,00g LXDThinPool                                                                         12,50                                  
  images_bfb2c59415c520100db4b6231fe3386ee84f2c98c5d1bda00f734226084e834a vgssd1    Vwi-a-tz--  10,00g LXDThinPool                                                                         8,80                                   
  images_dd2dcba03b23a9ceac220e49c972fe4270cabe1a30eefec5d8387d7ee2e87c12 vgssd1    Vwi-a-tz--  10,00g LXDThinPool                                                                         6,22                           
@stgraber

This comment has been minimized.

Copy link
Member

commented Feb 1, 2019

When manually unpacking, were you unpacking on a thin LV of 10GB on the same VG as LXD?

There's really not much else to what we do to unpack squashfs-es, so it sounds like LVM somehow causing write failures somehow.

@stgraber stgraber added the Incomplete label Feb 1, 2019

@falstaff1288

This comment has been minimized.

Copy link
Author

commented Feb 1, 2019

Just tested unpacking an image in a thin LV created on the same VG as LXD. It worked without any issues.

$ lvcreate -n ThinLVTest1 -V 10G --thinpool vgssd1/LXDThinPool
[sudo] Mot de passe de admin : 
  WARNING: Sum of all thin volume sizes (180,00 GiB) exceeds the size of thin pool vgssd1/LXDThinPool and the size of whole volume group (<100,00 GiB).
  WARNING: You have not turned on protection against thin pools running out of space.
  WARNING: Set activation/thin_pool_autoextend_threshold below 100 to trigger automatic extension of thin pools before they get full.
  Logical volume "ThinLVTest1" created.

$ mkfs.ext4 /dev/vgssd1/ThinLVTest1 
mke2fs 1.44.5 (15-Dec-2018)
Rejet des blocs de périphérique : complété                        
En train de créer un système de fichiers avec 2621440 4k blocs et 655360 i-noeuds.
UUID de système de fichiers=4a9de6d2-7c99-451f-836d-b9b6d8a71bfe
Superblocs de secours stockés sur les blocs : 
	32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632

Allocation des tables de groupe : complété                        
Écriture des tables d'i-noeuds : complété                        
Création du journal (16384 blocs) : complété
Écriture des superblocs et de l'information de comptabilité du système de
fichiers : complété

$ mount /dev/mapper/vgssd1-ThinLVTest1 /mnt/

$ unsquashfs -d /mnt/test /var/lib/lxd/images/df7155e205ba18e619d72ec7e8bff9d558d693a94d512cbb6ad56b09955d92ad.rootfs
Parallel unsquashfs: Using 4 processors
9676 inodes (10928 blocks) to write

[=========================================================================================================================================================================-] 10928/10928 100%

created 8186 files
created 944 directories
created 1474 symlinks
created 7 devices
created 0 fifos

$ ls -l /mnt/test/
total 80
drwxr-xr-x  2 root root 4096 27 jun  2018 bin
drwxr-xr-x  2 root root 4096 30 mai  2016 boot
drwxr-xr-x  3 root root 4096 27 jun  2018 dev
drwxr-xr-x 35 root root 4096 27 jun  2018 etc
drwxr-xr-x  2 root root 4096 30 mai  2016 home
drwxr-xr-x  7 root root 4096 22 jun  2012 lib
drwxr-xr-x  2 root root 4096 27 jun  2018 lib64
drwxr-xr-x  2 root root 4096 27 jun  2018 media
drwxr-xr-x  2 root root 4096 30 mai  2016 mnt
drwxr-xr-x  2 root root 4096 27 jun  2018 opt
drwxr-xr-x  2 root root 4096 30 mai  2016 proc
drwx------  2 root root 4096 27 jun  2018 root
drwxr-xr-x  2 root root 4096 27 jun  2018 run
drwxr-xr-x  2 root root 4096 27 jun  2018 sbin
drwxr-xr-x  2 root root 4096 10 jun  2012 selinux
drwxr-xr-x  2 root root 4096 27 jun  2018 srv
drwxr-xr-x  2 root root 4096 14 jui  2013 sys
drwxrwxrwt  2 root root 4096 27 jun  2018 tmp
drwxr-xr-x 10 root root 4096 27 jun  2018 usr
drwxr-xr-x 11 root root 4096 27 jun  2018 var

$ [root@nas mnt]# lvs
  LV                                                                      VG        Attr       LSize   Pool        Origin                Data%  Meta%  Move Log Cpy%Sync Convert
[...]                                                                         
  LXDThinPool                                                             vgssd1    twi-aotz-- <98,00g                                   7,37   2,03                            
  ThinLVTest1                                                             vgssd1    Vwi-aotz--  10,00g LXDThinPool                       2,24

Will keep investigating on my end.

@stgraber

This comment has been minimized.

Copy link
Member

commented Feb 1, 2019

That's pretty odd as that's effectively what LXD would do itself... I wonder what ends up being different then.

@falstaff1288

This comment has been minimized.

Copy link
Author

commented Feb 2, 2019

Additional verbose from journalctl when attempting to create a image that I didn't shared in my previous posts :

fév 02 14:41:20 nas kernel: EXT4-fs (dm-12): mounted filesystem with ordered data mode. Opts: discard
fév 02 14:41:20 nas systemd[1120]: var-lib-lxd-storage\x2dpools-LVM-images-d7838712001c535542b0a51a2721f797cd998eb25d4d3de4b2f23ffefa34c219.mount: Succeeded.
fév 02 14:41:20 nas systemd[1]: var-lib-lxd-storage\x2dpools-LVM-images-d7838712001c535542b0a51a2721f797cd998eb25d4d3de4b2f23ffefa34c219.mount: Succeeded.
fév 02 14:41:21 nas.maison.lan lxd[1050]: t=2019-02-02T14:41:21-0500 lvl=eror msg="Failed to create LVM storage volume for container \"test\" on storage pool \"LVM\": Image create: Unpack failed, Failed to run: unsquashfs -f -d /var/lib/lxd/storage-pools/LVM/images/d7838712001c535542b0a51a2721f797cd998eb25d4d3de4b2f23ffefa34c219/rootfs -n /var/lib/lxd/images/d7838712001c535542b0a51a2721f797cd998eb25d4d3de4b2f23ffefa34c219.rootfs: FATAL ERROR:Data queue size is too large.  FATAL ERROR:Data queue size is too large"
@falstaff1288

This comment has been minimized.

Copy link
Author

commented Feb 3, 2019

Might have found a lead. With unsquash, the maximum queue size set by default is 256MB :

$ unsquashfs 
SYNTAX: unsquashfs [options] filesystem [directories or files to extract]
	-v[ersion]		print version, licence and copyright information
[...]
	-da[ta-queue] <size>	Set data queue to <size> Mbytes.  Default 256
				Mbytes 

The queue size for all the dm block devices in my system is 128MB :

$ cat /sys/block/dm-*/queue/nr_requests
128
128
128
128
128
128
128
128
128
128
128
128
128
128
128
128
128
128
128

Has the max queue depth value has changed for the I/O schedule since the last system updates? I'm not sure.

Right now, I am looking for a solution to set the value for dm block devices to 256MB at boot time and see if this resolves the issue. I'm also wondering if there is a way to change the arguments used by LXD when executing the unsquash command. Any advise on the latter option would be welcomed.

Refs:
https://www.redhat.com/archives/linux-lvm/2004-February/msg00115.html
http://yoshinorimatsunobu.blogspot.com/2009/04/linux-io-scheduler-queue-size-and.html

@falstaff1288

This comment has been minimized.

Copy link
Author

commented Feb 3, 2019

  • Figured out how to add the -da 128 & -fr 128 variables to the command unsquash after inspecting the source code for lxd. unsquash still fails to unpack an image. Same when the max queue size is 64MB, which is the nr_requests value of the physical disk :
$ lxc launch images:archlinux test
Creating test
Error: Failed container creation: Create container from image: Image create: Unpack failed, Failed to run: unsquashfs -f -d /var/lib/lxd/storage-pools/LVM/images/66798231655d6c1ab1d42678b47557f7992126dfd941cb14bbcdd11db21726c3/rootfs -n -da 64 -fr 64 -p 1 /var/lib/lxd/images/66798231655d6c1ab1d42678b47557f7992126dfd941cb14bbcdd11db21726c3.rootfs: FATAL ERROR:Data queue size is too large.  FATAL ERROR:Data queue size is too large
  • Changed the backend to dir instead of LVM, the issue persist. Therefore, the problem doesn't seems to be related to LVM. Will remove the LVM backend tag from the title based on the latest information I've gathered.

@falstaff1288 falstaff1288 changed the title [LVM storage] Error: Failed container creation: Create container from image: Image create: Unpack failed, Failed to run: unsquashfs Error: Failed container creation: Create container from image: Image create: Unpack failed, Failed to run: unsquashfs Feb 3, 2019

@stgraber

This comment has been minimized.

Copy link
Member

commented Feb 3, 2019

Do you know if your distro updated squashfs-tools recently? Could there have been a change in there that may explain this change of behavior?

It's still odd that you're only seeing this through LXD, I wonder if maybe systemd is placing some kind of process limits on LXD which are then inherited by unsquashfs, explaining why you're not seeing this from a regular shell with the same arguments.

@stgraber

This comment has been minimized.

Copy link
Member

commented Feb 3, 2019

Comparing:

  • cat /proc/PID/limits (with PID being the LXD pid)
  • cat /proc/self/limits (for your current shell)

May give us some hints as to what's going on.

@falstaff1288

This comment has been minimized.

Copy link
Author

commented Feb 3, 2019

Output :

$ pidof lxd
19047

$ cat /proc/19047/limits 
Limit                     Soft Limit           Hard Limit           Units     
Max cpu time              unlimited            unlimited            seconds   
Max file size             unlimited            unlimited            bytes     
Max data size             unlimited            unlimited            bytes     
Max stack size            8388608              unlimited            bytes     
Max core file size        unlimited            unlimited            bytes     
Max resident set          unlimited            unlimited            bytes     
Max processes             unlimited            unlimited            processes 
Max open files            1073741816           1073741816           files     
Max locked memory         67108864             67108864             bytes     
Max address space         unlimited            unlimited            bytes     
Max file locks            unlimited            unlimited            locks     
Max pending signals       63652                63652                signals   
Max msgqueue size         819200               819200               bytes     
Max nice priority         0                    0                    
Max realtime priority     0                    0                    
Max realtime timeout      unlimited            unlimited            us  
      
$ cat /proc/self/limits 
Limit                     Soft Limit           Hard Limit           Units     
Max cpu time              unlimited            unlimited            seconds   
Max file size             unlimited            unlimited            bytes     
Max data size             unlimited            unlimited            bytes     
Max stack size            8388608              unlimited            bytes     
Max core file size        unlimited            unlimited            bytes     
Max resident set          unlimited            unlimited            bytes     
Max processes             63652                63652                processes 
Max open files            1024                 524288               files     
Max locked memory         67108864             67108864             bytes     
Max address space         unlimited            unlimited            bytes     
Max file locks            unlimited            unlimited            locks     
Max pending signals       63652                63652                signals   
Max msgqueue size         819200               819200               bytes     
Max nice priority         0                    0                    
Max realtime priority     0                    0                    
Max realtime timeout      unlimited            unlimited            us  
@falstaff1288

This comment has been minimized.

Copy link
Author

commented Feb 3, 2019

I don't know how different it is in the back-end to create containers with LXC, but I had no issue creating one :

$ lxc-create -t download -n test1
Setting up the GPG keyring
Downloading the image index

---
DIST	RELEASE	ARCH	VARIANT	BUILD
---
alpine	3.4	amd64	default	20180627_17:50
alpine	3.4	armhf	default	20180627_17:50
alpine	3.4	i386	default	20180627_17:50
alpine	3.5	amd64	default	20190203_13:01
alpine	3.5	arm64	default	20190203_13:04
alpine	3.5	armhf	default	20190203_13:04
alpine	3.5	i386	default	20190203_13:01
alpine	3.6	amd64	default	20190203_13:01
alpine	3.6	arm64	default	20190202_13:02
alpine	3.6	armhf	default	20190203_13:04
alpine	3.6	i386	default	20190203_13:01
alpine	3.7	amd64	default	20190203_13:01
alpine	3.7	arm64	default	20190203_13:02
alpine	3.7	armhf	default	20190203_13:04
alpine	3.7	i386	default	20190203_13:08
alpine	3.8	amd64	default	20190203_13:01
alpine	3.8	arm64	default	20190203_13:02
alpine	3.8	armhf	default	20190202_13:03
alpine	3.8	i386	default	20190203_13:01
alpine	3.8	ppc64el	default	20190203_13:02
alpine	3.8	s390x	default	20190203_13:02
alpine	edge	amd64	default	20190203_13:01
alpine	edge	arm64	default	20190203_13:03
alpine	edge	armhf	default	20190203_13:03
alpine	edge	i386	default	20190203_13:08
alpine	edge	ppc64el	default	20190203_13:02
alpine	edge	s390x	default	20190203_13:02
archlinux	current	amd64	default	20190203_01:27
centos	6	amd64	default	20190203_08:13
centos	6	i386	default	20190203_08:13
centos	7	amd64	default	20190203_08:13
centos	7	arm64	default	20190203_08:15
centos	7	armhf	default	20190203_08:16
centos	7	i386	default	20190203_08:20
centos	7	ppc64el	default	20190203_08:14
debian	buster	amd64	default	20190203_05:25
debian	buster	arm64	default	20190203_05:28
debian	buster	armel	default	20190203_05:28
debian	buster	armhf	default	20190203_05:28
debian	buster	i386	default	20190203_05:25
debian	buster	ppc64el	default	20190203_05:26
debian	buster	s390x	default	20190203_05:26
debian	jessie	amd64	default	20190203_05:25
debian	jessie	arm64	default	20180626_05:25
debian	jessie	armel	default	20190203_05:27
debian	jessie	armhf	default	20190203_05:42
debian	jessie	i386	default	20190203_05:32
debian	jessie	powerpc	default	20180626_05:25
debian	jessie	ppc64el	default	20180626_05:25
debian	jessie	s390x	default	20180626_05:25
debian	sid	amd64	default	20190203_05:25
debian	sid	arm64	default	20190203_05:25
debian	sid	armel	default	20190203_05:28
debian	sid	armhf	default	20190203_05:29
debian	sid	i386	default	20190203_05:25
debian	sid	powerpc	default	20180708_05:25
debian	sid	ppc64el	default	20190203_05:26
debian	sid	s390x	default	20190203_05:26
debian	stretch	amd64	default	20190203_05:25
debian	stretch	arm64	default	20190203_05:29
debian	stretch	armel	default	20190203_05:28
debian	stretch	armhf	default	20190203_05:28
debian	stretch	i386	default	20190203_05:32
debian	stretch	ppc64el	default	20190203_05:26
debian	stretch	s390x	default	20190203_05:26
debian	wheezy	amd64	default	20180627_05:24
debian	wheezy	armel	default	20180627_05:27
debian	wheezy	armhf	default	20180627_05:26
debian	wheezy	i386	default	20180627_05:25
debian	wheezy	powerpc	default	20180627_05:25
debian	wheezy	s390x	default	20180627_05:25
fedora	26	amd64	default	20181102_01:27
fedora	26	i386	default	20181102_01:27
fedora	27	amd64	default	20190203_01:27
fedora	27	i386	default	20190203_01:27
fedora	28	amd64	default	20190203_01:27
fedora	28	i386	default	20190203_01:27
fedora	29	amd64	default	20190203_01:27
fedora	29	i386	default	20190203_01:27
gentoo	current	amd64	default	20190202_14:12
gentoo	current	i386	default	20190202_14:12
opensuse	15.0	amd64	default	20190203_00:53
opensuse	42.3	amd64	default	20190203_00:53
oracle	6	amd64	default	20190203_11:40
oracle	6	i386	default	20190203_11:40
oracle	7	amd64	default	20190203_11:40
plamo	5.x	amd64	default	20180816_21:36
plamo	5.x	i386	default	20180816_21:36
plamo	6.x	amd64	default	20190202_21:36
plamo	6.x	i386	default	20190202_21:36
plamo	7.x	amd64	default	20190202_21:36
sabayon	current	amd64	default	20190203_07:38
ubuntu	bionic	amd64	default	20190203_08:14
ubuntu	bionic	arm64	default	20190203_08:16
ubuntu	bionic	armhf	default	20190203_08:16
ubuntu	bionic	i386	default	20190203_08:14
ubuntu	bionic	ppc64el	default	20190203_08:14
ubuntu	bionic	s390x	default	20190203_08:14
ubuntu	cosmic	amd64	default	20190203_08:13
ubuntu	cosmic	arm64	default	20190203_08:16
ubuntu	cosmic	armhf	default	20190203_08:16
ubuntu	cosmic	i386	default	20190203_08:22
ubuntu	cosmic	ppc64el	default	20190203_08:15
ubuntu	cosmic	s390x	default	20190203_08:14
ubuntu	disco	amd64	default	20190203_08:14
ubuntu	disco	arm64	default	20190203_08:14
ubuntu	disco	armhf	default	20190203_08:16
ubuntu	disco	i386	default	20190203_08:13
ubuntu	disco	ppc64el	default	20190203_08:14
ubuntu	disco	s390x	default	20190203_08:14
ubuntu	trusty	amd64	default	20190203_08:13
ubuntu	trusty	arm64	default	20190203_08:14
ubuntu	trusty	armhf	default	20190203_08:16
ubuntu	trusty	i386	default	20190203_08:22
ubuntu	trusty	powerpc	default	20180824_07:43
ubuntu	trusty	ppc64el	default	20190203_08:15
ubuntu	xenial	amd64	default	20190203_08:14
ubuntu	xenial	arm64	default	20190203_08:14
ubuntu	xenial	armhf	default	20190203_08:16
ubuntu	xenial	i386	default	20190203_08:14
ubuntu	xenial	powerpc	default	20180824_07:44
ubuntu	xenial	ppc64el	default	20190203_08:14
ubuntu	xenial	s390x	default	20190203_08:14
---

Distribution: 
archlinux 
Release: 
current
Architecture: 
amd64

Using image from local cache
Unpacking the rootfs

---
You just created an ArchLinux container (release=current, arch=amd64, variant=default)


For security reason, container images ship without user accounts
and without a root password.

Use lxc-attach or chroot directly into the rootfs to set a root password
or create user accounts.


$ lxc-start -n test1 -F
systemd 240 running in system mode. (+PAM +AUDIT -SELINUX -IMA -APPARMOR +SMACK -SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=hybrid)
Detected virtualization lxc.
Detected architecture x86-64.

Welcome to Arch Linux!

Set hostname to <test1>.
/usr/lib/systemd/system/auditd.service:12: PIDFile= references path below legacy directory /var/run/, updating /var/run/auditd.pid → /run/auditd.pid; please update the unit file accordingly.
[  OK  ] Listening on Device-mapper event daemon FIFOs.
[  OK  ] Listening on Journal Audit Socket.
[  OK  ] Listening on LVM2 metadata daemon socket.
[  OK  ] Started Forward Password Requests to Wall Directory Watch.
[  OK  ] Created slice system-getty.slice.
[  OK  ] Reached target Remote File Systems.
[  OK  ] Created slice system-container\x2dgetty.slice.
[  OK  ] Reached target Swap.
[  OK  ] Started Dispatch Password Requests to Console Directory Watch.
[  OK  ] Reached target Paths.
[  OK  ] Created slice User and Session Slice.
[  OK  ] Reached target Slices.
[  OK  ] Listening on Journal Socket (/dev/log).
[  OK  ] Reached target Local Encrypted Volumes.
[  OK  ] Listening on Process Core Dump Socket.
[  OK  ] Listening on Journal Socket.
         Mounting Temporary Directory (/tmp)...
         Mounting Huge Pages File System...
         Starting Apply Kernel Variables...
         Mounting POSIX Message Queue File System...
         Starting Journal Service...
         Starting Remount Root and Kernel File Systems...
         Starting Monitoring of LVM2 mirrors, snaps…tc. using dmeventd or progress polling...
[  OK  ] Listening on Network Service Netlink Socket.
[  OK  ] Listening on LVM2 poll daemon socket.
[  OK  ] Listening on initctl Compatibility Named Pipe.
[  OK  ] Mounted Temporary Directory (/tmp).
[  OK  ] Mounted Huge Pages File System.
[  OK  ] Mounted POSIX Message Queue File System.
[  OK  ] Started Apply Kernel Variables.
[  OK  ] Started Remount Root and Kernel File Systems.
         Starting Create System Users...
[  OK  ] Started LVM2 metadata daemon.
[  OK  ] Started Journal Service.
         Starting Flush Journal to Persistent Storage...
[  OK  ] Started Create System Users.
         Starting Network Service...
[  OK  ] Started Flush Journal to Persistent Storage.
[  OK  ] Started Network Service.
[  OK  ] Started Monitoring of LVM2 mirrors, snapsh… etc. using dmeventd or progress polling.
[  OK  ] Reached target Local File Systems (Pre).
[  OK  ] Reached target Local File Systems.
         Starting Rebuild Journal Catalog...
         Starting Create Volatile Files and Directories...
         Starting Rebuild Dynamic Linker Cache...
[  OK  ] Started Rebuild Journal Catalog.
[  OK  ] Started Create Volatile Files and Directories.
         Starting Update UTMP about System Boot/Shutdown...
         Starting Network Name Resolution...
[  OK  ] Started Rebuild Dynamic Linker Cache.
         Starting Update is Completed...
[  OK  ] Started Update UTMP about System Boot/Shutdown.
[  OK  ] Started Update is Completed.
[  OK  ] Reached target System Initialization.
[  OK  ] Started Daily rotation of log files.
[  OK  ] Started Daily verification of password and group files.
[  OK  ] Started Daily Cleanup of Temporary Directories.
[  OK  ] Reached target Timers.
[  OK  ] Listening on D-Bus System Message Bus Socket.
[  OK  ] Reached target Sockets.
[  OK  ] Reached target Basic System.
         Starting Login Service...
[  OK  ] Started D-Bus System Message Bus.
[  OK  ] Started Login Service.
[  OK  ] Started Network Name Resolution.
[  OK  ] Reached target Host and Network Name Lookups.
[  OK  ] Reached target Network.
         Starting Permit User Sessions...
[  OK  ] Started Permit User Sessions.
[  OK  ] Started Getty on lxc/tty4.
[  OK  ] Started Container Getty on /dev/pts/1.
[  OK  ] Started Getty on lxc/tty6.
[  OK  ] Started Getty on lxc/tty3.
[  OK  ] Started Getty on lxc/tty2.
[  OK  ] Started Container Getty on /dev/pts/2.
[  OK  ] Started Container Getty on /dev/pts/3.
[  OK  ] Started Getty on tty1.
[  OK  ] Started Getty on lxc/tty1.
[  OK  ] Started Console Getty.
[  OK  ] Started Container Getty on /dev/pts/0.
[  OK  ] Started Getty on lxc/tty5.
[  OK  ] Reached target Login Prompts.
[  OK  ] Reached target Multi-User System.

Arch Linux 4.19.19-1-lts (console)

test1 login: 

@stgraber

This comment has been minimized.

Copy link
Member

commented Feb 3, 2019

LXC doesn't use squashfs, so the creation process is pretty different.

@stgraber

This comment has been minimized.

Copy link
Member

commented Feb 3, 2019

I don't suppose you can setup such a system in a way that we could access it to take a look?

I'm kinda running out of ideas at this point, so being able to reproduce it reliably or having access to an affected system would help quite a bit.

@falstaff1288

This comment has been minimized.

Copy link
Author

commented Feb 3, 2019

  • I guess we'll need to look deeper into the relation of LXD and unsquashfs
  • Sadly, I cannot grant access to this server.
@falstaff1288

This comment has been minimized.

Copy link
Author

commented Feb 3, 2019

To answer a question you've asked earlier, here the history for the squash packages in the server :

$ cat /var/log/pacman.log | grep squas 
[2018-06-07 15:32] [PACMAN] Running 'pacman --noconfirm --asdeps -S -- lxc squashfs-tools'
[2018-06-07 15:33] [PACMAN] Running 'pacman --noconfirm --asdeps -S -- lxc squashfs-tools'
[2018-06-07 15:33] [ALPM] installed squashfs-tools (4.3-5)
[2018-10-04 17:17] [PACMAN] Running 'pacman -S squashfs'
[2018-10-04 17:21] [PACMAN] Running 'pacman -U --noconfirm --config /etc/pacman.conf -- /home/admin/.cache/yay/squashfuse-git/squashfuse-git-340.0b48352-1-x86_64.pkg.tar.xz'
[2018-10-04 17:21] [ALPM] installed squashfuse-git (340.0b48352-1)
[2018-12-25 09:33] [ALPM] upgraded squashfs-tools (4.3-5 -> 4.3-8)
[2019-02-03 00:24] [PACMAN] Running 'pacman -U --noconfirm --config /etc/pacman.conf -- /home/admin/.cache/yay/squashfuse-git/squashfuse-git-340.0b48352-1-x86_64.pkg.tar.xz'
[2019-02-03 00:24] [ALPM] reinstalled squashfuse-git (340.0b48352-1)
[2019-02-03 09:23] [PACMAN] Running 'pacman -U --config /etc/pacman.conf -- /home/admin/.cache/yay/squashfs-tools-git/squashfs-tools-git-1808.6e242dc-1-x86_64.pkg.tar.xz'
[2019-02-03 09:23] [ALPM] removed squashfs-tools (4.3-8)
[2019-02-03 09:23] [ALPM] installed squashfs-tools-git (1808.6e242dc-1)
[2019-02-03 09:38] [PACMAN] Running 'pacman -U http://ftp5.gwdg.de/pub/linux/archlinux/community/os/x86_64//squashfs-tools-4.3-8-x86_64.pkg.tar.xz'
[2019-02-03 09:38] [ALPM] removed squashfs-tools-git (1808.6e242dc-1)
[2019-02-03 09:38] [ALPM] installed squashfs-tools (4.3-8)

  • Creating containers was working fine until the Jan 31st
$ last -x reboot
reboot   system boot  4.19.19-1-lts    Sat Feb  2 20:33   still running
reboot   system boot  4.19.19-1-lts    Sat Feb  2 20:29 - 20:32  (00:02)
reboot   system boot  4.19.18-1-lts    Fri Feb  1 11:05 - 20:29 (1+09:23)
reboot   system boot  4.19.18-1-lts    Fri Feb  1 07:31 - 11:04  (03:32)
reboot   system boot  4.20.5-arch1-1-A Thu Jan 31 14:26 - 07:30  (17:03)
reboot   system boot  4.19.12-arch1-1- Thu Jan 17 13:10 - 14:26 (14+01:16)
reboot   system boot  4.19.12-arch1-1- Thu Jan 17 12:02 - 13:09  (01:06)
reboot   system boot  4.19.12-arch1-1- Tue Dec 25 14:53 - 13:09 (22+22:16)

I'll see if reverting the kernet to 4.19.12-arch1-1 helps any.

@stgraber

This comment has been minimized.

Copy link
Member

commented Feb 3, 2019

@monstermunchkin is this something you can try on your Arch system easily enough? Basically creating a LVM storage pool and trying to throw an ubuntu:xenial container at it, see if that works. Also make sure you have squashfs-tools installed as LXD would fallback to tar.xz if it's not available.

@monstermunchkin

This comment has been minimized.

Copy link
Member

commented Feb 4, 2019

I cannot confirm this error on my Arch system. Creating a Xenial container works fine for me.

@stgraber

This comment has been minimized.

Copy link
Member

commented Feb 4, 2019

@monstermunchkin on a LVM storage pool too?

@falstaff1288

This comment has been minimized.

Copy link
Author

commented Feb 4, 2019

I installed LXD from the AUR on my laptop running an up-to-date ArchLinux (ext4/dir backend only) and everything works fine.

The question for me is how to isolate the issue on my server. Again, I get the same issue whether I use LVM or dir as a storage backend which is weird.

@monstermunchkin

This comment has been minimized.

Copy link
Member

commented Feb 4, 2019

@stgraber Yes, with LVM.

@stgraber

This comment has been minimized.

Copy link
Member

commented Feb 4, 2019

@falstaff1288 can you confirm that your two systems are running the same version of squashfs-tools and kernel? You could compare LXD and LXC versions too, but that really shouldn't matter... Might be worth comparing systemd version too maybe?

@falstaff1288

This comment has been minimized.

Copy link
Author

commented Feb 5, 2019

  • Other than the kernel versions (installed linux-lts on my server in order to see if it fixed the issue), squashfs-tools, systemd, lxc and lxd are the same versions.
  • Tried with a btrfs backend by mounting a btrfs partition to /var/lib/lxd, started the service and ran lxd init for the initial configuration. Still getting the same result with the unsquashfs error.
$ lxc launch images:alpine/3.8 test
Creating test
Error: Failed container creation: Create container from image: Failed to create image volume: Unpack failed, Failed to run: unsquashfs -f -d /var/lib/lxd/storage-pools/default/images/5331a5e28e6be9316a52e3cb24c602d4734b375c010fafeb7062cdf6e2457256_tmp/rootfs -n /var/lib/lxd/images/5331a5e28e6be9316a52e3cb24c602d4734b375c010fafeb7062cdf6e2457256.rootfs: FATAL ERROR:Data queue size is too large.  FATAL ERROR:Data queue size is too large

$ lxc list
+------+-------+------+------+------+-----------+
| NAME | STATE | IPV4 | IPV6 | TYPE | SNAPSHOTS |
+------+-------+------+------+------+-----------+

$ lxc storage show default
config:
  source: /var/lib/lxd/storage-pools/default
  volatile.initial_source: /var/lib/lxd/storage-pools/default
description: ""
name: default
driver: btrfs
used_by:
- /1.0/images/5331a5e28e6be9316a52e3cb24c602d4734b375c010fafeb7062cdf6e2457256
- /1.0/images/ee33f784ee1fecbb5be50b6952dacbb8a150fb378b321c0b935f00bdb4a0db16
- /1.0/profiles/default
status: Created
locations:
- none

$ lxc image list
+-------+--------------+--------+-----------------------------------+--------+----------+-----------------------------+
| ALIAS | FINGERPRINT  | PUBLIC |            DESCRIPTION            |  ARCH  |   SIZE   |         UPLOAD DATE         |
+-------+--------------+--------+-----------------------------------+--------+----------+-----------------------------+
|       | 5331a5e28e6b | no     | Alpine 3.8 amd64 (20190204_13:01) | x86_64 | 2.34MB   | Feb 5, 2019 at 1:06pm (UTC) |
+-------+--------------+--------+-----------------------------------+--------+----------+-----------------------------+
|       | ee33f784ee1f | no     | Centos 7 amd64 (20190205_07:09)   | x86_64 | 124.86MB | Feb 5, 2019 at 1:11pm (UTC) |
+-------+--------------+--------+-----------------------------------+--------+----------+-----------------------------+

$ btrfs filesystem usage /var/lib/lxd 
Overall:
    Device size:		 100.00GiB
    Device allocated:		   2.02GiB
    Device unallocated:		  97.98GiB
    Device missing:		     0.00B
    Used:			 130.25MiB
    Free (estimated):		  98.86GiB	(min: 98.86GiB)
    Data ratio:			      1.00
    Metadata ratio:		      1.00
    Global reserve:		  16.00MiB	(used: 0.00B)

Data,single: Size:1.01GiB, Used:129.88MiB
   /dev/sdb2	   1.01GiB

Metadata,single: Size:1.01GiB, Used:368.00KiB
   /dev/sdb2	   1.01GiB

System,single: Size:4.00MiB, Used:16.00KiB
   /dev/sdb2	   4.00MiB

Unallocated:
   /dev/sdb2	  97.98GiB

$ btrfs subvolume show /var/lib/lxd 
/
	Name: 			<FS_TREE>
	UUID: 			39609623-c2c9-4d3d-8922-e574c5355cca
	Parent UUID: 		-
	Received UUID: 		-
	Creation time: 		2019-02-05 08:04:18 -0500
	Subvolume ID: 		5
	Generation: 		18
	Gen at creation: 	0
	Parent ID: 		0
	Top level ID: 		0
	Flags: 			-
	Snapshot(s):

$ btrfs subvolume list /var/lib/lxd 
ID 257 gen 13 top level 5 path storage-pools/default
ID 258 gen 9 top level 257 path storage-pools/default/containers
ID 259 gen 10 top level 257 path storage-pools/default/containers-snapshots
ID 260 gen 18 top level 257 path storage-pools/default/images
ID 261 gen 12 top level 257 path storage-pools/default/custom
ID 262 gen 13 top level 257 path storage-pools/default/custom-snapshots

  • Yesterday, I gave incomplete test results regarding trying LXD on my laptop. It worked but didn't used LVM as a storage back-end; it's a bare ext4 partition with dir. Just wanted to precise this detail although it is kind of irrelevant as I am getting the same error on my server no matter which backend is used.
$ lsblk -f
NAME   FSTYPE LABEL UUID                                 FSAVAIL FSUSE% MOUNTPOINT
sda                                                                     
├─sda1 ext2         d01fe79c-189e-4003-9e47-a1ba2bffbd96    424M    11% /boot
├─sda2 ext4         ddb8edcc-cd67-4944-96cd-74a64ce70932   30.4G    33% /
├─sda3 ext4         99d4186b-6aba-4dcb-9e0c-52fe5910d164  132.8G     4% /home
└─sda4      

$ sudo lxc storage show default
config:
  source: /var/lib/lxd/storage-pools/default
description: ""
name: default
driver: dir
used_by:
- /1.0/containers/docker
- /1.0/profiles/default
status: Created
locations:
- none
                                                           

I'll continue with further research in hope to isolate the issue.

@timlepes

This comment has been minimized.

Copy link

commented Feb 7, 2019

I am getting the same unsquashfs errors using the lxc create command. I'm running Arch with the 4.20.6-zen1-1-zen kernel. Both lxc and lxd are at version 3.9, storage is btrfs at version 4.19.1. Let me know if you want any diagnostics from me as well.

I first noticed problems yesterday or the day before when I went to delete several containers I had made a few weeks ago when playing around with the networking options. The lxbr0 containers deleted fine, but the macvlan ones took FOREVER to remove... I actually thought the command hung so I killed the terminal after a while, but was able to see that the containers had actually been removed (using lxc list). So after deleteing the half dozen containers I no longer needed, I went to create a new container and got the error message; searching the net landed me here.

@timlepes

This comment has been minimized.

Copy link

commented Feb 7, 2019

Here is some basic info in case it is useful. Let me know if more is needed.

[tlepes@mothership ~]$ sudo lxc launch ubuntu:18.04 monica-server
[sudo] password for tlepes:
Creating monica-server
Error: Failed container creation: Create container from image: Failed to create image volume: Unpack failed, Failed to run: unsquashfs -f -d /var/lib/lxd/storage-pools/default/images/b7c4dbea897f09f29474c8597c511b57c3b9c0d6f98dc42f257c64e76fea8c92_tmp/rootfs -n /var/lib/lxd/images/b7c4dbea897f09f29474c8597c511b57c3b9c0d6f98dc42f257c64e76fea8c92.rootfs: FATAL ERROR:Data queue size is too large. FATAL ERROR:Data queue size is too large
[tlepes@mothership ~]$ lxc info
config:
core.https_address: '[::]:8443'
core.trust_password: true
api_extensions:

  • storage_zfs_remove_snapshots
  • container_host_shutdown_timeout
  • container_stop_priority
  • container_syscall_filtering
  • auth_pki
  • container_last_used_at
  • etag
  • patch
  • usb_devices
  • https_allowed_credentials
  • image_compression_algorithm
  • directory_manipulation
  • container_cpu_time
  • storage_zfs_use_refquota
  • storage_lvm_mount_options
  • network
  • profile_usedby
  • container_push
  • container_exec_recording
  • certificate_update
  • container_exec_signal_handling
  • gpu_devices
  • container_image_properties
  • migration_progress
  • id_map
  • network_firewall_filtering
  • network_routes
  • storage
  • file_delete
  • file_append
  • network_dhcp_expiry
  • storage_lvm_vg_rename
  • storage_lvm_thinpool_rename
  • network_vlan
  • image_create_aliases
  • container_stateless_copy
  • container_only_migration
  • storage_zfs_clone_copy
  • unix_device_rename
  • storage_lvm_use_thinpool
  • storage_rsync_bwlimit
  • network_vxlan_interface
  • storage_btrfs_mount_options
  • entity_description
  • image_force_refresh
  • storage_lvm_lv_resizing
  • id_map_base
  • file_symlinks
  • container_push_target
  • network_vlan_physical
  • storage_images_delete
  • container_edit_metadata
  • container_snapshot_stateful_migration
  • storage_driver_ceph
  • storage_ceph_user_name
  • resource_limits
  • storage_volatile_initial_source
  • storage_ceph_force_osd_reuse
  • storage_block_filesystem_btrfs
  • resources
  • kernel_limits
  • storage_api_volume_rename
  • macaroon_authentication
  • network_sriov
  • console
  • restrict_devlxd
  • migration_pre_copy
  • infiniband
  • maas_network
  • devlxd_events
  • proxy
  • network_dhcp_gateway
  • file_get_symlink
  • network_leases
  • unix_device_hotplug
  • storage_api_local_volume_handling
  • operation_description
  • clustering
  • event_lifecycle
  • storage_api_remote_volume_handling
  • nvidia_runtime
  • container_mount_propagation
  • container_backup
  • devlxd_images
  • container_local_cross_pool_handling
  • proxy_unix
  • proxy_udp
  • clustering_join
  • proxy_tcp_udp_multi_port_handling
  • network_state
  • proxy_unix_dac_properties
  • container_protection_delete
  • unix_priv_drop
  • pprof_http
  • proxy_haproxy_protocol
  • network_hwaddr
  • proxy_nat
  • network_nat_order
  • container_full
  • candid_authentication
  • backup_compression
  • candid_config
  • nvidia_runtime_config
  • storage_api_volume_snapshots
  • storage_unmapped
  • projects
  • candid_config_key
  • network_vxlan_ttl
  • container_incremental_copy
  • usb_optional_vendorid
  • snapshot_scheduling
  • container_copy_project
  • clustering_server_address
  • clustering_image_replication
  • container_protection_shift
    api_status: stable
    api_version: "1.0"
    auth: trusted
    public: false
    auth_methods:
  • tls
    environment:
    addresses:
    • 172.16.1.150:8443
    • 10.165.73.1:8443
    • '[fd42:4d7c:53f5:5417::1]:8443'
    • 10.0.0.2:8443
      architectures:
    • x86_64
    • i686
      certificate: |
      -----BEGIN CERTIFICATE-----
      ** REDACTED **
      -----END CERTIFICATE-----
      certificate_fingerprint: ** REDACTED **
      driver: lxc
      driver_version: 3.1.0
      kernel: Linux
      kernel_architecture: x86_64
      kernel_version: 4.20.6-zen1-1-zen
      server: lxd
      server_pid: 979
      server_version: "3.9"
      storage: btrfs
      storage_version: 4.19.1
      server_clustered: false
      server_name: mothership
      project: default
      [tlepes@mothership ~]$
@stgraber

This comment has been minimized.

Copy link
Member

commented Feb 7, 2019

@timlepes @falstaff1288 the best bet for us to figure this out at this point is to either have reliable reproduction steps on a clean ArchLinux install, or provide us with a VM image that's got the issue happening inside it or provide us with access to an affected system.

The only hits for this error are this issue and the squashfs source code and looking at the code, it's pretty unclear what's going on, so we need to be able to reproduce this issue.

@falstaff1288

This comment has been minimized.

Copy link
Author

commented Feb 9, 2019

I'll try to reproduce the issue in a VM.

@stgraber

This comment has been minimized.

Copy link
Member

commented Feb 15, 2019

@falstaff1288 any luck with that?

@falstaff1288

This comment has been minimized.

Copy link
Author

commented Feb 15, 2019

I've been trying to reproduce my infrastructure in a VM through KVM using a LVM + macvlan, since this is the setup @timlepes has as well. I figured out for the LVM part, still working on the macvlan one.

Will keep you posted with more details.

@falstaff1288

This comment has been minimized.

Copy link
Author

commented Feb 22, 2019

I have not been able to reproduce the issue in a VM; my attempt to configure a macvlan bridge have been unsuccessful. I have not been able either to fix the issue on my server either. Running out of ideas.

@stgraber

This comment has been minimized.

Copy link
Member

commented Feb 22, 2019

Hmm, anyone else running into this and who has an idea of what the issue may be or can give us access to look around and track this down?

@timlepes

This comment has been minimized.

Copy link

commented Feb 22, 2019

@blair-0

This comment has been minimized.

Copy link

commented Feb 23, 2019

same problem was accrued after upgrade archlinux, I check the squashfs-tools code and found some information, dose this useful for you to troubleshoot?

if (max_files != -1) {
if(add_overflow(data_buffer_size, max_files) ||
add_overflow(data_buffer_size, max_files * 2))
EXIT_UNSQUASH("Data queue size is too large\n");

if(shift_overflow(data_buffer_size, 20 - block_log))
EXIT_UNSQUASH("Data queue size is too large\n");
else
data_buffer_size <<= 20 - block_log;

url: https://sourceforge.net/p/squashfs/code/ci/master/tree/squashfs-tools/unsquashfs.c#l2749

@stgraber

This comment has been minimized.

Copy link
Member

commented Feb 23, 2019

Not really, especially doesn't explain why running the same command outside of LXD would succeed...

@stgraber

This comment has been minimized.

Copy link
Member

commented Feb 23, 2019

I did read that part of the squashfs code but since it doesn't look like squashfs was changed just before this issue started, it seems unlikely to be the source.

@github-usr-name

This comment has been minimized.

Copy link

commented Mar 2, 2019

Just encountered the same:

root@cygnus:~# lxc launch images:alpine/3.9 Creating the container Error: Failed container creation: Create container from image: Unpack failed, Failed to run: unsquashfs -f -d /var/lib/lxd/storage-pools/default/images/991367651/rootfs -n /var/lib/lxd/images/40dd95add1657a574cc076982ce5f1e0bb3a44522b91eeb1b365460e7cd04f2c.rootfs: FATAL ERROR:Data queue size is too large

System setup is comparable to OP [Arch Linux, similar upgrade date & lxc info output]

@monstermunchkin

This comment has been minimized.

Copy link
Member

commented Mar 4, 2019

How are you running LXD? Are you using the AUR package? If so, please try running LXD from the command line (sudo lxd --group lxd --debug) , instead of using the systemd service. Does this change anything?

@github-usr-name

This comment has been minimized.

Copy link

commented Mar 4, 2019

@monstermunchkin Thanks for the help :). Yes, I'm using the AUR package. Interestingly enough, with LXD running through the CLI as above, things are working properly. Output from systemctl cat lxd:

[Unit]
Description=REST API, command line tool and OpenStack integration plugin for LXC.
After=network.target

[Service]
Type=simple
ExecStart=/usr/bin/lxd --group lxd
ExecStop=/usr/bin/lxd shutdown
KillMode=process
LimitNOFILE=1048576
LimitNPROC=infinity
TasksMax=infinity

[Install]
WantedBy=multi-user.target

# /etc/systemd/system/lxd.service.d/override.conf
[Service]
LimitNOFILE=infinity
LimitNPROC=infinity
TasksMax=infinity
@github-usr-name

This comment has been minimized.

Copy link

commented Mar 4, 2019

and... after removing the override, LXD is behaving properly again :)

@monstermunchkin

This comment has been minimized.

Copy link
Member

commented Mar 4, 2019

@falstaff1288 @timlepes does this solve your problem as well?

@falstaff1288

This comment has been minimized.

Copy link
Author

commented Mar 4, 2019

@monstermunchkin I confirm that removing /etc/systemd/system/lxd.service.d/override.conf fixed the issue with unsquashfs. Good find @github-usr-name. So is it the parameter LimitNOFILE=infinity that causes the issue?

@github-usr-name

This comment has been minimized.

Copy link

commented Mar 4, 2019

@falstaff1288 Based on empirical observation, yes - adding an override with just the LimitNOFILE=infinity caused the issue to re-occur. I suspect an upstream bug in squashfs around line 2315 where it's checking for an overflow; if the value of LimitNOFILE is exactly equal to INT_MAX then it looks as if the behaviour will be triggered.

@stgraber

This comment has been minimized.

Copy link
Member

commented Mar 4, 2019

Oh, that's interesting, not sure why squashfs would care about any of those prlimit bumps, might be interesting to figure out which one it is and file a bug against squashfs indeed.

Closing as this is clearly no a LXD bug then :)

@stgraber stgraber closed this Mar 4, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.