cos-upgrade uses COS_STATE for temporary files during upgrades

[bk201]

cos-toolkit version:
RancherOSv2 with utils/installer-0.8+9.

Describe the bug
Fail to upgrade OS from a registry.

To Reproduce

• Setup self-signed SSL things for private registry and accept the certs on nodes.
• Push OS image to the registry.
• Upgrade the OS from the registry. e.g.,

cos-upgrade --docker-image 192.168.2.106/rancher/harvester-os:c22323d-dirty

And the output:

Upgrading system..
-> Upgrade target: active.img
3240+0 records in
3240+0 records out
3397386240 bytes (3.4 GB, 3.2 GiB) copied, 2.10756 s, 1.6 GB/s
mke2fs 1.43.8 (1-Jan-2018)
Discarding device blocks: done
Creating filesystem with 829440 4k blocks and 207584 inodes
Filesystem UUID: 27619c9c-dd09-4eff-afe1-d439dabe734f
Superblock backups stored on blocks:
	32768, 98304, 163840, 229376, 294912, 819200

Allocating group tables: done
Writing inode tables: done
Writing superblocks and filesystem accounting information: done

mkdir: cannot create directory ‘/tmp/upgrade/usr/local’: No such file or directory
Upgrading from container image: 192.168.2.106/rancher/harvester-os:c22323d-dirty
+ luet util unpack --enable-logfile --logfile /tmp/luet.log --plugin luet-mtree 192.168.2.106/rancher/harvester-os:c22323d-dirty /usr/local/tmp/rootfs
 Downloading 192.168.2.106/rancher/harvester-os:c22323d-dirty to /usr/local/tmp/rootfs
 failed listing images: copying of parent failed: failed to copy files: copy file range failed: no space left on device
+ cleanup
+ EXIT=1
+ cleanup2
+ return 1

Expected behavior
The image content should be writtend into active.img.

Logs

Additional context
The space of /run/initramfs/cos-state is exhausted during the upgrade (the free space 11GB should be enough for the 1.27GB image and the new 3G active.img).

If I modify the code to this, the upgrade succeeds:

diff --git a/packages/installer/upgrade.sh b/packages/installer/upgrade.sh
index 1920bf9..351203c 100644
--- a/packages/installer/upgrade.sh
+++ b/packages/installer/upgrade.sh
@@ -162,7 +162,6 @@ upgrade() {
     # FIXME: Define default /var/tmp as tmpdir_base in default luet config file
     export XDG_RUNTIME_DIR=$temp_upgrade
     export TMPDIR=$temp_upgrade
-    export LUET_PRIVILEGED_EXTRACT=true

     args=""
     if [ -z "$VERIFY" ] || [ "$VERIFY" == true ]; then
@@ -181,6 +180,7 @@ upgrade() {
     else
         echo "Upgrading from container image: $UPGRADE_IMAGE"
         set -x
+        mkdir -p /usr/local/tmp/rootfs
         luet util unpack $args $UPGRADE_IMAGE /usr/local/tmp/rootfs
         set +x
         rsync -aqzAX --exclude='mnt' --exclude='proc' --exclude='sys' --exclude='dev' --exclude='tmp' /usr/local/tmp/rootfs/ $TARGET

The image will be extraced to /usr/local/tmp/rootfs rather than $STATEDIR/tmp/upgrade. And it only consumes the as much as the image size (~1.27GB).

[mudler]

👀 On first place, the upgrade process should use the temporary space in COS_PERSISTENT, so this is a bug. About the LUET_PRIVILEGED_EXTRACT is something required as for now as disabling makes our test suite to fail (#260), and that needs further investigation

[kkaempf]

eyes On first place, the upgrade process should use the temporary space in COS_PERSISTENT, so this is a bug. About the LUET_PRIVILEGED_EXTRACT is something required as for now as disabling makes our test suite to fail (#260), and that needs further investigation

So this is two separate issues. Can you please create the second one, @mudler ?

[mudler]

eyes On first place, the upgrade process should use the temporary space in COS_PERSISTENT, so this is a bug. About the LUET_PRIVILEGED_EXTRACT is something required as for now as disabling makes our test suite to fail (#260), and that needs further investigation

So this is two separate issues. Can you please create the second one, @mudler ?

I was already on my way :)

[mudler]

More context about LUET_PRIVILEGED_EXTRACT is provided here: #481 and why we have that turned on (and required)