Fedora v2.4.0 out of space during install

[Itxaka]

Kairos version:

NAME="Fedora Linux"
VERSION="38 (Container Image)"
ID=fedora
VERSION_ID=38
VERSION_CODENAME=""
PLATFORM_ID="platform:f38"
PRETTY_NAME="Fedora Linux 38 (Container Image)"
ANSI_COLOR="0;38;2;60;110;180"
LOGO=fedora-logo-icon
CPE_NAME="cpe:/o:fedoraproject:fedora:38"
DEFAULT_HOSTNAME="fedora"
HOME_URL="https://fedoraproject.org/"
DOCUMENTATION_URL="https://docs.fedoraproject.org/en-US/fedora/f38/system-administrators-guide/"
SUPPORT_URL="https://ask.fedoraproject.org/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"
REDHAT_BUGZILLA_PRODUCT="Fedora"
REDHAT_BUGZILLA_PRODUCT_VERSION=38
REDHAT_SUPPORT_PRODUCT="Fedora"
REDHAT_SUPPORT_PRODUCT_VERSION=38
SUPPORT_END=2024-05-14
VARIANT="Container Image"
VARIANT_ID=container
KAIROS_NAME="kairos-kairos-fedora"
KAIROS_VERSION="v2.4.0-k3sv1.25.11+k3s1"
KAIROS_ID="kairos"
KAIROS_ID_LIKE="kairos-kairos-fedora"
KAIROS_VERSION_ID="v2.4.0-k3sv1.25.11+k3s1"
KAIROS_PRETTY_NAME="kairos-kairos-fedora v2.4.0-k3sv1.25.11+k3s1"
KAIROS_BUG_REPORT_URL="https://github.com/kairos-io/kairos/issues"
KAIROS_HOME_URL="https://github.com/kairos-io/kairos"
KAIROS_IMAGE_REPO="quay.io/kairos/kairos-fedora"
KAIROS_IMAGE_LABEL=""
KAIROS_GITHUB_REPO="kairos-io/provider-kairos"
KAIROS_VARIANT="kairos"
KAIROS_FLAVOR="fedora"

CPU architecture, OS, and Version:

Describe the bug
Installing Fedora fails during rsync
rsync: [receiver] write failed on "/run/cos/active/usr/src/zfs-2.1.12/tests/zfs-tests/tests/functional/redacted_send/redacted_volume.ksh": No space left on device (28)

To Reproduce
Get any kairos-standard-fedora-v2.4.0 iso, install it.

Expected behavior
It installs

Logs

Additional context
Images in the spec look good:

  Active: v1.Image{
    File: "/run/cos/state/cOS/active.img",
    Label: "COS_ACTIVE",
    Size: 1747,
    FS: "ext2",
    Source: &v1.ImageSource{},
    MountPoint: "/run/cos/active",
    LoopDevice: "",
  },
  Recovery: v1.Image{
    File: "/run/cos/recovery/cOS/recovery.img",
    Label: "COS_SYSTEM",
    Size: 1747,
    FS: "ext2",
    Source: &v1.ImageSource{},
    MountPoint: "",
    LoopDevice: "",
  },
  Passive: v1.Image{
    File: "/run/cos/state/cOS/passive.img",
    Label: "COS_PASSIVE",
    Size: 1747,
    FS: "ext2",
    Source: &v1.ImageSource{},
    MountPoint: "",
    LoopDevice: "",
  },

Partitions also look good with proper size:

  Partitions: v1.ElementalPartitions{
    BIOS: &v1.Partition{
      Name: "bios",
      FilesystemLabel: "",
      Size: 1,
      FS: "",
      Flags: []string{
        "bios_grub",
      },
      MountPoint: "",
      Path: "",
      Disk: "",
    },
    EFI: nil,
    OEM: &v1.Partition{
      Name: "oem",
      FilesystemLabel: "COS_OEM",
      Size: 64,
      FS: "ext4",
      Flags: []string{}, // p0
      MountPoint: "/run/cos/oem",
      Path: "",
      Disk: "",
    },
    Recovery: &v1.Partition{
      Name: "recovery",
      FilesystemLabel: "COS_RECOVERY",
      Size: 3694,
      FS: "ext4",
      Flags: p0,
      MountPoint: "/run/cos/recovery",
      Path: "",
      Disk: "",
    },
    State: &v1.Partition{
      Name: "state",
      FilesystemLabel: "COS_STATE",
      Size: 6241,
      FS: "ext4",
      Flags: p0,
      MountPoint: "/run/cos/state",
      Path: "",
      Disk: "",
    },
    Persistent: &v1.Partition{
      Name: "persistent",
      FilesystemLabel: "COS_PERSISTENT",
      Size: 0,
      FS: "ext4",
      Flags: p0,
      MountPoint: "/run/cos/persistent",
      Path: "",
      Disk: "",
    },
  },

And they were created with proper sizes:

vda       20G 
├─vda1     1M 
├─vda2    64M COS_OEM
├─vda3   3.6G COS_RECOVERY
├─vda4   6.1G COS_STATE
└─vda5  10.2G COS_PERSISTENT

Aos, not sure why we are bundling /usr/src in the images??? Useless space that we should drop.

[Itxaka]

Image was properly constructed with the proper size:

[root@fedora cOS]# ls -ltra
total 1778304
drwxr-xr-x 4 root root       4096 Sep 21 13:15 ..
drwxr-xr-x 2 root root       4096 Sep 21 13:15 .
-rw-r--r-- 1 root root 1831862272 Sep 21 13:16 active.img

Maybe the source is bigger that what we checked for some reason? But df doesnt seem to think that:

root@fedora ~]# du -chs /run/rootfsbase/
1.6G	/run/rootfsbase/
1.6G	total

[Itxaka]

There is definitely enough space in the partition to hold active.img+passive.img so that calculation is okay.

[Itxaka]

umm, image is really full to the brim:

[root@fedora kaka2]# df -h .
Filesystem      Size  Used Avail Use% Mounted on
/dev/loop1      1.7G  1.7G     0 100% /tmp/kaka2

[Itxaka]

Im wondering if its due to symlinks or something because the sizes doesnt match here somehow.

/run/rootfsbase/ -> 1.6G
active.img -> 1.7Gb

Should be enough...

[Itxaka]

Im wondering if somewhere there is a wrong symlink that makes it go out of the root dir that we need to copy....gonna check rsync commands as there is a way of avoiding this IIRC.

[Itxaka]

Workaround for now, add this to your cloud-config to override the automatic sizes:

install:
  system:
    size: 2500
  passive:
    size: 2500
  recovery-system:
    size: 2500
  partitions:
    state:
      size: 9000
    recovery:
      size: 6000
upgrade:
  system:
    size: 2500
  passive:
    size: 2500
  recovery-system:
    size: 2500
  partitions:
    state:
      size: 9000
    recovery:
      size: 6000
reset:
  system:
    size: 2500
  passive:
    size: 2500
  recovery-system:
    size: 2500
  partitions:
    state:
      size: 9000
    recovery:
      size: 6000

[Itxaka]

With a size of 1780Mb It works.....
Auto calculation is 1747Mb ....

So somehow Im missing 23Mb somewhere. Maybe the rounding up has to be expanded to add half a gig or something.

[mudler]

With a size of 1780Mb It works..... Auto calculation is 1747Mb ....

So somehow Im missing 23Mb somewhere. Maybe the rounding up has to be expanded to add half a gig or something.

having a rounding up sounds good, at least to be in the safe side as well 👍

[mordax7]

@Itxaka Having the same issues with the OpenSUSE Alpine image.

Did you have to build it again with some extra parameters or did you overwrite the partitions with the above mentioned cloud-conig?

[Itxaka]

@Itxaka Having the same issues with the OpenSUSE Alpine image.

Did you have to build it again with some extra parameters or did you overwrite the partitions with the above mentioned cloud-conig?

Cloud config workaround until we do a new patch release with this fixed

[mauromorales]

@Itxaka how do you get the value of the rootfsbase? I'm getting something much smaller:

[root@fedora rootfsbase]# df -kh .
Filesystem      Size  Used Avail Use% Mounted on
/dev/loop0      658M  658M     0 100% /run/rootfsbase

[mauromorales]

Implemented a solution in kairos-io/kairos-agent#153 which also counts the symlinks but that increases the size of the image by more than the 20 something Megs that were mentioned before:

Creating file system image /run/cos/state/cOS/active.img with size 2530Mb 

[mudler]

Implemented a solution in kairos-io/kairos-agent#153 which also counts the symlinks but that increases the size of the image by more than the 20 something Megs that were mentioned before:

Creating file system image /run/cos/state/cOS/active.img with size 2530Mb 

good catch !