How to install CoreOS to a new gen T2 MacMini

[tseglevskiy]

Hi, guys!

It’s not an issue. It’s a story about adding new generation MacMini (I mean with T2 chip) to a k8s cluster. It requires a few workarounds. I found it and I'm happy. But frankly, I’d like to have more clean way, so you can use it as material for creating or solving of some other issues. :)

So, we have k8s cluster, installed by Terraform, Matchbox and iPXE. Some time ago iPXE stopped to work with built it MacMini’s ethernet adapters, so we bought a box of USB dongles and continue to use the same solution. But Apple made new improvement, and on new generation MacMini iPXE doesn’t work with any adapter, and internal SSD isn’t available.

I took Ubuntu installer on USB stick and booted. It demonstrated that boot from external drive is available. Sure, you need to enable it somewhere in MacMini’s settings, but it’s well known action.

Then I bought 256GB external USB-C SSD drive and installed CoreOS there. Old MacMini can boot from this drive, but new one cannot. Some problem with GRUB, it get frozen. I took GRUB from Ubuntu’s installer (do you remember, it booted?). Interesting, but Ubuntu’s GRUB is a little different: has different command arguments, and looks for config in different place (details are below). I fixed it, and CoreOS started.

Second problem with /usr partition. Originally in CoreOS it’s encrypted and mounted through mapper. Somehow using of Ubuntu’s GRUB broke this functionality: I booted it on old MacMini and got the same problem. No ideas, why? So, I recreated EXT4 filesystem on the partition 3, copied content of /usr there again, and pathed GRUB’s config to mount it without mapper.

It’s enough to add T2 MacMini to k8s cluster. Below are scripts and patches.

Patch for original grub.conf to adopt it for Ubuntu’s GRUB:

62c62
<    search --no-floppy --set oem --label OEM "$root"
---
>    search --no-floppy --set oem --part-label OEM --hint "$root"
74c74,75
< search --no-floppy --set randomize_disk_guid --fs-uuid 00000000-0000-0000-0000-000000000001
---
> search --no-floppy --set randomize_disk_guid \
>        --disk-uuid 00000000-0000-0000-0000-000000000001

Second patch to remove partition mapper:

129c129
<     set gptprio_cmdline="mount.usr=/dev/mapper/usr verity.usr=PARTUUID=$usr_uuid"
---
>     set gptprio_cmdline="mount.usr=PARTUUID=$usr_uuid"
143c143
<    linux$suf /coreos/vmlinuz-a mount.usr=/dev/mapper/usr verity.usr=PARTLABEL=USR-A $linux_cmdline
---
>    linux$suf /coreos/vmlinuz-a mount.usr=PARTLABEL=USR-A $linux_cmdline
147c147
<    linux$suf /coreos/vmlinuz-b mount.usr=/dev/mapper/usr verity.usr=PARTLABEL=USR-B $linux_cmdline
---
>    linux$suf /coreos/vmlinuz-b mount.usr=PARTLABEL=USR-B $linux_cmdline

The full grub.conf is here: https://gist.github.com/tseglevskiy/676e0ffedae141f1303f794897da40ea

And there is the script to install CoreOS to external SSD. I'm using already installed CoreOS for it (as well as Matchbox to get Ignition files and OS images).

Usage is sudo ./inst.sh aa:bb:cc:dd:ee:ff /dev/sdb

#!/usr/bin/env bash

set -x 

MAC=$1
DEVICE=$2

MATCHBOX=http://matchbox.cluster.local

if [[ -z "${MAC}" ]]; then
        echo "$0: MAC required" >&2
	exit 1
fi

# Device is required, must not be a partition, must be writable
if [[ -z "${DEVICE}" ]]; then
    echo "$0: No target block device provided, -d is required." >&2
    exit 1
fi

if ! [[ $(lsblk -n -d -o TYPE "${DEVICE}") =~ ^(disk|loop|lvm)$ ]]; then
    echo "$0: Target block device (${DEVICE}) is not a full disk." >&2
    exit 1
fi

if [[ ! -w "${DEVICE}" ]]; then
    echo "$0: Target block device (${DEVICE}) is not writable (are you root?)" >&2
    exit 1
fi

MAC=$(echo ${MAC} | sed 's/:/-/g')
IGNITION=${MAC}-ignition.json
VERSION=$(gawk --field-separator '=' '/COREOS_RELEASE_VERSION=/ { print $2 }' /usr/share/coreos/release)
CHANNEL=$(gawk --field-separator '=' '/GROUP=/ { print $2 }' /etc/coreos/update.conf)

echo MAC: ${MAC}
echo DEVICE: ${DEVICE}
echo VERSION: ${VERSION}
echo CHANNEL: ${CHANNEL}

toolbox dnf -y install jq

UUID=$(cat /proc/sys/kernel/random/uuid)

rm -f ${IGNITION}
curl "${MATCHBOX}/ignition?uuid=${UUID}&mac=${MAC}&os=installed" | \
  toolbox jq '.systemd.units += [{ "mask": true, "name": "update-engine.service" }]' | \
  tee ${IGNITION}

if [ -z ${IGNITION} ]; then
  echo ignition ${IGNITION} is empty. something goes wrong >&2
  exit 1
fi

/bin/coreos-install \
  -d ${DEVICE} \
  -C ${CHANNEL}\
  -V ${VERSION} \
  -o "" \
  -b ${MATCHBOX}/assets/coreos \
  -i ${IGNITION}

mkdir -p mmm
mount -t vfat ${DEVICE}1 mmm

rm mmm/coreos/grub/grub.cfg.tar
rm mmm/EFI/boot/*

cp efi/* mmm/EFI/boot/
cp grub.cfg mmm/boot/grub/grub.cfg

umount mmm

yes | mkfs.ext4 ${DEVICE}3
mount ${DEVICE}3 mmm

(cd /usr; tar cf - *) | tar xf - -C mmm
ls mmm

umount mmm

rmdir mmm

[bgilbert]

Thanks for your report. This issue tracker is for Fedora CoreOS, and you appear to be talking about CoreOS Container Linux. (The terminology is confusing, we know.) I'll move this issue into coreos/bugs, the tracker for Container Linux.