Help updating /etc/hosts and installing drivers

[mightymouse2045]

Hi,

I have updated my seedimage to add entries in /etc/hosts, but this doesn't carry across to the OS post install. I've tried updating it in the after-install-chroot stage which is also where I am trying to install RPM's https://elemental.docs.rancher.com/customizing#adding-extra-driver-binaries-into-the-iso-example

But the commands I am running don't appear to do anything - after reboot there are no host entries and the rpm packages I have attempted to install are not there?

Not sure what I'm doing wrong? Can anyone help out with how they've done it?

Also is there a way to get back into chroot manually - I have disabled the auto reboot/shutdown after the OS install, so wanted to manually go back into the chroot environment and see what has or hasn't been applied and troubleshoot it further that way.

[kkaempf]

I have updated my seedimage to add entries in /etc/hosts, but this doesn't carry across to the OS post install.

Please share more details on this. How do you "update your seedimage" ?

where I am trying to install RPMs

You shouldn't do this post-install but rather during image build.

[mightymouse2045]

I just delete the existing seedimage and create another with kubectl apply -f seedimage.yaml against the rancher cluster:

apiVersion: elemental.cattle.io/v1beta1
kind: SeedImage
metadata:
generation: 3
name: pch-seedimage
namespace: fleet-default
ownerReferences:
- apiVersion: elemental.cattle.io/v1beta1
kind: MachineRegistration
name: vg-registration
uid: 399bc63c-21c1-42fa-862d-f9e234f2eeda
spec:
baseImage: registry.suse.com/suse/sle-micro-iso/5.5:2.0.2
cleanupAfterMinutes: 60
cloud-config:
runcmd:
- echo "1.1.1.1 some.host.com" >> /etc/hosts
registrationRef:
name: vg-registration
namespace: fleet-default

This then generates an updated image which I download and install with that.

Is there another way to build the image and spit out an ISO that I can include installation of extra rpm's as well? I had seen mention of it with elemental toolkit but I have mainly been following guides such as:
https://elemental.docs.rancher.com/customizing#adding-extra-driver-binaries-into-the-iso-example

But if there is a better way can you point me to that?

[mightymouse2045]

Ah ok - I get it the docker part didn't sink in when I first read through. So I should create a custom docker build and then use that as the base for the iso image I'm building. Yeah let me try that out.

[mightymouse2045]

ok I've progressed to creating the ISO image and it is failing with:

Failed fetching EFI data: stat /tmp/elemental-iso680850754/rootfs/usr/share/grub2/x86_64-efi/grub.efi: no such file or directory

This happens with the instructions in here and with my custom docker image https://elemental.docs.rancher.com/customizing

[frelon]

Hi @mightymouse2045!

Could you share your Dockerfile and the commands used to build the iso?

[mightymouse2045]

I have managed to copy the grub files into there and now getting booted and successfully registered into Rancher, however the changes I made to my custom docker image for /etc/hosts haven't come over and I can't work out how to write to fstab in the ISO I have install-hook.yaml with:
name: "Create mount point and update fstab"
stages:
before-install:
- commands:
- mkdir /run/cos/persistent/eVol1
after-install-chroot:
- commands:
- echo "LABEL=eVol1 /usr/local/eVol1 xfs defaults 0 0" >> /etc/fstab
-
This is not working to mount my extra disk once I reboot there is only the default items in fstab.

I've also tried with the I yaml file under oem:
name: "Mount LVM volumes"
stages:
initramfs:
- name: "Extend fstab to mount LVM logical volumes at boot"
commands:
- |
echo "LABEL=eVol1 /usr/local/eVol1 xfs defaults 0 0" >> /etc/fstab

But neither are working

[frelon]

Files directly under /etc are going to behave strangely since they are on an ephemeral filesystem. To add lines to /etc/hosts we are going to want a cloud-init file in /oem that writes to it on every boot.

To place that file in /oem during post-install we need another file in the ISO.

This should be possible using the following seedimage:

apiVersion: elemental.cattle.io/v1beta1
kind: SeedImage
metadata:
  name: fire-img
  namespace: fleet-default
spec:
  cleanupAfterMinutes: 0
  baseImage: registry.suse.com/suse/sle-micro-iso/5.5:2.0.2
  registrationRef:
    apiVersion: elemental.cattle.io/v1beta1
    kind: MachineRegistration
    name: my-nodes
    namespace: fleet-default
  cloud-config:
    users:
      - name: root
        passwd: linux
    runcmd:
      - echo "1.1.1.1 some.host.com" >> /etc/hosts
    write_files:
      - permissions: "0755"
        path: /usr/local/cloud-config/hosts.yaml
        content: |
          name: "Add hosts configuration"
          stages:
            post-install:
              - files:
                - path: /run/cos/oem/90_update-hosts.yaml
                  permissions: 0777
                  content: | 
                    name: "Add hosts configuration"
                    stages:
                      initramfs:
                        - commands:
                          - echo "1.1.1.1 some.host.com" >> /etc/hosts

It's not a pretty solution, but it works... We are currently looking into how we can make this process smoother!

[mightymouse2045]

Docker file:

FROM registry.suse.com/suse/sle-micro/5.5:2.0.2
COPY repos/* /etc/zypp/repos.d/
RUN zypper --no-gpg-checks --gpg-auto-import-keys install -y cifs-utils bind-utils
ARG IMAGE_REPO='merepo/elemental'
ARG IMAGE_TAG='v1.1.1'
RUN
sed -i -e "s/^IMAGE_REPO=./IMAGE_REPO="${IMAGE_REPO}"/g" /etc/os-release &&
sed -i -e "s/^IMAGE_TAG=./IMAGE_TAG="${IMAGE_TAG}"/g" /etc/os-release &&
sed -i -e "s/^IMAGE=.*/IMAGE="${IMAGE_REPO}:${IMAGE_TAG}"/g" /etc/os-release &&
echo '1.1.1.1 somehost.com' >> /etc/hosts

I am then user this Dockerfile building the ISO:

FROM merepo/elemental:v1.1.1 AS os
FROM merepo/elemental:v1.1.1 AS builder

ARG TARGETARCH
WORKDIR /iso
COPY --from=os / rootfs
COPY EFI/BOOT rootfs/usr/share/grub2/x86_64-efi

RUN --mount=type=bind,source=./,target=/output,rw
elemental build-iso
dir:rootfs
--bootloader-in-rootfs
--squash-no-compression
-o /output -n "elemental-${TARGETARCH}"

Building with:
podman build --tag merepo/test:v1.1.1 .

I then copy all my overlay files into the iso image output from the build and then build the VM with that ISO.

[mightymouse2045]

Thanks for the other workaround just push that and testing it shortly
They are self signed certs on the rancher management cluster
Version 2.8.3 Provider: RKE2
Kubernetes Version: v1.28.8 +rke2r1

On a previous attempt I looked in /var/lib/rancher/rke2/server/tls and saw some of them were there, I tried copying the kube.scheduler crt and key (which for some reason had copied across) to the kube-controller-manager.crt and .key (missing) and that got it progressed but caused me issues later...

But this time round they are all complaining that they are missing.

[mightymouse2045]

Just finally able to try now with the new iso build, so it is fully built, hosts and fstab entries working nicely, deleted cluster and the old machine entry and started a new cluster.

Get the same errors as above and looking at /var/lib/rancher/rke2/server there is no tls directory at all in there. I'm happy to add the files manually if I know where to copy them from (would copying them from an existing non teal cluster work?). Have you managed to reproduce this on your side?

rancher-8767:~ # ls -al /var/lib/rancher/rke2/server/
total 12
drwxr-xr-x 3 root root 4096 May 4 17:59 .
drwxr-xr-x 3 root root 4096 May 4 17:59 ..
drwxr-xr-x 3 root root 4096 May 4 17:59 manifests

[mightymouse2045]

ok finally worked it out - /etc/resolv.conf is empty, so it defaults to 127.0.0.1:53 for resolution which is failing. My hosts entries were getting me part way there, but obviously it couldn't resolve everything it needed.

I also worked out how to prepopulate the hosts file without the seedimage, in the custom iso build you just have to sed or echo the contents into rootfs/etc/hosts and that appears permanently.

Added the relevant entries to resolv.conf and that got the cluster responding however it is now stuck at :
Waiting for probes: calico, kube-controller-manager

Any help working this out, if I list pods I get these running:

NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system cloud-controller-manager-rancher-12079 1/1 Running 0 12m
kube-system etcd-rancher-12079 1/1 Running 0 11m
kube-system kube-apiserver-rancher-12079 1/1 Running 0 11m
kube-system kube-scheduler-rancher-12079 1/1 Running 0 11m

If I look at the status of rancher-system-agent I am now getting this:

level=error msg="error loading CA cert for probe (kube-controller-manager) /var/lib/rancher/rke2/server/tls/kube-controller-manager/kube-controller-manager.crt: open /var/lib/rancher/rke2/server/tls/kube-controller-manager/kube-controller-manager.crt: no such file or directory"
level=error msg="error while appending ca cert to pool for probe kube-controller-manager"

ls -al /var/lib/rancher/rke2/server/tls

drwxr-xr-x 2 root root 4096 May 4 23:50 kube-controller-manager
drwxr-xr-x 2 root root 4096 May 4 23:50 kube-scheduler

I can see crt and key in kube-scheduler, but nothing in kube-controller-manager

How can I sort this one out?

[mightymouse2045]

Nevermind - worked this one out as well - it is resource starvation HERE

Upped the memory on the VM from 2GB to 8GB next issue space - but yeah sorted it

Thanks for the help guys

[frelon]

Glad to hear you solved it! And nice work documenting your findings! 👍