Skip to content

Commit

Permalink
✨ custom user bind and ephemeral mounts (#692)
Browse files Browse the repository at this point in the history
* ✨ custom user bind and ephemeral mounts

Users can now specify custom and ephemeral mounts in cloud-init under
the `install` section, e.g.:

```
users:
 - name: kairos
...
install:
  auto: true
  device: "auto"
  bind_mounts:
  - /mnt/bind1
  - /mnt/bind2
  ephemeral_mounts:
  - /mnt/ephemeral
  - /mnt/ephemeral2
...
```
Ephemeral mounts are mounted as RW - but changes are discarded when the
machine is restart.
Bind mounts will persist changes after restarted.

This is a fix for #210

Signed-off-by: Oz Tiram <oz@spectrocloud.com>

* Add the custom user mount to /cos/run/cos-layout.env

Signed-off-by: Oz Tiram <oz@spectrocloud.com>

* Add docs for custom user mounts in configuration example

Signed-off-by: Oz Tiram <oz@spectrocloud.com>

* Reuse test_install function

DRY the code, change how we call the function install_test

Signed-off-by: Oz Tiram <oz@spectrocloud.com>

* Enable custom mount tests and install tests

Signed-off-by: Oz Tiram <oz@spectrocloud.com>

* Enable tests in CI

Signed-off-by: Oz Tiram <oz@spectrocloud.com>

* Remove duplicate if check

Signed-off-by: Oz Tiram <oz@spectrocloud.com>

* Remove uneeded fmt.Println ...

Signed-off-by: Oz Tiram <oz@spectrocloud.com>

* Use separate label for custom mounts

Signed-off-by: Oz Tiram <oz@spectrocloud.com>

* 🔧 Earthfile - DRY ginkgo

We repeat this a more than twice so just extract to own
target...

Signed-off-by: Oz Tiram <oz@spectrocloud.com>

* Correct return type for  ContainElements

Signed-off-by: Oz Tiram <oz@spectrocloud.com>

* Remove CLOUD_INIT from custom mounts test

This is not needed here.

Signed-off-by: Oz Tiram <oz@spectrocloud.com>

* Fix qemu-test-* earthly targets

Signed-off-by: Oz Tiram <oz@spectrocloud.com>

* 🔧 Fix CPU passing to VM

Signed-off-by: Oz Tiram <oz@spectrocloud.com>

* 🔧 remove apt cache after install qemu

Helps deal with space running out in the CI.

Signed-off-by: Oz Tiram <oz@spectrocloud.com>

* 📖 Document custom mounts

Signed-off-by: Oz Tiram <oz@spectrocloud.com>

---------

Signed-off-by: Oz Tiram <oz@spectrocloud.com>
Co-authored-by: Ettore Di Giacinto <mudler@users.noreply.github.com>
  • Loading branch information
oz123 and mudler committed Feb 3, 2023
1 parent 4dabf47 commit f8aef9e
Show file tree
Hide file tree
Showing 11 changed files with 295 additions and 72 deletions.
18 changes: 18 additions & 0 deletions .github/workflows/image.yaml
Expand Up @@ -189,6 +189,24 @@ jobs:
./earthly.sh +datasource-iso --CLOUD_CONFIG=tests/assets/autoinstall.yaml
./earthly.sh +run-qemu-datasource-tests --FLAVOR=${{ matrix.flavor }} --SSH_PORT=${{ matrix.port }}
qemu-custom-mount-tests:
needs:
- build
runs-on: ubuntu-latest
strategy:
fail-fast: false
matrix:
include:
- flavor: "opensuse-leap"
steps:
- uses: actions/checkout@v3
- name: Download artifacts
uses: actions/download-artifact@v3
with:
name: kairos-${{ matrix.flavor }}.iso.zip
- run: |
./earthly.sh +run-qemu-custom-mount-tests --FLAVOR=${{ matrix.flavor }}
qemu-bundles-tests:
needs:
- build
Expand Down
79 changes: 50 additions & 29 deletions Earthfile
Expand Up @@ -42,14 +42,24 @@ go-deps:
SAVE ARTIFACT go.mod AS LOCAL go.mod
SAVE ARTIFACT go.sum AS LOCAL go.sum

test:

ginkgo:
FROM +go-deps
WORKDIR /build
RUN go get github.com/onsi/gomega/...
RUN go get github.com/onsi/ginkgo/v2/ginkgo/internal@v2.1.4
RUN go get github.com/onsi/ginkgo/v2/ginkgo/generators@v2.1.4
RUN go get github.com/onsi/ginkgo/v2/ginkgo/labels@v2.1.4
RUN go install -mod=mod github.com/onsi/ginkgo/v2/ginkgo

test:
FROM +ginkgo
WORKDIR /build
RUN go get github.com/onsi/gomega/...
RUN go get github.com/onsi/ginkgo/v2/ginkgo/internal@v2.1.4
RUN go get github.com/onsi/ginkgo/v2/ginkgo/generators@v2.1.4
RUN go get github.com/onsi/ginkgo/v2/ginkgo/labels@v2.1.4
RUN go install -mod=mod github.com/onsi/ginkgo/v2/ginkgo
COPY +luet/luet /usr/bin/luet
COPY . .
ENV ACK_GINKGO_DEPRECATIONS=2.5.1
Expand Down Expand Up @@ -437,13 +447,13 @@ linux-bench-scan:
###
# usage e.g. ./earthly.sh +run-qemu-datasource-tests --FLAVOR=alpine-opensuse-leap --FROM_ARTIFACTS=true
run-qemu-datasource-tests:
FROM opensuse/leap
FROM +ginkgo
RUN apt install -y qemu-system-x86 qemu-utils golang git
WORKDIR /test
RUN zypper in -y qemu-x86 qemu-arm qemu-tools go git
ARG FLAVOR
ARG TEST_SUITE=autoinstall-test
ENV FLAVOR=$FLAVOR
ENV SSH_PORT=60022
ENV SSH_PORT=60023
ENV CREATE_VM=true
ARG CLOUD_CONFIG="./tests/assets/autoinstall.yaml"
ENV USE_QEMU=true
Expand All @@ -466,19 +476,34 @@ run-qemu-datasource-tests:
ELSE
ENV DATASOURCE=/test/build/datasource.iso
END
RUN go get github.com/onsi/gomega/...
RUN go get github.com/onsi/ginkgo/v2/ginkgo/internal@v2.1.4
RUN go get github.com/onsi/ginkgo/v2/ginkgo/generators@v2.1.4
RUN go get github.com/onsi/ginkgo/v2/ginkgo/labels@v2.1.4
RUN go install -mod=mod github.com/onsi/ginkgo/v2/ginkgo

ENV CLOUD_INIT=/tests/tests/$CLOUD_CONFIG

RUN PATH=$PATH:$GOPATH/bin ginkgo --label-filter "$TEST_SUITE" --fail-fast -r ./tests/
RUN PATH=$PATH:$GOPATH/bin ginkgo -v --label-filter "$TEST_SUITE" --fail-fast -r ./tests/

run-qemu-netboot-test:
FROM ubuntu
run-qemu-custom-mount-tests:
FROM +ginkgo
RUN apt install -y qemu-system-x86 qemu-utils git && apt clean
ARG FLAVOR

COPY . .
RUN ls -liah
IF [ -e /build/kairos.iso ]
ENV ISO=/build/kairos.iso
ELSE
COPY +iso/kairos.iso kairos.iso
ENV ISO=/build/kairos.iso
END

ENV GOPATH="/go"
ARG TEST_SUITE=custom-mounts-test
ENV SSH_PORT=60024
ENV CREATE_VM=true
ENV USE_QEMU=true
RUN pwd && ls -liah
RUN PATH=$PATH:$GOPATH/bin ginkgo -v --label-filter custom-mounts-test --fail-fast -r ./tests/

run-qemu-netboot-test:
FROM +ginkgo
COPY . /test
WORKDIR /test

Expand All @@ -487,7 +512,7 @@ run-qemu-netboot-test:
ARG VERSION=$(cat VERSION)

RUN apt update
RUN apt install -y qemu qemu-utils qemu-system golang git
RUN apt install -y qemu qemu-utils qemu-system git && apt clean

# This is the IP at which qemu vm can see the host
ARG IP="10.0.2.2"
Expand All @@ -504,7 +529,7 @@ run-qemu-netboot-test:
ENV USE_QEMU=true
ARG TEST_SUITE=netboot-test
ENV GOPATH="/go"
RUN go install -mod=mod github.com/onsi/ginkgo/v2/ginkgo


# TODO: use --pull or something to cache the python image in Earthly
WITH DOCKER
Expand All @@ -514,9 +539,8 @@ run-qemu-netboot-test:
END

run-qemu-test:
FROM opensuse/leap
WORKDIR /test
RUN zypper in -y qemu-x86 qemu-arm qemu-tools go git
FROM +ginkgo
RUN apt install -y qemu-system-x86 qemu-utils git && apt clean
ARG FLAVOR
ARG TEST_SUITE=upgrade-with-cli
ARG CONTAINER_IMAGE
Expand All @@ -528,18 +552,15 @@ run-qemu-test:

ENV GOPATH="/go"


COPY . .
RUN go get github.com/onsi/gomega/...
RUN go get github.com/onsi/ginkgo/v2/ginkgo/internal@v2.1.4
RUN go get github.com/onsi/ginkgo/v2/ginkgo/generators@v2.1.4
RUN go get github.com/onsi/ginkgo/v2/ginkgo/labels@v2.1.4
RUN go install -mod=mod github.com/onsi/ginkgo/v2/ginkgo

ARG ISO=$(ls /test/build/*.iso)
ENV ISO=$ISO

RUN PATH=$PATH:$GOPATH/bin ginkgo --label-filter "$TEST_SUITE" --fail-fast -r ./tests/
IF [ -e /build/kairos.iso ]
ENV ISO=/build/kairos.iso
ELSE
COPY +iso/kairos.iso kairos.iso
ENV ISO=/build/kairos.iso
END
RUN pwd && ls -l && ls -l build
RUN PATH=$PATH:$GOPATH/bin ginkgo -v --label-filter "$TEST_SUITE" --fail-fast -r ./tests/

###
### Artifacts targets
Expand Down
42 changes: 42 additions & 0 deletions docs/content/en/docs/Advanced/customizing.md
Expand Up @@ -115,3 +115,45 @@ If you are using an Alpine-based distribution, modifying the kernel is only poss
{{% /alert %}}

After you have modified the kernel and initrd, you can use the kairos-agent upgrade command to update your nodes, or [within Kubernetes](/docs/upgrade/kubernetes).


## Customizing the file system hierarchy using custom mounts.


### Bind mounts

For clusters that needs to mount network block storage you might want to add
custom mount point that bind mounted to your system. For example, when using
Ceph file system, the OS mounts drives to `/var/lib/ceph` (for example).

To achieve this you need to add the key `bind_mounts` to the `install` section
you pass the install, and specify a list of one or more bind mounts path.

```
install:
auto: true
device: "auto"
# changes persist reboot - mount as BIND
bind_mounts:
- /var/lib/ceph
...
```


### Ephemeral mounts

One can also specifying custom mounts which are ephemeral. These are writable,
however changes are discarded at boot (like `/etc/` already does).
```
install:
auto: true
device: "auto"
# changes persist reboot - mount as BIND
bind_mounts:
- /var/lib/ceph
ephemeral_mounts:
- /opt/scratch/
...
```
Note, that these paths should exist in the container file-system used to create the ISO.
See [ISO customization](/docs/Advanced/customizing/) above.
9 changes: 8 additions & 1 deletion docs/content/en/docs/Reference/configuration.md
Expand Up @@ -50,6 +50,13 @@ install:
# Environmental variable to set to the installer calls
env:
- foo=bar
# custom user mounts
# bind mounts, can be read and modified, changes persist reboots
bind_mounts:
- /mnt/bind1
- /mnt/bind2
# ephemeral mounts, can be read and modified, changed are discarded at reboot
ephemeral_mounts:

k3s:
# Additional env/args for k3s server instances
Expand Down Expand Up @@ -1055,4 +1062,4 @@ stages:
group: 0
timeout: 0
owner_string: "root"
```
```
1 change: 1 addition & 0 deletions internal/agent/hooks/hook.go
Expand Up @@ -12,6 +12,7 @@ var AfterInstall = []Interface{
&RunStage{}, // Shells out to stages defined from the container image
&GrubOptions{}, // Set custom GRUB options
&BundleOption{},
&CustomMounts{},
&Kcrypt{},
&Lifecycle{}, // Handles poweroff/reboot by config options
}
Expand Down
59 changes: 59 additions & 0 deletions internal/agent/hooks/mounts.go
@@ -0,0 +1,59 @@
package hook

import (
"fmt"
"os"
"path/filepath"
"strings"

config "github.com/kairos-io/kairos/pkg/config"
"github.com/kairos-io/kairos/pkg/machine"
"github.com/mudler/yip/pkg/schema"
yip "github.com/mudler/yip/pkg/schema"
"gopkg.in/yaml.v1"
)

type CustomMounts struct{}

func saveCloudConfig(name config.Stage, yc yip.YipConfig) error {
yipYAML, err := yaml.Marshal(yc)
if err != nil {
return err
}
return os.WriteFile(filepath.Join("/oem", fmt.Sprintf("10_%s.yaml", name)), yipYAML, 0400)
}

// Read the keys sections ephemeral_mounts and bind mounts from install key in the cloud config.
// If not empty write an environment file to /run/cos/custom-layout.env.
// That env file is in turn read by /overlay/files/system/oem/11_persistency.yaml in fs.after stage.
func (cm CustomMounts) Run(c config.Config) error {

//fmt.Println("Custom mounts hook")
//fmt.Println(strings.Join(c.Install.BindMounts, " "))
//fmt.Println(strings.Join(c.Install.EphemeralMounts, " "))

if len(c.Install.BindMounts) == 0 && len(c.Install.EphemeralMounts) == 0 {
return nil
}

machine.Mount("COS_OEM", "/oem") //nolint:errcheck
defer func() {
machine.Umount("/oem") //nolint:errcheck
}()

var mountsList = map[string]string{}

mountsList["CUSTOM_BIND_MOUNTS"] = strings.Join(c.Install.BindMounts, " ")
mountsList["CUSTOM_EPHEMERAL_MOUNTS"] = strings.Join(c.Install.EphemeralMounts, " ")

config := yip.YipConfig{Stages: map[string][]schema.Stage{
"rootfs": []yip.Stage{{
Name: "user_custom_mounts",
EnvironmentFile: "/run/cos/custom-layout.env",
Environment: mountsList,
}},
}}

saveCloudConfig("user_custom_mounts", config) //nolint:errcheck
return nil
}
15 changes: 15 additions & 0 deletions overlay/files/system/oem/11_persistency.yaml
Expand Up @@ -36,6 +36,21 @@ stages:
/usr/share/pki/trust/anchors
/var/lib/ca-certificates
PERSISTENT_STATE_BIND: "true"
- if: |
[ -r /run/cos/custom-layout.env ]
name: "append custom bind and ephemeral mounts to /run/cos/cos-layout.env"
commands:
- |
source /run/cos/cos-layout.env
source /run/cos/custom-layout.env
PERSISTENT_STATE_PATHS="${CUSTOM_EPHEMERAL_MOUNTS} ${PERSISTENT_STATE_PATHS}"
RW_PATHS="${CUSTOM_BIND_MOUNTS} ${RW_PATHS}"
echo CUSTOM_BIND_MOUNTS=\"${CUSTOM_BIND_MOUNTS}\" >> /run/cos/cos-layout.env
echo CUSTOM_EPHEMERAL_MOUNTS=\"${CUSTOM_EPHEMERAL_MOUNTS}\" >> /run/cos/cos-layout.env
echo "# rw paths with user bind mounts" >> /run/cos/cos-layout.env
echo RW_PATHS=\"${RW_PATHS}\" >> /run/cos/cos-layout.env
echo "# persistent state paths with user ephemeral mounts" >> /run/cos/cos-layout.env
echo PERSISTENT_STATE_PATHS=\"${PERSISTENT_STATE_PATHS}\" >> /run/cos/cos-layout.env
- if: |
cat /proc/cmdline | grep -q "kairos.boot_live_mode"
name: "Layout configuration"
Expand Down
21 changes: 12 additions & 9 deletions pkg/config/config.go
Expand Up @@ -23,15 +23,18 @@ import (
const DefaultWebUIListenAddress = ":8080"

type Install struct {
Auto bool `yaml:"auto,omitempty"`
Reboot bool `yaml:"reboot,omitempty"`
Device string `yaml:"device,omitempty"`
Poweroff bool `yaml:"poweroff,omitempty"`
GrubOptions map[string]string `yaml:"grub_options,omitempty"`
Bundles Bundles `yaml:"bundles,omitempty"`
Encrypt []string `yaml:"encrypted_partitions,omitempty"`
Env []string `yaml:"env,omitempty"`
Image string `yaml:"image,omitempty"`
Auto bool `yaml:"auto,omitempty"`
Reboot bool `yaml:"reboot,omitempty"`
Device string `yaml:"device,omitempty"`
Poweroff bool `yaml:"poweroff,omitempty"`
GrubOptions map[string]string `yaml:"grub_options,omitempty"`
Bundles Bundles `yaml:"bundles,omitempty"`
Encrypt []string `yaml:"encrypted_partitions,omitempty"`
SkipEncryptCopyPlugins bool `yaml:"skip_copy_kcrypt_plugin,omitempty"`
Env []string `yaml:"env,omitempty"`
Image string `yaml:"image,omitempty"`
EphemeralMounts []string `yaml:"ephemeral_mounts,omitempty"`
BindMounts []string `yaml:"bind_mounts,omitempty"`
}

type Config struct {
Expand Down

0 comments on commit f8aef9e

Please sign in to comment.