bootstrap-containers: migrate to using configuration file #3724

jmt-lab · 2024-01-22T18:19:13Z

Issue number: #3626

Description of changes:
This migrates the bootstrap-containers tool to fetch its settings via a generated toml file.

log:

add migrations and package changes
migrate bootstrap-containers to load from config file

Testing done:

Test Report

Initial_Build_ID.Output

{
  "os": {
    "arch": "aarch64",
    "build_id": "bcfb7156",
    "pretty_name": "Bottlerocket OS 1.20.0 (aws-ecs-2)",
    "variant_id": "aws-ecs-2",
    "version_id": "1.20.0"
  }
}

Initial_Config.Output

["bear"]
source = "public.ecr.aws/h1i5r3v0/jmt-bottletest:latest"
mode = "off"
user-data = "ypXCt82h4bSlwrfKlA=="

configuration-files.bootstrap-containers-toml

{
  "configuration-files": {
    "bootstrap-containers-toml": {
      "path": "/etc/bootstrap-containers/bootstrap-containers.toml",
      "template-path": "/usr/share/templates/bootstrap-containers-toml"
    }
  }
}

services.bootstrap-containers

{
  "services": {
    "bootstrap-containers": {
      "configuration-files": [
        "host-ctr-toml",
        "bootstrap-containers-toml"
      ],
      "restart-commands": [
        "/usr/bin/bootstrap-containers create-containers"
      ]
    }
  }
}

Downgrade.Output

{
  "active_partition": {
    "image": {
      "arch": "aarch64",
      "variant": "aws-ecs-2",
      "version": "1.19.3"
    },
    "next_to_boot": true
  },
  "available_updates": [
    "1.19.3",
    "1.19.2"
  ],
  "chosen_update": {
    "arch": "aarch64",
    "variant": "aws-ecs-2",
    "version": "1.19.2"
  },
  "most_recent_command": {
    "cmd_status": "Success",
    "cmd_type": "refresh",
    "exit_status": 0,
    "stderr": "",
    "timestamp": "2024-03-18T20:24:18.772874378Z"
  },
  "staging_partition": null,
  "update_state": "Available"
}

20:24:17 [INFO] Refreshing updates...
20:24:18 [INFO] Downloading and applying update to disk...
20:24:23 [INFO] Still waiting for updated status, will wait up to 594.5s longer...
20:24:28 [INFO] Still waiting for updated status, will wait up to 589.5s longer...
20:24:33 [INFO] Still waiting for updated status, will wait up to 584.5s longer...
20:24:36 [INFO] Setting the update active so it will apply on the next reboot...
20:24:37 [INFO] Rebooting, goodbye...

Downgrade_Build_ID.Output

{
  "os": {
    "arch": "aarch64",
    "build_id": "f097c617",
    "pretty_name": "Bottlerocket OS 1.19.3 (aws-ecs-2)",
    "variant_id": "aws-ecs-2",
    "version_id": "1.19.3"
  }
}

Downgrade_Config.Output

cat: /etc/bootstrap-containers/bootstrap-containers.toml: No such file or directory

configuration-files.bootstrap-containers-toml

{}

services.bootstrap-containers

{
  "services": {
    "bootstrap-containers": {
      "configuration-files": [
        "host-ctr-toml"
      ],
      "restart-commands": [
        "/usr/bin/bootstrap-containers create-containers"
      ]
    }
  }
}

Upgrade.Output

{
  "active_partition": {
    "image": {
      "arch": "aarch64",
      "variant": "aws-ecs-2",
      "version": "1.19.3"
    },
    "next_to_boot": true
  },
  "available_updates": [
    "1.20.0",
    "1.19.3"
  ],
  "chosen_update": {
    "arch": "aarch64",
    "variant": "aws-ecs-2",
    "version": "1.20.0"
  },
  "most_recent_command": {
    "cmd_status": "Success",
    "cmd_type": "refresh",
    "exit_status": 0,
    "stderr": "",
    "timestamp": "2024-03-28T20:08:53.656541162Z"
  },
  "staging_partition": null,
  "update_state": "Available"
}

20:08:52 [INFO] Refreshing updates...
20:08:53 [INFO] Downloading and applying update to disk...
20:08:58 [INFO] Still waiting for updated status, will wait up to 594.5s longer...
20:09:03 [INFO] Still waiting for updated status, will wait up to 589.5s longer...
20:09:08 [INFO] Still waiting for updated status, will wait up to 584.5s longer...
20:09:13 [INFO] Still waiting for updated status, will wait up to 579.5s longer...
20:09:14 [INFO] Setting the update active so it will apply on the next reboot...
20:09:15 [INFO] Rebooting, goodbye...

Upgrade_Build_ID.Output

{
  "os": {
    "arch": "aarch64",
    "build_id": "bcfb7156",
    "pretty_name": "Bottlerocket OS 1.20.0 (aws-ecs-2)",
    "variant_id": "aws-ecs-2",
    "version_id": "1.20.0"
  }
}

Upgrade_Config.Output

["bear"]
source = "public.ecr.aws/h1i5r3v0/jmt-bottletest:latest"
mode = "off"
user-data = "ypXCt82h4bSlwrfKlA=="

configuration-files.bootstrap-containers-toml

{
  "configuration-files": {
    "bootstrap-containers-toml": {
      "path": "/etc/bootstrap-containers/bootstrap-containers.toml",
      "template-path": "/usr/share/templates/bootstrap-containers-toml"
    }
  }
}

services.bootstrap-containers

{
  "services": {
    "bootstrap-containers": {
      "configuration-files": [
        "host-ctr-toml",
        "bootstrap-containers-toml"
      ],
      "restart-commands": [
        "/usr/bin/bootstrap-containers create-containers"
      ]
    }
  }
}

Change_Settings.Output

apiclient set --json '{"bootstrap-containers": { "bear": { "user-data": "SGVsbG8gZnJvbSBib290c3RyYXAtY29udGFpbmVycw==" } } }'

["bear"]
source = "public.ecr.aws/h1i5r3v0/jmt-bottletest:latest"
mode = "off"
user-data = "SGVsbG8gZnJvbSBib290c3RyYXAtY29udGFpbmVycw=="

Terms of contribution:

By submitting this pull request, I agree that this contribution is dual-licensed under the terms of both the Apache License, version 2.0, and the MIT license.

sources/api/bootstrap-containers/src/main.rs

webern

Isn't there is systemd unit that needs to change in order to pass the new argument?

sources/api/bootstrap-containers/src/main.rs

packages/os/os.spec

sources/api/bootstrap-containers/src/main.rs

packages/os/bootstrap-containers-toml

sources/api/migration/migrations/v1.18.1/bootstrap-containers-config-file-v0-1-0/src/main.rs

sources/models/shared-defaults/defaults.toml

webern · 2024-02-07T21:11:16Z

Let's merge this first #3768 then rebase your PR to align on 1.19.2

packages/os/bootstrap-containers-toml

jmt-lab · 2024-02-28T22:56:59Z

rebase on latest

jmt-lab · 2024-03-08T00:42:35Z

Finished changes and testing run

webern

Verified this commit from the testing report 65bc4fc7. 👍

packages/os/bootstrap-containers-toml

sources/api/bootstrap-containers/src/main.rs

jmt-lab · 2024-03-18T20:37:06Z

Addressed comments

arnaldo2792 · 2024-03-20T23:15:44Z

packages/os/bootstrap-containers-toml

+bootstrap-containers = "v1"
+++
+{{#if settings.bootstrap-containers}}
+{{#each settings.bootstrap-containers}}


Wouldn't it be more readable (and probably more maintainable in the long run) to have this rendered by a handlebars helper? e.g.

{{bootstrap-containers settings.bootstrap-containers}}

Then you can just use Rust and strong typing to generate the toml tables. . I want us to avoid another "generate on-the-flight complex structures" and use the Rust to guarantee the correctness of the serialized format. (@bcressey what do you think?)

I like this approach, this could just be a generic settings-to-toml helper which we could use for all of these PRs where we substitute reading from the API for reading from a config file

Wouldn't it be more readable (and probably more maintainable in the long run) to have this rendered by a handlebars helper?

It doesn't really help in the long run where the goal is to decouple templates and settings extensions so they can be maintained independently.

I like this approach, this could just be a generic settings-to-toml helper which we could use for all of these PRs where we substitute reading from the API for reading from a config file

I wasn't too keen on settings-to-json when that put in an appearance recently.

I am a little more positively inclined on something like settings-to-toml if the helper is producing a 1:1 match with a subset of the external system settings API, if only because that API is covered by a strong covenant with the end user that it won't change.

However I'm still not very enthusiastic. I'd like most of these agents to be able to be configured in ways that aren't 100% reflected in the external API - like kubelet and ecs-agent are not - since some behaviors may make sense to hard-code in certain variants.

Ultimately an agent whose config is nothing but the external API is in a precarious position: it makes more sense to me to have use a combination of Bottlerocket features and systemd functionality to dispense with that agent entirely (as with ecs-settings-applier in the linked PR) where we can, than to add helpers which special-case just that agent rather than answering a more general need.

sources/api/bootstrap-containers/src/main.rs

webern

Remove bear.toml

jmt-lab · 2024-03-22T22:44:11Z

Adjusted template to use new if_not_null helper
Adjusted call to apiclient to use command() helper

Still need to remove tokio

jmt-lab · 2024-03-22T22:44:58Z

removed tokio

jmt-lab · 2024-03-26T21:42:50Z

Rebased off latest and moved migrations to 1.20.0 will rerun the tests to be safe

This migrates the bootstrap-containers tool to fetch its settings via a generated toml file. log: * add migrations and package changes * migrate bootstrap-containers to load from config file * add mount with secret_t

jmt-lab · 2024-03-27T23:41:23Z

Release.toml was wrong resulting in migrations not occuring, fixed this

bcressey

LGTM with one additional testing callout.

bcressey · 2024-04-02T02:54:46Z

sources/api/bootstrap-containers/src/main.rs

-        apiclient::set::set(socket_path, &settings)
-            .await
-            .context(error::SetSnafu)?;
+        command("apiclient", ["set", formatted.as_str()])?;


Can you verify that this actually works as expected? I.e. a bootstrap container that is set to "once" mode only runs once on first boot and not multiple times - either on the same boot (which can happen when the systemd target changes) or on subsequent boots.

I'd suggest a bootstrap container that runs something like:

#!/bin/bash TS="$(date +%s)" apiclient get settings.bootstrap-containers > /.bottlerocket/rootfs/local/${TS}.txt

And confirm that only one file is created and it shows the expected value.

Tested and it only returned one file, logs also show correct behavior

arnaldo2792 reviewed Jan 22, 2024

View reviewed changes

sources/api/bootstrap-containers/src/main.rs Show resolved Hide resolved

webern reviewed Jan 24, 2024

View reviewed changes

sources/api/bootstrap-containers/src/main.rs Outdated Show resolved Hide resolved

bcressey reviewed Jan 25, 2024

View reviewed changes

sam-berning reviewed Feb 14, 2024

View reviewed changes

packages/os/bootstrap-containers-toml Outdated Show resolved Hide resolved

jmt-lab force-pushed the jmt/bootstrap-containers/config branch 2 times, most recently from 77e9407 to 60c3928 Compare February 28, 2024 22:56

jmt-lab marked this pull request as draft February 29, 2024 21:49

jmt-lab force-pushed the jmt/bootstrap-containers/config branch from 60c3928 to 65bc4fc Compare March 6, 2024 23:32

jmt-lab marked this pull request as ready for review March 8, 2024 00:42

jmt-lab requested review from bcressey and webern March 8, 2024 18:11

webern approved these changes Mar 8, 2024

View reviewed changes

rpkelly approved these changes Mar 15, 2024

View reviewed changes

bcressey reviewed Mar 15, 2024

View reviewed changes

packages/os/bootstrap-containers-toml Outdated Show resolved Hide resolved

sources/api/bootstrap-containers/src/main.rs Show resolved Hide resolved

sources/api/bootstrap-containers/src/main.rs Outdated Show resolved Hide resolved

jmt-lab force-pushed the jmt/bootstrap-containers/config branch from 65bc4fc to e5bf382 Compare March 18, 2024 20:33

sam-berning mentioned this pull request Mar 20, 2024

schnauzer: add if_not_null template helper #3838

Merged

arnaldo2792 reviewed Mar 20, 2024

View reviewed changes

webern requested changes Mar 21, 2024

View reviewed changes

jmt-lab force-pushed the jmt/bootstrap-containers/config branch from e5bf382 to ad83e08 Compare March 22, 2024 22:40

jmt-lab force-pushed the jmt/bootstrap-containers/config branch from ad83e08 to f7f12a5 Compare March 22, 2024 22:44

jmt-lab force-pushed the jmt/bootstrap-containers/config branch 3 times, most recently from 4a467fa to b40aebb Compare March 26, 2024 21:41

bootstrap-containers: migrate to using configuration file

bcfb715

This migrates the bootstrap-containers tool to fetch its settings via a generated toml file. log: * add migrations and package changes * migrate bootstrap-containers to load from config file * add mount with secret_t

jmt-lab force-pushed the jmt/bootstrap-containers/config branch from b40aebb to bcfb715 Compare March 27, 2024 23:41

jmt-lab requested review from webern and bcressey April 1, 2024 16:54

bcressey approved these changes Apr 2, 2024

View reviewed changes

webern approved these changes Apr 2, 2024

View reviewed changes

jmt-lab merged commit 9d208db into bottlerocket-os:develop Apr 3, 2024
50 checks passed

jmt-lab deleted the jmt/bootstrap-containers/config branch April 3, 2024 19:31

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

bootstrap-containers: migrate to using configuration file #3724

bootstrap-containers: migrate to using configuration file #3724

jmt-lab commented Jan 22, 2024 •

edited

webern left a comment

webern commented Feb 7, 2024

jmt-lab commented Feb 28, 2024

jmt-lab commented Mar 8, 2024

webern left a comment

jmt-lab commented Mar 18, 2024

arnaldo2792 Mar 20, 2024

sam-berning Mar 20, 2024

bcressey Apr 2, 2024

webern left a comment

jmt-lab commented Mar 22, 2024

jmt-lab commented Mar 22, 2024

jmt-lab commented Mar 26, 2024

jmt-lab commented Mar 27, 2024

bcressey left a comment

bcressey Apr 2, 2024

jmt-lab Apr 3, 2024

bootstrap-containers: migrate to using configuration file #3724

bootstrap-containers: migrate to using configuration file #3724

Conversation

jmt-lab commented Jan 22, 2024 • edited

webern left a comment

Choose a reason for hiding this comment

webern commented Feb 7, 2024

jmt-lab commented Feb 28, 2024

jmt-lab commented Mar 8, 2024

webern left a comment

Choose a reason for hiding this comment

jmt-lab commented Mar 18, 2024

arnaldo2792 Mar 20, 2024

Choose a reason for hiding this comment

sam-berning Mar 20, 2024

Choose a reason for hiding this comment

bcressey Apr 2, 2024

Choose a reason for hiding this comment

webern left a comment

Choose a reason for hiding this comment

jmt-lab commented Mar 22, 2024

jmt-lab commented Mar 22, 2024

jmt-lab commented Mar 26, 2024

jmt-lab commented Mar 27, 2024

bcressey left a comment

Choose a reason for hiding this comment

bcressey Apr 2, 2024

Choose a reason for hiding this comment

jmt-lab Apr 3, 2024

Choose a reason for hiding this comment

jmt-lab commented Jan 22, 2024 •

edited