kata-manager: Allow hypervisor to be changed #9059

jodh-intel · 2024-02-08T16:21:18Z

kata-manager: Allow hypervisor to be changed

Add new options to allow the configured hypervisor to be changed:

-L: List available packaged hypervisor config names (filename fragments).
-e: List available local hypervisor config names (filename fragments).
-H <hypervisor>: Install Kata then switch to the specified hypervisor (by filename fragment).
-S <hypervisor>: Switch to the specified hypervisor (by filename fragment [Errors if Kata not installed]).

For example, to install Kata and configure it to use Cloud Hypervisor
with the golang Kata runtime:

$ kata-manager.sh -H clh

To switch back to the default hypervisor:

$ kata-manager.sh -S default

To show details of the available packaged configs:

$ kata-manager.sh -L

To show details of the local configs:

$ kata-manager.sh -e

Notes:
This change only applies to the current default (golang) Kata runtime.
Although this is mainly for users wishing to switch hypervisor (by
changing the Kata config file to another of the packaged config files
provided for specific hypervisors), strictly it allows users to change
to any config file. For example, if the user has a config file called
/etc/kata-containers/configuration-my-custom-config.toml, they could
switch to this by running:
$ kata-manager.sh -S my-custom-config

Fixes: #8305.

Signed-off-by: James O. D. Hunt james.o.hunt@intel.com

jodh-intel · 2024-02-08T16:27:30Z

Note that running with -L won't work unless you are using a very new installation of Kata (newer than the official 3.3.0-alpha0 release: https://github.com/kata-containers/kata-containers/releases. The reason being that #8483 isn't yet in any official releases.

/cc @fidencio, @amshinde, @cmaf, @dborquez.

jodh-intel · 2024-02-09T11:20:27Z

Two thoughts:

I don't particularly like the option "names": -L and -e are related but you'd never know from the option letters. Maybe we should switch to using long options with getopt(1) for greater clarity?
We could drop one of -H or -S and just have a single option that will dtrt. Thoughts?

jodh-intel · 2024-02-09T11:21:46Z

I've updated the branch today to add in -e (show local config details). But the change also removes the hard-coded default hypervisor from the script which now derives it from the packaged config file directory.

jodh-intel · 2024-02-09T16:32:39Z

As demonstrated by the doc changes on this PR, using the kata-manager script from GitHub is unwieldy. I've raised #9066 so we can consider adding it to the release.

cmaf

@jodh-intel this LGTM although I have not tested it locally

jodh-intel · 2024-02-14T08:54:08Z

Thanks, @cmaf!

@jodh-intel this LGTM although I have not tested it locally

What you can do is run the code using curl. For example, to see the help statement:

$ bash -c "$(curl -fsSL https://raw.githubusercontent.com/kata-containers/kata-containers/3dcd5341637fd60d7b9f86602acb0312dcf005d0/utils/kata-manager.sh) -h"

Note: The commit in the curl URL is the latest commit currently on this PR (3dcd534).

See https://github.com/kata-containers/kata-containers/blob/main/utils/README.md for other options not added on this PR.

jodh-intel · 2024-02-15T08:06:55Z

/test

fidencio

@jodh-intel, I have a few suggestions, none of those are code related so far.
Please, let me know your thoights.

fidencio · 2024-02-15T20:49:46Z

utils/README.md

+
+### Choose a Hypervisor
+
+Kata works with different [hypervisors](../docs/hypervisors.md). When you install a Kata system, the default hypervisor


I understand the motivation on linking to the hypervisors page, but the user will not know which name to use to select, for instance, "Cloud Hypervisor".

We need a way to have the list of supported hypervisors being displayed without installing kata.

It's not explained in the commits, but I tried to minimise what we hard-code in the script as it's better to rely on the structure of the packaged config files to determine the logic. Further, it will simplify our lives when we tackle #8677 and #9060.

I've now updated the branch so that the hypervisors.md page shows the short names, which allows the users to determine the names used by the script.

fidencio · 2024-02-15T20:54:17Z

utils/README.md

+$ bash -c "$(curl -fsSL https://raw.githubusercontent.com/kata-containers/kata-containers/main/utils/kata-manager.sh) -H clh"
+```
+
+> **Note:** See the [List available hypervisors](#list-available-hypervisors) section


This note is not that much useful as we don't have a way to actually list the hypervisors before we install Kata Containers. :-/

The doc now lists the "short names".

However, I am wondering about removing the -H option since, effectively, that option just installs Kata, then calls -S. There isn't much benefit to providing -H since we still need to install Kata. Then the -S just does the config switch. wdyt to removing -H?

fidencio · 2024-02-15T20:56:01Z

utils/README.md

+#### List available hypervisors
+
+Run the following on an installed system:
+
+```bash
+$ bash -c "$(curl -fsSL https://raw.githubusercontent.com/kata-containers/kata-containers/main/utils/kata-manager.sh) -L"
+```
+
+#### Show the default packaged hypervisor
+
+To show the default packaged hypervisor, run the following on an
+installed system:
+
+```bash
+$ bash -c "$(curl -fsSL https://raw.githubusercontent.com/kata-containers/kata-containers/main/utils/kata-manager.sh) -L" | grep default
+```
+
+#### Show the locally configured hypervisor
+
+`kata-manager.sh` will create a "local" copy of the packaged Kata configuration
+file.
+
+To show details of the _local_ copy of the configuration files on an
+installed system, run:
+
+```bash
+$ bash -c "$(curl -fsSL https://raw.githubusercontent.com/kata-containers/kata-containers/main/utils/kata-manager.sh) -e"
+```
+
+> **Note:** This command can only be run once Kata has been installed.
+
+> See the [configuration documentation](https://github.com/kata-containers/kata-containers#configuration)
+> for further information.


I wonder if those 3 options could be combined into a single one which would output something like:

HYPERVISOR | ARGUMENT NAME | DEFAULT Cloud Hypervisor | clh | yes QEMU | qemu | no ,,,,

Or something towards that line ...
NOTE: After the next release e should remove all the mentions to $ bash -c ... and just point to the script directly.

I wonder if those 3 options could be combined into a single one

I'm not totally clear what you mean by this comment but I have updated the hypervisors doc to include more hypervisor config details including the "short name" used by kata-manager.

NOTE: After the next release e should remove all the mentions to $ bash -c ... and just point to the script directly.

Yep - see #9091.

fidencio · 2024-02-15T21:14:38Z

A few more notes here:

ubuntu@kata-manager:~/kata-containers/utils$ ./kata-manager.sh -L
acrn;;packaged;golang
clh;;packaged;golang
dragonball;;packaged;golang
fc;;packaged;golang
qemu-nvidia-gpu;;packaged;golang
qemu-sev;;packaged;golang
qemu-snp;;packaged;golang
qemu-tdx;;packaged;golang
qemu;default;packaged;golang

This output is not user friendly, and dragonball is not golang. :-)
It's hard for people to know what each column means.

Also, what's the value of having packaged there?

fidencio · 2024-02-15T21:17:05Z

I noticed you'be been changing the default configuration on /etc/kata-containers, wouldn't be easier to just set the default in /opt/kata/...? That's what we already do with kata-deploy.

jodh-intel · 2024-02-16T16:34:05Z

I noticed you'be been changing the default configuration on /etc/kata-containers, wouldn't be easier to just set the default in /opt/kata/...? That's what we already do with kata-deploy.

It is admittedly more "fiddly" to deal with, but kata-manager isn't just for kata-deploy environments, so it's safer and cleaner to modify the local config files in /etc/kata-containers/ as that also works for stateless environments, read-only disk environments, etc. So, once kata-manager installs Kata, it never touches the pristine files below /opt/kata/. Plus, I don't think we want to drop the concept that if you futz up your local config somehow, you can either move / rename your local config files, or even sudo rm -rf /etc/kata-containers/, and the system will reset to "factory defaults" 😄

jodh-intel · 2024-02-20T07:52:00Z

A few more notes here:

ubuntu@kata-manager:~/kata-containers/utils$ ./kata-manager.sh -L
acrn;;packaged;golang
clh;;packaged;golang
dragonball;;packaged;golang
fc;;packaged;golang
qemu-nvidia-gpu;;packaged;golang
qemu-sev;;packaged;golang
qemu-snp;;packaged;golang
qemu-tdx;;packaged;golang
qemu;default;packaged;golang

This output is not user friendly

It's designed to be parseable, but we could add in a tab separated format too.

, and dragonball is not golang. :-)

Correct. See my note at #9059 (comment). That issue is caused by the fact that the Kata project hasn't produced a new release in four months. Related: #9064.

It's hard for people to know what each column means.

I can update the usage.

Also, what's the value of having packaged there?

I could drop that. It's there to show these are packaged config short names. We're not showing the full paths (since they may differ, are quite long and would make the output even worse unless you're happy with line wrapping or happen to have a wide terminal), so need an indicator as to whether these are the pristine packaged config files, or if they are the local config file copies.

amshinde · 2024-02-20T08:09:03Z

utils/kata-manager.sh

@@ -737,6 +737,7 @@ configure_kata()
 	sudo install -o root -g root -m 0644 "$cfg_from" "$cfg_to"

 	sudo sed -i \
+		--follow-symlinks \


Great catch @jodh-intel !
I wasnt aware myself about this behaviour myself. Reading about this, it shows that GNU sed has --follow-symlinks, while BSD sed does not. It will be useful to atleast document this here, in case someone tries to use this/port this to MacOS. We afterall do support macos to some extent.

We could mention something like "If you're on macOS or another BSD variant, you'll need to install GNU sed via your package manager (brew, pkg, etc)"

amshinde · 2024-02-20T08:19:15Z

utils/kata-manager.sh

+
+  > **Note:** For details of each hypervisor, see [3].
+
+- If an invalid hypervisor name is specified with the '-H' option, an


Personally I do not like the concept of designating any partcular hypervisor as default. I feel all the hypervisors should be treated the same with the user choosing one based on his/her needs.
(While I do understand that configuration.toml points to qemu conf by default, we should limit the usage of default and not designate any one hypervisor as default)

If there is a failure on configuration here, could we just go back to previous configuration instead of default?

Hi, @amshinde - I don't disagree with what you've said, but we need some way to designate "the currently configured hypervisor". How about just configured?

@jodh-intel Yes that wording works for me.

@amshinde - Switching everything to configured unfortunately results in a less clear approach. For example, how does the user "undo all changes" and revert to the pristine default configured hypervisor?

The current approach is:

$ kata-manager.sh -S default

But that just looks odd if we change the name to configured as it just doesn't express the same intent:

$ kata-manager.sh -S configured

That's confusing and ambiguous imho as it doesn't suggest the value specified to the -S option means "revert to the original configured hypervisor".

The best I can come up with as a compromise is:

$ kata-manager.sh -S pristine

wdyt?

/cc @fidencio

amshinde · 2024-02-20T08:21:00Z

utils/kata-manager.sh

@@ -23,6 +23,8 @@ readonly containerd_slug="containerd/containerd"
 readonly kata_project_url="https://github.com/${kata_slug}"


Adding my comment here, but this is for the commiot message. Just reading the commit message, the usage of filename fragments is not really clear to me. Could we reword it?

jodh-intel · 2024-02-29T16:20:26Z

I've also changed the format so that it now uses tabs rather than semi-colons so that it's still parseable but looks a bit less "busy" 😄

jodh-intel · 2024-02-29T16:45:24Z

/test

jodh-intel · 2024-03-01T16:07:59Z

/test

Remove extraneous whitespace. Signed-off-by: James O. D. Hunt <james.o.hunt@intel.com>

Ensure the usage statement lists all options in alphabetical order. Signed-off-by: James O. D. Hunt <james.o.hunt@intel.com>

Add an info message just before the archive file is checked. This keeps the user informed about what is happening as it can take a few seconds to perform the checks on slower systems. Signed-off-by: James O. D. Hunt <james.o.hunt@intel.com>

The `configure_kata()` function modifies the configuration file to enable debug. But it was doing this by calling `sed -i` which, by default, creates a new _file_ from the `configuration.toml` symbolic link. This defeated the point of the symbolic link which is supposed to resolve to the local copy of the pristine config file, so we now use the GNU sed(1) specific `---follow-symlinks` option to retain the sym-link. Signed-off-by: James O. D. Hunt <james.o.hunt@intel.com>

Add new options to allow the configured hypervisor to be changed: - `-L`: List available _packaged_ hypervisor config short names. - `-e`: List available _local_ hypervisor config names. - `-H <hypervisor>`: Install Kata then switch to the specified hypervisor. - `-S <hypervisor>`: Switch to the specified hypervisor (by config short name [Errors if Kata not installed]). For example, to install Kata and configure it to use Cloud Hypervisor with the golang Kata runtime: ```bash $ kata-manager.sh -H clh ``` To switch back to the default hypervisor: ```bash $ kata-manager.sh -S default ``` To show details of the available packaged configs: ```bash $ kata-manager.sh -L ``` To show details of the local configs: ```bash $ kata-manager.sh -e ``` > **Notes:** > > - This change **only** applies to the current default (golang) Kata runtime. > > - Although this is mainly for users wishing to switch hypervisor (by > changing the Kata config file to another of the packaged config files > provided for specific hypervisors), strictly it allows users to change > to _any_ config file. For example, if the user has a config file called > `/etc/kata-containers/configuration-my-custom-config.toml`, they could > switch to this by running: > > ```bash > $ kata-manager.sh -S my-custom-config > ``` > > - The "config short names" are the hypervisor specific part of the configuration file name. > For example, the config short name for file `configuration-qemu.toml` is > `qemu` and the config short name for `configuration-clh.toml` is `clh`. Fixes: kata-containers#8305. Signed-off-by: James O. D. Hunt <james.o.hunt@intel.com>

Remove extraneous whitespace from hypervisors doc. Signed-off-by: James O. D. Hunt <james.o.hunt@intel.com>

Add details to the README for `kata-manager` showing how to list available hypervisor configs (packaged and local), and switch between the configurations. Also, update the hypervisors page to show a lot more detail about the hypervisor configurations, including the "short name" used by `kata-manager` for switching hypervisor config. > **Note:** > > These changes only apply to the current default golang runtime. Signed-off-by: James O. D. Hunt <james.o.hunt@intel.com>

jodh-intel · 2024-03-04T12:25:13Z

/test

jodh-intel · 2024-03-04T15:58:16Z

The ARM CI (kata-containers-2.0-ubuntu20.04-ARM-main) is failing:

4:21:11 Reading package lists...
14:21:11 E: The repository 'http://packages.cloud.google.com/apt kubernetes-xenial-unstable Release' no longer has a Release file.
14:21:11 W: Target Packages (stable/binary-arm64/Packages) is configured multiple times in /etc/apt/sources.list:5 and /etc/apt/sources.list.d/docker.list:1
14:21:11 W: Target Packages (stable/binary-all/Packages) is configured multiple times in /etc/apt/sources.list:5 and /etc/apt/sources.list.d/docker.list:1
14:21:11 W: Target Packages (stable/binary-arm64/Packages) is configured multiple times in /etc/apt/sources.list:5 and /etc/apt/sources.list.d/docker.list:1
14:21:11 W: Target Packages (stable/binary-all/Packages) is configured multiple times in /etc/apt/sources.list:5 and /etc/apt/sources.list.d/docker.list:1
14:21:11 [setup_env_ubuntu.sh:15] ERROR: sudo -E apt update
14:21:11 [jenkins_job_build.sh:248] ERROR: ci/setup.sh
14:21:13 Build step 'Execute shell' marked build as failure

/cc @jongwu.

katacontainersbot added the size/large Task of significant size label Feb 8, 2024

jodh-intel force-pushed the kata-manager-add-hypervisor-option branch from 5c84b0c to a45b9a5 Compare February 9, 2024 11:17

katacontainersbot added size/huge Largest and most complex task (probably needs breaking into small pieces) and removed size/large Task of significant size labels Feb 9, 2024

jodh-intel mentioned this pull request Feb 9, 2024

packaging: Add kata-manager script #9066

Closed

jodh-intel requested a review from a team as a code owner February 9, 2024 16:45

jodh-intel force-pushed the kata-manager-add-hypervisor-option branch 3 times, most recently from 94056f5 to 3dcd534 Compare February 9, 2024 17:08

jodh-intel mentioned this pull request Feb 12, 2024

kata-manager: Allow switching between rust runtime configs #9060

Open

cmaf approved these changes Feb 14, 2024

View reviewed changes

jodh-intel mentioned this pull request Feb 14, 2024

packaging: Add the kata manager script #9091

Merged

jodh-intel added safe-to-test Add to PR after manually reviewing to allow certain extra checks to run ok-to-test labels Feb 15, 2024

fidencio reviewed Feb 15, 2024

View reviewed changes

jodh-intel force-pushed the kata-manager-add-hypervisor-option branch from 3dcd534 to 54fd20f Compare February 16, 2024 14:46

amshinde reviewed Feb 20, 2024

View reviewed changes

jodh-intel force-pushed the kata-manager-add-hypervisor-option branch from 54fd20f to a4f4999 Compare February 28, 2024 15:28

GabyCT approved these changes Feb 28, 2024

View reviewed changes

amshinde approved these changes Feb 29, 2024

View reviewed changes

jodh-intel force-pushed the kata-manager-add-hypervisor-option branch from a4f4999 to 0429771 Compare February 29, 2024 16:42

jodh-intel force-pushed the kata-manager-add-hypervisor-option branch from 0429771 to 9ddd823 Compare March 1, 2024 16:07

jodh-intel added 4 commits March 4, 2024 11:15

kata-manager: Whitespace fixes

159d296

Remove extraneous whitespace. Signed-off-by: James O. D. Hunt <james.o.hunt@intel.com>

kata-manager: Sort options in usage

ce35045

Ensure the usage statement lists all options in alphabetical order. Signed-off-by: James O. D. Hunt <james.o.hunt@intel.com>

kata-manager: Show message when checking file

455637b

Add an info message just before the archive file is checked. This keeps the user informed about what is happening as it can take a few seconds to perform the checks on slower systems. Signed-off-by: James O. D. Hunt <james.o.hunt@intel.com>

jodh-intel added the do-not-merge PR has problems or depends on another label Mar 4, 2024

jodh-intel added 2 commits March 4, 2024 12:18

docs: Whitespace fix

4f6fef1

Remove extraneous whitespace from hypervisors doc. Signed-off-by: James O. D. Hunt <james.o.hunt@intel.com>

jodh-intel force-pushed the kata-manager-add-hypervisor-option branch from 9ddd823 to 5d21040 Compare March 4, 2024 12:21

jodh-intel force-pushed the kata-manager-add-hypervisor-option branch from 5d21040 to 7af892f Compare March 4, 2024 12:24

jodh-intel removed the do-not-merge PR has problems or depends on another label Mar 4, 2024

jodh-intel merged commit b761a80 into kata-containers:main Mar 5, 2024
286 of 297 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

kata-manager: Allow hypervisor to be changed #9059

kata-manager: Allow hypervisor to be changed #9059

jodh-intel commented Feb 8, 2024 •

edited

jodh-intel commented Feb 8, 2024

jodh-intel commented Feb 9, 2024

jodh-intel commented Feb 9, 2024

jodh-intel commented Feb 9, 2024

cmaf left a comment

jodh-intel commented Feb 14, 2024

jodh-intel commented Feb 15, 2024

fidencio left a comment

fidencio Feb 15, 2024

fidencio Feb 15, 2024

jodh-intel Feb 16, 2024

fidencio Feb 15, 2024

jodh-intel Feb 16, 2024

fidencio Feb 15, 2024

jodh-intel Feb 16, 2024

fidencio commented Feb 15, 2024 •

edited

fidencio commented Feb 15, 2024

jodh-intel commented Feb 16, 2024

jodh-intel commented Feb 20, 2024

amshinde Feb 20, 2024

jodh-intel Feb 29, 2024

amshinde Feb 20, 2024

jodh-intel Feb 21, 2024

amshinde Feb 21, 2024

jodh-intel Feb 26, 2024

amshinde Feb 20, 2024

jodh-intel Feb 29, 2024

jodh-intel commented Feb 29, 2024

jodh-intel commented Feb 29, 2024

jodh-intel commented Mar 1, 2024

jodh-intel commented Mar 4, 2024

jodh-intel commented Mar 4, 2024 •

edited


		### Choose a Hypervisor

		Kata works with different [hypervisors](../docs/hypervisors.md). When you install a Kata system, the default hypervisor


		> Note: For details of each hypervisor, see [3].

		- If an invalid hypervisor name is specified with the '-H' option, an

		@@ -23,6 +23,8 @@ readonly containerd_slug="containerd/containerd"
		readonly kata_project_url="https://github.com/${kata_slug}"

kata-manager: Allow hypervisor to be changed #9059

kata-manager: Allow hypervisor to be changed #9059

Conversation

jodh-intel commented Feb 8, 2024 • edited

jodh-intel commented Feb 8, 2024

jodh-intel commented Feb 9, 2024

jodh-intel commented Feb 9, 2024

jodh-intel commented Feb 9, 2024

cmaf left a comment

Choose a reason for hiding this comment

jodh-intel commented Feb 14, 2024

jodh-intel commented Feb 15, 2024

fidencio left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

fidencio commented Feb 15, 2024 • edited

fidencio commented Feb 15, 2024

jodh-intel commented Feb 16, 2024

jodh-intel commented Feb 20, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jodh-intel commented Feb 29, 2024

jodh-intel commented Feb 29, 2024

jodh-intel commented Mar 1, 2024

jodh-intel commented Mar 4, 2024

jodh-intel commented Mar 4, 2024 • edited

jodh-intel commented Feb 8, 2024 •

edited

fidencio commented Feb 15, 2024 •

edited

jodh-intel commented Mar 4, 2024 •

edited