Platform Request: Microsoft Hyper-V #1411

baude · 2023-02-06T19:44:29Z

Why is the platform important? Who uses it?

Microsoft HyperV is a widely known and adopted virtualization platform for Windows. The Podman team would also like to being supporting HyperV with their podman machine function which uses FCOS as its base OS. I understand separately FCOS would like to support HyperV as well presumably due to user demand.

What is the official name of the platform? Is there a short name that's commonly used in client API implementations?

Microsoft HyperV

How can the OS retrieve instance userdata? What happens if no userdata is provided?

HyperV implements a key-value-pair (KVP) through its hypervisor using Windows Management Implementation (WMI).

On the host side, the injection occurs by setting values in a given registry. The KVP does have limitations; the key is limited to 512 bytes and the value 2048 bytes. Because this is Windows UTF halves those values to 256 and 1024.

On the Linux guest, the KVPs are shared by WMI and are written to the Linux kernel. Using code from the already existing hypervkvpd (daemon), we can extract the information we need. Because of the 1K value limit, the values will need to be split up on the host and re-assembled on the guest.

Does the platform provide a way to configure SSH keys for the instance? How can the OS retrieve them? What happens if none are provided?

I do not know. I want to use ignition for this.

How can the OS retrieve network configuration? Is DHCP sufficient, or is there some other network-accessible metadata service?

DHCP is sufficient

In particular, how can the OS retrieve the system hostname?

The host hostname is also provided as a KVP automatically. Otherwise the expectation is that DHCP delivers this.

Does the platform require the OS to have a specific console configuration?

I do not believe so, but we should clarify what exactly this question is asking ...

Is there a mechanism for the OS to report to the platform that it has successfully booted? Is the mechanism required?

It could be done if really needed through KVPs as well but it would require code on the host looking for it. The mechanism is not required.

Does the platform have an agent that runs inside the instance? Is it required? What does it do? What language is it implemented in, and where is the source code repository?

There are such things. Early analysis suggests we do not need it and therefore is optional.

How are VM images uploaded to the platform and published to other users? Is there an API? What disk image format is expected?

the disk format is vhd/vhdx. the long term expectation is that these images would be produced upstream

Are there any other platform quirks we should know about?

ah! it's windows!

The text was updated successfully, but these errors were encountered:

miabbott · 2023-02-06T20:52:14Z

Previous investigation done by the CoreOS team in 2020 - https://issues.redhat.com/browse/COS-842

bgilbert · 2023-02-06T21:45:00Z

Thanks for filing this. coreos/ignition#1538 tracks the Ignition portion.

For the platform name, hyperv is the obvious choice, and there's precedent for it.
I wouldn't expect the platform to provide special mechanisms for SSH keys (usually that happens on clouds), so we can likely ignore that part.
The console configuration question means: does the Hyper-V UI expect the VM to produce console output on a graphical console? Serial port? Both?
We should decide whether to produce VHD or VHDX.

On the Linux guest, the KVPs are shared by WMI and are written to the Linux kernel. Using code from the already existing hypervkvpd (daemon), we can extract the information we need.

Re-emphasizing, for safety, that this needs to be new code for license compatibility reasons.

The host hostname is also provided as a KVP automatically. Otherwise the expectation is that DHCP delivers this.

Does DHCP actually always deliver this? If not (e.g. if Hyper-V doesn't control the DHCP server) we might need to teach Afterburn to read the hostname key.

bgilbert · 2023-02-06T22:20:32Z

Our Hyper-V docs should include a PowerShell script that takes a VM identifier and an Ignition config, chunks the config into appropriately-sized pieces, and creates the KVP keys.

Add hyperV support to cosa including a buildextend command to create vhdx images. Supports coreos/fedora-coreos-tracker#1411 Signed-off-by: Brent Baude <bbaude@redhat.com>

This PR adds initial support for the Microsoft HyperV platform. It uses HyperV and MS WMI to read key/value pairs as provided by the Windows host. Because the size allocation of the value is quite small, the ignition values will likely be split into at least 2 and more like more parts which must be read and reassembled in ignition. Supports coreos/fedora-coreos-tracker#1411 Signed-off-by: Brent Baude <bbaude@redhat.com>

Add hyperV support to cosa including a buildextend command to create vhdx/hyperv images. Supports coreos/fedora-coreos-tracker#1411 Signed-off-by: Brent Baude <bbaude@redhat.com>

This PR adds initial support for the Microsoft HyperV platform. It uses HyperV and MS WMI to read key/value pairs as provided by the Windows host. Because the size allocation of the value is quite small, the ignition values will likely be split into at least 2 and more like more parts which must be read and reassembled in ignition. Supports coreos/fedora-coreos-tracker#1411 Signed-off-by: Brent Baude <bbaude@redhat.com>

bgilbert · 2023-02-15T20:54:05Z

Implementation checklist in #1424.

Add a buildextend-hyperv command that creates a `hyperv` image in VHDX format. Supports coreos/fedora-coreos-tracker#1411. Supports coreos/fedora-coreos-tracker#1411 Signed-off-by: Brent Baude <bbaude@redhat.com>

Add a buildextend-hyperv command that creates a `hyperv` image in VHDX format. Supports coreos/fedora-coreos-tracker#1411.

bgilbert · 2023-02-15T22:04:46Z

@baude pointed out that Windows can't decompress .xz (or indeed .gz) files by default. We should probably consider compressing into a .zip file instead.

Add a buildextend-hyperv command that creates a `hyperv` image in VHDX format. Supports coreos/fedora-coreos-tracker#1411.

This PR adds initial support for the Microsoft HyperV platform. It uses HyperV and MS WMI to read key/value pairs as provided by the Windows host. Because the size allocation of the value is quite small, the ignition values will likely be split into at least 2 and more like more parts which must be read and reassembled in ignition. Supports coreos/fedora-coreos-tracker#1411 Signed-off-by: Brent Baude <bbaude@redhat.com>

buildextend --compress supports gzip as an alternate compression over xz using a key called "gzip" where the value is a bool. This is now refactored to a key called "compression" with a value of "gzip", "skip", or "zip" (see below). The "skip-compression" key is no longer applicable. HyperV images which are used on Windows need to be compressed with zip because xz and gzip are not supported natively. Added zip support to buildextend and set the hyperv platform compression to "zip". Supports: * coreos/fedora-coreos-tracker#1411 * coreos/fedora-coreos-tracker#1421 Signed-off-by: Brent Baude <bbaude@redhat.com>

Supports: * coreos/fedora-coreos-tracker#1411 * coreos/fedora-coreos-tracker#1421 Signed-off-by: Brent Baude <bbaude@redhat.com>

This PR adds initial support for the Microsoft HyperV platform. It uses HyperV and MS WMI to read key/value pairs as provided by the Windows host. Because the size allocation of the value is quite small, the ignition values will likely be split into at least 2 and more like more parts which must be read and reassembled in ignition. Supports coreos/fedora-coreos-tracker#1411 Signed-off-by: Brent Baude <bbaude@redhat.com>

This PR adds initial support for the Microsoft HyperV platform. It uses HyperV and MS WMI to read key/value pairs as provided by the Windows host. Because the size allocation of the value is quite small, the ignition values will likely be split into at least 2 and more like more parts which must be read and reassembled in ignition. Supports: * coreos/fedora-coreos-tracker#1411 * coreos/fedora-coreos-tracker#1424 Signed-off-by: Brent Baude <bbaude@redhat.com>

Read the Ignition config from the Hyper-V Data Exchange Service ("KVP"), which can be configured on the host via WMI. KVP on Linux is normally handled by a combination of a kernel module (hv_utils) and a daemon shipped with the kernel source (hv_kvp_daemon). The latter writes world-readable files to /var/lib/hyperv; these contain an array of binary structs. The host-guest protocol has the guest OS as the passive peer; hv_kvp_daemon connects and is sent all the host's key-value pairs. We don't want to require hv_kvp_daemon as a dependency, so we use a pure Go reimplementation (libhvee) that supports just enough of the protocol to receive the KVPs. However, if we disconnect and then hv_kvp_daemon reconnects later, the host won't re-send the KVPs, so we also need to save them to /var/lib/hyperv in the same format used by the daemon. Unlike the daemon, we set the files to mode 600, since the Ignition config is potentially sensitive; the daemon does not change the permissions of existing files. Small configs can be stored in the `ignition.config` KVP. However, individual values have severe length limitations (~1 KiB of UTF-8), so we also support concatenating `ignition.config.0`, `ignition.config.1`, etc. to form a larger config. See also: coreos/fedora-coreos-tracker#1411 coreos/fedora-coreos-tracker#1424 Almost all of this work was done by Brent Baude; I just added the final polish. Thanks Brent! Co-authored-by: Brent Baude <bbaude@redhat.com>

Supports: * coreos/fedora-coreos-tracker#1411 * coreos/fedora-coreos-tracker#1424 Co-authored-by: Brent Baude <bbaude@redhat.com>

Read the Ignition config from the Hyper-V Data Exchange Service ("KVP"), which can be configured on the host via WMI. KVP on Linux is normally handled by a combination of a kernel module (hv_utils) and a daemon shipped with the kernel source (hv_kvp_daemon). The latter writes world-readable files to /var/lib/hyperv; these contain an array of binary structs. The host-guest protocol has the guest OS as the passive peer; hv_kvp_daemon connects and is sent all the host's key-value pairs. We don't want to require hv_kvp_daemon as a dependency, so we use a pure Go reimplementation (libhvee) that supports just enough of the protocol to receive the KVPs. However, if we disconnect and then hv_kvp_daemon reconnects later, the host won't re-send the KVPs, so we also need to save them to /var/lib/hyperv in the same format used by the daemon. Unlike the daemon, we set the files to mode 600, since the Ignition config is potentially sensitive; the daemon does not change the permissions of existing files. Small configs can be stored in the `ignition.config` KVP. However, individual values have severe length limitations (~1 KiB of UTF-8), so we also support concatenating `ignition.config.0`, `ignition.config.1`, etc. to form a larger config. See also: coreos/fedora-coreos-tracker#1411 coreos/fedora-coreos-tracker#1424 Almost all of this work was done by Brent Baude; I just added the final polish. Thanks Brent! Co-authored-by: Brent Baude <bbaude@redhat.com>

bgilbert · 2023-06-28T14:08:36Z

I've looked through the Hyper-V Manager and Virtual Machine Connection UIs in Windows Server 2024, and serial port setup is not surfaced anywhere. I think it's safe to stick with graphical console only for now.

Add example metadata for new platform support of hyperv. This supports: * coreos#1411 * coreos#1424

Add example metadata for hyperv platform. This supports: * coreos#1411 * coreos#1424 Co-authored-by: Brent Baude <bbaude@redhat.com>

Read the Ignition config from the Hyper-V Data Exchange Service ("KVP"), which can be configured on the host via WMI. KVP on Linux is normally handled by a combination of a kernel module (hv_utils) and a daemon shipped with the kernel source (hv_kvp_daemon). The latter writes world-readable files to /var/lib/hyperv; these contain an array of binary structs. The host-guest protocol has the guest OS as the passive peer; hv_kvp_daemon connects and is sent all the host's key-value pairs. We don't want to require hv_kvp_daemon as a dependency, so we use a pure Go reimplementation (libhvee) that supports just enough of the protocol to receive the KVPs. However, if we disconnect and then hv_kvp_daemon reconnects later, the host won't re-send the KVPs, so we also need to save them to /var/lib/hyperv in the same format used by the daemon. Unlike the daemon, we set the files to mode 600, since the Ignition config is potentially sensitive; the daemon does not change the permissions of existing files. Small configs can be stored in the `ignition.config` KVP. However, individual values have severe length limitations (~1 KiB of UTF-8), so we also support concatenating `ignition.config.0`, `ignition.config.1`, etc. to form a larger config. See also: coreos/fedora-coreos-tracker#1411 coreos/fedora-coreos-tracker#1424 Almost all of this work was done by Brent Baude; I just added the final polish. Thanks Brent! Co-authored-by: Brent Baude <bbaude@redhat.com>

Part of coreos/fedora-coreos-tracker#1411

dustymabe · 2023-08-11T14:16:52Z

The fix for this went into next stream release 38.20230806.1.0. Please try out the new release and report issues.

dustymabe · 2023-08-11T14:17:01Z

The fix for this went into testing stream release 38.20230806.2.0. Please try out the new release and report issues.

dustymabe · 2023-08-11T14:17:10Z

The fix for this went into stable stream release 38.20230722.3.0.

dustymabe · 2023-08-11T14:18:23Z

Please see our docs for more information on how to use this new image!

Follow-up to coreos/fedora-coreos-tracker#1411 Signed-off-by: Jason T. Greene <jason.greene@redhat.com>

Reflux0301 · 2023-12-15T20:06:51Z

Is it possible we can add the Hyper-V specific tools to the builds?

For example, hyperv-daemons, hyperv-daemons-license, hyperv-tools, hypervkvpd (provides the host with information about the VM) and hypervvssd (useful for backup replication).

dustymabe · 2023-12-16T01:10:35Z

Ususally we don't ship agents. see #12

If you'd like to carry this conversation further please open a new issue.

Follow-up to coreos/fedora-coreos-tracker#1411 Signed-off-by: Jason T. Greene <jason.greene@redhat.com>

baude added area/platforms kind/enhancement labels Feb 6, 2023

bgilbert mentioned this issue Feb 6, 2023

platform: add support for Hyper-V #251

Closed

baude mentioned this issue Feb 13, 2023

Add support for building Hyperv images coreos/coreos-assembler#3350

Merged

baude added a commit to baude/coreos-assembler that referenced this issue Feb 13, 2023

Add hyperV support

d111363

Add hyperV support to cosa including a buildextend command to create vhdx images. Supports coreos/fedora-coreos-tracker#1411 Signed-off-by: Brent Baude <bbaude@redhat.com>

baude added a commit to baude/coreos-assembler that referenced this issue Feb 13, 2023

Add hyperV support

175a78b

Add hyperV support to cosa including a buildextend command to create vhdx images. Supports coreos/fedora-coreos-tracker#1411 Signed-off-by: Brent Baude <bbaude@redhat.com>

baude mentioned this issue Feb 13, 2023

Add support for Hyper-V platform coreos/ignition#1555

Merged

baude added a commit to baude/coreos-assembler that referenced this issue Feb 15, 2023

Add support for building Hyperv images

b44bf2e

Add a buildextend-hyperv command that creates a `hyperv` image in VHDX format. Supports coreos/fedora-coreos-tracker#1411.

dustymabe added the meeting topics for meetings label Feb 15, 2023

bgilbert pushed a commit to coreos/coreos-assembler that referenced this issue Feb 16, 2023

Add support for building Hyperv images

8b224ab

Add a buildextend-hyperv command that creates a `hyperv` image in VHDX format. Supports coreos/fedora-coreos-tracker#1411.

baude mentioned this issue Feb 16, 2023

Refactor buildextend compression and add zip coreos/coreos-assembler#3363

Merged

baude added a commit to baude/fedora-coreos-browser that referenced this issue Feb 16, 2023

HyperV added to browser list

ddccdd5

Supports: * coreos/fedora-coreos-tracker#1411 * coreos/fedora-coreos-tracker#1421 Signed-off-by: Brent Baude <bbaude@redhat.com>

baude mentioned this issue Feb 16, 2023

HyperV added to browser list coreos/fedora-coreos-browser#45

Merged

baude added a commit to baude/fedora-coreos-pipeline that referenced this issue Feb 16, 2023

Add hyperv to default artifacts

0c3a032

Supports: * coreos/fedora-coreos-tracker#1411 * coreos/fedora-coreos-tracker#1421 Signed-off-by: Brent Baude <bbaude@redhat.com>

baude mentioned this issue Feb 16, 2023

Add hyperv to default artifacts coreos/fedora-coreos-pipeline#815

Closed

bgilbert added a commit to baude/fedora-coreos-browser that referenced this issue Jun 28, 2023

Add Hyper-V to browser list

83b6c69

Supports: * coreos/fedora-coreos-tracker#1411 * coreos/fedora-coreos-tracker#1424 Co-authored-by: Brent Baude <bbaude@redhat.com>

bgilbert pushed a commit to baude/fedora-coreos-tracker that referenced this issue Jun 28, 2023

Add HyperV metadata

668b2f2

Add example metadata for new platform support of hyperv. This supports: * coreos#1411 * coreos#1424

bgilbert added a commit to baude/fedora-coreos-tracker that referenced this issue Jun 28, 2023

Add Hyper-V metadata

d33efd9

Add example metadata for hyperv platform. This supports: * coreos#1411 * coreos#1424 Co-authored-by: Brent Baude <bbaude@redhat.com>

dustymabe added a commit to dustymabe/fedora-coreos-pipeline that referenced this issue Jul 14, 2023

config: start building hyperv image

1637cc0

Part of coreos/fedora-coreos-tracker#1411

dustymabe mentioned this issue Jul 14, 2023

config: start building hyperv image coreos/fedora-coreos-pipeline#891

Merged

dustymabe added a commit to coreos/fedora-coreos-pipeline that referenced this issue Jul 15, 2023

config: start building hyperv image

eae1a5d

Part of coreos/fedora-coreos-tracker#1411

bgilbert added status/pending-testing-release Fixed upstream. Waiting on a testing release. status/pending-stable-release Fixed upstream and in testing. Waiting on stable release. status/pending-next-release Fixed upstream. Waiting on a next release. and removed status/decided labels Jul 21, 2023

dustymabe closed this as completed Aug 11, 2023

dustymabe removed status/pending-testing-release Fixed upstream. Waiting on a testing release. status/pending-stable-release Fixed upstream and in testing. Waiting on stable release. status/pending-next-release Fixed upstream. Waiting on a next release. labels Aug 11, 2023

n1hility added a commit to n1hility/fedora-coreos-pipeline that referenced this issue Nov 8, 2023

config: start building hyperv image for aarch64

2970aa1

Follow-up to coreos/fedora-coreos-tracker#1411 Signed-off-by: Jason T. Greene <jason.greene@redhat.com>

n1hility mentioned this issue Nov 8, 2023

config: start building hyperv image for aarch64 coreos/fedora-coreos-pipeline#923

Merged

dustymabe pushed a commit to coreos/fedora-coreos-pipeline that referenced this issue Nov 14, 2023

config: start building hyperv image for aarch64

d3bc79d

Follow-up to coreos/fedora-coreos-tracker#1411 Signed-off-by: Jason T. Greene <jason.greene@redhat.com>

aaradhak pushed a commit to aaradhak/fedora-coreos-pipeline that referenced this issue May 14, 2024

config: start building hyperv image for aarch64

e8d0db1

Follow-up to coreos/fedora-coreos-tracker#1411 Signed-off-by: Jason T. Greene <jason.greene@redhat.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Platform Request: Microsoft Hyper-V #1411

Platform Request: Microsoft Hyper-V #1411

baude commented Feb 6, 2023

miabbott commented Feb 6, 2023

bgilbert commented Feb 6, 2023

bgilbert commented Feb 6, 2023

bgilbert commented Feb 15, 2023 •

edited

Loading

bgilbert commented Feb 15, 2023

bgilbert commented Jun 28, 2023

dustymabe commented Aug 11, 2023

dustymabe commented Aug 11, 2023

dustymabe commented Aug 11, 2023

dustymabe commented Aug 11, 2023

Reflux0301 commented Dec 15, 2023

dustymabe commented Dec 16, 2023

Platform Request: Microsoft Hyper-V #1411

Platform Request: Microsoft Hyper-V #1411

Comments

baude commented Feb 6, 2023

Why is the platform important? Who uses it?

What is the official name of the platform? Is there a short name that's commonly used in client API implementations?

How can the OS retrieve instance userdata? What happens if no userdata is provided?

Does the platform provide a way to configure SSH keys for the instance? How can the OS retrieve them? What happens if none are provided?

How can the OS retrieve network configuration? Is DHCP sufficient, or is there some other network-accessible metadata service?

In particular, how can the OS retrieve the system hostname?

Does the platform require the OS to have a specific console configuration?

Is there a mechanism for the OS to report to the platform that it has successfully booted? Is the mechanism required?

Does the platform have an agent that runs inside the instance? Is it required? What does it do? What language is it implemented in, and where is the source code repository?

How are VM images uploaded to the platform and published to other users? Is there an API? What disk image format is expected?

Are there any other platform quirks we should know about?

miabbott commented Feb 6, 2023

bgilbert commented Feb 6, 2023

bgilbert commented Feb 6, 2023

bgilbert commented Feb 15, 2023 • edited Loading

bgilbert commented Feb 15, 2023

bgilbert commented Jun 28, 2023

dustymabe commented Aug 11, 2023

dustymabe commented Aug 11, 2023

dustymabe commented Aug 11, 2023

dustymabe commented Aug 11, 2023

Reflux0301 commented Dec 15, 2023

dustymabe commented Dec 16, 2023

bgilbert commented Feb 15, 2023 •

edited

Loading