Add stream metadata for RHCOS

This implements part of the plan from:
openshift/os#477

When we originally added the pinned RHCOS metadata `rhcos.json`
to the installer, we also changed the coreos-assembler `meta.json`
format into an arbitrary new format in the name of some cleanups.
In retrospect, this was a big mistake because we now have two
formats.

Then Fedora CoreOS appeared and added streams JSON as a public API.

We decided to unify on streams metadata; there's now a published
Go library for it: https://github.com/coreos/stream-metadata-go

Among other benefits, it is a single file that supports multiple
architectures.

UPI installs should now use stream metadata, particularly
to find public cloud images.  This is exposed via a new
`openshift-install coreos print-stream-json` command.

This is an important preparatory step for exposing this via
`oc` as well as having something in the cluster update to
it.

HOWEVER as a (really hopefully temporary) hack, we *duplicate*
the metadata so that IPI installs use the new stream format,
and UPI CI jobs can still use the old format (with different RHCOS versions).

We will port the UPI docs and CI jobs after this merges.

Co-authored-by: Matthew Staebler <staebler@redhat.com>

docs/user/aws/install_upi.md

@@ -21,8 +21,8 @@ $ openshift-install create install-config
 ### Optional: Create Encrypted AMIs
 
 The IPI-based installer creates an encrypted AMI by default. If you wish to have an encrypted AMI for UPI-based
-installs, you will need to create it directly. You can find a list of the appropriate base AMIs 
-[here](../../../data/data/rhcos.json).
+installs, you will need to create it directly.  See [CoreOS bootimages](../overview.md#coreos-bootimages) for more information
+about bootimages, including how to find the AMI identifiers.
 
 You will make an encrypted copy of the AMI according to the [AWS documentation][encrypted-copy].
 

docs/user/metal/customization_ipi.md

@@ -24,13 +24,15 @@
 
 When doing a disconnected installation, the baremetal platform has the
 additional requirement that we have locations to download the RHCOS
-images. The installer downloads these from a location described in
-[/data/data/rhcos.json](/data/data/rhcos.json), but they can be
+images. The installer downloads these from a CoreOS stream metadata
+embedded in the installer code, but they can be
 overridden to point to a local mirror.
 
-The SHA256 parameter in the URLs are required, and should match the
-uncompressed SHA256 from rhcos.json.
+See [CoreOS bootimages](../overview.md#coreos-bootimages) for more information
+about bootimages.
 
+The SHA256 parameter in the URL is required, and should match the
+uncompressed SHA256 from the stream metadata JSON.
 
 * `bootstrapOSImage` (optional string): Override the image used for the
     bootstrap virtual machine.

docs/user/openstack/customization.md

@@ -112,13 +112,18 @@ sshKey: ssh-ed25519 AAAA...
 
 ## Image Overrides
 
-Normally the installer downloads the RHCOS image from a predetermined location described in [data/data/rhcos.json](/data/data/rhcos.json)). But the download URL can be overridden, notably for disconnected installations.
+The OpenShift installer pins the version of RHEL CoreOS and normally handles uploading the image to the target OpenStack instance.
 
-To do so and upload binary data from a custom location the user may set `clusterOSImage` parameter in the install config that points to that location, and then start the installation. In all other respects the process will be consistent with the default.
+If you want to download the image manually, see [CoreOS bootimages](../overview.md#coreos-bootimages) for more information
+about bootimages.  This is useful, for example, to perform a disconnected installation.  To do this,
+download the `qcow2` and host it at a custom location.  Then set the `openstack.clusterOSImage`
+parameter field in the install config to point to that location.   The install process will
+then use that mirrored image.
+In all other respects the process will be consistent with the default.
 
 **NOTE:** For this to work, the parameter value must be a valid http(s) URL.
 
-**NOTE:** The optional `sha256` query parameter can be attached to the URL, which will force the installer to check the image file checksum before uploading it into Glance.
+**NOTE:** The optional `sha256` query parameter can be attached to the URL. This will force the installer to check the uncompressed image file checksum before uploading it into Glance.
 
 Example:
 

docs/user/overview.md

@@ -84,3 +84,33 @@ As the unstable warning suggests, the presence of `manifests` and the names and
 It is occasionally useful to make alterations like this as one-off changes, but don't expect them to work on subsequent installer releases.
 
 [cluster-version]: https://github.com/openshift/cluster-version-operator/blob/master/docs/dev/clusterversion.md
+
+### CoreOS bootimages
+
+The `openshift-install` binary contains pinned versions of RHEL CoreOS "bootimages" (e.g. OpenStack `qcow2`, AWS AMI, bare metal `.iso`).
+Fully automated installs use these by default.
+
+For UPI (User Provisioned Infrastructure) installs, you can use the `openshift-install coreos print-stream-json` command to access information
+about the bootimages in [CoreOS Stream Metadata](https://github.com/coreos/stream-metadata-go) format.
+
+For example, this command will print the `x86_64` AMI for `us-west-1`:
+
+```
+$ openshift-install coreos print-stream-json | jq -r '.architectures.x86_64.images.aws.regions["us-west-1"].image'
+ami-0c548bdf93b74cd59
+```
+
+For on-premise clouds (e.g. OpenStack) with UPI installs, you may need to manually copy
+a bootimage into the infrastructure.  Here's an example command to print the `x86_64` `qcow2` file for `openstack`:
+
+```
+$ openshift-install coreos print-stream-json | jq -r '.architectures.x86_64.artifacts.openstack.formats["qcow2.gz"]'
+{
+  "disk": {
+    "location": "https://releases-art-rhcos.svc.ci.openshift.org/art/storage/releases/rhcos-4.8/48.83.202102230316-0/x86_64/rhcos-48.83.202102230316-0-openstack.x86_64.qcow2.gz",
+    "signature": "https://releases-art-rhcos.svc.ci.openshift.org/art/storage/releases/rhcos-4.8/48.83.202102230316-0/x86_64/rhcos-48.83.202102230316-0-openstack.x86_64.qcow2.gz.sig",
+    "sha256": "abc2add9746eb7be82e6919ec13aad8e9eae8cf073d8da6126d7c95ea0dee962",
+    "uncompressed-sha256": "9ed73a4e415ac670535c2188221e5a4a5f3e945bc2e03a65b1ed4fc76e5db6f2"
+  }
+}
+```

hack/update-rhcos-bootimage.py

@@ -1,4 +1,23 @@
 #!/usr/bin/env python3
+# As of 4.8 we are aiming to switch to stream metadata:
+# https://github.com/openshift/enhancements/pull/679
+# That transition hasn't yet fully completed; there are two copies of the
+# RHCOS metadata:
+# 
+# - data/data/rhcos-4.8.json (stream format, 4.8+)
+# - data/data/rhcos-$arch.json (openshift/installer specific, 4.7 and below)
+# 
+# See https://github.com/coreos/coreos-assembler/pull/2000 in particular.
+# 
+# The initial file data/data/rhcos-4.8 was generated this way:
+# 
+# $ plume cosa2stream --name rhcos-4.8 --distro rhcos  x86_64=48.83.202102230316-0 s390x=47.83.202102090311-0 ppc64le=47.83.202102091015-0 > data/data/rhcos-4.8.json
+# 
+# To update the bootimage for one or more architectures, use e.g.
+# 
+# $ plume cosa2stream --target data/data/rhcos-4.8.json --distro rhcos  x86_64=48.83.202102230316-0 s390x=47.83.202102090311-0 ppc64le=47.83.202102091015-0
+#
+# To update the legacy metadata, use:
 # Usage: ./hack/update-rhcos-bootimage.py https://releases-art-rhcos.svc.ci.openshift.org/art/storage/releases/rhcos-4.6/46.82.202008260918-0/x86_64/meta.json amd64
 import codecs,os,sys,json,argparse
 import urllib.parse

pkg/asset/cluster/tfvars.go

@@ -9,6 +9,7 @@ import (
 	"strings"
 
 	igntypes "github.com/coreos/ignition/v2/config/v3_2/types"
+	coreosarch "github.com/coreos/stream-metadata-go/arch"
 	gcpprovider "github.com/openshift/cluster-api-provider-gcp/pkg/apis/gcpprovider/v1beta1"
 	kubevirtprovider "github.com/openshift/cluster-api-provider-kubevirt/pkg/apis/kubevirtprovider/v1alpha1"
 	kubevirtutils "github.com/openshift/cluster-api-provider-kubevirt/pkg/utils"
@@ -338,16 +339,30 @@ func (t *TerraformVariables) Generate(parents asset.Parents) error {
 		}
 		preexistingnetwork := installConfig.Config.GCP.Network != ""
 
-		imageRaw, err := rhcospkg.GCPRaw(ctx, installConfig.Config.ControlPlane.Architecture)
+		archName := coreosarch.RpmArch(string(installConfig.Config.ControlPlane.Architecture))
+		st, err := rhcospkg.FetchCoreOSBuild(ctx)
 		if err != nil {
-			return errors.Wrap(err, "failed to find Raw GCP image URL")
+			return err
+		}
+		streamArch, err := st.GetArchitecture(archName)
+		if err != nil {
+			return err
+		}
+
+		img := streamArch.Images.Gcp
+		if img == nil {
+			return fmt.Errorf("%s: No GCP build found", st.FormatPrefix(archName))
 		}
+		// For backwards compatibility, we generate this URL to the image (only applies to RHCOS, not FCOS/OKD)
+		// right now.  It will only be used if nested virt or other licenses are enabled, which we
+		// really should deprecate and remove - xref https://github.com/openshift/installer/pull/4696
+		imageURL := fmt.Sprintf("https://storage.googleapis.com/rhcos/rhcos/%s.tar.gz", img.Name)
 		data, err := gcptfvars.TFVars(
 			gcptfvars.TFVarsSources{
 				Auth:               auth,
 				MasterConfigs:      masterConfigs,
 				WorkerConfigs:      workerConfigs,
-				ImageURI:           imageRaw,
+				ImageURI:           imageURL,
 				ImageLicenses:      installConfig.Config.GCP.Licenses,
 				PublicZoneName:     publicZoneName,
 				PublishStrategy:    installConfig.Config.Publish,

pkg/asset/rhcos/bootstrap_image.go

@@ -3,8 +3,11 @@ package rhcos
 
 import (
 	"context"
+	"fmt"
 	"time"
 
+	"github.com/coreos/stream-metadata-go/arch"
+
 	"github.com/openshift/installer/pkg/asset"
 	"github.com/openshift/installer/pkg/asset/installconfig"
 	"github.com/openshift/installer/pkg/rhcos"
@@ -37,28 +40,44 @@ func (i *BootstrapImage) Generate(p asset.Parents) error {
 	p.Get(ic)
 	config := ic.Config
 
-	var osimage string
-	var err error
 	ctx, cancel := context.WithTimeout(context.TODO(), 30*time.Second)
 	defer cancel()
+
 	switch config.Platform.Name() {
 	case baremetal.Name:
-		// Check for RHCOS image URL override
-		if boi := config.Platform.BareMetal.BootstrapOSImage; boi != "" {
-			osimage = boi
-			break
+		archName := arch.RpmArch(string(config.ControlPlane.Architecture))
+		st, err := rhcos.FetchCoreOSBuild(ctx)
+		if err != nil {
+			return err
+		}
+		streamArch, err := st.GetArchitecture(archName)
+		if err != nil {
+			return err
 		}
 
+		// Check for CoreOS image URL override
+		if boi := config.Platform.BareMetal.BootstrapOSImage; boi != "" {
+			*i = BootstrapImage(boi)
+			return nil
+		}
 		// Baremetal IPI launches a local VM for the bootstrap node
 		// Hence requires the QEMU image to use the libvirt backend
-		osimage, err = rhcos.QEMU(ctx, config.ControlPlane.Architecture)
+		if a, ok := streamArch.Artifacts["qemu"]; ok {
+			u, err := rhcos.FindArtifactURL(a)
+			if err != nil {
+				return err
+			}
+			*i = BootstrapImage(u)
+			return nil
+		}
+		return fmt.Errorf("%s: No qemu build found", st.FormatPrefix(archName))
 	default:
 		// other platforms use the same image for all nodes
-		osimage, err = osImage(config)
-	}
-	if err != nil {
-		return err
+		u, err := osImage(config)
+		if err != nil {
+			return err
+		}
+		*i = BootstrapImage(u)
+		return nil
 	}
-	*i = BootstrapImage(osimage)
-	return nil
 }

pkg/asset/rhcos/image.go

@@ -7,7 +7,7 @@ import (
 	"os"
 	"time"
 
-	"github.com/pkg/errors"
+	"github.com/coreos/stream-metadata-go/arch"
 	"github.com/sirupsen/logrus"
 
 	"github.com/openshift/installer/pkg/asset"
@@ -66,67 +66,95 @@ func (i *Image) Generate(p asset.Parents) error {
 }
 
 func osImage(config *types.InstallConfig) (string, error) {
-	arch := config.ControlPlane.Architecture
-
-	var osimage string
-	var err error
 	ctx, cancel := context.WithTimeout(context.TODO(), 30*time.Second)
 	defer cancel()
+
+	archName := arch.RpmArch(string(config.ControlPlane.Architecture))
+
+	st, err := rhcos.FetchCoreOSBuild(ctx)
+	if err != nil {
+		return "", err
+	}
+	streamArch, err := st.GetArchitecture(archName)
+	if err != nil {
+		return "", err
+	}
 	switch config.Platform.Name() {
 	case aws.Name:
 		if len(config.Platform.AWS.AMIID) > 0 {
-			osimage = config.Platform.AWS.AMIID
-			break
+			return config.Platform.AWS.AMIID, nil
 		}
 		region := config.Platform.AWS.Region
 		if !configaws.IsKnownRegion(config.Platform.AWS.Region) {
 			region = "us-east-1"
 		}
-		osimage, err = rhcos.AMI(ctx, arch, region)
+		osimage, err := st.GetAMI(archName, region)
+		if err != nil {
+			return "", err
+		}
 		if region != config.Platform.AWS.Region {
 			osimage = fmt.Sprintf("%s,%s", osimage, region)
 		}
+		return osimage, nil
 	case gcp.Name:
-		osimage, err = rhcos.GCP(ctx, arch)
+		if streamArch.Images.Gcp != nil {
+			img := streamArch.Images.Gcp
+			return fmt.Sprintf("projects/%s/global/images/%s", img.Project, img.Name), nil
+		}
+		return "", fmt.Errorf("%s: No GCP build found", st.FormatPrefix(archName))
 	case libvirt.Name:
-		osimage, err = rhcos.QEMU(ctx, arch)
-	case openstack.Name:
-		if oi := config.Platform.OpenStack.ClusterOSImage; oi != "" {
-			osimage = oi
-			break
+		// 𝅘𝅥𝅮 Everything's going to be a-ok 𝅘𝅥𝅮
+		if a, ok := streamArch.Artifacts["qemu"]; ok {
+			return rhcos.FindArtifactURL(a)
+		}
+		return "", fmt.Errorf("%s: No qemu build found", st.FormatPrefix(archName))
+	case ovirt.Name, kubevirt.Name, openstack.Name:
+		op := config.Platform.OpenStack
+		if op != nil {
+			if oi := op.ClusterOSImage; oi != "" {
+				return oi, nil
+			}
+		}
+		if a, ok := streamArch.Artifacts["openstack"]; ok {
+			return rhcos.FindArtifactURL(a)
 		}
-		osimage, err = rhcos.OpenStack(ctx, arch)
-	case ovirt.Name:
-		osimage, err = rhcos.OpenStack(ctx, arch)
-	case kubevirt.Name:
-		osimage, err = rhcos.OpenStack(ctx, arch)
+		return "", fmt.Errorf("%s: No openstack build found", st.FormatPrefix(archName))
 	case azure.Name:
-		osimage, err = rhcos.VHD(ctx, arch)
+		ext := streamArch.RHELCoreOSExtensions
+		if ext == nil {
+			return "", fmt.Errorf("%s: No azure build found", st.FormatPrefix(archName))
+		}
+		azd := ext.AzureDisk
+		if azd == nil {
+			return "", fmt.Errorf("%s: No azure build found", st.FormatPrefix(archName))
+		}
+		return azd.URL, nil
 	case baremetal.Name:
-		// Check for RHCOS image URL override
+		// Check for image URL override
 		if oi := config.Platform.BareMetal.ClusterOSImage; oi != "" {
-			osimage = oi
-			break
+			return oi, nil
 		}
 
 		// Note that baremetal IPI currently uses the OpenStack image
 		// because this contains the necessary ironic config drive
 		// ignition support, which isn't enabled in the UPI BM images
-		osimage, err = rhcos.OpenStack(ctx, arch)
+		if a, ok := streamArch.Artifacts["openstack"]; ok {
+			return rhcos.FindArtifactURL(a)
+		}
+		return "", fmt.Errorf("%s: No openstack build found", st.FormatPrefix(archName))
 	case vsphere.Name:
-		// Check for RHCOS image URL override
+		// Check for image URL override
 		if config.Platform.VSphere.ClusterOSImage != "" {
-			osimage = config.Platform.VSphere.ClusterOSImage
-			break
+			return config.Platform.VSphere.ClusterOSImage, nil
 		}
 
-		osimage, err = rhcos.VMware(ctx, arch)
+		if a, ok := streamArch.Artifacts["vmware"]; ok {
+			return rhcos.FindArtifactURL(a)
+		}
+		return "", fmt.Errorf("%s: No vmware build found", st.FormatPrefix(archName))
 	case none.Name:
+		return "", nil
 	default:
-		return "", errors.New("invalid Platform")
-	}
-	if err != nil {
-		return "", err
+		return "", fmt.Errorf("invalid platform %v", config.Platform.Name())
 	}
-	return osimage, nil
 }