Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

RFE: openshift-install download-bootimage #1399

Closed
cgwalters opened this issue Mar 11, 2019 · 13 comments
Closed

RFE: openshift-install download-bootimage #1399

cgwalters opened this issue Mar 11, 2019 · 13 comments

Comments

@cgwalters
Copy link
Member

In order to support e.g. bare metal and other user-provisioned infrastructure platforms, let's add a high level command to download the "bootimages" (qemu qcow2, bare metal UEFI, VMWare vmdk, and for that matter the AWS VMDK).

Something like:

$ openshift-install download-bootimage --platform=metal-uefi
Downloading redhat-coreos-metal-uefi-4.72.raw ...done.
@ghost
Copy link

ghost commented Mar 12, 2019

@cgwalter,

it sounds good !
I think the same for libvirt use with bridge

Regards,
Fábio Sbano

@abhinavdahiya
Copy link
Contributor

abhinavdahiya commented Mar 12, 2019 •
edited
Loading

In order to support e.g. bare metal and other user-provisioned infrastructure platforms, let's add a high level command to download the "bootimages" (qemu qcow2, bare metal UEFI, VMWare vmdk, and for that matter the AWS VMDK).

Something like:

$ openshift-install download-bootimage --platform=metal-uefi
Downloading redhat-coreos-metal-uefi-4.72.raw ...done.

I do not think that the the installer should be downloading the bootimages

  • how is the installer supposed tied the recommended platform flag to the set of images required. for example for metal some people will use the iso, while some want the iso (for pxe assets) and raw image for installation.
  • recommended platform flag is already used by installer and creates confusion...
  • The installer needs to move to release image for all source of truth, the intermediate direction is pulling the release image pkg/rhcos/release: Extract RHCOS build from release image #1286. But in the long term it would be nice to move all the image information onto the bootstrap node and installer only chooses the release image.

@cgwalters
Copy link
Member Author

how is the installer supposed tied the recommended platform flag to the set of images required. for example for metal some people will use the iso, while some want the iso (for pxe assets) and raw image for installation.

I think the simplest is:

openshift-install download-bootimage --platform=metal

which would do all of the metal stuff. The ISO is tiny, and anyone installing on metal surely has enough bandwidth to grab both metal-uefi and metal-bios.

This was referenced Apr 3, 2019
Merged
Closed
@wking
Copy link
Member

wking commented Apr 4, 2019

I've taken a stab at something similar in #1529, if folks want to take a look.

This was referenced Apr 25, 2019
Merged
Merged
@cgwalters
Copy link
Member Author

Today RHCOS is available here: https://mirror.openshift.com/pub/openshift-v4/dependencies/rhcos/4.1/

However, it still must be downloaded manually.

If you want something other than that, today you can find the download URL used by the installer in git:

$ rg releases-art
data/data/rhcos.json
49:    "baseURI": "https://releases-art-rhcos.svc.ci.openshift.org/art/storage/releases/ootpa/410.8.20190508.1/",

Ideally this would be an implementation detail of the installer, but in the short term that URL isn't going away.

@wking wking mentioned this issue May 22, 2019
Merged
cgwalters added a commit to cgwalters/installer that referenced this issue Jun 4, 2019
@cgwalters
version: Output RHEL CoreOS bootimage version
1720bd9 
There are a lot of reasons to want to know the RHCOS bootimage version
associated with an installer - the installer pins it today, but doesn't
make the information easy to find particularly in a CI/development
environment.

This is a very small step towards something like:
openshift#1399
@cgwalters cgwalters mentioned this issue Jun 4, 2019
Closed
cgwalters added a commit to cgwalters/installer that referenced this issue Jun 4, 2019
@cgwalters
version: Output RHEL CoreOS bootimage version
982210e 
There are a lot of reasons to want to know the RHCOS bootimage version
associated with an installer - the installer pins it today, but doesn't
make the information easy to find particularly in a CI/development
environment.

This is a very small step towards something like:
openshift#1399
cgwalters added a commit to cgwalters/installer that referenced this issue Jun 5, 2019
@cgwalters
version: Output RHEL CoreOS bootimage version
f2f7256 
There are a lot of reasons to want to know the RHCOS bootimage version
associated with an installer - the installer pins it today, but doesn't
make the information easy to find particularly in a CI/development
environment.

This is a very small step towards something like:
openshift#1399
cgwalters added a commit to cgwalters/installer that referenced this issue Jun 13, 2019
@cgwalters
version: Output RHEL CoreOS bootimage version
d3cab5e 
There are a lot of reasons to want to know the RHCOS bootimage version
associated with an installer - the installer pins it today, but doesn't
make the information easy to find particularly in a CI/development
environment.

This is a very small step towards something like:
openshift#1399
@cgwalters
Copy link
Member Author

cgwalters commented Jun 14, 2019 •
edited
Loading

Maybe this would be better:

$ openshift-install bootimage-info
{
    "amis": {
        "ap-northeast-1": {
            "hvm": "ami-0c63b39219b8123e5"
        },
        "ap-northeast-2": {
            "hvm": "ami-073cba0913d2250a4"
        },
        "ap-south-1": {
            "hvm": "ami-0270be11430101040"
        },
        "ap-southeast-1": {
            "hvm": "ami-06eb9d35ede4f08a3"
        },
        "ap-southeast-2": {
            "hvm": "ami-0d980796ce258b5d5"
        },
        "ca-central-1": {
            "hvm": "ami-0f907257d1686e3f7"
        },
        "eu-central-1": {
            "hvm": "ami-02fdd627029c0055b"
        },
        "eu-west-1": {
            "hvm": "ami-0d4839574724ed3fa"
        },
        "eu-west-2": {
            "hvm": "ami-053073b95aa285347"
        },
        "eu-west-3": {
            "hvm": "ami-09deb5deb6567bcd5"
        },
        "sa-east-1": {
            "hvm": "ami-068a2000546e1889d"
        },
        "us-east-1": {
            "hvm": "ami-046fe691f52a953f9"
        },
        "us-east-2": {
            "hvm": "ami-0649fd5d42859bdfc"
        },
        "us-west-1": {
            "hvm": "ami-0c1d2b5606111ac8c"
        },
        "us-west-2": {
            "hvm": "ami-00745fcbb14a863ed"
        }
    },
    "baseURI": "https://releases-art-rhcos.svc.ci.openshift.org/art/storage/releases/ootpa/410.8.20190520.0/",
    "buildid": "410.8.20190520.0",
    "images": {
        "aws": {
            "path": "rhcos-410.8.20190520.0-aws.vmdk",
            "sha256": "71f4190916b5cb26b0230adb54809b10f8f694245ec3e743d1773efc0359f027",
            "size": "709180655",
            "uncompressed-sha256": "eef6764cc8f571a87fff715430220711bd0a90f4f88cc6141cfdbd412b654f81",
            "uncompressed-size": 742703616
        },
        "initramfs": {
            "path": "rhcos-410.8.20190520.0-installer-initramfs.img",
            "sha256": "9cbbdc92b687f87b511191ea866fc8dd080598954bd1a9ab7ed7157ac6a2adcc"
        },
        "iso": {
            "path": "rhcos-410.8.20190520.0-installer.iso",
            "sha256": "92fdbc5c95410fea73c6b200f6a10d4a8e043dd6933628237a5aed093f88701b"
        },
        "kernel": {
            "path": "rhcos-410.8.20190520.0-installer-kernel",
            "sha256": "fc984907fdd3674f8412e3ba5d9cdad461f04b7f9ea5e2ca142e3292997ed6bb"
        },
        "metal-bios": {
            "path": "rhcos-410.8.20190520.0-metal-bios.raw.gz",
            "sha256": "662fe69b6db719717d36a547828eb69b8f6a7787a99bcd4fdce408e083fc2ef4",
            "size": "719954527",
            "uncompressed-sha256": "312bcfca4ea966ee5167ac0c63f795768e7eb421a751e2d3d189b2e2ddd3d3e9",
            "uncompressed-size": "3317694464"
        },
        "metal-uefi": {
            "path": "rhcos-410.8.20190520.0-metal-uefi.raw.gz",
            "sha256": "ecf494ff4ef8c709a089e72a164e675739bc92c833f435d2b639285017c2004d",
            "size": "721567045",
            "uncompressed-sha256": "eb98b9d2e527117e692c4b7abe4be72bea9e7705bd572924155e62337bafccda",
            "uncompressed-size": "3317694464"
        },
        "openstack": {
            "path": "rhcos-410.8.20190520.0-openstack.qcow2",
            "sha256": "f2c22be84f262c60dfd75f0696987af683d1d384bb13b0cf95dc920986703dd6",
            "size": "719500193",
            "uncompressed-sha256": "280b1d7dce6bc67ffc03569b242ba741dbf2c7b645bdcb4e0f20b71792c45e77",
            "uncompressed-size": "2018967552"
        },
        "qemu": {
            "path": "rhcos-410.8.20190520.0-qemu.qcow2",
            "sha256": "d22b308631bf9a13329f402fbb574dd4d07005a3c4c5d6f1505ecf2246a86676",
            "size": "719492230",
            "uncompressed-sha256": "e92ecd8ebe0c06a58cecf004733cf12fc835650beb2a6914c672d57e82d9011a",
            "uncompressed-size": "2018902016"
        },
        "vmware": {
            "path": "rhcos-410.8.20190520.0-vmware.ova",
            "sha256": "eff088ec361f94258a4cd9400e8483b3fc14be7e205e645b372003e77de7ec3a",
            "size": "742717440"
        }
    },
    "oscontainer": {
        "digest": "sha256:53389c9b4a00d7afebb98f7bd9d20348deb1d77ca4baf194f0ae1b582b7e965b",
        "image": "quay.io/openshift-release-dev/ocp-v4.0-art-dev"
    },
    "ostree-commit": "d969dcf9e106044ba42a7dca0d445f5c2f1ba755276f433b4d166e6297627254",
    "ostree-version": "410.8.20190520.0"
}
$

We can't easily right now do the "download" part without having an online service which maps from e.g. ostree-version to OpenShift versions.

@cgwalters cgwalters mentioned this issue Jul 1, 2019
Closed
@cgwalters
Copy link
Member Author

Maybe...what we should do for this is treat RHCOS the same as the release image - and particularly for official installer releases we binary patch the binary so that it refers to https://mirror.openshift.com/pub/openshift-v4/dependencies/rhcos/ or so.

@sferich888
Copy link
Contributor

@abhinavdahiya what do we need to do to progress this as I see this as a huge win for our user experience in a few IPI use cases and mainly in all UPI use cases (as it helps simplify much of the complicated and cumbersome documentation points we have now).

@cgwalters
Copy link
Member Author

Another appoach here is to expand on the "redirector service" model that we already use on the RHCOS side. A team like SD would run a dynamic service mirror.openshift.com/pub/openshift-v4/dependencies/rhcos?sha256=ff088ec361f94258a4cd9400e8483b3fc14be7e205e645b372003e77de7ec3a.

To avoid duplication, the existing "static" content would just become redirects to the dynamic one which fetched by sha256sum.

To say this another way, there would be a unified way to fetch bootimage binaries that is exactly the same between dev and release - which is part of achieving the important goal of making releases just like "git tag" and not a special series of steps.

@cgwalters
Copy link
Member Author

xref https://bugzilla.redhat.com/show_bug.cgi?id=1730910

cgwalters added a commit to cgwalters/installer that referenced this issue Jul 25, 2019
@cgwalters
Add new rhcos-metadata command
2a324b0 
Today the installer pins RHCOS, and that's not likely
to change.  The embedded version isn't easy to know in
pre-release/development scenarios.

Further, the way we do releases via CI, we don't know
what the "re-branded OpenShift version" (e.g. 4.1.0)
will be until it's released.

For other things like the AMIs, today the documentation
writers copy them into the docs manually.

Since the installer is the canonical source of this data,
let's make it easy to extract.

We're not documenting this much yet, but having this
data easily available is going to be extremely helpful
for the variety of cases mentioned above and allow
other teams to have a canonical way to get it.

Related: openshift#1399
@cgwalters cgwalters mentioned this issue Jul 25, 2019
Closed
This was referenced Oct 7, 2019
Merged
Merged
@cgwalters
Copy link
Member Author

This was auto-linked from Github but just to clarify, I tried a small PR related to this in
#2092
that didn't get merged as of yet.

@cgwalters cgwalters mentioned this issue Oct 21, 2019
Open
@yuqi-zhang yuqi-zhang mentioned this issue Nov 4, 2019
Merged
@hardys hardys mentioned this issue Nov 22, 2019
Closed
@abhinavdahiya
Copy link
Contributor

closing this in favor of openshift/enhancements#201

/close

@openshift-ci-robot
Copy link
Contributor

@abhinavdahiya: Closing this issue.

In response to this:

closing this in favor of openshift/enhancements#201

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@wking @cgwalters @sferich888 @abhinavdahiya @openshift-ci-robot