Add KubeVirt containerdisk build and upload #2750

rmohr · 2022-03-11T13:12:03Z

Create a KubeVirt containerdisk ociarchive and upload it to arbitrary container registries. buildah is used for building the ociarchive, skopeo is used to push the image to a container registry.

Build the ociarchive

$ cosa buildextend-kubevirt
[...]
+ qemu-img convert -f qcow2 -O qcow2 /srv/tmp/tmpjibmiol_/fedora-coreos-35.20220401.dev.0-qemu.x86_64.qcow2.working /srv/tmp/tmpjibmiol_/fedora-coreos-35.20220401.dev.0-kubevirt.x86_64.qcow2
+ qemu-img info --output=json /srv/tmp/tmpjibmiol_/fedora-coreos-35.20220401.dev.0-kubevirt.x86_64.qcow2
[INFO]: Processing work image callback
+ buildah from scratch             
+ buildah add --chmod 0555 working-container /srv/tmp/tmpjibmiol_/fedora-coreos-35.20220401.dev.0-kubevirt.x86_64.qcow2 /disk/coreos.img             
c386adf15887856a35dfbae95eb8983c6ac173e697515be147ca7332cf13e071
+ buildah commit working-container           
Getting image source signatures
Copying blob d1f3d1874ef8 done  
Copying config dd323e4c49 done  
Writing manifest to image destination
Storing signatures
+ buildah push --format oci dd323e4c49fc353503a5c80e265025c144c2a31e8ce5a4493b7471263c7ab3e6 oci-archive:/srv/builds/35.20220401.dev.0/x86_64/fedora-coreos-35.20220401.dev.0-kubevirt.x86_64.ociarchive           
Getting image source signatures
Copying blob d1f3d1874ef8 done  
Copying config dd323e4c49 done  
Writing manifest to image destination
Storing signatures
[INFO]: Calculating metadata for fedora-coreos-35.20220401.dev.0-kubevirt.x86_64.ociarchive
[INFO]: Finished building artifacts
[INFO]: finishing ore processing
+ rc=0
+ set +x

Upload the OCI archive

[rmohr@fedora fcos]$ cosa buildextend-kubevirt --upload --repository "quay.io/rmohr"
[...]
[INFO]: Processed build for: Fedora CoreOS testing-devel (FEDORA-COREOS-x86_64) 35.20220401.dev.0
[INFO]: operating on fedora-coreos-35.20220401.dev.0-kubevirt.x86_64.ociarchive
[INFO]: executing upload commands for ore
+ skopeo inspect oci-archive:builds/35.20220401.dev.0/x86_64/fedora-coreos-35.20220401.dev.0-kubevirt.x86_64.ociarchive -f {{.Digest}}
[INFO]: pushing quay.io/rmohr/fedora-coreos:35.20220401.dev.0-x86_64 with digest sha256:2040a78a66d3b2189d6511c45a0cf4eedb21db556338056cd18c3c3d42543810
+ skopeo copy oci-archive:builds/35.20220401.dev.0/x86_64/fedora-coreos-35.20220401.dev.0-kubevirt.x86_64.ociarchive docker://quay.io/rmohr/fedora-coreos:35.20220401.dev.0-x86_64
Getting image source signatures
Copying blob 1e305a849202 done  
Copying config dd323e4c49 done  
Writing manifest to image destination
Storing signatures
[INFO]: finishing ore processing
+ rc=0
+ set +x

Generate and inspect metadata

[rmohr@fedora fcos]$ cosa generate-release-meta
COREOS_ASSEMBLER_CONTAINER=localhost/coreos-assembler
COREOS_ASSEMBLER_GIT=/home/rmohr/coreos-assembler
+ podman run --rm -ti --security-opt label=disable --privileged --uidmap=1000:0:1 --uidmap=0:1:1000 --uidmap 1001:1001:64536 -v /home/rmohr/fcos:/srv/ --device /dev/kvm --device /dev/fuse -v /home/rmohr/.docker/config.json:/home/builder/a
Creating release.json for build 35.20220401.dev.0
testing-devel stream
  x86_64 images:
   - kubevirt
   - qemu
Successfully wrote release file at builds/35.20220401.dev.0/release.json
+ rc=0
+ set +x
[rmohr@fedora fcos]$ cat builds/35.20220401.dev.0/release.json | jq
{
  "release": "35.20220401.dev.0",
  "stream": "testing-devel",
  "architectures": {
    "x86_64": {
      "media": {
        "kubevirt": {
          "artifacts": {
            "ociarchive": {
              "disk": {
                "location": "https://builds.coreos.fedoraproject.org/prod/streams/testing-devel/builds/35.20220401.dev.0/x86_64/fedora-coreos-35.20220401.dev.0-kubevirt.x86_64.ociarchive",
                "signature": "https://builds.coreos.fedoraproject.org/prod/streams/testing-devel/builds/35.20220401.dev.0/x86_64/fedora-coreos-35.20220401.dev.0-kubevirt.x86_64.ociarchive.sig",
                "sha256": "d24d902a7729a81f351f3cb040891a91d7f1ab7a6c5bb2a4ddeddb02e16506f6",
                "uncompressed-sha256": null
              }
            }
          },
          "image": {
            "image": "quay.io/rmohr/fedora-coreos@sha256:2040a78a66d3b2189d6511c45a0cf4eedb21db556338056cd18c3c3d42543810"
          }
        },
[...]
}

openshift-ci · 2022-03-11T13:12:14Z

Hi @rmohr. Thanks for your PR.

I'm waiting for a coreos member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

rmohr · 2022-03-11T13:24:37Z

/cc @miabbott
/cc @cgwalters

Would be happy to get your feedback on this (@cgwalters due to the interest in publishing for fcos). The build and the push seems to work as expected. I plan to tackle the rhcos.json modification in a follow-up.

rmohr · 2022-03-11T13:30:27Z

/cc @davidvossel FYI

openshift-ci · 2022-03-11T13:30:29Z

@rmohr: GitHub didn't allow me to request PR reviews from the following users: davidvossel.

Note that only coreos members and repo collaborators can review this PR, and authors cannot review their own PRs.

In response to this:

/cc @davidvossel

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

miabbott · 2022-03-11T13:56:14Z

/ok-to-test

rmohr · 2022-03-11T14:22:02Z

/retest

rmohr · 2022-03-11T15:48:29Z

/retest

schema/cosa/schema_doc.go

miabbott · 2022-03-11T20:12:06Z

/retest

jlebon · 2022-03-14T19:14:11Z

Is there a link with more background on this? Is this part of an OCP epic?

miabbott · 2022-03-14T20:07:11Z

@jlebon This artifact is useful for OCP Virtualization + Hypershift: https://issues.redhat.com/browse/COS-1232

See also: https://issues.redhat.com/browse/RFE-2501

rmohr · 2022-03-15T13:21:01Z

/hold

~~needs coreos/stream-metadata-go#41 merged and referenced first.~~

Edit: Need an official release tag, but referencing now the merged code already.

mantle/cmd/ore/kubevirt/build.go

cgwalters · 2022-03-15T15:36:33Z

mantle/go.mod

@@ -1,56 +1,107 @@
 module github.com/coreos/mantle

-go 1.12
+go 1.17


This is probably worth factoring out as a separate preparatory commit.

Do you mean a preparatory PR, or just creating a dedicated commit with the vendoring in a follow-up commit to make it more visible?

Separate commit in the same PR should be fine.

cgwalters · 2022-03-15T15:36:45Z

mantle/go.mod

+	github.com/godbus/dbus/v5 v5.0.4 // indirect
+	github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da // indirect
+	github.com/golang/protobuf v1.5.2 // indirect
+	github.com/google/go-containerregistry v0.8.0


Offhand this library looks sane, but it also duplicates a lot of what we already have vendored in the repositories from github.com/containers.

(Now personally, I would say we should stop vendoring all of podman in this repo, but that's a different discussion)

A possible alternative is to fork off buildah; we already do that for the other containers.

One specific reason to do that instead of using this library is to ensure that we have a single code path for e.g. authentication, and fewer places we need to change for things like "upload in oci format" (xref #2726 ). I'm guessing go-containerregistry only does v2s2?

But to be clear, I'm OK with this as is.

(Now personally, I would say we should stop vendoring all of podman in this repo, but that's a different discussion)

A possible alternative is to fork off buildah; we already do that for the other containers.

If you do that already I can definitely change it. I missed that you already build containers in go in coreos-assembler.

But to be clear, I'm OK with this as is.

Oh nice :)

bgilbert · 2022-03-15T19:25:49Z

stream-metadata-go update in #2754.

jlebon

LGTM!

jlebon · 2022-04-11T14:51:00Z

(I restarted CI so that it builds on top of #2808, which will verify that the schema was updated correctly.)

rmohr · 2022-04-11T15:18:26Z

(I restarted CI so that it builds on top of #2808, which will verify that the schema was updated correctly.)

And it detected issues. Fixing. 👍

Build the containerdisk based on a qcow2 image and wrap it into an oci archive as final artifact. This artifact can be uploaded to an arbitrary container registry with the upload step. Signed-off-by: Roman Mohr <rmohr@redhat.com>

Signed-off-by: Roman Mohr <rmohr@redhat.com>

rmohr · 2022-04-11T15:33:25Z

Can you also add it as an artifact to the .cci.jenkinsfile so it's tested by CI?

@jlebon can it be that the pipeline changes only get picked up after a merge?

jlebon · 2022-04-11T16:03:56Z

Can you also add it as an artifact to the .cci.jenkinsfile so it's tested by CI?

@jlebon can it be that the pipeline changes only get picked up after a merge?

Ahh yup, indeed. Jenkins is wary of you. :)

11:30:07  Loading trusted files from base branch main at f6991e5fedf40426b82f9c296bd6760dce8a7769 rather than 808b53c5ddaa6c4d71990f49da643c952f429b37
11:30:07  Obtained .cci.jenkinsfile from f6991e5fedf40426b82f9c296bd6760dce8a7769
11:30:07  ‘.cci.jenkinsfile’ has been modified in an untrusted revision

I think we just need to add you to the org. Let me sort that out.

Signed-off-by: Roman Mohr <rmohr@redhat.com>

jlebon · 2022-04-11T21:28:02Z

[2022-04-11T17:29:59.796Z] error adding content to container "working-container": error mounting build
container "4b745604dcfcf72d6a4b27ade77c70a8b194827be838510e0961670bfce60167": error creating overlay m
ount to /home/jenkins/agent/workspace/coreos-assembler_PR-2750/.local/share/containers/storage/overlay
/5fa288fbb3d22588f79bc8c7925e4f555e56f893aedbe2a2c2c95b0df4c107f5/merged, mount_data=",lowerdir=/home/
jenkins/agent/workspace/coreos-assembler_PR-2750/.local/share/containers/storage/overlay/5fa288fbb3d22
588f79bc8c7925e4f555e56f893aedbe2a2c2c95b0df4c107f5/empty,upperdir=/home/jenkins/agent/workspace/coreo
s-assembler_PR-2750/.local/share/containers/storage/overlay/5fa288fbb3d22588f79bc8c7925e4f555e56f893ae
dbe2a2c2c95b0df4c107f5/diff,workdir=/home/jenkins/agent/workspace/coreos-assembler_PR-2750/.local/shar
e/containers/storage/overlay/5fa288fbb3d22588f79bc8c7925e4f555e56f893aedbe2a2c2c95b0df4c107f5/work,vol
atile": using mount program /usr/bin/fuse-overlayfs: fuse: device not found, try 'modprobe fuse' first
[2022-04-11T17:29:59.796Z] fuse-overlayfs: cannot mount: No such file or directory
[2022-04-11T17:29:59.796Z] : exit status 1
[2022-04-11T17:29:59.796Z] time="2022-04-11T17:29:57Z" level=error msg="exit status 125"
[2022-04-11T17:29:59.796Z] Error running command buildah

OK right, this won't work. Hmm, but how does the existing buildah bits in the oscontainer code work in the RHCOS pipeline currently then? Ahhh OK, I think it's this bit:

coreos-assembler/src/oscontainer.py

Lines 119 to 121 in f6991e5

    
           if os.environ.get('container') is not None: 
        
               print("Using nested container mode due to container environment variable") 
        
               buildah_base_argv.extend(NESTED_BUILD_ARGS)

coreos-assembler/src/oscontainer.py

Line 27 in f6991e5

NESTED_BUILD_ARGS = ['--storage-driver', 'vfs']

@rmohr Can you try adding that to write_oci?

Ensure that buildah is called for kubevirt with the same base arguments like for the oscontainer. Signed-off-by: Roman Mohr <rmohr@redhat.com>

rmohr · 2022-04-12T12:37:51Z

@rmohr Can you try adding that to write_oci?

That did the trick! 👍

jlebon

Nice work!

openshift-ci bot added the needs-rebase label Mar 11, 2022

openshift-ci bot added the needs-ok-to-test label Mar 11, 2022

rmohr force-pushed the kubevirt-build-and-upload branch from e30b451 to b9d79f1 Compare March 11, 2022 13:13

openshift-ci bot removed the needs-rebase label Mar 11, 2022

openshift-ci bot requested review from cgwalters and miabbott March 11, 2022 13:24

openshift-ci bot added ok-to-test and removed needs-ok-to-test labels Mar 11, 2022

rmohr force-pushed the kubevirt-build-and-upload branch from b9d79f1 to ef90ea4 Compare March 11, 2022 14:01

rmohr force-pushed the kubevirt-build-and-upload branch from ef90ea4 to e7318d3 Compare March 11, 2022 14:47

travier reviewed Mar 11, 2022

View reviewed changes

schema/cosa/schema_doc.go Outdated Show resolved Hide resolved

rmohr force-pushed the kubevirt-build-and-upload branch from e7318d3 to 48913b9 Compare March 14, 2022 09:43

This was referenced Mar 14, 2022

Add KubeVirt platform artifact and image information coreos/stream-metadata-go#41

Merged

[kubevirt] Add auto-detection of the correct RHCOS base image openshift/hypershift#994

Closed

rmohr force-pushed the kubevirt-build-and-upload branch from 177ea8a to 2c330e8 Compare March 14, 2022 16:10

openshift-ci bot added the do-not-merge/hold label Mar 15, 2022

rmohr force-pushed the kubevirt-build-and-upload branch from 4c046fa to a8ddadd Compare March 15, 2022 14:30

cgwalters reviewed Mar 15, 2022

View reviewed changes

openshift-ci bot added the needs-rebase label Apr 9, 2022

rmohr dismissed jlebon’s stale review via a87ed90 April 11, 2022 13:31

rmohr force-pushed the kubevirt-build-and-upload branch from f72977d to a87ed90 Compare April 11, 2022 13:31

openshift-ci bot removed the needs-rebase label Apr 11, 2022

jlebon previously approved these changes Apr 11, 2022

View reviewed changes

jlebon enabled auto-merge (rebase) April 11, 2022 14:51

rmohr added 4 commits April 11, 2022 17:21

Implement build and upload of KubeVirt containerdisks

997f17a

Build the containerdisk based on a qcow2 image and wrap it into an oci archive as final artifact. This artifact can be uploaded to an arbitrary container registry with the upload step. Signed-off-by: Roman Mohr <rmohr@redhat.com>

Document providing KubeVirt secrets for uploads

99680c2

Signed-off-by: Roman Mohr <rmohr@redhat.com>

Update schemata and populate the kubevirt release section

1da8019

Signed-off-by: Roman Mohr <rmohr@redhat.com>

Test KubeVirt builds on CI

dbcf8d1

Signed-off-by: Roman Mohr <rmohr@redhat.com>

auto-merge was automatically disabled April 11, 2022 15:29
Head branch was pushed to by a user without write access

rmohr dismissed jlebon’s stale review via 808b53c April 11, 2022 15:29

rmohr force-pushed the kubevirt-build-and-upload branch from 8cbfdec to 808b53c Compare April 11, 2022 15:29

Add missing -y to yum install in Dockerfile build

6f7af75

Signed-off-by: Roman Mohr <rmohr@redhat.com>

rmohr force-pushed the kubevirt-build-and-upload branch from 808b53c to 6f7af75 Compare April 11, 2022 16:52

jlebon previously approved these changes Apr 11, 2022

View reviewed changes

jlebon enabled auto-merge (rebase) April 11, 2022 16:54

jlebon disabled auto-merge April 11, 2022 21:28

Pick up buildah base build arguments for container builds

1acb609

Ensure that buildah is called for kubevirt with the same base arguments like for the oscontainer. Signed-off-by: Roman Mohr <rmohr@redhat.com>

rmohr dismissed jlebon’s stale review via 1acb609 April 12, 2022 07:35

jlebon approved these changes Apr 12, 2022

View reviewed changes

jlebon merged commit 6b50db1 into coreos:main Apr 12, 2022

cgwalters mentioned this pull request Jun 7, 2022

schema: Add entry for base-oscontainer #2907

Merged

cgwalters mentioned this pull request Oct 30, 2023

Consume OCI images for machine image containers/podman#20538

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add KubeVirt containerdisk build and upload #2750

Add KubeVirt containerdisk build and upload #2750

rmohr commented Mar 11, 2022 •

edited

Loading

openshift-ci bot commented Mar 11, 2022

rmohr commented Mar 11, 2022

rmohr commented Mar 11, 2022 •

edited

Loading

openshift-ci bot commented Mar 11, 2022

miabbott commented Mar 11, 2022

rmohr commented Mar 11, 2022

rmohr commented Mar 11, 2022

miabbott commented Mar 11, 2022

jlebon commented Mar 14, 2022

miabbott commented Mar 14, 2022

rmohr commented Mar 15, 2022 •

edited

Loading

cgwalters Mar 15, 2022

rmohr Mar 15, 2022

bgilbert Mar 15, 2022

rmohr Mar 16, 2022

cgwalters Mar 15, 2022

cgwalters Mar 15, 2022

rmohr Mar 15, 2022

rmohr Mar 15, 2022

bgilbert commented Mar 15, 2022

jlebon left a comment

jlebon commented Apr 11, 2022

rmohr commented Apr 11, 2022

rmohr commented Apr 11, 2022

jlebon commented Apr 11, 2022

jlebon commented Apr 11, 2022

rmohr commented Apr 12, 2022

jlebon left a comment

Add KubeVirt containerdisk build and upload #2750

Add KubeVirt containerdisk build and upload #2750

Conversation

rmohr commented Mar 11, 2022 • edited Loading

openshift-ci bot commented Mar 11, 2022

rmohr commented Mar 11, 2022

rmohr commented Mar 11, 2022 • edited Loading

openshift-ci bot commented Mar 11, 2022

miabbott commented Mar 11, 2022

rmohr commented Mar 11, 2022

rmohr commented Mar 11, 2022

miabbott commented Mar 11, 2022

jlebon commented Mar 14, 2022

miabbott commented Mar 14, 2022

rmohr commented Mar 15, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bgilbert commented Mar 15, 2022

jlebon left a comment

Choose a reason for hiding this comment

jlebon commented Apr 11, 2022

rmohr commented Apr 11, 2022

rmohr commented Apr 11, 2022

jlebon commented Apr 11, 2022

jlebon commented Apr 11, 2022

rmohr commented Apr 12, 2022

jlebon left a comment

Choose a reason for hiding this comment

rmohr commented Mar 11, 2022 •

edited

Loading

rmohr commented Mar 11, 2022 •

edited

Loading

rmohr commented Mar 15, 2022 •

edited

Loading