podvm: update qemu for rhel s390x #1741

Saripalli-lavanya · 2024-03-13T12:15:29Z

Packer is picking qemu-system-x86 instead of s390x as qemu binaries for rhel podvm image build

2024/03/13 11:36:43 machine readable: error-count []string{"1"}
==> Some builds didn't complete successfully and had errors:
2024/03/13 11:36:43 machine readable: qemu.rhel,error []string{"Failed creating Qemu driver: exec: \"qemu-system-x86_64\": executable file not found in $PATH"}
==> Builds finished but no artifacts were created.
2024/03/13 11:36:43 [INFO] (telemetry) Finalizing.
Build 'qemu.rhel' errored after 793 microseconds: Failed creating Qemu driver: exec: "qemu-system-x86_64": executable file not found in $PATH

==> Wait completed after 900 microseconds

==> Some builds didn't complete successfully and had errors:
--> qemu.rhel: Failed creating Qemu driver: exec: "qemu-system-x86_64": executable file not found in $PATH

==> Builds finished but no artifacts were created.
2024/03/13 11:36:43 waiting for all plugin processes to complete...
2024/03/13 11:36:43 /usr/local/bin/packer: plugin process exited
2024/03/13 11:36:43 /usr/local/bin/packer: plugin process exited
2024/03/13 11:36:43 /usr/local/bin/packer: plugin process exited
2024/03/13 11:36:43 /usr/local/bin/packer: plugin process exited
2024/03/13 11:36:43 /usr/local/bin/packer: plugin process exited
2024/03/13 11:36:43 /usr/local/bin/packer: plugin process exited
2024/03/13 11:36:43 /usr/local/bin/packer: plugin process exited
2024/03/13 11:36:43 /usr/local/bin/packer: plugin process exited
2024/03/13 11:36:43 /usr/local/bin/packer: plugin process exited
2024/03/13 11:36:43 /usr/local/bin/packer: plugin process exited
make: *** [Makefile:77: podvm-e5a6fb8-dirty-s390x.qcow2] Error 1

Fixes: #1740

stevenhorsman · 2024-03-15T10:04:28Z

@bpradipt @wainersm - IIRC we don't have any current testing of the RHEL based image in open-source? Do you have any internal tests that we can check this with?

stevenhorsman · 2024-03-18T09:53:38Z

@Saripalli-lavanya - can you post the command you used (I guess ARCH=s390x PODVM_DISTRO=rhel make podvm-builder podvm-binaries podvm-image?) and a summary of the output that you tested it with as that will help reviewers understand the testing of the PR better. Thanks

Saripalli-lavanya · 2024-03-18T14:05:49Z

Hi @stevenhorsman , i've used below commands as per redhat doc here. on an s390x machine.

podman build -t podvm_builder_rhel_s390x:test-1 --build-arg ORG_ID=$ORG_ID --build-arg ACTIVATION_KEY=$ACTIVATION_KEY --build-arg ARCH="s390x" --build-arg GO_VERSION="1.21.3" --build-arg PROTOC_VERSION="25.1" --build-arg PACKER_VERSION="v1.9.4" --build-arg RUST_VERSION="1.72.0" --build-arg YQ_VERSION="v4.35.1" --build-arg YQ_CHECKSUM="sha256:4e6324d08630e7df733894a11830412a43703682d65a76f1fc925aac08268a45" -f podvm/Dockerfile.podvm_builder.rhel .
podman build -t podvm_binaries_rhel_s390x:test-1 --build-arg BUILDER_IMG="podvm_builder_rhel_s390x:test-1" --build-arg ARCH=s390x -f podvm/Dockerfile.podvm_binaries.rhel .
podman build -t podvm_rhel_s390x --build-arg ARCH=s390x --build-arg CLOUD_PROVIDER=libvirt --build-arg BUILDER_IMG="localhost/podvm_builder_rhel_s390x:test-1" --build-arg BINARIES_IMG="localhost/podvm_binaries_rhel_s390x:test-1" -v ${IMAGE_URL}:/tmp/rhel.qcow2:Z --build-arg IMAGE_URL="/tmp/rhel.qcow2" --build-arg IMAGE_CHECKSUM=${IMAGE_CHECKSUM} -f podvm/Dockerfile.podvm.rhel .

Note: podman doesn't support ADD --checksum=${YQ_CHECKSUM} flag so I have updated Dockerfile.podvm_builder.rhel file with below line in my local fork

RUN current_checksum=$(sha256sum /usr/local/bin/yq | cut -d' ' -f1) && if [ "$current_checksum" != "$YQ_CHECKSUM" ]; then echo "Checksum verification failed"; exit 1; fi

Below is the packer build command from the logs of podvm_rhel_s390x image build.

packer build -var qemu_image_name=podvm-22da7b0-dirty-s390x.qcow2 -var cloud_image_url=/tmp/rhel.qcow2 -var cloud_image_checksum=9fdcceceb6bce1697117444ba683c74c98b5b42d70630744cc499af2b0af822a -var disk_size=11144 -var machine_type=s390-ccw-virtio -var qemu_binary=qemu-system-s390x qcow2/rhel
Warning: Bundled plugins used

This template relies on the use of plugins bundled into the Packer binary.
The practice of bundling external plugins into Packer will be removed in an
upcoming version.

To remove this warning, add the following section to your template:

packer {
  required_plugins {
    qemu = {
      source  = "github.com/hashicorp/qemu"
      version = "~> 1"
    }
  }
}

Then run 'packer init' to manage installation of the plugins


qemu.rhel: output will be in this color.

==> qemu.rhel: Retrieving ISO
==> qemu.rhel: Trying /tmp/rhel.qcow2
==> qemu.rhel: Trying /tmp/rhel.qcow2?checksum=sha256%3A9fdcceceb6bce1697117444ba683c74c98b5b42d70630744cc499af2b0af822a
==> qemu.rhel: /tmp/rhel.qcow2?checksum=sha256%3A9fdcceceb6bce1697117444ba683c74c98b5b42d70630744cc499af2b0af822a => /tmp/rhel.qcow2
    qemu.rhel: File extension already matches desired output format. Skipping qemu-img convert step
==> qemu.rhel: Resizing hard drive...
==> qemu.rhel: Found port for communicator (SSH, WinRM, etc): 3611.
==> qemu.rhel: Looking for available port between 5900 and 6000 on 127.0.0.1
==> qemu.rhel: Starting VM, booting disk image
    qemu.rhel: The VM will be run headless, without a GUI. If you want to
    qemu.rhel: view the screen of the VM, connect via VNC without a password to
    qemu.rhel: vnc://127.0.0.1:5990
==> qemu.rhel: Overriding default Qemu arguments with qemuargs template option...
    qemu.rhel: Not using a NetBridge -- skipping StepWaitGuestAddress
==> qemu.rhel: Using SSH communicator to connect: 127.0.0.1
==> qemu.rhel: Waiting for SSH to become available...
==> qemu.rhel: Connected to SSH!
==> qemu.rhel: Running local shell script: /tmp/packer-shell2351065329
==> qemu.rhel: Uploading ./toupload => /tmp/
==> qemu.rhel: Provisioning with shell script: /tmp/packer-shell1119613630
==> qemu.rhel: Uploading qcow2/copy-files.sh => ~/copy-files.sh
==> qemu.rhel: Provisioning with shell script: /tmp/packer-shell1509913767
==> qemu.rhel: Uploading qcow2/selinux_relabel.sh => ~/selinux_relabel.sh
==> qemu.rhel: Provisioning with shell script: /tmp/packer-shell3567759185
    qemu.rhel:
    qemu.rhel:
    qemu.rhel:
==> qemu.rhel: Uploading qcow2/misc-settings.sh => ~/misc-settings.sh
==> qemu.rhel: Provisioning with shell script: /tmp/packer-shell40305848
    qemu.rhel: Locking password for user peerpod.
    qemu.rhel: passwd: Success
==> qemu.rhel: Removed /etc/systemd/system/multi-user.target.wants/kdump.service.
==> qemu.rhel: Removed /etc/systemd/system/multi-user.target.wants/tuned.service.
==> qemu.rhel: Failed to disable unit: Unit file firewalld.service does not exist.
==> qemu.rhel: Gracefully halting virtual machine...
==> qemu.rhel: Converting hard drive...
Build 'qemu.rhel' finished after 9 minutes 14 seconds.

==> Wait completed after 9 minutes 14 seconds

==> Builds finished. The artifacts of successful builds are:
--> qemu.rhel: VM files in directory: output
rm -fr toupload
rm -f cloud-init.img
--> 45473f89dec2
[3/3] STEP 1/2: FROM scratch
[3/3] STEP 2/2: COPY --from=podvm_builder /src/cloud-api-adaptor/podvm/output/*.qcow2 /
[3/3] COMMIT podvm_rhel_s390x
--> 2f18dc13d2aa
Successfully tagged localhost/podvm_rhel_s390x:latest
2f18dc13d2aa9b868d73b41701a2f5658c8e621fc2868f6189a2aeb943aa648d

Below is the result:

[root@a3elp61 cloud-api-adaptor]# podman images
REPOSITORY                                                                              TAG               IMAGE ID      CREATED            SIZE
localhost/podvm_rhel_s390x                                                              latest            2f18dc13d2aa  13 minutes ago     1.4 GB
localhost/podvm_binaries_rhel_s390x                                                     test-1            b551e8152beb  2 hours ago        317 MB
localhost/podvm_builder_rhel_s390x                                                      test-1            50aea8143238  3 hours ago        7.03 GB

huoqifeng

LGTM, I think @wyuany and @Saripalli-lavanya both tested it on s390x OCP. Thanks! @Saripalli-lavanya

Saripalli-lavanya · 2024-03-21T06:32:55Z

Thank you @huoqifeng,

snir911 · 2024-03-21T07:44:19Z

podvm/qcow2/rhel/qemu-rhel.pkr.hcl

@@ -14,12 +14,13 @@ source "qemu" "rhel" {
  iso_checksum     = "${var.cloud_image_checksum}"
  iso_url          = "${var.cloud_image_url}"
  output_directory = "output"
-  qemuargs         = [["-m", "${var.memory}"], ["-smp", "cpus=${var.cpus}"], ["-cdrom", "${var.cloud_init_image}"], ["-serial", "mon:stdio"], ["-cpu", "Cascadelake-Server"]]


Did you have a chance to test with x86? i recall we needed to specify cpu type for CentOS otherwise it failed, see 90ccfc3
cc @bpradipt as he wrote this line

stevenhorsman · 2024-03-21T11:25:34Z

I might be worth noting that the Dockerfile.podvm_builder.rhel is not valid, so when I tried to test it based on open-source commands it failed as ${ARCH/x86_64/amd64} syntax is not POSIX compatible, so doesn't work with docker.

$ ARCH=s390x PODVM_DISTRO=rhel make podvm-builder podvm-binaries podvm-image
...
Dockerfile.podvm_builder.rhel:34
--------------------
  32 |         device-mapper-devel qemu-kvm unzip wget libassuan-devel genisoimage cloud-utils-growpart cloud-init;
  33 |
  34 | >>> ADD --checksum=${YQ_CHECKSUM} https://github.com/mikefarah/yq/releases/download/${YQ_VERSION}/yq_linux_${ARCH/x86_64/amd64} /usr/local/bin/yq
  35 |     RUN chmod a+x /usr/local/bin/yq && \
  36 |         curl https://dl.google.com/go/go${GO_VERSION}.linux-${ARCH/x86_64/amd64}.tar.gz -o go${GO_VERSION}.linux-${ARCH/x86_64/amd64}.tar.gz && \
--------------------
ERROR: failed to solve: failed to process "https://github.com/mikefarah/yq/releases/download/${YQ_VERSION}/yq_linux_${ARCH/x86_64/amd64}": missing ':' in substitution
make: *** [Makefile:210: podvm-builder] Error 1

So I think #1531 was not valid.

Saripalli-lavanya · 2024-03-25T12:09:28Z

I might be worth noting that the Dockerfile.podvm_builder.rhel is not valid, so when I tried to test it based on open-source commands it failed as ${ARCH/x86_64/amd64} syntax is not POSIX compatible, so doesn't work with docker.

$ ARCH=s390x PODVM_DISTRO=rhel make podvm-builder podvm-binaries podvm-image
...
Dockerfile.podvm_builder.rhel:34
--------------------
  32 |         device-mapper-devel qemu-kvm unzip wget libassuan-devel genisoimage cloud-utils-growpart cloud-init;
  33 |
  34 | >>> ADD --checksum=${YQ_CHECKSUM} https://github.com/mikefarah/yq/releases/download/${YQ_VERSION}/yq_linux_${ARCH/x86_64/amd64} /usr/local/bin/yq
  35 |     RUN chmod a+x /usr/local/bin/yq && \
  36 |         curl https://dl.google.com/go/go${GO_VERSION}.linux-${ARCH/x86_64/amd64}.tar.gz -o go${GO_VERSION}.linux-${ARCH/x86_64/amd64}.tar.gz && \
--------------------
ERROR: failed to solve: failed to process "https://github.com/mikefarah/yq/releases/download/${YQ_VERSION}/yq_linux_${ARCH/x86_64/amd64}": missing ':' in substitution
make: *** [Makefile:210: podvm-builder] Error 1

So I think #1531 was not valid.

Hi @stevenhorsman , could you please try with the latest code using below command?

for s390x :-

docker build -t podvm_builder_rhel_s390x \
  --build-arg ARCH="s390x" \
  --build-arg GO_VERSION="1.21.3" \
  --build-arg PROTOC_VERSION="25.1" \
  --build-arg PACKER_VERSION="v1.9.4" \
  --build-arg RUST_VERSION="1.72.0" \
  --build-arg YQ_VERSION="v4.35.1" \
  --build-arg YQ_CHECKSUM="4e6324d08630e7df733894a11830412a43703682d65a76f1fc925aac08268a45" \
  -f podvm/Dockerfile.podvm_builder.rhel .

for x86_64:-

podman build -t podvm_builder_rhel_amd64 \
  --build-arg ARCH="x86_64" \
  --build-arg GO_VERSION="1.21.3" \
  --build-arg PROTOC_VERSION="25.1" \
  --build-arg PACKER_VERSION="v1.9.4" \
  --build-arg RUST_VERSION="1.72.0" \
  --build-arg YQ_VERSION="v4.35.1" \
  --build-arg YQ_CHECKSUM="bd695a6513f1196aeda17b174a15e9c351843fb1cef5f9be0af170f2dd744f08" \
  -f podvm/Dockerfile.podvm_builder.rhel .

src/cloud-api-adaptor/podvm/Dockerfile.podvm_builder.rhel

bpradipt · 2024-03-25T12:51:08Z

src/cloud-api-adaptor/podvm/qcow2/rhel/qemu-rhel.pkr.hcl

@@ -14,12 +14,13 @@ source "qemu" "rhel" {
  iso_checksum     = "${var.cloud_image_checksum}"
  iso_url          = "${var.cloud_image_url}"
  output_directory = "output"
-  qemuargs         = [["-m", "${var.memory}"], ["-smp", "cpus=${var.cpus}"], ["-cdrom", "${var.cloud_init_image}"], ["-serial", "mon:stdio"], ["-cpu", "Cascadelake-Server"]]
+  qemuargs         = [["-m", "${var.memory}"], ["-smp", "cpus=${var.cpus}"], ["-cdrom", "${var.cloud_init_image}"], ["-serial", "mon:stdio"]]


Has this also been tested on Intel system? I recall issues when not specifying the cpu model

stevenhorsman · 2024-03-25T13:21:12Z

docker build -t podvm_builder_rhel_s390x \
  --build-arg ARCH="s390x" \
  --build-arg GO_VERSION="1.21.3" \
  --build-arg PROTOC_VERSION="25.1" \
  --build-arg PACKER_VERSION="v1.9.4" \
  --build-arg RUST_VERSION="1.72.0" \
  --build-arg YQ_VERSION="v4.35.1" \
  --build-arg YQ_CHECKSUM="4e6324d08630e7df733894a11830412a43703682d65a76f1fc925aac08268a45" \
  -f podvm/Dockerfile.podvm_builder.rhel .

This doesn't work for me, I assume it's something to do with needing a license server key?

Saripalli-lavanya · 2024-03-25T13:30:56Z

docker build -t podvm_builder_rhel_s390x \
  --build-arg ARCH="s390x" \
  --build-arg GO_VERSION="1.21.3" \
  --build-arg PROTOC_VERSION="25.1" \
  --build-arg PACKER_VERSION="v1.9.4" \
  --build-arg RUST_VERSION="1.72.0" \
  --build-arg YQ_VERSION="v4.35.1" \
  --build-arg YQ_CHECKSUM="4e6324d08630e7df733894a11830412a43703682d65a76f1fc925aac08268a45" \
  -f podvm/Dockerfile.podvm_builder.rhel .

This doesn't work for me, I assume it's something to do with needing a license server key?

ohh my bad, I missed that point we need to use RHEL subscribed machine or below command will work provided ORG_ID & ACTIVATION_KEY

podman build -t podvm_builder_rhel_s390x \
--build-arg ORG_ID=$ORG_ID \
--build-arg ACTIVATION_KEY=$ACTIVATION_KEY \
--build-arg ARCH="s390x" \
--build-arg GO_VERSION="1.21.3" \
--build-arg PROTOC_VERSION="25.1" \
--build-arg PACKER_VERSION="v1.9.4" \
--build-arg RUST_VERSION="1.72.0" \
--build-arg YQ_VERSION="v4.35.1" \
--build-arg YQ_CHECKSUM="4e6324d08630e7df733894a11830412a43703682d65a76f1fc925aac08268a45" \
-f podvm/Dockerfile.podvm_builder.rhel .

stevenhorsman · 2024-03-25T14:22:57Z

podman build -t podvm_builder_rhel_s390x
--build-arg ORG_ID=$ORG_ID
--build-arg ACTIVATION_KEY=$ACTIVATION_KEY
--build-arg ARCH="s390x"
--build-arg GO_VERSION="1.21.3"
--build-arg PROTOC_VERSION="25.1"
--build-arg PACKER_VERSION="v1.9.4"
--build-arg RUST_VERSION="1.72.0"
--build-arg YQ_VERSION="v4.35.1"
--build-arg YQ_CHECKSUM="4e6324d08630e7df733894a11830412a43703682d65a76f1fc925aac08268a45"
-f podvm/Dockerfile.podvm_builder.rhel .

I have the pdovm builder working with docker, not podman, for both s390x and x86, but I'm not really happy about overriding all the build-args. We have a versions.yaml that is supposed to set the go, rust etc versions, that are supported and tested with the latest code, so I'd rather we get this working in the standard way with ARCH=s390x PODVM_DISTRO=rhel make podvm-builder podvm-binaries podvm-image, or at least the minimal set of build-args?

Saripalli-lavanya · 2024-03-28T06:36:02Z

@huoqifeng, @stevenhorsman, @snir911, @bpradipt Thank you for reviewing the PR, Could you please review the latest changes and suggest?

@stevenhorsman with these latest changes I was able to build image using command make podvm-builder podvm-binaries podvm-image PODVM_DISTRO=rhel PODVM_TAG=latest ORG_ID=**** ACTIVATION_KEY="***" CLOUD_PROVIDER=libvirt ARCH="x86_64" IMAGE_URL=rhel-9.2-x86_64-kvm.qcow2 Could you please try now? Note: i have downloaded rhel-9.2-x86_64-kvm.qcow2 from [here](url: https://access.redhat.com/downloads/content/479/ver=/rhel---9/9.2/x86_64/product-software) into podvm directory

Saripalli-lavanya · 2024-03-28T06:41:54Z

Test Results for amd64 podvm image:

PODVM_DISTRO=rhel make podvm-builder podvm-binaries podvm-image PODVM_TAG=latest ORG_ID=**** ACTIVATION_KEY="***"  CLOUD_PROVIDER=libvirt ARCH="x86_64" IMAGE_URL=rhel-9.2-x86_64-kvm.qcow2
make: sha256sum: Command not found
docker buildx build -t quay.io/confidential-containers/podvm-builder-rhel:latest -f podvm/Dockerfile.podvm_builder.rhel \
    --build-arg GO_VERSION=1.21.8 \
    --build-arg ORG_ID=**** \
    --build-arg ARCH=x86_64 \
    --build-arg ACTIVATION_KEY=*** \
    --build-arg PACKER_VERSION=v1.9.4 \
    --build-arg PROTOC_VERSION=3.15.0 \
    --build-arg RUST_VERSION=1.72.0 \
    --build-arg YQ_VERSION=v4.35.1 \
    --build-arg YQ_CHECKSUM="sha256:bd695a6513f1196aeda17b174a15e9c351843fb1cef5f9be0af170f2dd744f08" \
    --load  .
[+] Building 7.1s (18/18) FINISHED                                                                                                         docker:rancher-desktop
 => [internal] load build definition from Dockerfile.podvm_builder.rhel                                                                                      0.0s
 => => transferring dockerfile: 3.25kB                                                                                                                       0.0s
 => [internal] load .dockerignore                                                                                                                            0.0s
 => => transferring context: 254B                                                                                                                            0.0s
 => resolve image config for docker.io/docker/dockerfile:1.5-labs                                                                                            4.4s
 => CACHED docker-image://docker.io/docker/dockerfile:1.5-labs@sha256:f2e91734a84c0922ff47aa4098ab775f1dfa932430d2888dd5cad5251fafdac4                       0.0s
 => [internal] load metadata for registry.access.redhat.com/ubi9/ubi:9.2                                                                                     2.5s
 => [ 1/12] FROM registry.access.redhat.com/ubi9/ubi:9.2@sha256:2f168398c538b287fd705519b83cd5b604dc277ef3d9f479c28a2adb4d830a49                             0.0s
 => CACHED [ 2/12] RUN if [[ -n "****" && -n "***" ]]; then     subscription-manager register --org=*** --activationkey=****;     fi       0.0s
 => CACHED [ 3/12] RUN subscription-manager repos --enable codeready-builder-for-rhel-9-x86_64-rpms;     dnf groupinstall -y 'Development Tools';     dnf i  0.0s
 => CACHED [ 4/12] RUN curl -L -o /usr/local/bin/yq https://github.com/mikefarah/yq/releases/download/${YQ_VERSION}/yq_linux_${ARCH/x86_64/amd64}     && ec  0.0s
 => CACHED [ 5/12] RUN chmod a+x /usr/local/bin/yq &&     curl https://dl.google.com/go/go${GO_VERSION}.linux-${ARCH/x86_64/amd64}.tar.gz -o go${GO_VERSION  0.0s
 => CACHED [ 6/12] RUN if [ "x86_64" == "s390x" ]; then     git clone --depth 1 --single-branch https://github.com/hashicorp/packer.git -b v1.9.4;     cd p  0.0s
 => CACHED [ 7/12] RUN ln -s /usr/libexec/qemu-kvm /usr/bin/qemu-system-$(uname -m)                                                                          0.0s
 => CACHED [ 8/12] RUN git clone https://github.com/canonical/cloud-utils                                                                                    0.0s
 => CACHED [ 9/12] RUN cd cloud-utils && make install                                                                                                        0.0s
 => CACHED [10/12] RUN curl https://sh.rustup.rs -sSf | sh -s -- -y --default-toolchain "1.72.0"                                                             0.0s
 => CACHED [11/12] RUN wget https://github.com/protocolbuffers/protobuf/releases/download/v${PROTOC_VERSION}/protoc-${PROTOC_VERSION}-linux-${ARCH/s390x/s3  0.0s
 => CACHED [12/12] WORKDIR /src                                                                                                                              0.0s
 => exporting to image                                                                                                                                       0.0s
 => => exporting layers                                                                                                                                      0.0s
 => => writing image sha256:118d97c4321a87dd876b370238ba34f044b6f656939d7028a356ab3a1edaf1da                                                                 0.0s
 => => naming to quay.io/confidential-containers/podvm-builder-rhel:latest                                                                                   0.0s
cp -rf ../../.git .git
cd ../ && docker buildx build -t quay.io/confidential-containers/podvm-binaries-rhel-x86_64:latest -f cloud-api-adaptor/podvm/Dockerfile.podvm_binaries.rhel \
    --build-arg BUILDER_IMG=quay.io/confidential-containers/podvm-builder-rhel:latest \
    --build-arg PODVM_DISTRO=rhel \
    --build-arg ARCH=x86_64 \
    --build-arg AA_KBC= \
     \
    --load  .
[+] Building 1959.3s (15/15) FINISHED                                                                                                      docker:rancher-desktop
 => [internal] load build definition from Dockerfile.podvm_binaries.rhel                                                                                     0.0s
 => => transferring dockerfile: 984B                                                                                                                         0.0s
 => [internal] load .dockerignore                                                                                                                            0.0s
 => => transferring context: 2B                                                                                                                              0.0s
 => [internal] load metadata for registry.access.redhat.com/ubi9/ubi:9.2                                                                                     1.2s
 => [internal] load metadata for quay.io/confidential-containers/podvm-builder-rhel:latest                                                                   0.0s
 => [internal] load build context                                                                                                                            0.4s
 => => transferring context: 27.50MB                                                                                                                         0.4s
 => CACHED [podvm_builder 1/6] FROM quay.io/confidential-containers/podvm-builder-rhel:latest                                                                0.0s
 => CACHED [stage-1 1/3] FROM registry.access.redhat.com/ubi9/ubi:9.2@sha256:2f168398c538b287fd705519b83cd5b604dc277ef3d9f479c28a2adb4d830a49                0.0s
 => [podvm_builder 2/6] COPY . /src                                                                                                                          4.6s
 => [podvm_builder 3/6] WORKDIR /src/cloud-api-adaptor/podvm                                                                                                 0.0s
 => [podvm_builder 4/6] RUN LIBC=gnu make binaries                                                                                                        1934.8s
 => [podvm_builder 5/6] RUN tar czvf /podvm-binaries.tar.gz -C /src/cloud-api-adaptor/podvm/files usr/ etc/                                                  8.6s
 => [podvm_builder 6/6] RUN tar czvf /pause-bundle.tar.gz -C /src/cloud-api-adaptor/podvm/files pause_bundle/                                                0.5s 
 => [stage-1 2/3] COPY --from=podvm_builder /podvm-binaries.tar.gz /                                                                                         0.2s 
 => [stage-1 3/3] COPY --from=podvm_builder /pause-bundle.tar.gz /                                                                                           0.0s 
 => exporting to image                                                                                                                                       0.4s 
 => => exporting layers                                                                                                                                      0.4s 
 => => writing image sha256:1e4156f426bf958a84461b5c8ae5411fcea1282309d1c27783a1ae1d8e087095                                                                 0.0s 
 => => naming to quay.io/confidential-containers/podvm-binaries-rhel-x86_64:latest                                                                           0.0s
rm -rf .git
cp -rf ../../.git .git
cd ../ && docker buildx build -t quay.io/confidential-containers/podvm-libvirt-rhel-x86_64:latest -f cloud-api-adaptor/podvm/Dockerfile.podvm.rhel \
        --build-arg BUILDER_IMG=quay.io/confidential-containers/podvm-builder-rhel:latest \
        --build-arg BINARIES_IMG=quay.io/confidential-containers/podvm-binaries-rhel-x86_64:latest \
        --build-arg PODVM_DISTRO=rhel \
        --build-arg ARCH=x86_64 \
        --build-arg IMAGE_URL=rhel-9.2-x86_64-kvm.qcow2 \
        --build-arg CLOUD_PROVIDER=libvirt \
        --load  .
[+] Building 1723.4s (17/17) FINISHED                                                                            docker:rancher-desktop
 => [internal] load build definition from Dockerfile.podvm.rhel                                                                    0.0s
 => => transferring dockerfile: 1.72kB                                                                                             0.0s
 => [internal] load .dockerignore                                                                                                  0.0s
 => => transferring context: 2B                                                                                                    0.0s
 => [internal] load metadata for quay.io/confidential-containers/podvm-builder-rhel:latest                                         0.0s
 => [internal] load metadata for quay.io/confidential-containers/podvm-binaries-rhel-x86_64:latest                                 0.0s
 => [podvm_builder 1/9] FROM quay.io/confidential-containers/podvm-builder-rhel:latest                                             0.0s
 => [podvm_binaries 1/1] FROM quay.io/confidential-containers/podvm-binaries-rhel-x86_64:latest                                    0.0s
 => [internal] load build context                                                                                                  0.5s
 => => transferring context: 27.51MB                                                                                               0.5s
 => CACHED [podvm_builder 2/9] COPY --from=podvm_binaries /podvm-binaries.tar.gz /src/cloud-api-adaptor/podvm/files/podvm-binarie  0.0s
 => CACHED [podvm_builder 3/9] RUN tar xvf /src/cloud-api-adaptor/podvm/files/podvm-binaries.tar.gz -C /src/cloud-api-adaptor/pod  0.0s
 => CACHED [podvm_builder 4/9] COPY --from=podvm_binaries /pause-bundle.tar.gz /src/cloud-api-adaptor/podvm/files/pause-bundle.ta  0.0s
 => CACHED [podvm_builder 5/9] RUN tar xvf /src/cloud-api-adaptor/podvm/files/pause-bundle.tar.gz -C /src/cloud-api-adaptor/podvm  0.0s
 => [podvm_builder 6/9] COPY . /src                                                                                                3.9s
 => [podvm_builder 7/9] WORKDIR /src/cloud-api-adaptor/podvm                                                                       0.0s
 => [podvm_builder 8/9] RUN if [ "x86_64" == "x86_64" || "x86_64" == "amd64"]; then     packer plugins install github.com/hashico  0.3s
 => [podvm_builder 9/9] RUN LIBC=gnu make image                                                                                 1697.7s
 => [stage-2 1/1] COPY --from=podvm_builder /src/cloud-api-adaptor/podvm/output/*.qcow2 /                                          2.1s 
 => exporting to image                                                                                                             7.3s 
 => => exporting layers                                                                                                            7.3s 
 => => writing image sha256:fd795f42592360b02fa00bd2ec5bdab3849e7ee6827a08e1c49d4193f09d8c9d                                       0.0s 
 => => naming to quay.io/confidential-containers/podvm-libvirt-rhel-x86_64:latest                                                  0.0s 
rm -rf .git                                                                                                                                                                                                             
saripallilavanya@Lavanyas-MacBook-Pro cloud-api-adaptor %

stevenhorsman · 2024-03-28T16:59:07Z

I managed to run: ORG_ID="17498941" ACTIVATION_KEY="shorsman-rhel" PODVM_DISTRO=rhel ARCH="x86_64" make podvm-builder podvm-binaries podvm-image IMAGE_URL=rhel-9.2-x86_64-kvm.qcow2
which created the image successfully:

# docker images
REPOSITORY                                                    TAG                                                                IMAGE ID       CREATED              SIZE
quay.io/confidential-containers/podvm-generic-rhel-x86_64     c4b61afdaea6fbb1a21d937d295f566186d1c1880be6558b80fda9d38bdff796   8bfd839731d2   About a minute ago   1.08GB

I've not got an environment to test it though.

It's definitely better than it was, but the user experience is quite tricky to setting this up and there isn't any doc, but most of it is linked to the RHEL licensing issue, so I'm not sure if we should be worrying about this upstream.

snir911

LGTM overall, thanks! added some comments for your consideration

snir911 · 2024-03-31T08:03:53Z

src/cloud-api-adaptor/podvm/qcow2/rhel/qemu-rhel.pkr.hcl

@@ -14,14 +14,15 @@ source "qemu" "rhel" {
  iso_checksum     = "${var.cloud_image_checksum}"
  iso_url          = "${var.cloud_image_url}"
  output_directory = "output"
-  qemuargs         = [["-m", "${var.memory}"], ["-smp", "cpus=${var.cpus}"], ["-cdrom", "${var.cloud_init_image}"], ["-serial", "mon:stdio"], ["-cpu", "Cascadelake-Server"]]
+  qemuargs         = [["-m", "${var.memory}"], ["-smp", "cpus=${var.cpus}"], ["-cdrom", "${var.cloud_init_image}"], ["-serial", "mon:stdio"], ["-cpu", "${var.cpu_type}"]]


nit, if this cpu types are usually constants for the architectures, you can do something like the following instead:
cpu_model = "${var.os_arch}" == "x86_64" ? "Cascadelake-Server" : "max"

I tried it but seems like there is a failure. I also tried passing os_arch = s390x as a variable, seems no luck

snir911 · 2024-03-31T08:12:08Z

src/cloud-api-adaptor/Makefile

@@ -22,6 +22,7 @@ SOURCES     := $(shell find $(SOURCEDIRS) -name '*.go' -print)
 TEST_E2E_TIMEOUT ?= 60m

 RESOURCE_CTRL ?= true
+YQ_CHECKSUM_${ARCH} ?= $(YQ_CHECKSUM)


would it make sense to drop YQ_CHECKSUM in favor of YQ_CHECKSUM_x86_64 and YQ_CHECKSUM_s390x?

Hi @snir911, Line 10 substitutes 'x86_64' with 'amd64' in the 'ARCH' value if it's not provided, potentially introducing inconsistency? Could you please suggest

ARCH ?= $(subst x86_64,amd64,$(shell uname -m))

That's odd, isn't the Docker file always expects x86_64? we are passing it amd64

Yeah, Docker file for RHEL always expects x86_64, for now i have updated code to use YQ_CHECKSUM_x86_64.

But, when i don't provide Arch value as x86_64, it is getting substituted by amd64 due to line-10

PODVM_DISTRO=rhel make podvm-builder podvm-binaries podvm-image PODVM_TAG=latest ORG_ID=*** ACTIVATION_KEY="***" CLOUD_PROVIDER=libvirt IMAGE_URL="rhel-9.2-x86_64-kvm.qcow2" make: sha256sum: Command not found docker buildx build -t quay.io/confidential-containers/podvm-builder-rhel:latest -f podvm/Dockerfile.podvm_builder.rhel \ --build-arg GO_VERSION=1.21.8 \ --build-arg ORG_ID=*** \ --build-arg ARCH=amd64 \ --build-arg ACTIVATION_KEY=*** \ --build-arg PACKER_VERSION=v1.9.4 \ --build-arg PROTOC_VERSION=3.15.0 \ --build-arg RUST_VERSION=1.72.0 \ --build-arg YQ_VERSION=v4.35.1 \ --build-arg YQ_CHECKSUM= \ --load . make: *** [podvm-builder] Error 1 saripallilavanya@Lavanyas-MacBook-Pro cloud-api-adaptor %

Please suggest here.

src/cloud-api-adaptor/podvm/Makefile

src/cloud-api-adaptor/podvm/qcow2/rhel/qemu-rhel.pkr.hcl

Variable substitution is failing with ADD in docker file Fixes: confidential-containers#1740 Signed-off-by: Saripalli Lavanya <Saripalli.Lavanya@ibm.com>

QEMU plugin installation for RHEL x86 podvm image build. Signed-off-by: Saripalli Lavanya <Saripalli.Lavanya@ibm.com>

This change will help fix checksum failure issue & perform checksum work with Docker and Podman. Signed-off-by: Saripalli Lavanya <Saripalli.Lavanya@ibm.com>

This fix streamlines RHEL PodVM image creation via Makefile integration with version.yaml. Signed-off-by: Saripalli Lavanya <Saripalli.Lavanya@ibm.com>

stevenhorsman

I was able to get the build working for rhel and the tests are passing, so from my side this is okay to merge. Thanks

bpradipt

/lgtm
Thanks @Saripalli-lavanya

Saripalli-lavanya force-pushed the qemu-var-s390x-bin branch from 4ee8ace to 22da7b0 Compare March 13, 2024 12:36

Saripalli-lavanya force-pushed the qemu-var-s390x-bin branch from 22da7b0 to 883fe87 Compare March 18, 2024 13:45

huoqifeng approved these changes Mar 21, 2024

View reviewed changes

huoqifeng requested a review from stevenhorsman March 21, 2024 06:35

snir911 reviewed Mar 21, 2024

View reviewed changes

Saripalli-lavanya force-pushed the qemu-var-s390x-bin branch 2 times, most recently from b8cea41 to f7e84bc Compare March 25, 2024 08:41

bpradipt reviewed Mar 25, 2024

View reviewed changes

src/cloud-api-adaptor/podvm/Dockerfile.podvm_builder.rhel Show resolved Hide resolved

bpradipt reviewed Mar 25, 2024

View reviewed changes

Saripalli-lavanya force-pushed the qemu-var-s390x-bin branch 2 times, most recently from a44e85c to 89c644b Compare March 28, 2024 06:27

stevenhorsman added the test_e2e_libvirt Run Libvirt e2e tests label Mar 28, 2024

Saripalli-lavanya force-pushed the qemu-var-s390x-bin branch 3 times, most recently from fda85ec to 4b5a884 Compare March 29, 2024 12:01

snir911 approved these changes Mar 31, 2024

View reviewed changes

Saripalli-lavanya force-pushed the qemu-var-s390x-bin branch from 4b5a884 to 6029f09 Compare April 4, 2024 06:43

snir911 reviewed Apr 4, 2024

View reviewed changes

src/cloud-api-adaptor/podvm/qcow2/rhel/qemu-rhel.pkr.hcl Outdated Show resolved Hide resolved

Saripalli-lavanya force-pushed the qemu-var-s390x-bin branch 4 times, most recently from d3cee98 to 28c2c1d Compare April 8, 2024 09:13

Saripalli-lavanya added 2 commits April 10, 2024 11:08

podvm: update Docker file podvm builder for rhel to fix

e56c180

Variable substitution is failing with ADD in docker file Fixes: confidential-containers#1740 Signed-off-by: Saripalli Lavanya <Saripalli.Lavanya@ibm.com>

podvm: QEMU plugin installation for RHEL x86 podvm build

618773f

QEMU plugin installation for RHEL x86 podvm image build. Signed-off-by: Saripalli Lavanya <Saripalli.Lavanya@ibm.com>

Saripalli-lavanya force-pushed the qemu-var-s390x-bin branch from 28c2c1d to 32f8e4c Compare April 10, 2024 05:39

Saripalli-lavanya added 2 commits April 10, 2024 11:20

podvm: Fix variable substitution in rhel podvm builder.

c8ed18a

This change will help fix checksum failure issue & perform checksum work with Docker and Podman. Signed-off-by: Saripalli Lavanya <Saripalli.Lavanya@ibm.com>

podvm: enhancement for RHEL PodVM image creation

f8cdb2d

This fix streamlines RHEL PodVM image creation via Makefile integration with version.yaml. Signed-off-by: Saripalli Lavanya <Saripalli.Lavanya@ibm.com>

Saripalli-lavanya force-pushed the qemu-var-s390x-bin branch from 32f8e4c to f8cdb2d Compare April 10, 2024 06:22

stevenhorsman approved these changes Apr 10, 2024

View reviewed changes

Saripalli-lavanya requested a review from bpradipt April 10, 2024 10:56

bpradipt approved these changes Apr 10, 2024

View reviewed changes

bpradipt merged commit bb6d3d2 into confidential-containers:main Apr 10, 2024
27 checks passed

stevenhorsman mentioned this pull request Apr 26, 2024

podvm: podvm_binaries - Revert s390x breakage #1823

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

podvm: update qemu for rhel s390x #1741

podvm: update qemu for rhel s390x #1741

Saripalli-lavanya commented Mar 13, 2024 •

edited

stevenhorsman commented Mar 15, 2024

stevenhorsman commented Mar 18, 2024

Saripalli-lavanya commented Mar 18, 2024

huoqifeng left a comment

Saripalli-lavanya commented Mar 21, 2024

snir911 Mar 21, 2024

stevenhorsman commented Mar 21, 2024

Saripalli-lavanya commented Mar 25, 2024 •

edited

bpradipt Mar 25, 2024

stevenhorsman commented Mar 25, 2024

Saripalli-lavanya commented Mar 25, 2024

stevenhorsman commented Mar 25, 2024 •

edited

Saripalli-lavanya commented Mar 28, 2024

Saripalli-lavanya commented Mar 28, 2024

stevenhorsman commented Mar 28, 2024

snir911 left a comment

snir911 Mar 31, 2024

Saripalli-lavanya Apr 4, 2024

snir911 Mar 31, 2024

Saripalli-lavanya Apr 1, 2024

snir911 Apr 4, 2024

Saripalli-lavanya Apr 8, 2024

stevenhorsman left a comment

bpradipt left a comment

podvm: update qemu for rhel s390x #1741

podvm: update qemu for rhel s390x #1741

Conversation

Saripalli-lavanya commented Mar 13, 2024 • edited

stevenhorsman commented Mar 15, 2024

stevenhorsman commented Mar 18, 2024

Saripalli-lavanya commented Mar 18, 2024

huoqifeng left a comment

Choose a reason for hiding this comment

Saripalli-lavanya commented Mar 21, 2024

Choose a reason for hiding this comment

stevenhorsman commented Mar 21, 2024

Saripalli-lavanya commented Mar 25, 2024 • edited

Choose a reason for hiding this comment

stevenhorsman commented Mar 25, 2024

Saripalli-lavanya commented Mar 25, 2024

stevenhorsman commented Mar 25, 2024 • edited

Saripalli-lavanya commented Mar 28, 2024

Saripalli-lavanya commented Mar 28, 2024

stevenhorsman commented Mar 28, 2024

snir911 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

stevenhorsman left a comment

Choose a reason for hiding this comment

bpradipt left a comment

Choose a reason for hiding this comment

Saripalli-lavanya commented Mar 13, 2024 •

edited

Saripalli-lavanya commented Mar 25, 2024 •

edited

stevenhorsman commented Mar 25, 2024 •

edited