providers/qemu: support Ignition block device on s390x and ppc64le #905

jlebon · 2019-12-18T16:56:15Z

Add experimental support for fetching Ignition configs via a virtio
block device with serial/ID ignition.

The main advantage of this is that it is cross-platform. But for now, we
only use it on platforms which don't support the QEMU firmware config
approach, which are (of those we care about) s390x and ppc64le. We may
end up using it across the remaining platforms.

See related discussions in:
#928

jlebon · 2019-12-18T17:03:12Z

See coreos/coreos-assembler#1004 (comment)
Not sure if we actually need to get this in, or if we can just wait until #825 is fixed.

Prashanth684 · 2019-12-18T17:15:02Z

we need the same mechanism for ppc too since ppc doesn't support fw_cfg ..should the qemu provider just support fetching config from the config-drive for non x86 arches ?

Of course for ppc , we still need the cluster-api-provider libvirt to support config drive just like for s390x so we can make use of it - but i am bringing this up for the sake of a long term discussion

ajeddeloh · 2020-01-02T22:24:57Z

If we introduce a hack, we need an exit strategy too. What's the plan for when we want to remove this?

jaypoulz · 2020-02-19T21:18:18Z

@dbenoit17 we need to re-initiate this discussion so that we can get this to work for s390x

Prashanth684 · 2020-02-20T22:07:45Z

@jlebon , i don't think #825 is going to be fixed anytime soon as nobody seems to be actively looking at it. Could we get this PR in and while we're at it also add ppc64le to it.Thanks.

jlebon · 2020-02-26T22:02:53Z

@Prashanth684 Were you able to test this?

Once this goes in, it's going to be harder to move to something cleaner. Just quickly, has the alternative @lucab mentioned here been investigated? #825 (comment).

Prashanth684 · 2020-02-26T22:20:27Z

@Prashanth684 Were you able to test this?

Once this goes in, it's going to be harder to move to something cleaner. Just quickly, has the alternative @lucab mentioned here been investigated? #825 (comment).

@jlebon No I have not tested this fix yet, we have just been using the openstack image itself to run the CI. I will build try to test this fix and let you know.

As for #825 , we need some kvm experts from IBM to look into this and give their feedback. I will take this as an action item to talk to some of their kvm experts and get some suggestions from them, but I'm guessing it won't be done anytime soon as #825 has been open for a while now. We need this change for CI on Openshift 4.3 for s390x and that means this fix is our only viable option for now.

Prashanth684 · 2020-02-27T18:21:19Z

@Prashanth684 Were you able to test this?
Once this goes in, it's going to be harder to move to something cleaner. Just quickly, has the alternative @lucab mentioned here been investigated? #825 (comment).

@jlebon No I have not tested this fix yet, we have just been using the openstack image itself to run the CI. I will build try to test this fix and let you know.

As for #825 , we need some kvm experts from IBM to look into this and give their feedback. I will take this as an action item to talk to some of their kvm experts and get some suggestions from them, but I'm guessing it won't be done anytime soon as #825 has been open for a while now. We need this change for CI on Openshift 4.3 for s390x and that means this fix is our only viable option for now.

Just tested this on s390x and it works. @jlebon another alternative to getting the fix upstream is to just patch the downstream Ignition rpm spec for RHCOS since this is specifically needed only in the RHCOS image used by Openshift CI s390x pipeline. This patch would have to carried-forward downstream till #825 is addressed.

internal/platform/platform.go

lucab · 2020-02-28T09:07:51Z

I don't honestly see a feasible plan (actions+timeline) to ever get out of this terrible hack without breaking the whole world; does that plan exist somewhere?

Additionally, there has been literally no effort in investigating how to do this in a non-hackish way with platform-specific capabilities of qemu-system-s390x (I guess because, to the best of my knowledge, #825 hasn't been properly prioritized & planned on the roadmap of openshift-s390x development team).

jlebon · 2020-02-28T21:11:55Z

We're discussing changing the semantics for QEMU here: #928

Prashanth684 · 2020-02-28T21:55:34Z

I don't honestly see a feasible plan (actions+timeline) to ever get out of this terrible hack without breaking the whole world; does that plan exist somewhere?

Additionally, there has been literally no effort in investigating how to do this in a non-hackish way with platform-specific capabilities of qemu-system-s390x (I guess because, to the best of my knowledge, #825 hasn't been properly prioritized & planned on the roadmap of openshift-s390x development team).

I think the problem was when #825 was opened, it was not opened with the motive to find a solution in qemu itself but rather to work around it. The IBM team went with the config drive solution for the Openshift installer, the problem being that with the config drive approach, it would only work with Openstack platform. So for most of their and our CI testing, we were using the Openstack image. But now with the automated CI, we only have the option of using the qemu image (since this was turned down: openshift/installer#2092). We are also discussing an alternative to somehow use the Openstack image for CI instead of the qemu image which would be a better way to do it.

crawford · 2020-03-05T17:31:17Z

Can we move forward with this and just put a big "experimental" label on it? Within OpenShift, our customers don't actually make use of this mechanism anyway (we recommend using coreos.inst.ignition_url). We only use this in CI, where we don't need to worry about backward compatibility as much.

jlebon · 2020-03-05T18:35:22Z

Can we move forward with this and just put a big "experimental" label on it?

Is there a way to do this at the technical level? Or is it more of a policy thing? I guess for now we can just print scary warnings on the console or something.

crawford · 2020-03-05T19:34:11Z

I think it would be a combination of policy and a s̰̮̉ͮ̇c̞̠̮̤͍͔̊̚ͅa͈̟̘̬̲̟̅̾͋̆r̳̘͎̭͚̦̐̉̀͑͐̋̓y̆͋̓ͫ ̣̺̤͈̪͛̊͂ͩ̃̉̚w̞͍̣̍̓̓̌a͉̗͇̻͈͉̻r̜̜͖͎͖̜͛̈́ͅniͣn͖̼̹̓̓̓͋̐gͪͧ͊͗ͮͪ̂. We could bother with creating a feature flag mechanism eventually, but that seems overkill for this.

jlebon · 2020-03-06T16:17:16Z

#928 (comment)

So here's a suggestion: we're currently in dire need of getting something to work for better OCP CI coverage on s390x. It doesn't have to be stabilized and we don't expect production use of whatever mechanism we choose.

Thus, I think this is a good opportunity to try out this approach to see how it looks and get feedback from it. I've updated #905 for this. I've tested it successfully on amd64 (by tweaking the conditional build flags).

@Prashanth684 Can you test this on s390x and ppc64le?

Prashanth684 · 2020-03-06T16:27:38Z

#928 (comment)

So here's a suggestion: we're currently in dire need of getting something to work for better OCP CI coverage on s390x. It doesn't have to be stabilized and we don't expect production use of whatever mechanism we choose.
Thus, I think this is a good opportunity to try out this approach to see how it looks and get feedback from it. I've updated #905 for this. I've tested it successfully on amd64 (by tweaking the conditional build flags).

@Prashanth684 Can you test this on s390x and ppc64le?

Thanks @jlebon . Will do. @crawford going ahead with this option would require changes to the terraform-libvirt-provider which would then have to be bumped in the Openshift installer FYI. So we have to be prepared for that. And the cluster-api-provider-libvirt too.

internal/providers/qemu/qemu_blockdev.go

Prashanth684 · 2020-03-06T22:19:45Z

#928 (comment)

So here's a suggestion: we're currently in dire need of getting something to work for better OCP CI coverage on s390x. It doesn't have to be stabilized and we don't expect production use of whatever mechanism we choose.
Thus, I think this is a good opportunity to try out this approach to see how it looks and get feedback from it. I've updated #905 for this. I've tested it successfully on amd64 (by tweaking the conditional build flags).

@Prashanth684 Can you test this on s390x and ppc64le?

works on s390x. i was able to bring up an rhcos VM up with terraform. will also test ppc64le

Prashanth684 · 2020-03-10T13:26:25Z

#928 (comment)

So here's a suggestion: we're currently in dire need of getting something to work for better OCP CI coverage on s390x. It doesn't have to be stabilized and we don't expect production use of whatever mechanism we choose.
Thus, I think this is a good opportunity to try out this approach to see how it looks and get feedback from it. I've updated #905 for this. I've tested it successfully on amd64 (by tweaking the conditional build flags).

@Prashanth684 Can you test this on s390x and ppc64le?

works on s390x. i was able to bring up an rhcos VM up with terraform. will also test ppc64le

works on ppc64le as well.

internal/providers/qemu/qemu_blockdev.go

internal/providers/qemu/qemu_fwcfg.go

internal/providers/qemu/qemu_blockdev.go

crawford · 2020-03-10T18:22:19Z

@crawford going ahead with this option would require changes to the terraform-libvirt-provider which would then have to be bumped in the Openshift installer FYI.

That's okay. We had to do that in the 4.2 branch as well. Fortunately, libvirt is not a supported platform, so there is relatively little risk.

jlebon · 2020-03-10T21:28:52Z

Had a chat with @lucab and we've agreed that re-using the same context.Context pattern from the other providers is not very appealing. Went with a simpler approach now that just uses a vanilla chan + select.

internal/providers/qemu/qemu_blockdev.go

Add experimental support for fetching Ignition configs via a virtio block device with serial `ignition`. The main advantage of this is that it is cross-platform. But for now, we only use it on platforms which don't support the QEMU firmware config approach, which are (of those we care about) s390x and ppc64le. We may end up using it across the remaining platforms. See related discussions in: coreos#928

lucab

Code LGTM.

jlebon · 2020-03-12T14:10:09Z

@bgilbert Make sense? Or did you want to keep exploring options in #928?

Prashanth684 · 2020-03-16T13:28:07Z

@jlebon we can continue exploring better options in #928 if needed, but do you think it makes sense to get this in now so we have a solution that unblocks us?

jlebon · 2020-03-16T13:48:10Z

@jlebon we can continue exploring better options in #928 if needed, but do you think it makes sense to get this in now so we have a solution that unblocks us?

Yes, agreed. Let's get this in, given that it's experimental and only applies to previously unsupported architectures.

This finally unifies the advantages of `cosa run` and `kola spawn`. I kept getting annoyed by how serial console sizing is broken (e.g. trying to use `less` etc.). Using `ssh` via `kola spawn` addresses that, but it means you can't debug the initramfs. Now things work in an IMO pretty cool way; if you do e.g. `cosa run --kargs ignition.config.url=blah://` (or inject a bad Ignition config) to cause a failure in the initramfs, you'll see a nice error (building on coreos/ignition-dracut#146 ) telling you to rerun with `cosa run --devshell-console`. Things are also wired up cleanly so that we support rebooting with the equivalent of `kola spawn --reconnect` (which we should probably remove now). You can exit via *either* quitting SSH cleanly or using `poweroff`, and the lifecycle of ssh and qemu is wired together. And finally, if we detect a cosa workdir we also bind it in by default. More to come here, such as auto-injecting debugging tools and containers.

cgwalters · 2020-04-22T17:43:51Z

What's the qemu incantation to use this? The "convenience" -drive method seems to reject using serial.

jlebon · 2020-04-22T18:46:02Z

What's the qemu incantation to use this? The "convenience" -drive method seems to reject using serial.

Yeah, serial is a property of the virtio-blk device driver, so you have to associate with an explicitly defined virtio-blk device. E.g.:

-drive file=foobar.ign,if=none,format=raw,readonly=on,id=ignition \
  -device virtio-blk,serial=ignition,drive=ignition

(This is also what kola does for primary-disk.)

…fault Move from injecting ingnition directly in to the VM image on non-fw_cfg architectures to passing it via virtio-blk device. It should be more performant and should be more robust going forward. See: coreos/ignition#905

dustymabe · 2022-09-27T16:22:09Z

For anyone in the future who stumbles upon this like I did:

@jebon pointed me at #928 (comment)

so this should work instead of passing --qemu-commandline= directly:

virt-install ... --disk path=$PWD/config.ign,format=raw,readonly=on,serial=ignition

jlebon mentioned this pull request Dec 18, 2019

Updated platformid to force openstack for s390x on 4.2 coreos/coreos-assembler#1004

Merged

jaypoulz mentioned this pull request Dec 19, 2019

Forcing PLATFORM_ID to openstack on s390x systems coreos/ignition-dracut#145

Closed

lucab suggested changes Feb 28, 2020

View reviewed changes

internal/platform/platform.go Outdated Show resolved Hide resolved

jlebon mentioned this pull request Feb 28, 2020

Provide cross-architecture Ignition QEMU support #928

Open

Prashanth684 mentioned this pull request Mar 5, 2020

Bug 1818140: Backport adding groundwork for multi-arch images openshift/installer#3233

Merged

jlebon force-pushed the pr/s390x-qemu branch from 8efc56f to 13c60e5 Compare March 6, 2020 16:11

jlebon changed the title ~~platform: add hack to use openstack fetcher for s390x qemu~~ providers/qemu: support Ignition block device on s390x and ppc64le Mar 6, 2020

jlebon force-pushed the pr/s390x-qemu branch from 13c60e5 to f354ca5 Compare March 6, 2020 16:16

jlebon force-pushed the pr/s390x-qemu branch from f354ca5 to 8f98edb Compare March 6, 2020 16:18

jlebon requested a review from lucab March 6, 2020 16:34

cgwalters reviewed Mar 6, 2020

View reviewed changes

internal/providers/qemu/qemu_blockdev.go Show resolved Hide resolved

lucab reviewed Mar 10, 2020

View reviewed changes

jlebon force-pushed the pr/s390x-qemu branch from 8f98edb to 0785eb6 Compare March 10, 2020 21:26

jlebon requested a review from lucab March 11, 2020 17:38

ashcrow approved these changes Mar 11, 2020

View reviewed changes

lucab reviewed Mar 12, 2020

View reviewed changes

internal/providers/qemu/qemu_blockdev.go Outdated Show resolved Hide resolved

jlebon force-pushed the pr/s390x-qemu branch from 0785eb6 to 3b930b4 Compare March 12, 2020 13:23

lucab approved these changes Mar 12, 2020

View reviewed changes

jlebon requested a review from bgilbert March 12, 2020 14:10

jlebon merged commit a25df69 into coreos:master Mar 16, 2020

jlebon mentioned this pull request Mar 16, 2020

providers/qemu: support Ignition block device on s390x and ppc64le #936

Merged

jlebon mentioned this pull request Mar 24, 2020

providers/qemu: fix default to fwcfg #944

Merged

jlebon mentioned this pull request Apr 22, 2020

cmdlib: Use kola qemuexec, delete qemu-kvm code coreos/coreos-assembler#1289

Merged

This was referenced Apr 24, 2020

cmdlib: only use index=N for supermin root coreos/coreos-assembler#1398

Merged

Redundant / incorrect kernel module check coreos/ignition-dracut#176

Closed

jlebon mentioned this pull request May 27, 2020

kola/qemu: Pass ignition via blk device on non-fw_cfg platforms by default coreos/coreos-assembler#1484

Merged

dustymabe mentioned this pull request Sep 27, 2022

failed to boot fedora coreos on s390x: fw_cfg option doesn't work coreos/fedora-coreos-tracker#1307

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

providers/qemu: support Ignition block device on s390x and ppc64le #905

providers/qemu: support Ignition block device on s390x and ppc64le #905

jlebon commented Dec 18, 2019 •

edited

Loading

jlebon commented Dec 18, 2019

Prashanth684 commented Dec 18, 2019 •

edited

Loading

ajeddeloh commented Jan 2, 2020

jaypoulz commented Feb 19, 2020

Prashanth684 commented Feb 20, 2020

jlebon commented Feb 26, 2020

Prashanth684 commented Feb 26, 2020 •

edited

Loading

Prashanth684 commented Feb 27, 2020

lucab commented Feb 28, 2020

jlebon commented Feb 28, 2020

Prashanth684 commented Feb 28, 2020

crawford commented Mar 5, 2020 •

edited

Loading

jlebon commented Mar 5, 2020

crawford commented Mar 5, 2020

jlebon commented Mar 6, 2020

Prashanth684 commented Mar 6, 2020 •

edited

Loading

Prashanth684 commented Mar 6, 2020

Prashanth684 commented Mar 10, 2020

crawford commented Mar 10, 2020

jlebon commented Mar 10, 2020

lucab left a comment

jlebon commented Mar 12, 2020

Prashanth684 commented Mar 16, 2020 •

edited

Loading

jlebon commented Mar 16, 2020

cgwalters commented Apr 22, 2020

jlebon commented Apr 22, 2020

dustymabe commented Sep 27, 2022

providers/qemu: support Ignition block device on s390x and ppc64le #905

providers/qemu: support Ignition block device on s390x and ppc64le #905

Conversation

jlebon commented Dec 18, 2019 • edited Loading

jlebon commented Dec 18, 2019

Prashanth684 commented Dec 18, 2019 • edited Loading

ajeddeloh commented Jan 2, 2020

jaypoulz commented Feb 19, 2020

Prashanth684 commented Feb 20, 2020

jlebon commented Feb 26, 2020

Prashanth684 commented Feb 26, 2020 • edited Loading

Prashanth684 commented Feb 27, 2020

lucab commented Feb 28, 2020

jlebon commented Feb 28, 2020

Prashanth684 commented Feb 28, 2020

crawford commented Mar 5, 2020 • edited Loading

jlebon commented Mar 5, 2020

crawford commented Mar 5, 2020

jlebon commented Mar 6, 2020

Prashanth684 commented Mar 6, 2020 • edited Loading

Prashanth684 commented Mar 6, 2020

Prashanth684 commented Mar 10, 2020

crawford commented Mar 10, 2020

jlebon commented Mar 10, 2020

lucab left a comment

Choose a reason for hiding this comment

jlebon commented Mar 12, 2020

Prashanth684 commented Mar 16, 2020 • edited Loading

jlebon commented Mar 16, 2020

cgwalters commented Apr 22, 2020

jlebon commented Apr 22, 2020

dustymabe commented Sep 27, 2022

jlebon commented Dec 18, 2019 •

edited

Loading

Prashanth684 commented Dec 18, 2019 •

edited

Loading

Prashanth684 commented Feb 26, 2020 •

edited

Loading

crawford commented Mar 5, 2020 •

edited

Loading

Prashanth684 commented Mar 6, 2020 •

edited

Loading

Prashanth684 commented Mar 16, 2020 •

edited

Loading