qemu: Add rng virtio device #676

jcvenegas · 2018-08-30T16:40:16Z

Kata Containers does not provide good entropy. Enable a paravirtual rng device to provide a random number generator source.

Depens on: kata-containers/govmm#45

Fixes: #445

jodh-intel · 2018-08-30T16:51:53Z

lgtm

opendev-zuul · 2018-08-30T17:01:44Z

Build failed (third-party-check pipeline) integration testing with
OpenStack. For information on how to proceed, see
http://docs.openstack.org/infra/manual/developers.html#automated-testing

kata-runsh : RETRY_LIMIT in 5m 25s

katacontainersbot · 2018-08-30T17:42:32Z

PSS Measurement:
Qemu: 165775 KB
Proxy: 4085 KB
Shim: 8761 KB

Memory inside container:
Total Memory: 2043464 KB
Free Memory: 2003728 KB

devimc · 2018-08-30T19:46:34Z

I wonder if this works with vm templating
cc @bergwolf

WeiZhang555 · 2018-08-31T02:55:12Z

Retrigger CI now...

bergwolf · 2018-08-31T07:00:29Z

Does it require a new version of QEMU? I'm getting this error
docker: Error response from daemon: OCI runtime create failed: qemu-system-x86_64-v2.11.1-template: Property '.static-prt' not found: unknown
w/ or w/o vm templating.

jodh-intel · 2018-08-31T10:37:03Z

@bergwolf - looks like you are using a "special build" of qemu? fwics, you have to use qemu-lite or qemu-vanilla for this feature.

bergwolf · 2018-08-31T10:41:15Z

@jodh-intel I was using self-built qemu-lite. I guess I was missing some qemu feature configs. I'll retry with kata's qemu-lite package.

jodh-intel · 2018-08-31T10:43:38Z

@bergwolf - ack. OOI did you build using https://github.com/kata-containers/packaging/blob/master/scripts/configure-hypervisor.sh?

jodh-intel · 2018-08-31T10:46:13Z

@bergwolf - see #445 (comment).

bergwolf · 2018-08-31T11:30:04Z

@jodh-intel I was using q35 not pc yet still hit the problem described in #445 (comment). So qemu-lite breaks virtio rng device?

jodh-intel · 2018-08-31T12:54:08Z

On an Ubuntu 18.04 test box, this PR:

works with qemu-lite.
works with qemu-vanilla.
fails with the same error you got using the Ubuntu-packaged version of qemu (qemu-system-x86 version 1:2.11+dfsg-1ubuntu7.4).

jcvenegas · 2018-08-31T20:08:37Z

@bergwolf @jodh-intel yes qemu-lite breaks virtio-rng, meanwhile is fixed, I added created this PR:
kata-containers/qemu#3

opendev-zuul · 2018-08-31T22:34:24Z

Build succeeded (third-party-check pipeline).

kata-runsh : SUCCESS in 56m 42s

opendev-zuul · 2018-09-03T21:40:28Z

Build succeeded (third-party-check pipeline).

kata-runsh : SUCCESS in 55m 59s

sboeuf · 2018-09-04T17:14:07Z

virtcontainers/qemu_amd64.go

@@ -20,7 +20,7 @@ const defaultQemuPath = "/usr/bin/qemu-system-x86_64"

 const defaultQemuMachineType = QemuPC

-const defaultQemuMachineOptions = "accel=kvm,kernel_irqchip,nvdimm"
+const defaultQemuMachineOptions = "accel=kvm,kernel_irqchip,nvdimm,nostatic_prt"


@jcvenegas why do you define nostatic_prt as a default option for pc machine type? I thought you said this was only introduced by the very specific qemu-lite branch of qemu. What if we use a vanilla qemu?
I just want to make sure we won't get an error from a vanilla Qemu because we're using a non defined flag by default here.

jcvenegas · 2018-09-04T18:06:02Z

@sboeuf agree, now that qemu-lite branch static_prt is false by default that would not be an issue, I removed the commit, now I need to debug why is failing the CI. I see my PR working locally. And it used to work the first day I created the PR.

opendev-zuul · 2018-09-04T18:33:11Z

Build failed (third-party-check pipeline) integration testing with
OpenStack. For information on how to proceed, see
http://docs.openstack.org/infra/manual/developers.html#automated-testing

kata-runsh : RETRY_LIMIT in 8m 50s

codecov · 2018-09-04T20:45:28Z

Codecov Report

❗ No coverage uploaded for pull request base (master@8f5fec8). Click here to learn what that means.
The diff coverage is 0%.

@@            Coverage Diff            @@
##             master     #676   +/-   ##
=========================================
  Coverage          ?   64.97%           
=========================================
  Files             ?       85           
  Lines             ?    10925           
  Branches          ?        0           
=========================================
  Hits              ?     7098           
  Misses            ?     3133           
  Partials          ?      694

opendev-zuul · 2018-09-04T21:11:11Z

Build failed (third-party-check pipeline) integration testing with
OpenStack. For information on how to proceed, see
http://docs.openstack.org/infra/manual/developers.html#automated-testing

kata-runsh : RETRY_LIMIT in 8m 32s

opendev-zuul · 2018-09-04T21:48:53Z

Build failed (third-party-check pipeline) integration testing with
OpenStack. For information on how to proceed, see
http://docs.openstack.org/infra/manual/developers.html#automated-testing

kata-runsh : RETRY_LIMIT in 9m 47s

jcvenegas · 2018-09-05T17:26:05Z

update, the PR is failing randomly due to sometimes the boot times take about 30 seconds in the CI machines.

[    0.354048] ACPI: PCI Interrupt Link [LNKB] enabled at IRQ 11
[    0.354752] virtio-pci 0000:00:06.0: virtio_pci: leaving for legacy driver
[    0.389049] Serial: 8250/16550 driver, 4 ports, IRQ sharing disabled
[    0.441705] console [hvc0] enabled
[   29.446221] brd: module loaded
[   29.448989] loop: module loaded
[   29.450840]  pmem0: p1
[   29.451326] pmem0: detected capacity change from 0 to 536870912
[   29.461476] scsi host0: Virtio SCSI HBA
[   29.532665] random: fast init done
[   29.546490] tun: Universal TUN/TAP device driver, 1.6
[   29.585697] ixgbe: Intel(R) 10 Gigabit PCI Express Network Driver - version 5.1.0-k
[   29.586723] ixgbe: Copyright (c) 1999-2016 Intel Corporation.
[   29.588834] ixgbevf: Intel(R) 10 Gigabit PCI Express Virtual Function Network Driver - version 4.1.0-k

I took a look to the kernel logs and I see the logs jump from [ 0.441705] console [hvc0] enabled
to [ 29.446221] brd: module loaded .

@liujing2 any hint why it is taking so long, the CI are VMs so this is nested virtualization.

katacontainersbot · 2018-09-06T17:21:10Z

PSS Measurement:
Qemu: 167799 KB
Proxy: 4179 KB
Shim: 8751 KB

Memory inside container:
Total Memory: 2043460 KB
Free Memory: 2003688 KB

opendev-zuul · 2018-09-06T18:20:17Z

Build failed (third-party-check pipeline) integration testing with
OpenStack. For information on how to proceed, see
http://docs.openstack.org/infra/manual/developers.html#automated-testing

kata-runsh : RETRY_LIMIT in 9m 15s

opendev-zuul · 2018-09-06T19:23:17Z

Build failed (third-party-check pipeline) integration testing with
OpenStack. For information on how to proceed, see
http://docs.openstack.org/infra/manual/developers.html#automated-testing

kata-runsh : RETRY_LIMIT in 11m 12s

katacontainersbot · 2018-09-06T19:39:22Z

PSS Measurement:
Qemu: 169661 KB
Proxy: 4139 KB
Shim: 8853 KB

Memory inside container:
Total Memory: 2043460 KB
Free Memory: 2003556 KB

jcvenegas · 2018-09-06T22:41:31Z

@jodh-intel @bergwolf @sboeuf kata-containers/govmm#45 is ready to merge, needed to update vendor here.

sboeuf · 2018-09-10T18:18:26Z

@jcvenegas kata-containers/govmm#45 has been merged, please revendor accordingly :)

Changes: - qemu/qmp: support query-memory-devices qmp command. - qemu: Add virtio RNG device. Signed-off-by: Jose Carlos Venegas Munoz <jose.carlos.venegas.munoz@intel.com>

katacontainersbot · 2018-09-10T19:27:50Z

PSS Measurement:
Qemu: 156013 KB
Proxy: 4031 KB
Shim: 8793 KB

Memory inside container:
Total Memory: 2043460 KB
Free Memory: 2003628 KB

jcvenegas · 2018-09-10T19:35:44Z

@sboeuf updated, note that I am not adding a way to restrict the entropy bandwidth of the host, I wonder if this is something is supported via cgroups (for example if runc support it ). In case this is not possible to do it with cgroups, a possible option is to add it as part of our configuration.toml, this could a nice feature to allow administrators restrict the amount of entropy at node level.

opendev-zuul · 2018-09-10T19:44:00Z

Build failed (third-party-check pipeline) integration testing with
OpenStack. For information on how to proceed, see
http://docs.openstack.org/infra/manual/developers.html#automated-testing

kata-runsh : RETRY_LIMIT in 9m 46s

sboeuf

@jcvenegas one comment, but looks fine otherwise!

sboeuf · 2018-09-10T21:20:09Z

virtcontainers/qemu.go

@@ -488,6 +488,11 @@ func (q *qemu) createSandbox() error {
 	if ioThread != nil {
 		qemuConfig.IOThreads = []govmmQemu.IOThread{*ioThread}
 	}
+	// Add RNG device to hypervisor
+	rngDev := config.RNGDev{
+		ID: "rng0",


Maybe you could define a const or var at the top of this file to avoid hardcoding this directly from the code. WDYT?

updated, lets wait for the CI again. After that should be ready do merge.

Kata Containers does not have provide a good entropy level, make use of a paravirtual rng device to solve this problem. Fixes: kata-containers#445 Signed-off-by: Jose Carlos Venegas Munoz <jose.carlos.venegas.munoz@intel.com>

Update qemu-lite this disable static PRT on pc platform, needed to use devices like virtio-rng. Signed-off-by: Jose Carlos Venegas Munoz <jose.carlos.venegas.munoz@intel.com>

katacontainersbot · 2018-09-10T22:30:01Z

PSS Measurement:
Qemu: 172014 KB
Proxy: 4100 KB
Shim: 8832 KB

Memory inside container:
Total Memory: 2043460 KB
Free Memory: 2002448 KB

opendev-zuul · 2018-09-10T22:51:29Z

Build failed (third-party-check pipeline) integration testing with
OpenStack. For information on how to proceed, see
http://docs.openstack.org/infra/manual/developers.html#automated-testing

kata-runsh : RETRY_LIMIT in 10m 03s

jodh-intel · 2018-09-11T07:20:16Z

Restarted F27 CI (which fell over due to the problem fixed by #710)...

Add haveged will allow have entropy for PR. kata-containers/runtime#676 Fixes: kata-containers#712 Signed-off-by: Jose Carlos Venegas Munoz <jose.carlos.venegas.munoz@intel.com>

egernst added the review label Aug 30, 2018

jcvenegas added the do-not-merge label Aug 30, 2018

jcvenegas mentioned this pull request Aug 31, 2018

pc: static_prt disabled by default. kata-containers/qemu#3

Merged

jcvenegas force-pushed the rng-device branch from f61a158 to f622035 Compare August 31, 2018 21:36

jcvenegas force-pushed the rng-device branch from f622035 to 2fc8cae Compare September 3, 2018 20:43

sboeuf reviewed Sep 4, 2018

View reviewed changes

jcvenegas force-pushed the rng-device branch from 2fc8cae to eb8c0a8 Compare September 4, 2018 18:02

jcvenegas force-pushed the rng-device branch from 3679ecd to 7df8021 Compare September 4, 2018 21:12

jcvenegas force-pushed the rng-device branch from 7df8021 to c860ee3 Compare September 5, 2018 18:41

jcvenegas force-pushed the rng-device branch from 8ed2991 to 3781551 Compare September 6, 2018 17:04

jcvenegas force-pushed the rng-device branch from 3781551 to dba6841 Compare September 6, 2018 17:45

vendor: Update govmm

f1829d0

Changes: - qemu/qmp: support query-memory-devices qmp command. - qemu: Add virtio RNG device. Signed-off-by: Jose Carlos Venegas Munoz <jose.carlos.venegas.munoz@intel.com>

jcvenegas force-pushed the rng-device branch 2 times, most recently from 3f88c1c to ebe54a6 Compare September 10, 2018 19:10

jcvenegas mentioned this pull request Sep 10, 2018

tests: Add test to verify container entropy. kata-containers/tests#734

Closed

jcvenegas removed the do-not-merge label Sep 10, 2018

sboeuf approved these changes Sep 10, 2018

View reviewed changes

jcvenegas added 2 commits September 10, 2018 17:11

vc: hypervisor: qemu: Add rng device.

bf7fd2b

Kata Containers does not have provide a good entropy level, make use of a paravirtual rng device to solve this problem. Fixes: kata-containers#445 Signed-off-by: Jose Carlos Venegas Munoz <jose.carlos.venegas.munoz@intel.com>

versions: update qemu-lite version.

44e7631

Update qemu-lite this disable static PRT on pc platform, needed to use devices like virtio-rng. Signed-off-by: Jose Carlos Venegas Munoz <jose.carlos.venegas.munoz@intel.com>

jcvenegas force-pushed the rng-device branch from ebe54a6 to 44e7631 Compare September 10, 2018 22:13

sboeuf merged commit e02695b into kata-containers:master Sep 11, 2018

egernst removed the review label Sep 11, 2018

sboeuf added the feature New functionality label Sep 12, 2018

jcvenegas deleted the rng-device branch January 29, 2019 18:33

jcvenegas restored the rng-device branch September 30, 2019 18:48

jcvenegas deleted the rng-device branch January 23, 2020 19:44

qemu: Add rng virtio device #676

qemu: Add rng virtio device #676

Conversation

jcvenegas commented Aug 30, 2018

jodh-intel commented Aug 30, 2018 • edited by amshinde

opendev-zuul bot commented Aug 30, 2018

katacontainersbot commented Aug 30, 2018

devimc commented Aug 30, 2018

WeiZhang555 commented Aug 31, 2018

bergwolf commented Aug 31, 2018

jodh-intel commented Aug 31, 2018

bergwolf commented Aug 31, 2018

jodh-intel commented Aug 31, 2018

jodh-intel commented Aug 31, 2018

bergwolf commented Aug 31, 2018

jodh-intel commented Aug 31, 2018

jcvenegas commented Aug 31, 2018

opendev-zuul bot commented Aug 31, 2018

opendev-zuul bot commented Sep 3, 2018

sboeuf Sep 4, 2018

Choose a reason for hiding this comment

jcvenegas commented Sep 4, 2018

opendev-zuul bot commented Sep 4, 2018

codecov bot commented Sep 4, 2018 • edited

Codecov Report

opendev-zuul bot commented Sep 4, 2018

opendev-zuul bot commented Sep 4, 2018

jcvenegas commented Sep 5, 2018 • edited

katacontainersbot commented Sep 6, 2018

opendev-zuul bot commented Sep 6, 2018

opendev-zuul bot commented Sep 6, 2018

katacontainersbot commented Sep 6, 2018

jcvenegas commented Sep 6, 2018

sboeuf commented Sep 10, 2018

katacontainersbot commented Sep 10, 2018

jcvenegas commented Sep 10, 2018

opendev-zuul bot commented Sep 10, 2018

sboeuf left a comment

Choose a reason for hiding this comment

sboeuf Sep 10, 2018

Choose a reason for hiding this comment

jcvenegas Sep 10, 2018

Choose a reason for hiding this comment

katacontainersbot commented Sep 10, 2018

opendev-zuul bot commented Sep 10, 2018

jodh-intel commented Sep 11, 2018

jodh-intel commented Aug 30, 2018 •

edited by amshinde

codecov bot commented Sep 4, 2018 •

edited

jcvenegas commented Sep 5, 2018 •

edited