Improve entropy collection in VMs

[marmarek]

Reported by joanna on 15 Nov 2012 19:03 UTC
While this is only my feeling, I suspect that the entropy collection daemon in our VMs needs some improvements.. This is because of the limited interaction with the physical world of each VM (e.g. mouse events go via vchan instead of via kernel module in a VM).

This can be easily noticed when one tries to generate a new GPG key in a VM -- the gpg would complain about inadequate entropy that is available and will hang until more is produced. One can produce more entropy via various disk activities (e.g. grep through the filesystem), however this:

1. Isn't very convenient
2. It's questionable whether such entropy is of "first-class freshness", or is it somehow inferior to the entropy that could be collected with the help of mouse movements, etc.?

It would probably be desirable to create some entropy producing device that would run in each of the VMs, and to feed this device from Dom0 or other domains exposed to physical hardware (netvm, usbvm?). One should be careful, however, not to distribute the same "entropy bits" to more than one domain, as this would likely compromise domain isolation.

Migrated-From: https://wiki.qubes-os.org/ticket/673

[marmarek]

Comment by joanna on 15 Nov 2012 23:25 UTC
Ok, I see two simple solutions:

1. We run a set of daemons in Dom0 (one for each VM) that essentially do this in a loop:

read_a_chunk_of_bytes (/dev/ranomd);
send_bytes_to_VM(); // via qrexec
sleep (...) // let other read some Dom0's entropy also

Then, in the VM, there is a code that reads the transmitted bytes and sends them into the kernel's rng using the RNDADDENTROPY IOCTL on /dev/random:

http://git.kernel.org/?p=linux/kernel/git/stable/linux-stable.git;a=blob;f=drivers/char/random.c;h=b86eae9b77dfaeb04dd2d4efefd6ebc01b9e0a93;hb=HEAD#l1265

2. We just enable haveged in each VM (it gathers entopry from measuring internal CPU state):

http://www.issihosts.com/haveged/index.html
http://www.irisa.fr/caps/projects/hipsor/

Note 1: haveged is incredibly fast! Just seem to be a bit TOO fast for me... So, I think I would feel better with the option #1 I think...

Note 2: Dom0 entropy seems pretty reasonable (thanks to mouse and keyboard!), so it's not unreasonable to share it among all the VMs. But perhaps we could allow to manually exclude some VMs from getting the entropy from Dom0 (e.g. those that are not very sensitive). E.g. I have almost 30 domains on my laptop, while there are maybe 4 only that are used for key generation and those are the only ones that need fresh entropy from Dom0.

[marmarek]

Comment by joanna on 15 Nov 2012 23:25 UTC
Ok, I see two simple solutions:

1. We run a set of daemons in Dom0 (one for each VM) that essentially do this in a loop:

read_a_chunk_of_bytes (/dev/ranomd);
send_bytes_to_VM(); // via qrexec
sleep (...) // let other read some Dom0's entropy also

Then, in the VM, there is a code that reads the transmitted bytes and sends them into the kernel's rng using the RNDADDENTROPY IOCTL on /dev/random:

http://git.kernel.org/?p=linux/kernel/git/stable/linux-stable.git;a=blob;f=drivers/char/random.c;h=b86eae9b77dfaeb04dd2d4efefd6ebc01b9e0a93;hb=HEAD#l1265

2. We just enable haveged in each VM (it gathers entopry from measuring internal CPU state):

http://www.issihosts.com/haveged/index.html
http://www.irisa.fr/caps/projects/hipsor/

Note 1: haveged is incredibly fast! Just seem to be a bit TOO fast for me... So, I think I would feel better with the option #1 I think...

Note 2: Dom0 entropy seems pretty reasonable (thanks to mouse and keyboard!), so it's not unreasonable to share it among all the VMs. But perhaps we could allow to manually exclude some VMs from getting the entropy from Dom0 (e.g. those that are not very sensitive). E.g. I have almost 30 domains on my laptop, while there are maybe 4 only that are used for key generation and those are the only ones that need fresh entropy from Dom0.

[marmarek]

Comment by joanna on 21 Nov 2012 10:25 UTC
Some comments are in this thread:

https://groups.google.com/group/qubes-devel/browse_thread/thread/e7023cca06daa219

[marmarek]

Comment by joanna on 21 Nov 2012 10:25 UTC
Some comments are in this thread:

https://groups.google.com/group/qubes-devel/browse_thread/thread/e7023cca06daa219

[marmarek]

Modified by joanna on 8 Feb 2013 12:59 UTC

[marmarek]

Modified by joanna on 8 Feb 2013 12:59 UTC

[marmarek]

Modified by joanna on 30 Aug 2013 17:21 UTC

[marmarek]

Modified by joanna on 30 Aug 2013 17:21 UTC

[marmarek]

Modified by joanna on 20 Apr 2014 17:04 UTC

[marmarek]

Modified by joanna on 20 Apr 2014 17:04 UTC

[marmarek]

Comment by joanna on 3 Jul 2014 12:07 UTC
For now (R2 release) we should just ensure haveged in the default template, I think.

[marmarek]

Comment by joanna on 3 Jul 2014 12:07 UTC
For now (R2 release) we should just ensure haveged in the default template, I think.

[marmarek]

Comment by marmarek on 4 Jul 2014 02:45 UTC
Why this can't be the final solution? I don't believe we ever implement any other solution for this...

Also - do we want to cover by this fix also updates to template (which would mean hard dependency on haveged from qubes-core-vm)? Or installing it in new templates would be enough (so on R2 ISO)?

[marmarek]

Comment by marmarek on 4 Jul 2014 02:45 UTC
Why this can't be the final solution? I don't believe we ever implement any other solution for this...

Also - do we want to cover by this fix also updates to template (which would mean hard dependency on haveged from qubes-core-vm)? Or installing it in new templates would be enough (so on R2 ISO)?

[marmarek]

Comment by joanna on 4 Jul 2014 09:37 UTC
I think just the new template, no need to issue updates. This is no a security problem, rather a usability -- i.e. if read() on /dev/random hangs, it's an annoyance to the user.

I agree to closing this ticket with haveged.

[marmarek]

Comment by joanna on 4 Jul 2014 09:37 UTC
I think just the new template, no need to issue updates. This is no a security problem, rather a usability -- i.e. if read() on /dev/random hangs, it's an annoyance to the user.

I agree to closing this ticket with haveged.

[marmarek]

Comment by marmarek on 4 Jul 2014 10:02 UTC
http://git.qubes-os.org/?p=marmarek/linux-template-builder.git;a=commit;h=e416c1a5b3b5a97525881d38efc0eef39659b48c

[marmarek]

Comment by marmarek on 4 Jul 2014 10:02 UTC
http://git.qubes-os.org/?p=marmarek/linux-template-builder.git;a=commit;h=e416c1a5b3b5a97525881d38efc0eef39659b48c

[adrelanos]

From https://wiki.archlinux.org/index.php/Haveged#Virtual_Machines

As discussed at Is it appropriate to use haveged as a source of entropy on virtual machines?, it can be contested whether haveged provides quality entropy within a virtual environment.

---

It's not as simple as writing to /dev/random. From man random(4)

This differs from writing to /dev/random or /dev/urandom, which only adds some data but does not increment the entropy count. The following structure is used:

---

Looks like this could be implemented by reading from /dev/random and forwarding that entropy qrexec, ioctl(2), RNDADDENTROPY to VMs.

[adrelanos]

From https://wiki.archlinux.org/index.php/Haveged#Virtual_Machines

As discussed at Is it appropriate to use haveged as a source of entropy on virtual machines?, it can be contested whether haveged provides quality entropy within a virtual environment.

---

It's not as simple as writing to /dev/random. From man random(4)

This differs from writing to /dev/random or /dev/urandom, which only adds some data but does not increment the entropy count. The following structure is used:

---

Looks like this could be implemented by reading from /dev/random and forwarding that entropy qrexec, ioctl(2), RNDADDENTROPY to VMs.

[marmarek]

Program attached there returns numbers between 10 and 100 (30-50 on my
system), so theoretically it means that VMs have access to real rdtsc.
This is on R3rc1 (Xen 4.4.2), not sure how about R2 (Xen 4.1.2). I'll
check Xen documentation later - I think I've seen some option for this.

Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?

[marmarek]

Program attached there returns numbers between 10 and 100 (30-50 on my
system), so theoretically it means that VMs have access to real rdtsc.
This is on R3rc1 (Xen 4.4.2), not sure how about R2 (Xen 4.1.2). I'll
check Xen documentation later - I think I've seen some option for this.

Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?

[adrelanos]

https://github.com/mirage/xentropyd

[adrelanos]

https://github.com/mirage/xentropyd

[jpouellet]

xref: https://groups.google.com/d/topic/qubes-devel/6qAR1lTBzn8/discussion

[jpouellet]

xref: https://groups.google.com/d/topic/qubes-devel/6qAR1lTBzn8/discussion

[adrelanos]

haveged is discouraged in VMs by Andre Seznec, one of haveged's main authors. Source:
BetterCrypto/Applied-Crypto-Hardening@cf7cef7#commitcomment-23006392

Entropy is needed before systemd / systemd-random-seed.service / haveged by the kernel (and possibly others, which is not researched). (Wrote a bit about that here #2704 (comment).)

Please reopen. @andrewdavidwong

[adrelanos]

haveged is discouraged in VMs by Andre Seznec, one of haveged's main authors. Source:
BetterCrypto/Applied-Crypto-Hardening@cf7cef7#commitcomment-23006392

Entropy is needed before systemd / systemd-random-seed.service / haveged by the kernel (and possibly others, which is not researched). (Wrote a bit about that here #2704 (comment).)

Please reopen. @andrewdavidwong

[andrewdavidwong]

Relevant to this issue is the exchange between @rustybird and @adrelanos in #2704 that starts here.

[andrewdavidwong]

Relevant to this issue is the exchange between @rustybird and @adrelanos in #2704 that starts here.

[InnovativeInventor]

I think that it would be good to note that passing random data directly from dom0 to a vm is a bad idea. Linux's entropy pool is not designed to recover from pool compromise (source). This is a problem if we are seeding compromised vms with random data. Hashing/using a one-way function would protect other vms and dom0 if there is a bad vm receiving dom0's random data.

[InnovativeInventor]

I think that it would be good to note that passing random data directly from dom0 to a vm is a bad idea. Linux's entropy pool is not designed to recover from pool compromise (source). This is a problem if we are seeding compromised vms with random data. Hashing/using a one-way function would protect other vms and dom0 if there is a bad vm receiving dom0's random data.

[v6ak]

 Linux's entropy pool is not designed to recover from pool compromise (source)

The linked article mentions that **fast** recovery is not a requirement.

This is a problem if we are seeding compromised vms with random data.

It isn't, or at least it shouldn't be. Not leaking anything substantial about its internal state is a basic requirement of RNGs. I believe that output from the RNG is already hashed or somehow else processed to ensure this property. There should be no need for additional hashing.

[v6ak]

 Linux's entropy pool is not designed to recover from pool compromise (source)

The linked article mentions that **fast** recovery is not a requirement.

This is a problem if we are seeding compromised vms with random data.

It isn't, or at least it shouldn't be. Not leaking anything substantial about its internal state is a basic requirement of RNGs. I believe that output from the RNG is already hashed or somehow else processed to ensure this property. There should be no need for additional hashing.

[adrelanos]

Since Qubes is using HVM and Qubes now, could we use VirtIO RNG?

https://wiki.qemu.org/Features/VirtIORNG

---

(As @HulaHoopWhonix suggested... Reworded by me...)

[adrelanos]

Since Qubes is using HVM and Qubes now, could we use VirtIO RNG?

https://wiki.qemu.org/Features/VirtIORNG

---

(As @HulaHoopWhonix suggested... Reworded by me...)

[v6ak]

Qubes 4 primarily uses PVH, not HVM. HVM requires QEMU, which runs in a separate PV domain. This increases memory/CPU requirements and attack surface (consider HVM->QEMU and PV->dom0 exploit chain). Note that both PV and QEMU are considered to be weak (at least by Qubes team), so Qubes tries to limit impact of vulnerabilities there. HVM needs to be currently used for PCI-enabled VMs (like sys-net and sys-usb) and VMs where OS requires it (e.g., Windows), because we currently* have no better way to support them. So no, I don't think virtio can be used with most VMs without decreasing overall security. *) Once PCI support for PVH (PVHv2?) is implemented, we could get rid of HVM usage for PCI enabled devices. Furthermore, I hope that stubdoms will use PVH one day, which should dramatically reduce the security concern about chained exploitation of QEMU and PV.

[v6ak]

Qubes 4 primarily uses PVH, not HVM. HVM requires QEMU, which runs in a separate PV domain. This increases memory/CPU requirements and attack surface (consider HVM->QEMU and PV->dom0 exploit chain). Note that both PV and QEMU are considered to be weak (at least by Qubes team), so Qubes tries to limit impact of vulnerabilities there. HVM needs to be currently used for PCI-enabled VMs (like sys-net and sys-usb) and VMs where OS requires it (e.g., Windows), because we currently* have no better way to support them. So no, I don't think virtio can be used with most VMs without decreasing overall security. *) Once PCI support for PVH (PVHv2?) is implemented, we could get rid of HVM usage for PCI enabled devices. Furthermore, I hope that stubdoms will use PVH one day, which should dramatically reduce the security concern about chained exploitation of QEMU and PV.

[marmarek]

It isn't, or at least it shouldn't be. Not leaking anything substantial about its internal state is a basic requirement of RNGs.

Regardless of actual state here, the current solution (seeding VM's RNG from dom0 as one of boot services) do hash random data extracted in dom0.

[marmarek]

It isn't, or at least it shouldn't be. Not leaking anything substantial about its internal state is a basic requirement of RNGs.

Regardless of actual state here, the current solution (seeding VM's RNG from dom0 as one of boot services) do hash random data extracted in dom0.