DispVMs: support for in-RAM execution only (for anti-forensics)

[marmarek]

Reported by joanna on 25 Sep 2014 20:26 UTC
Currently volatile.img is being backed up on the fs. See: https://groups.google.com/forum/#!topic/qubes-devel/QwL5PjqPs-4/discussion

Migrated-From: https://wiki.qubes-os.org/ticket/904

[marmarek]

Two options are evaluated for it:

• use RAM only (including turning off swap)
• encrypt everything written to disk with disposable key

/cc @qubesuser

[marmarek]

Two options are evaluated for it:

• use RAM only (including turning off swap)
• encrypt everything written to disk with disposable key

/cc @qubesuser

[marmarek]

Pasting @v6ak comment from linked discussion:

I've some scripts that I use for a temporary swapfile and temporary filesystem. They use a random key (from /dev/random). The tmp filesystem uses some configuration for better performance by disabling some features like journaling. (We don't need journaling for filesystem that are expected to be unreadable after reboot…)

My current usage:

1. The swapfile is attached automatically added in the background after 120 seconds in order not to block by reading from /dev/random during the system boot.
2. The largetmp is mounted only when needed, i.e. manually. (Yes, the usage of sudo suggests that…) I usually use tmpfs, but when I need something large, I mount the largetmp. I was thinking about automount of largetmp, but I was unsure about safety and some other potential issues. (Moreover, largetmp lies on rotational HDD, so using it instead of tmpfs could cause much more HDD usage and more power consumption.)

Some security considerations:

1. It is essential to handle safely situation when largetmp is not mounted. This is the reason why I use /tmp/large and not /large/tmp. If I forget to mount it, the worst thing that can happen is writing large amount of data to RAM. If it was in /large/tmp, accidental writing to a less protected partition (i.e. /large or /, which depends on the setup) may happen if the largetmp1 is not mounted. (Which is what I've once accidentally done. It was followed by several days of continuous wiping…)
2. The /dev/random is usually seeded from saved random seed. When some wear-levelling or relocation is used, the random seed might be available to local forensics, which could reduce the effective entropy of the key in the considered case. (Fortunatelly, the Qubes login screen requires some keystrokes, which adds some entrophy.)

There are the scripts and crypttab lines: https://gist.github.com/v6ak/3171313bc2c22efc263d

[marmarek]

Pasting @v6ak comment from linked discussion:

I've some scripts that I use for a temporary swapfile and temporary filesystem. They use a random key (from /dev/random). The tmp filesystem uses some configuration for better performance by disabling some features like journaling. (We don't need journaling for filesystem that are expected to be unreadable after reboot…)

My current usage:

1. The swapfile is attached automatically added in the background after 120 seconds in order not to block by reading from /dev/random during the system boot.
2. The largetmp is mounted only when needed, i.e. manually. (Yes, the usage of sudo suggests that…) I usually use tmpfs, but when I need something large, I mount the largetmp. I was thinking about automount of largetmp, but I was unsure about safety and some other potential issues. (Moreover, largetmp lies on rotational HDD, so using it instead of tmpfs could cause much more HDD usage and more power consumption.)

Some security considerations:

1. It is essential to handle safely situation when largetmp is not mounted. This is the reason why I use /tmp/large and not /large/tmp. If I forget to mount it, the worst thing that can happen is writing large amount of data to RAM. If it was in /large/tmp, accidental writing to a less protected partition (i.e. /large or /, which depends on the setup) may happen if the largetmp1 is not mounted. (Which is what I've once accidentally done. It was followed by several days of continuous wiping…)
2. The /dev/random is usually seeded from saved random seed. When some wear-levelling or relocation is used, the random seed might be available to local forensics, which could reduce the effective entropy of the key in the considered case. (Fortunatelly, the Qubes login screen requires some keystrokes, which adds some entrophy.)

There are the scripts and crypttab lines: https://gist.github.com/v6ak/3171313bc2c22efc263d

[rootkovska]

FWIW, I really like the idea of encrypting volatile.img with a one-time key and then throw it away after the DispVM shutdown. Can we ensure the dm will never write the key to dom0 fs anywhere? Do we need to disable swap in dom0 for that?

[rootkovska]

FWIW, I really like the idea of encrypting volatile.img with a one-time key and then throw it away after the DispVM shutdown. Can we ensure the dm will never write the key to dom0 fs anywhere? Do we need to disable swap in dom0 for that?

[marmarek]

No need to disable dom0 swap - VM memory is never written to disk (AFAIR
it isn't even supported).
The tricky part would be to integrate it with our DispVM implementation

• based on savefiles, to each DispVM have different key. But that's
  surely doable.

And it should be trivial for normal AppVMs when #1308 got implemented.

Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?

[marmarek]

No need to disable dom0 swap - VM memory is never written to disk (AFAIR
it isn't even supported).
The tricky part would be to integrate it with our DispVM implementation

• based on savefiles, to each DispVM have different key. But that's
  surely doable.

And it should be trivial for normal AppVMs when #1308 got implemented.

Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?

[v6ak]

First, I see two levels of anti-forensic DVMs. Attacker is not able to get any data from the DVM, if:

1. …the whole computer is shut down and he cannot get any RAM dump, but attacker knows the encryption password. (Supposing that RAM does not persist. AFAIK this is sometimes not kept on older RAM types (see cold boot attacks), but it is practically kept on modern RAMs.)
2. …the DVM is shut down. (But the instance of Qubes might be still running) Attacker is able to get RAM dump (and thus e.g. extract encryption keys for full-disk-encryption), but this is not useful for extracting information about the DVM that is not running. Attacker also knows the encryption password.

The level 1 protection can be implemented IMHO relatively easily in dom0:

• All temporary files (e.g. volatile.img and COWs) are saved to some filesystem encrypted by a disposable key.
• Any dom0 swap should be encrypted by a disposable key. (I am not sure about default setup on Qubes.) If this is not done, it might be hard to ensure that no related sensitive data (e.g. encryption keys of the temporary filesystem above) might be swapped on HDD and thus potentially exposed to the attacker. This is likely doable, but much harder.

Both of them this might be also useful for standard VMs and also or performance reasons, as the disposable FS might be configured not to persist data reliably (e.g. data=writeback, disabled metadata journal and so on). But this is rather a nice side effect than primary goal of this issue.

By design, hibernation can't be supported.

The level 2 protection seems to be much harder:

• Even having some swap in dom0 is potentially an issue. Remember that encryption currently runs in dom0. Maybe this could be offloaded to some “stubvm” for these cases. (Note that enough entropy in such stubvm has to be ensured in some way.)
• When DVM shuts down, the related memory must be wiped immediately. I am not sure if Xen does that. It is likely that it does that lazily (in the same way that Linux kernel without proper GRSec config does) for performance reasons. But doing that lazily implies a potential problem.
• When the DVM releases some allocated memory (through Qmemman), it also must be wiped soon enough. (Theoretically, wiping might be delayed after VM shutdown, but not more.)
• Maybe a proper DVM shutdown indication has to be implemented.

Rather a side note at this point: I have to correct my statement about /dev/random and relocation. With my today's knowledge, this issue is true only for /dev/urandom, not for /dev/random, at least if standard systemd is used.

[v6ak]

First, I see two levels of anti-forensic DVMs. Attacker is not able to get any data from the DVM, if:

1. …the whole computer is shut down and he cannot get any RAM dump, but attacker knows the encryption password. (Supposing that RAM does not persist. AFAIK this is sometimes not kept on older RAM types (see cold boot attacks), but it is practically kept on modern RAMs.)
2. …the DVM is shut down. (But the instance of Qubes might be still running) Attacker is able to get RAM dump (and thus e.g. extract encryption keys for full-disk-encryption), but this is not useful for extracting information about the DVM that is not running. Attacker also knows the encryption password.

The level 1 protection can be implemented IMHO relatively easily in dom0:

• All temporary files (e.g. volatile.img and COWs) are saved to some filesystem encrypted by a disposable key.
• Any dom0 swap should be encrypted by a disposable key. (I am not sure about default setup on Qubes.) If this is not done, it might be hard to ensure that no related sensitive data (e.g. encryption keys of the temporary filesystem above) might be swapped on HDD and thus potentially exposed to the attacker. This is likely doable, but much harder.

Both of them this might be also useful for standard VMs and also or performance reasons, as the disposable FS might be configured not to persist data reliably (e.g. data=writeback, disabled metadata journal and so on). But this is rather a nice side effect than primary goal of this issue.

By design, hibernation can't be supported.

The level 2 protection seems to be much harder:

• Even having some swap in dom0 is potentially an issue. Remember that encryption currently runs in dom0. Maybe this could be offloaded to some “stubvm” for these cases. (Note that enough entropy in such stubvm has to be ensured in some way.)
• When DVM shuts down, the related memory must be wiped immediately. I am not sure if Xen does that. It is likely that it does that lazily (in the same way that Linux kernel without proper GRSec config does) for performance reasons. But doing that lazily implies a potential problem.
• When the DVM releases some allocated memory (through Qmemman), it also must be wiped soon enough. (Theoretically, wiping might be delayed after VM shutdown, but not more.)
• Maybe a proper DVM shutdown indication has to be implemented.

Rather a side note at this point: I have to correct my statement about /dev/random and relocation. With my today's knowledge, this issue is true only for /dev/urandom, not for /dev/random, at least if standard systemd is used.

[marmarek]

I was thinking about implementing that volatile.img encryption inside of
VM, not in dom0. Dom0 would not know that encryption key. In this case
we can be sure that encryption key will not land in dom0 swap. Generally
this give us rather nice property - every data in DispVM is either in VM
RAM (not dom0 RAM), or written to volatile.img, encrypted. This doesn't
include some intentional leaks, cover channels etc, but this is offtopic
here.

As said earlier - VM memory is never written to dom0 swap (or any other
dom0 part). This, I think, fully solve the first case.

As for the second case, we "just" need to ensure that VM memory is wiped
when VM is shut down. In practice, I think this is the case, because all
the memory released by VM will be redistributed to other VMs. And before
VM get such memory, it is scrubbed by Xen. But in theory qmemman leave
some small part of memory (50MB) unassigned just for Xen internal use.
Theoretically it can happen that Xen will give some part from that pool
to the VM, instead of reusing just released memory - we do not control
which memory page is assigned where.
this pool and

Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?

[marmarek]

I was thinking about implementing that volatile.img encryption inside of
VM, not in dom0. Dom0 would not know that encryption key. In this case
we can be sure that encryption key will not land in dom0 swap. Generally
this give us rather nice property - every data in DispVM is either in VM
RAM (not dom0 RAM), or written to volatile.img, encrypted. This doesn't
include some intentional leaks, cover channels etc, but this is offtopic
here.

As said earlier - VM memory is never written to dom0 swap (or any other
dom0 part). This, I think, fully solve the first case.

As for the second case, we "just" need to ensure that VM memory is wiped
when VM is shut down. In practice, I think this is the case, because all
the memory released by VM will be redistributed to other VMs. And before
VM get such memory, it is scrubbed by Xen. But in theory qmemman leave
some small part of memory (50MB) unassigned just for Xen internal use.
Theoretically it can happen that Xen will give some part from that pool
to the VM, instead of reusing just released memory - we do not control
which memory page is assigned where.
this pool and

Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?

[v6ak]

The DVM can handle encryption of volatile.img (well, actually a block device backed by dom0:volatile.img), but I don't think it can properly handle encryption of COWs without some non-trivial tweaks.

When I run xentop, it seems to usually show several dozens of megabytes free, but I also remember having about 1.5 GiB free when runing few VMs on laptop with 16GiB RAM. I however agree with the statement that the one issue is likely much less serious than I initially thought.

[v6ak]

The DVM can handle encryption of volatile.img (well, actually a block device backed by dom0:volatile.img), but I don't think it can properly handle encryption of COWs without some non-trivial tweaks.

When I run xentop, it seems to usually show several dozens of megabytes free, but I also remember having about 1.5 GiB free when runing few VMs on laptop with 16GiB RAM. I however agree with the statement that the one issue is likely much less serious than I initially thought.

[qubesuser]

For level 1 I think dom0 swap needs to be encrypted with a disposable key because it's possible for both keystrokes, window contents and audio data to be swapped to dom0 swap

It might be possible to completely avoid that by mlocking Xorg, kwin, pulseaudio, qubes-guid, pacat-simple-vchan, etc. but it seems a risky route.

Encrypting dom0 swap with a disposable key is not hard at all: all you need is to add an entry to /etc/crypttab with /dev/urandom as the key and point the swap to the resulting DM device.

For the same reason, I don't think "Level 2 protection" is easy.

You definitely need to at least restart the whole GUI since otherwise there could be window contents, keystrokes and audio data in the heap or unused stacks of all processes.

Fixing that is infeasible since it requires patching gcc to clear all parts of the stack when adjusting it, hacking glibc to scrub memory on free() and unused parts of all thread stacks on request

Even restarting the GUI you'd still need to patch the Linux kernel to scrub kernel memory and GPU memory.

I think it's probably worthwhile to do a "best effort level 2" where VM memory is not leaked, but the UI leaks are not plugged.

In this case, due to dom0 4GB mem max it is not true that qmemman will redistribute all memory, so it probably requires a simple patch to Xen to scrub memory no longer assigned to VMs (or maybe Xen can already do it?).

[qubesuser]

For level 1 I think dom0 swap needs to be encrypted with a disposable key because it's possible for both keystrokes, window contents and audio data to be swapped to dom0 swap

It might be possible to completely avoid that by mlocking Xorg, kwin, pulseaudio, qubes-guid, pacat-simple-vchan, etc. but it seems a risky route.

Encrypting dom0 swap with a disposable key is not hard at all: all you need is to add an entry to /etc/crypttab with /dev/urandom as the key and point the swap to the resulting DM device.

For the same reason, I don't think "Level 2 protection" is easy.

You definitely need to at least restart the whole GUI since otherwise there could be window contents, keystrokes and audio data in the heap or unused stacks of all processes.

Fixing that is infeasible since it requires patching gcc to clear all parts of the stack when adjusting it, hacking glibc to scrub memory on free() and unused parts of all thread stacks on request

Even restarting the GUI you'd still need to patch the Linux kernel to scrub kernel memory and GPU memory.

I think it's probably worthwhile to do a "best effort level 2" where VM memory is not leaked, but the UI leaks are not plugged.

In this case, due to dom0 4GB mem max it is not true that qmemman will redistribute all memory, so it probably requires a simple patch to Xen to scrub memory no longer assigned to VMs (or maybe Xen can already do it?).

[v6ak]

The X11 leaks is a good point. And it is hard to get rid of that. I basically agree with the whole @qubesuser's post.

Restarting the GUI is theoretically feasible. When you restart X11, Qubes seems to remember some state and be able to continue working. (It is not perfect, though. For example, minimization states don't seem to be remembered.) With a separate GUI domain, a whole VM reboot could be probably implemented. I am, however, not sure if it is worth the work. It is just idea meaned: When you feel it is worth the work, you might find the idea useful.

Where have you found the 4GB mem max for dom0? My experience with Qubes does not confirm that, I can see higher memory amounts assigned to dom0. However, after stopping some VMs, I can see 1446384k free in xentop, which is much more than ~50MiB.

[v6ak]

The X11 leaks is a good point. And it is hard to get rid of that. I basically agree with the whole @qubesuser's post.

Restarting the GUI is theoretically feasible. When you restart X11, Qubes seems to remember some state and be able to continue working. (It is not perfect, though. For example, minimization states don't seem to be remembered.) With a separate GUI domain, a whole VM reboot could be probably implemented. I am, however, not sure if it is worth the work. It is just idea meaned: When you feel it is worth the work, you might find the idea useful.

Where have you found the 4GB mem max for dom0? My experience with Qubes does not confirm that, I can see higher memory amounts assigned to dom0. However, after stopping some VMs, I can see 1446384k free in xentop, which is much more than ~50MiB.

[marmarek]

On Mon, Oct 12, 2015 at 12:35:20PM -0700, qubesuser wrote:

Encrypting dom0 swap with a disposable key is not hard at all: all you need is to add an entry to /etc/crypttab with /dev/urandom as the key and point the swap to the resulting DM device.

When you do this manually, it's easy. When you need to script this (in
installer), things get (slightly) more complicated. But still not that
hard.

I think it's probably worthwhile to do a "best effort level 2" where VM memory is not leaked, but the UI leaks are not plugged.

Yes.

In this case, due to dom0 4GB mem max it is not true that qmemman will redistribute all memory, so it probably requires a simple patch to Xen to scrub memory no longer assigned to VMs (or maybe Xen can already do it?).

I'm not sure. By default VM have maxmem as half of physical memory, so if you have at least two running VMs (and haven't altered that default significantly), it shouldn't be needed.

Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?

[marmarek]

On Mon, Oct 12, 2015 at 12:35:20PM -0700, qubesuser wrote:

Encrypting dom0 swap with a disposable key is not hard at all: all you need is to add an entry to /etc/crypttab with /dev/urandom as the key and point the swap to the resulting DM device.

When you do this manually, it's easy. When you need to script this (in
installer), things get (slightly) more complicated. But still not that
hard.

I think it's probably worthwhile to do a "best effort level 2" where VM memory is not leaked, but the UI leaks are not plugged.

Yes.

In this case, due to dom0 4GB mem max it is not true that qmemman will redistribute all memory, so it probably requires a simple patch to Xen to scrub memory no longer assigned to VMs (or maybe Xen can already do it?).

I'm not sure. By default VM have maxmem as half of physical memory, so if you have at least two running VMs (and haven't altered that default significantly), it shouldn't be needed.

Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?

[marmarek]

On Mon, Oct 12, 2015 at 01:33:51PM -0700, Vít Šesták wrote:

Where have you found the 4GB mem max for dom0? My experience with Qubes does not confirm that, I can see higher memory amounts assigned to dom0. However, after stopping some VMs, I can see 1446384k free in xentop, which is much more than ~50MiB.

Done for R3.1: #1313

Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?

[marmarek]

On Mon, Oct 12, 2015 at 01:33:51PM -0700, Vít Šesták wrote:

Where have you found the 4GB mem max for dom0? My experience with Qubes does not confirm that, I can see higher memory amounts assigned to dom0. However, after stopping some VMs, I can see 1446384k free in xentop, which is much more than ~50MiB.

Done for R3.1: #1313

Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?

[v6ak]

Some implementation for anti-forensic DVMs (for volatile.img): https://groups.google.com/forum/#!topic/qubes-users/X0BBZ-kfix0

Drawbacks:

• Logs are not covered.
• Not much fine-grained. You have to reboot in order to make the old volatile.img unreadable.
• Some small part of current implementation (moving location of volatile.img) is a bit hacky at the moment.

Advantages unrelated to this goal:

• Some anti-forensics for other VMs (e.g. for swap).
• Likely some performance benefits (mainly for HDD) and less wear (mainly for SSD).

[v6ak]

Some implementation for anti-forensic DVMs (for volatile.img): https://groups.google.com/forum/#!topic/qubes-users/X0BBZ-kfix0

Drawbacks:

• Logs are not covered.
• Not much fine-grained. You have to reboot in order to make the old volatile.img unreadable.
• Some small part of current implementation (moving location of volatile.img) is a bit hacky at the moment.

Advantages unrelated to this goal:

• Some anti-forensics for other VMs (e.g. for swap).
• Likely some performance benefits (mainly for HDD) and less wear (mainly for SSD).

[Rudd-O]

From #1819 :

This is a feature request.

User requests DisposableVM via UX interacton.

dom0 script in charge of DisposableVM setup sets up the root file system as a device-mapper device. Then it sets up the swap device and the home directory device in the following manner:

do the exact same thing being done right now to create the block devices
generate cryptographically secure random key, 256 bits of entropy; this program must mlockall() to prevent that data being swapped
luksFormat and luksOpen those devices using the secure random key (which will be held in RAM and will never be swapped to disk); check that the assumption holds that invocation of these programs won't leak to swap
make the filesystems and swap devices atop those block devices

(To be honest, the swap devices of all VMs should be made atop that).

Teardown of devices is the exact opposite -- once the VM is dead, the devices must be luksClosed and then luksWiped.

Presto correcto mundo — unrecoverable devices associated with DisposableVMs, so long as the user does not write to anything other than /home.

This should not be too much of a complication compared to DisposableVM setup today.

[Rudd-O]

From #1819 :

This is a feature request.

User requests DisposableVM via UX interacton.

dom0 script in charge of DisposableVM setup sets up the root file system as a device-mapper device. Then it sets up the swap device and the home directory device in the following manner:

do the exact same thing being done right now to create the block devices
generate cryptographically secure random key, 256 bits of entropy; this program must mlockall() to prevent that data being swapped
luksFormat and luksOpen those devices using the secure random key (which will be held in RAM and will never be swapped to disk); check that the assumption holds that invocation of these programs won't leak to swap
make the filesystems and swap devices atop those block devices

(To be honest, the swap devices of all VMs should be made atop that).

Teardown of devices is the exact opposite -- once the VM is dead, the devices must be luksClosed and then luksWiped.

Presto correcto mundo — unrecoverable devices associated with DisposableVMs, so long as the user does not write to anything other than /home.

This should not be too much of a complication compared to DisposableVM setup today.

[andrewdavidwong]

Just doing a routine check: Is it still correct that @rootkovska is assigned to this issue?

[andrewdavidwong]

Just doing a routine check: Is it still correct that @rootkovska is assigned to this issue?

[marmarek]

As you've probably guessed, no.

[marmarek]

As you've probably guessed, no.

[andrewdavidwong]

User suggestion regarding per-VM encryption:

https://groups.google.com/d/msgid/qubes-devel/38301f18-0d04-dad2-511f-e5c56f255135%40yahoo.com

[andrewdavidwong]

User suggestion regarding per-VM encryption:

https://groups.google.com/d/msgid/qubes-devel/38301f18-0d04-dad2-511f-e5c56f255135%40yahoo.com

[jpouellet]

@marmarek wrote:

I was thinking about implementing that volatile.img encryption inside of VM, not in dom0.

I think it is safe to say that users may have compromised VMs which they would still like to have resist local forensics. Putting the crypto inside makes violating confidentiality trivial for the adversary. Putting it outside makes it harder.

I had the same objection when reading the storage domain section of the arch spec.

Thoughts?

[jpouellet]

@marmarek wrote:

I was thinking about implementing that volatile.img encryption inside of VM, not in dom0.

I think it is safe to say that users may have compromised VMs which they would still like to have resist local forensics. Putting the crypto inside makes violating confidentiality trivial for the adversary. Putting it outside makes it harder.

I had the same objection when reading the storage domain section of the arch spec.

Thoughts?

[jpouellet]

If we are trying to make the storage domain (thing touching disks) untrusted, then clearly it can not be allowed to handle keys, but neither should the VM we are trying to protect.

Consider also the case where you have an HVM with an OS that does not have disk encryption that you trust, or where disk unlocking can not be bootstrapped via kernel/initramfs fed from xen because it is not linux. IMO these scenarios still deserve ensured confidentiality as a feature, but can not provide it inside the VM.

To me, this suggests a middle crypto VM as a preferred option.

[jpouellet]

If we are trying to make the storage domain (thing touching disks) untrusted, then clearly it can not be allowed to handle keys, but neither should the VM we are trying to protect.

Consider also the case where you have an HVM with an OS that does not have disk encryption that you trust, or where disk unlocking can not be bootstrapped via kernel/initramfs fed from xen because it is not linux. IMO these scenarios still deserve ensured confidentiality as a feature, but can not provide it inside the VM.

To me, this suggests a middle crypto VM as a preferred option.

[marmarek]

Indeed middle crypto VM would solve both cases. Not sure if this should
be per-VM crypto VM, or one for all. In any case, I'm worried about
performance (yet another chain of xen-blkfront/xen-blkback in the
storage path) and even bigger memory footprint (especially in case of
per-VM crypto VM).
For HVM domains, stubdomain (domain hosting qemu) could be used. At
least for systems without PV drivers installed (which is the case for
Windows - we exclude block PV driver there, for unrelated reason). This
would not solve all the cases (for example running some live-cd Linux
system there), but at least some.

In case of DispVM in non-storage-domain world, the problem is - it's
hard to handle the key in dom0 and be sure it didn't landed in swap or
such. Is relying on crypsetup properly mlock-ing memory good idea?

Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?

[marmarek]

Indeed middle crypto VM would solve both cases. Not sure if this should
be per-VM crypto VM, or one for all. In any case, I'm worried about
performance (yet another chain of xen-blkfront/xen-blkback in the
storage path) and even bigger memory footprint (especially in case of
per-VM crypto VM).
For HVM domains, stubdomain (domain hosting qemu) could be used. At
least for systems without PV drivers installed (which is the case for
Windows - we exclude block PV driver there, for unrelated reason). This
would not solve all the cases (for example running some live-cd Linux
system there), but at least some.

In case of DispVM in non-storage-domain world, the problem is - it's
hard to handle the key in dom0 and be sure it didn't landed in swap or
such. Is relying on crypsetup properly mlock-ing memory good idea?

Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?

[jpouellet]

On Mon, Nov 14, 2016 at 4:44 PM, Marek Marczykowski-Górecki wrote:

In case of DispVM in non-storage-domain world, the problem is - it's
hard to handle the key in dom0 and be sure it didn't landed in swap or
such.

This is also true for other user secrets such as keystrokes and frame/audio buffers. IMO that should be solved generally, and is not a disk-crypto-specific issue.

Is relying on crypsetup properly mlock-ing memory good idea?

Perhaps not, but I would definitely prefer mlocked cryptsetup in dom0 over compromised cryptsetup in domU.

[jpouellet]

On Mon, Nov 14, 2016 at 4:44 PM, Marek Marczykowski-Górecki wrote:

In case of DispVM in non-storage-domain world, the problem is - it's
hard to handle the key in dom0 and be sure it didn't landed in swap or
such.

This is also true for other user secrets such as keystrokes and frame/audio buffers. IMO that should be solved generally, and is not a disk-crypto-specific issue.

Is relying on crypsetup properly mlock-ing memory good idea?

Perhaps not, but I would definitely prefer mlocked cryptsetup in dom0 over compromised cryptsetup in domU.