Wipe RAM on shutdown

[adrelanos]

Some stuff that Tails is having in mind.

---

package:

http://git.tails.boum.org/wiperam/tree/

---

Tails currently has a few issues with it.

https://tails.boum.org/support/known_issues/index.en.html#index23h2

On rare systems (non-PAE with big amounts of memory), Tails does not consistently erase all system memory as it should.

The other issue is an obvious one. If shutdown fails for software or hardware reasons, RAM shutdown won't be executed.

https://labs.riseup.net/code/issues/6006

After following the steps for "erase memory on shutdown it turns out some patterns may survive the wipe which indicates that not all memory was wiped. One theory for this is that the new kernel loaded by kexec may allocate some buffer, which then is filled with the pattern and won't get cleared, since sdmem only wipes unallocated memory.

Hi, not sure how to comment so I will just add it here. The test is incorrect, since on a 32-bit architecture the address space of a given process is usually limited at 3 GiB (or less, depends on the kernel configuration). It's not easy to change the Python script to fill all available memory (it's possible, but complex due to kernel's OOM killer behavior), but your best bet would be to spawn many pattern filler processes simultaneously, hoping that at least some executions will overlap. Then, on a 4 GiB machine you will see about 1 GiB of non-wiped memory (and on systems with > 4 GiB RAM, with PAE enabled, you won't wipe that extra memory, too, of course). Sorry to burst your bubble -- wiping memory is not easy, and THC's SecureDelete package is mostly badly implemented snake oil based on wrongly interpreted old theoretical papers. Anyway, to see whether the 3 GiB limit has anything to do with the unwiped patterns you have seen thus far, retry on a system with less than 3 GiB RAM. --MK

In theory, it's supposed to work "most of the time", but it is not reliable as is: sdmem kills itself as soon as it's refused to allocate memory, so there's no guarantee several instances of it will be allocating enough memory at the same time to ensure all memory is erased. In the worst case, this change can make the memory erasure process 32 times longer, with no efficiency improvement.

And more.

---

Tails blueprint:

https://tails.boum.org/blueprint/more_efficient_memory_wipe/

---

Documentation on testing if wiping RAM works:

https://tails.boum.org/contribute/release_process/test/erase_memory_on_shutdown/

---

Test suite recipe:

https://github.com/vjrj/tails/blob/master/features/erase_memory.feature

---

Documentation:

https://tails.boum.org/doc/advanced_topics/cold_boot_attacks/index.en.html

---

Related:

• DisposableVMs: support for in-RAM execution only (for anti-forensics) #904
• Wipe Video RAM on Shutdown #1563

[tasket]

Good feature, but a little reminder: Some memory controllers scramble addresses and data using prng pattern, such as on Intel Core processors using ddr3. So far haven't seen anyone defeat this, so maybe some users will feel less vulnerable. See pp. 26-29 http://www.slideshare.net/codeblue_jp/igor-skochinsky-enpub

[awokd]

Looks like the kernel is compiled with CONFIG_XEN_SCRUB_PAGES=y. That should result in DomUs getting wiped on memory free operations, including a shutdown. Was this issue filed against the host itself? #2024 refers mostly to VMs.

[andrewdavidwong]

Updating title to explicitly include things mentioned in #4488.

[marmarek]

On memory balancing, the memory definitely is zeroed before returning it to Xen. On VM shutdown/crash/etc it is too, but I'm not sure if it's done synchronously, or asynchronously - IOW at which point in time it is guaranteed to be cleared already.

[andrewdavidwong]

On memory balancing, the memory definitely is zeroed before returning it to Xen. On VM shutdown/crash/etc it is too, but I'm not sure if it's done synchronously, or asynchronously - IOW at which point in time it is guaranteed to be cleared already.

Updated title to:

Wipe RAM of VMs synchronously when shut down and killed

(Feel free to change it further.)

[adrelanos]

Progress on this feature has been made. Source code review and suggestions for the sdmem parameters are welcome, see:
https://forums.whonix.org/t/is-ram-wipe-possible-inside-whonix-cold-boot-attack-defense/5596/39

This is now in the development version of security-misc. (#1885)

[adrelanos]

This progressed far, has now a dedicated source code repository and sub project page.

• https://www.kicksecure.com/wiki/Ram-wipe
• https://github.com/Kicksecure/ram-wipe
• https://www.kicksecure.com/wiki/Dev/RAM_Wipe

Nowadays dracut based. But could probably easily ported to mkosi-initrd if that ever becomes the default init system in Qubes dom0 host.

Remaining issue is actually a dracut issue, which is not easy to fix.

• dracut should unmount the root encrypted disk cryptsetup luksClose during shutdown dracutdevs/dracut#1888
• fix(crypt): close crypt devices to release encryption keys get on shutdown dracutdevs/dracut#2471

[renehoj]

What is the main scenario where this is useful, in what situation is the VM shutdown in a controlled manner, but you can't rely on Xen to clear the memory?

[adrelanos]

I was also wondering what I was thinking when ~9 years ago I set the title to

Wipe RAM of VMs synchronously when shut down and killed

which from nowadays perspective does not make a lot sense to me. Except after you asked, I checked the history and the title I've chosen was actually:

Wipe RAM on Shutdown

This would be a dom0 feature. This is what I was suggesting and what I still think makes sense. Why is it useful? It's for cold boot attack defense. Reference:
https://www.kicksecure.com/wiki/Cold_Boot_Attack_Defense

@adw changed the title Wipe RAM on Shutdown Wipe RAM of VM when it shuts down or is killed, and during memory balancing on Sep 14, 2019

Wipe RAM of VMs synchronously when shut down and killed

I don't think the title should have been changed or the scope of the ticket as complicated as it is by itself should have been extended.

• Wipe RAM of VM when it shuts down or is killed, and during memory balancing #4488 is in my opinion not a duplicate of this ticket.

To answer your question...

What is the main scenario where this is useful, in what situation is the VM shutdown in a controlled manner, but you can't rely on Xen to clear the memory?

I didn't suggest VM memory to be cleared when a VM is closed. Maybe that's #4488. This ticket should only be for Wipe RAM on Shutdown.

[renehoj]

This patch might be relevant, at least it confirms that scrubbing doesn't always get scheduled until the page is allocated.

https://github.com/OpenXT/xenclient-oe/blob/master/recipes-extended/xen/files/memory-scrub-on-domain-shutdown.patch

[brendanhoar]

My understanding is that the Xen default policy has been to scrub RAM only when allocating it to a VM or to some other internal Xen purpose.

Also this would not be a dom0 feature, it would need to be a Xen feature (it's possible there's already a policy flag for this?).

Deallocated VM RAM is released to and managed by Xen, not dom0. Dom0 itself is just a special (often management) VM.

B

[renehoj]

If dom0 just is a VM, does that mean the scrub-domheap option also apply to releasing the pages used by dom0?

[brendanhoar]

If dom0 just is a VM, does that mean the scrub-domheap option also apply to releasing the pages used by dom0?

Yes, when any domain is returning memory to Xen, including dom0.

I hadn't noticed the addition of the scrub-domheap option in 2019, thanks for pointing that out. Presumably this prevents leaks of vm secrets into Xen controlled memory.

My understanding still is that Xen already scrubs memory before giving it to a VM, perhaps this option is meant to ensure VM secrets don't spend any more time in RAM than the lifetime of the VM (system guaranteed full-scrubbed of secrets on VM shutdown).

[renehoj]

My understanding still is that Xen already scrubs memory before giving it to a VM, perhaps this option is meant to ensure VM secrets don't spend any more time in RAM than the lifetime of the VM (system guaranteed full-scrubbed of secrets on VM shutdown).

I read that scrubbing is done during the cpu idle time across all cores, if you have free memory and your cpu never idles, I don't think there is any guarantee the pages are scrubbed until the memory is used again.

As I understand it, scrub-domheap always scrub the memory when it's released, it doesn't wait for the cpu to be idle.