Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Changes to Automatic Partitioning and Swap to allow Hibernation Out-Of-The-Box #171

Open
eduncan911 opened this issue Jan 27, 2019 · 27 comments
Labels
bite-sized feature request A request for a new feature

Comments

@eduncan911
Copy link

eduncan911 commented Jan 27, 2019

This is related to #64 and possibly #63.

Distribution (run cat /etc/os-release):

NAME="Pop!_OS"
VERSION="18.10"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Pop!_OS 18.10"
VERSION_ID="18.10"
HOME_URL="https://system76.com/pop"
SUPPORT_URL="http://support.system76.com"
BUG_REPORT_URL="https://github.com/pop-os/pop/issues"
PRIVACY_POLICY_URL="https://system76.com/privacy"
VERSION_CODENAME=cosmic
UBUNTU_CODENAME=cosmic

Related Application and/or Package Version (run apt policy $PACKAGE NAME):

LUKS, LVM2

Issue/Bug Description:

18.10's partition schema does not allow hibernation. We get this:

  • 500MB EFI partition (cleartext)
  • 4GB Recovery?
  • LUKS containerA + LVM volume for / root partition
  • 4GB Swap (Cryptswap using random key on boot)

Two problems here:

  • The separate (outside-of-LVM-on-LUKS) 4GB cryptswap uses a random key. Attempting to use a static key would prompt user twice.
  • The swap partition is too small (could make an option during provisioning).

Steps to reproduce (if you know):

Install Pop!_OS 18.10 and select Erase Entire Disk.

Expected behavior:

  • 500MB EFI partition (cleartext)
  • 4GB Recovery?
  • LUKS containerA + LVM volume for / root partition
  • LUKS containerA + (size-of-mem+2GB) LVM volume for SWAP

Having the swap partition the size of ram + 2 GB (per specs) allows for hibernation.

Also, having it on the same LUKS container allows for only a single password to decrypt the swap partition alone with the root partition, which is the holy grail in Hibernation with LUKS!

For example, here's my successful LUKS + 2x LVM Pop!_OS modified volumes working with hibernation:

$ lsblk
nvme1n1         259:1    0 953.9G  0 disk  
├─nvme1n1p1     259:2    0   498M  0 part  /boot/efi
├─nvme1n1p2     259:3    0     4G  0 part  /recovery
└─nvme1n1p3     259:4    0 945.4G  0 part  
  └─cryptdata   253:0    0 945.4G  0 crypt 
    ├─data-root 253:1    0 911.4G  0 lvm   /
    └─data-swap 253:2    0    34G  0 lvm   [SWAP]

Note: I have 32 GB of ram - hence, the 34 GB swap partition on LUKS. This is a hard requirement for Linux to allow hibernation (aka suspend-to-disk).

If the user doesn't want hibernation, they can reclaim this space (resizing swap to 2 or 4 GB, and extending the root partition) in LVM2 much more easily than us than someone that wants to resize it larger to enable hibernation.

Other Notes:

I am writing a blog post on how to modify a Pop!_OS default install for this, as I got it working this weekend flawlessly.

Systemd now has a feature called "suspend-then-hibernate" that works in hibernation, as well as locking your screen between suspend and hibernation resume - including with LUKS (no other setup, even with pm-utils, was this clean and simple!). To use these built-in scripts for Hibernation, we have to change the partition layout. Doing this after the OS is installed is extremely difficult for the average user, and even worse when either LUKS or LVM is involved (my blog post will try to walk people through it).

The LUKS and swap partition arrangement can allow the use of the latest Systemd's Hibernation and Suspend-Hybrid features with a simple change to the OS. This is done by adding a single file:

# /etc/systemd/sleep.conf
[Sleep]
HibernateDelaySec=7200

Then it's up to the user to decide where to use this system. For example, on my system that I've modified the partitions for, I have enabled this in /etc/systemd/logind.conf using:

HandleSuspendKey=suspend-then-hibernate
HandleLidSwitch=suspend-then-hibernate
HandleLidSwitchExternalPower=suspend-then-hibernate

If we can instead partition a system differently ahead of time, it will be simple for users to enable systemd hibernation without any partition table and LUKS changes - which is very risky for them to perform.


/TL;DR

  • Drop the cryptswap at the end of the partition
  • Extend the existing LUKS container to the end of the disk
  • Add another volume to the LVM vgroup for a simple "swap", that must be the size of ram + 2 GB on the machine to hold the hibernation data for resume.

This, at a minimal, would allow users to:

  • Use a single password to unlock both the / and swap partition (yay!)
  • Allow the use of hibernation, with LUKS encrypted disks

Not a requirement due to the "can't use UUIDs, but direct device references with /dev/mapper/data-swap only" as this would break if users change their disks or install a new one potentially, but a nice to have would be:

  • Add the initramfs entry for resume, for any users that want it:

    $ cat /etc/initramfs-tools/conf.d/resume
    RESUME=/dev/mapper/data-swap

Which requires initramfs to be rebuilt.

@eduncan911
Copy link
Author

If someone can point me to the partitioning code, I'll take a stab at a PR and test it on a few machines here.

@jackpot51
Copy link
Member

We decided not to support hibernation in Pop!_OS due to the large number of machines we sell where the amount of RAM is a significant portion of the disk space.

@eduncan911
Copy link
Author

eduncan911 commented Jan 28, 2019

@jackpot51 i completely understand, given that situation

what I am originally proposing is to make it easier for someone to enable hibernation if they wanted. to do this, at a minimal, we would need to partition the entire disk using the LUKS volume for the entire data portion + swap portion:

nvme1n1         259:1    0 953.9G  0 disk  
├─nvme1n1p1     259:2    0   498M  0 part  /boot/efi
├─nvme1n1p2     259:3    0     4G  0 part  /recovery
└─nvme1n1p3     259:4    0 945.4G  0 part  
  └─cryptdata   253:0    0 945.4G  0 crypt 
    ├─data-root 253:1    0 911.4G  0 lvm   /
    └─data-swap 253:2    0     4G  0 lvm   [SWAP]

This means getting rid of the dedicated swap crypt partition, and moving it to a volume within the LVM2 vgroup. With just that change, things would be much easier for us who do want it to modify the LVM partitions only.

Now on the size of the swap disk... Yes, I can see systems sold that would be too large. So a 4GB swap partition would be fine. It's much easier to shrink the root LV and increase the size of the swap LV for us to have hibernation manually (and I'll write a doc for that for Pop!_OS).


Possibly one other proposal...

large number of machines we sell where the amount of RAM is a significant portion of the disk space

Could we add some simple logic to this statement? For example, let's say:

if (RAM / DISK) < THRESHOLD then
  swap_size = RAM + 2GB
else
  swap_size = 4GB
end

...and threshold can be something like "0.10" or 10%. For example, a 16 GB ram machine with 100 GB disk would be 16%, which is too high - that machine would get a standard 4 GB swap. However, a 16 GB machine with a 250 GB drive would only be using 6% of disk space - and would get a hibernation-size Swap partition of 18GB.

Or, we could just give the users the choice during provisioning with a checkbox.

@mmstick mmstick added bite-sized feature request A request for a new feature labels Mar 5, 2019
@flukejones
Copy link

I've been reading in to the hibernation aspect of Linux and the two things I find are:

  1. The kernel by default only requires 2/5ths of the size of the RAM to hibernate, eg: 16GB == 6.6GB hibernation image
  2. The kernel apparently compresses the image if required.

I have been using an 8GB swap with a 16GB RAM machine for a while now without any issues.

@flukejones
Copy link

flukejones commented Apr 13, 2019

Further, as discussed in this thread initially their could be options:

  • Encrypted hard-disk would require only the single password per the thread here,
  • Non-encrypted HDD would ask the user if they require hibernation:
    • Yes:
      • Encrypted + password
      • Non-encrypted
    • No:
      • Default random encryption

I wonder if it would be best to default to just two options? Encrypted HDD + hibernation, or non-encrypted + hibernation. The argument for this would be that users who understand why an encrypted swap would be required would be able to set it up themselves or they would use full disk encryption.

Swap size would be perhaps RAM / 5 * 2 + 2||4GB

Kernel doc references

@Alex-Muirhead
Copy link

I am writing a blog post

@eduncan911 Did you finish the blog post for this? I'd be rather interested in your solution.

@SuiNom
Copy link

SuiNom commented Jun 23, 2019

I am writing a blog post

@eduncan911 Did you finish the blog post for this? I'd be rather interested in your solution.

Seconded.

I got as far as creating the LVM on LUKS Following this guide:
https://wiki.archlinux.org/index.php/Dm-crypt/Encrypting_an_entire_system#LVM_on_LUKS
Up to but not including the "Preparing the boot partition" step - this gave me an encrypted set of LVs that I could decrypt with the Pop Installer natively.
Installed Pop!_OS and did a few bits of configuration, but I've been unable to get hibernate working successfully.
A blog post explaining it would be really helpful!

Edit: I've tried a few things for hibernation at the this point, including this guide which seems relatively thorough and includes all the steps i'd expect to make hibernation work, but still not dice.
https://outhereinthefield.wordpress.com/2019/05/21/enabling-hibernate-on-ubuntu-19-04-disco-dingo/

Edit 2: Turns out all I was missing was updating initramfs:
update-initramfs -u
I didn't manually run this, but it was run on my next kernel update and suddenly hibernation started working again.

Would be really good if pop supported this config out of the box - seems like a pretty key feature.

@sten0
Copy link

sten0 commented Jan 30, 2020

@jackpot51 @eduncan911, it's not necessary to have rootfs and swap in the same LUKS volume. The swap device can use a second LUKS volume, and that second LUKS volume can store a key derived from the rootfs LUKS volume in its (the LUKS volume for swap) second key slot. The effect is that only one password needs to be entered, during the pivot from initramfs to rootfs when the rootfs LUKS volume is unlocked. This becomes important if ever Pop! OS moves to btrfs, because IMHO btrfs layered on LVM is fragile, so it's a general solution that is future-compatible. It also works perfectly with systemd-boot, but I was unable to get it to work reliably enough with GRUB.

If unencrypted rootfs is desired, and the system needs encrypted swap to keep sensitive, decrypted data-in-RAM from being written to an SSD, and with the assumption that the user doesn't want to enter an additional password, then I suggest configuring a keyslot for the swap LUKS volume using a secondary passwordless method.

P.S. @jackpot51 I'm really impressed by what I've seen of Pop! OS so far. I'm in the process of enabling a new lease on life for a family member's early or mid 2012 Macbook Pro, and what impresses me is that Pop! OS has made most of the changes that I make for my Debian laptop, I'd like to commend your courage to digress from Ubuntu and Debian defaults (eg: by using systemd-boot). Your kernelstub package, plus the existing Debian facilities for using derived keys do 80% of the work, and what remains are associated changes to the installer (configuration of derived key stuff).

To support a wider variety of laptops, I recommend defaulting to the "shutdown" method rather than the often buggy "platform" one. If System76 laptops work with "platform" then this method could be automatically enabled in a post-install dpkg-configure hook using some kind of hardware ID whitelist. IMHO lack of hibernation support is genuine bug and represents a regression over Debian Stable, where it works out of the box on known-good hardware. For Pop! OS to grow in popularity I'd also suggest white-listing known-good ThinkPads, but I understand if that might conflict with your business model.

@GitJamz
Copy link

GitJamz commented May 24, 2020

Any news on this subject?
Is there any change for the 20.04 Pop-os?
I have a laptop with 8GB RAM that I would like to use hibernation together with my fully encrypted disk - from the installer picking automatic installation.

@dels07
Copy link

dels07 commented Sep 14, 2020

Any news on this?

@mewejo
Copy link

mewejo commented Jan 8, 2021

Is there any movement on this at all?

@sten0
Copy link

sten0 commented Jan 9, 2021

@dels07 @mewejo, it appears this isn't a priority for Pop!OS... :-( Here are is a solution I've tested, and one I'm planning to test.

Well tested, I've used it for years:

  1. Create ESP, 128MB "vault" partition, rootfs, and swap partitions.
  2. Encrypt the "vault" as LUKS1, and encrypt each of rootfs and swap as LUKS2 containers.
    • If I remember correctly, this is needed because LUKS2 doesn't allow derived keys.
  3. Move password in keyslot 0 of rootfs and swap LUKS2 containers to keyslot 1.
  4. Use the decrypt_derived script from cryptsetup to derive a key from the "vault", and insert this key in keyslot 0 of rootfs and swap LUKS2 containers.
  5. Use kernel command line arguments to use decrypt_derived when the system boots.
  6. Close the "vault" device.
  7. Clear keys from memory (does keyctl clear actually occur securely?)
  8. This works, despite https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=818158
  9. /etc/crypttab contains a single entry UUID=foo none luks
    • I'm not entirely sure if this is required, since everything should have been set up using kernel arguments. update-initramfs complains that there is a missing crypttab entry for the swapspace, but because kernel arguments are used, it works fine nonetheless. IIRC this was key to solving the systemd-related bug.

Not yet tested (unless I tested it years ago and it failed):

  1. Do the same thing, but without the "vault" LUKS1 container, and using decrypt_keyctl rather than decrypt_derived.
    • This method will also require kernel command line arguments, IIRC due to Bug #818158

Unfortunately, given that RAM is not encrypted, I suspect that neither of these options is truly secure, unless the point at 7 directly overwrites the memory block used to cache the passphrase or derived key.

@mewejo
Copy link

mewejo commented Jan 9, 2021

Thanks for the details @sten0

I did some more reading last night, found an article and gave it a go this morning. Turns out it works with a swapfile (I'm on ext4 and encrypted with a normal Pop!_OS install + kept using systemd-boot). I only get asked for the password once and then the kernel resumes from the image.

I have 32GB RAM, a 40GB swapfile is worth the cost on a 1TB drive for me. I admire what you've done with Pop and love it, but I think it should be the users choice to have hibernation at a cost of disk space at install time. IMO the policy of denying it to everyone because some laptops have small SSD's should be reviewed.

The article to setup encrypted hibernation with systemd-boot is here: https://abskmj.github.io/notes/posts/pop-os/enable-hibernate/ - huge credit to the author: @abskmj - thank you.

My next step is to find out if it's possible to close the lid, sleep and then only hibernate if required.

@flukejones
Copy link

@mewejo where do you get the idea that such a large swap is required? That used to be a rule two decades ago but much has changed. Kernel now performs highly efficient hibernation where the guide is it tries to reduce swap use to 2/5ths. You would be quite fine with swap set as half ram size.

@mewejo
Copy link

mewejo commented Jan 10, 2021

@flukejones primarily because I was fed up of hitting brick walls and wanted to rule out lack of space as one of the potential problems.

@eduncan911
Copy link
Author

eduncan911 commented Jan 11, 2021

Hello all. Apologies for not replying much sooner. My blog post and all notes were lost when my laptop was stolen and I was never able to post it.

Why is this a problem? See OP above: I outline what the current problem is (and the benefit) of a random key'd swap partition.

Note 1: Hibernation is currently spec'd in the linux kernel as requiring Memory + 2 GB of swap space. If you have 2GB or less of ram (e.g. RPi), then the spec is double the size of ram.

Note 2: I don't think you can use a swapfile. IIRC, it may have been a limitation of initramfs. So stick to swap partitions.

Note 3: Pop_OS! installer does correctly by creating LVM within LUKS. You could flip this around and create LUKS within LVM; however, some security experts recommend against this (me included); besides, you'll have two keys to deal with (or copies of keys to update twice). Let's stick to the default LUKS first -> then LVM volumes within that LUKS that Pop_OS sets up for us by default. That's what this guide below is for.

WARNING: Backup all data and be prepared to wipe your entire disk and start over. You may very well blow a gasket.

WARNING 2: The below is all from memory. I have done this 3 times though. Be prepared to DuckDuckGo each step.

  1. Wipe out the existing swap partition. If memory serves, it shows up as a /{disk}/cryptswap. Just delete it.
  2. Using parted, extend your root partition to all remaining disk (e.g. taking back the empty space left from step 1 above).
  3. Using cryptsetup, extend the cryptdata LUKS volume to total disk space (to reclaim the SWAP partition deletion). You can skip this, but you'll be missing those remaining GBs.
  4. Use LVM and shrink the Root / partition to make room for a larger SWAP partition (mem + 2GB in size). IIRC, this requires a little math and block sector calculations to use all because you now have some additional GB of space free, but you need to take that count and subtract the rest you need to equal a total of Mem+2GB free (I think have a few GB left over as I was conservative with my numbers).

NOTE: I can't recall if I needed to shrink the ext4 filesytem first though, before using LVM to shrink the volume. I do recall that this is a safe operation as you'll get errors if something is not resized first.

  1. Use LVM and now create your data-swap PV, which should be the size of your memory+2GB. I have 32GB of ram, so mine is 34GB in size.

Taking a pause here, step 5 is the trick to making hibernation work with a single key: we have a single LUKS unlocked partition, with a single passphrase. Any LVM volumes we create here will all benefit from the single LUKS passphase already unlocked. You are free to split up your /home, maybe /var/log, etc now. And all will work with a single LUKS passphrase.

  1. Follow existing steps when creating a new SWAP partition for Ubuntu (/etc/fstab, etc). Else, your system may not boot as it may be configured for the old swap location.

Now the fun stuff: setting booting to use the new swap partition during initd. LUKS I don';t think needs any further tweaking, as I think it inspects all partitions for any LUKS volumes during initd (which would prompt you for passphrases for each one). Since we only have 1, and the random swap is now gone, I think that's it.

There are a number of steps to take to make life a lot easier on yourself, especially during distro upgrades. I personally have upgraded two machines from Pop_OS! 18.04 to 18.10, 19.04, 19.10, 20.04, and 20.10 with zero issues. This only works because I have taken extra steps to ensure a smooth distro upgrade.

Unfortunately, I lost all of my notes to everything I did... So, I will try to recall from memory what all I did:

  • Look for a guide on setting up LUKS -> LVM with hibernation. IIRC, there is a step or two to take to edit initrd booting process to include LUKS to look for a system that has been hibernated in the swap partition. The key item is to have initd load any hibernation config/module to inspect /dev/mapper/data-swap at the very very beginning of the init process.
  • ^- When you are doing the above, ensure you are doing it the "upgrade" way - not manually. During kernel upgrades (part of normal system updates), you'll loose the changes - and loose hibernation! Apologies as I can't recall the procedure; but, there's a config file in /etc/ somewhere to edit so that when initramfs is built (which is now required after every kernel upgrade) it will always use your configuration. Basically, you do not want to run mkinitramfs after every kernel upgrade during system updates. There's a procedure to do this for you using a config file. This is what allows distro upgrades to be smooth and effortless.
  • Edit /etc/systemd/logind.conf to enable hibernation per laptop commands (PowerKey, SuspendKey, LidSwitch, LidSwitchDocked, etc).

My /etc/systemd/login.conf looks like (this is my preference, change as needed):

[Login]
#NAutoVTs=6
#ReserveVT=6
#KillUserProcesses=no
#KillOnlyUsers=
#KillExcludeUsers=root
#InhibitDelayMaxSec=5
HandlePowerKey=hibernate
HandleSuspendKey=suspend-then-hibernate
#HandleHibernateKey=hibernate
HandleLidSwitch=suspend-then-hibernate
HandleLidSwitchExternalPower=suspend-then-hibernate
HandleLidSwitchDocked=ignore
#PowerKeyIgnoreInhibited=no
#SuspendKeyIgnoreInhibited=no
#HibernateKeyIgnoreInhibited=no
#LidSwitchIgnoreInhibited=yes
HoldoffTimeoutSec=5s
#IdleAction=ignore
#IdleActionSec=30min
#RuntimeDirectorySize=10%
#RemoveIPC=yes
#InhibitorsMax=8192
#SessionsMax=8192

^- This really should be in the Gnome settings panel. I really don't like editing system configs like that.

  • Lastly, there's a Suspend-Then-Hibernation setting to edit. IIRC, you need to create this file:
# nvim /etc/systemd/sleep.conf
[Sleep]
HibernateDelaySec=43200

^- This really should be in the Gnome settings panel as well.

This setting is the timeout that the system switches from Suspend and then to Hibernation. NOTE: This works pretty much flawlessly. The only exception is that the system doesn't detect when it's battery is too low to stay in suspend - and just looses power and dies (taking the suspend-to-ram session with it). There is an option to hibernate-then-sleep with Ubuntu, but I haven't played with that yet.


This is why we should edit the Pop_OS Installer to do this FOR US at installation time.

@sten0
Copy link

sten0 commented Jan 11, 2021 via email

@chuck-alt-delete
Copy link

chuck-alt-delete commented Aug 18, 2021

Just want to add my opinion as a System76 laptop customer. I consider hibernation as table stakes for a modern laptop. It almost doesn’t matter how good the laptop is. If I can’t reliably open and close it whenever I want for weeks on end, it’s a huge hindrance to my user experience. Powering off between uses or running out of battery in two days of closed lid is not an acceptable user experience, period.

I enjoy my Lemur Pro, but if I had known it doesn’t have hibernation out of the box, I wouldn’t have bought it. I was a vocal advocate for System76 and Pop_OS, but now I can’t recommend it.

@mguthaus
Copy link

Really want this to become an option. I just messed up my partitions trying to repartition to support hibernation as I'm sure a bunch of other users did. I agree that hibernation is a basic thing that users just want working.

@ianhundere
Copy link

@mguthaus i'd try with a swapfile...it's easier and more straightforward especially if you want encryption.

@mguthaus
Copy link

@ianhundere Great suggestion. I just set this up and it is working great. yay!

@chuck-alt-delete
Copy link

@ianhundere Great suggestion. I just set this up and it is working great. yay!

Could you post a reference to a guide you followed for other who stumble upon this issue?

@leviport
Copy link
Member

@chuck-alt-delete pop-os/pop#1966 may be of some help

@ianhundere
Copy link

^ ou, that hibernation button is kinda cool.

that's one way to go about it. i can write something up on how to do it via a swapfile tho. i've since removed my swap partition.

@mguthaus
Copy link

mguthaus commented Oct 22, 2021

@chuck-alt-delete I used this for the swapfile creation:
https://abskmj.github.io/notes/posts/pop-os/enable-hibernate/

@ianhundere
Copy link

@chdenizard
Copy link

I would like to migrate to Pop Os, but hibernation is important to me..
Unfortunately I am going to postpone my Pop OS tryout.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bite-sized feature request A request for a new feature
Projects
None yet
Development

No branches or pull requests