Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Boot partition runs out of space within months #1714

Open
KloudKoder opened this issue May 28, 2021 · 45 comments
Open

Boot partition runs out of space within months #1714

KloudKoder opened this issue May 28, 2021 · 45 comments

Comments

@KloudKoder
Copy link

Distribution (run cat /etc/os-release):

20.10

Related Application and/or Package Version (run apt policy $PACKAGE NAME):

/boot

Issue/Bug Description:

After typically a few months of use, I get a warning saying that my /boot folder is running low on space. This is because too many old Linux images and associated files are being saved. At most, there only need to be 2 of them, namely, vmlinuz and vmlinuz.old (and associated files). Probably anything beyond that should be deleted as soon as the latest upgrade reboots successfully all the way to desktop.

Steps to reproduce (if you know):

Just keep getting updates until /boot fills up. Most users probably wouldn't know what to do about it, resulting in a costly reinstall.

Expected behavior:

Either delete tertiary OS images and associated files, or change the installer to reserve like 10X as much in /boot (which would still make it quite trivially sized).

Other Notes:

@jacobgkau
Copy link
Member

jacobgkau commented May 28, 2021

At least on EFI installations, the /boot directory is just part of the / partition. The /boot/efi subdirectory is the separate EFI System Partition (ESP) that can fill up separately from the rest of the system.

The ESP should be 512M by default, and it should only contain two sets of vmlinuz/initrd files, current and old.

│   ├── Pop_OS-92abb9cf-b62d-4c49-93c9-ef9a18a3abce
│   │   ├── cmdline
│   │   ├── initrd.img
│   │   ├── initrd.img-previous
│   │   ├── vmlinuz.efi
│   │   └── vmlinuz-previous.efi

The files in this directory aren't named with version numbers, the old ones just have -previous appended, so having more than two shouldn't happen by default. Did you install with a non-default partition setup, and are you dual-booting this machine with any other OS's that are keeping files in the ESP?

If you believe this is an issue with Pop!_OS rather than your configuration, please provide the output of the following commands (in a text file):

lsblk
df -h
sudo tree /boot
apt list --installed

The tree command might require installing tree first with sudo apt install tree.


System76 customers can reach out to support for technical assistance. For non-System76 hardware, you can seek community support on Reddit or Mattermost.

@KloudKoder
Copy link
Author

@jacobgkau Thanks for the feedback. Definitely my install is as routine as it gets. One partitiion, totally reformatted during the install process. No weird extra harddrives. And I'm not using any apps that should have anything to do with /boot whatsoever. So it's quite a mystery, especially since you mention "-previous", which is not present in any filename. Just ".old" on some of the links.

There is no /boot/efi. Does that mean that I'm somehow not using EFI?

I'm not really comfortable dumping fingerprinting info on Github, but I'm also not completely averse to sharing some info.

@jacobgkau
Copy link
Member

There is no /boot/efi. Does that mean that I'm somehow not using EFI?

Probably. You can confirm with ls /sys/firmware/efi/, the directory will exist if you're booted in UEFI mode and it will not exist if you're booted in legacy BIOS mode.

One partitiion, totally reformatted during the install process.

My question now would be how your boot partition could be filling up if you don't have a boot partition. You should not be receiving notifications for a "folder" filling up. Can you please provide a screenshot of the warning you're referring to?

I'm not really comfortable dumping fingerprinting info on Github,

I will explain why I need each output that I asked for:

  • lsblk: to confirm your partition configuration.
  • df -h: to confirm the low free space.
  • sudo tree /boot: to show what files are present that would be taking up the free space.
  • apt list --installed: to show which packages you have installed that would place files in your boot partition (typically kernel packages.)

Depending on the output of the final command, I would recommend trying a sudo apt autoremove --purge to clear out older kernel versions.

@KloudKoder
Copy link
Author

@jacobgkau Thanks for the thorough notes. It turns out that this is my particular firmware problem. At least, that's why it's not booting in EFI (and no, it's not System76). Therefore I'm probably in a small minority of users, so I'll just deal with it manually instead of asking the team to fix it. As to the error message about low space, it occurs in the PopShop, but it went away when I deleted the stale kernel images from /boot a while back.

The autoremove tip sounds like a good idea, but I did it once in another OS context and bricked the machine.

@jacobgkau
Copy link
Member

jacobgkau commented Jun 4, 2021

The autoremove tip sounds like a good idea, but I did it once in another OS context and bricked the machine.

The sudo apt autoremove command should always be safe to run (and should generally do nothing if you're running it periodically); if running that breaks your system, then other package management issues are preexisting. You can run sudo apt install pop-desktop system76-driver (or omit the second package if you're not on System76 hardware) after running an autoremove to try and ensure you have the necessary packages to boot your system installed.

As to the error message about low space, it occurs in the PopShop, but it went away when I deleted the stale kernel images from /boot a while back.

Please take a screenshot next time you see this error message/notification so we can see which one you're talking about.

@KloudKoder
Copy link
Author

@jacobgkau "Please take a screenshot next time you see this error message/notification so we can see which one you're talking about." OK it's a deal. Presumably within the next few months. Thanks for the detailed comments.

@KloudKoder
Copy link
Author

@jacobgkau It happened again in Pop!_Shop! (It actually happened at least twice but I forgot to copy down the message last time.) This interface won't allow me to attach a screenshot, but you can search the source code for the following error message (note the improper capitalization, which might help in searching, but ignore "boot" which is of course variable):

Low Disk Space on "boot"
The volume "boot" has only 0 bytes disk space remaining.

@KloudKoder
Copy link
Author

As before, I fixed the problem by going into /boot and doing "sudo rm" with a set of 3 outdated system binaries that were no longer the targets of any simlinks.

@jacobgkau
Copy link
Member

That is not a screenshot. Did you see the error message within the Pop!_Shop window, or was it at the top of the screen as a notification?

Did you try running an autoremove like I previously recommended this time, before manually deleting those files? That would tell us whether the package manager is capable of handling the problem.

@KloudKoder
Copy link
Author

@jacobgkau

That is not a screenshot.

See above: "This interface won't allow me to attach a screenshot". It's also gone now, but surely the error message can be found by searching the sourcecode for pieces of it. And yes, it was at the top of the screen as a notification.

My bad that I forgot to run autoremove. I'll hopefully remember if this occurs again.

@jacobgkau
Copy link
Member

See above: "This interface won't allow me to attach a screenshot".

The GitHub interface does allow attaching screenshots, either by dragging and dropping into your message or by clicking Attach files by dragging & dropping, selecting or pasting them. at the bottom of the message window:

image

My bad that I forgot to run autoremove. I'll hopefully remember if this occurs again.

Please let us know what you find.

@KloudKoder
Copy link
Author

@jacobgkau LOL I never looked down at the bottom of the response dialog because in literally every other forum tool I've ever used, there's an attachment button in the same toolbar as the one containing text style and linking icons. And that "Attach files..." thing looks like a grayed-out tooltip, not a button. But you're obviously correct in your assertion, so I'll use it next time this happens. Statistically speaking, I'll return with a screenshot in a few months. Sorry about that.

@KloudKoder
Copy link
Author

@jacobgkau I have an update on this. I tried autoremove and it DELETED my entire /boot folder, leaving all else intact. It then failed with an error, having successfully rendered my machine unusable and effectively unrecoverable. Fortunately I happen to have realtime backup so my data survived the disaster.

Anyways, with respect to the error message that came up when I ran out of space in the boot folder, see attached.
fail0
fail1

@jacobgkau
Copy link
Member

I tried autoremove and it DELETED my entire /boot folder, leaving all else intact. It then failed with an error, having successfully rendered my machine unusable and effectively unrecoverable.

Can you please provide the complete terminal output of this transaction, including the error you're referring to?

@jacobgkau
Copy link
Member

jacobgkau commented Jan 3, 2022

Fortunately I happen to have realtime backup so my data survived the disaster.

Your /boot folder being removed would not harm your data, unless you are storing personal data within the /boot folder, which would be strange. Should this happen again, I would suggest following the bootloader repair support article next time to see if that gets the system booting again. (If it doesn't, any issues you run into might be relevant for determining the cause of the initial issue.)

@KloudKoder
Copy link
Author

KloudKoder commented Jan 21, 2022

@jacobgkau Sorry for the delay. You're lucky that I thought to capture the autoremove output just before I shut down, only to find /boot deleted when I powered back up. And of course you're correct: the data was intact (independent of my backup). But thanks for the boot repair link, which I didn't know about. I'll use that if this happens again.

...oops, it's happening again right now! Which is why I remembered that I needed to get back to you. I'll just fix it by deleting unused files in /boot. (I'll rename them first, then reboot and make sure everything is OK, before actually deleting them.)

Anyways, here's your "complete terminal output of this transaction". Note where it starts going wonky because it has issues with cryptswap. (OMG! It's an encrypted drive! What do we do now?!)

Thanks as always. I hope this will give you some insights.

Sorry for the horrible formatting. I don't know how to fix it and "code" tags make things even worse. Github is picking up random text as formatting operations.


kloudkoder@cloud:/boot$ ls -l
total 391980
-rw-r--r-- 1 root root    256690 Sep 14 15:19 config-5.13.0-7614-generic
-rw-r--r-- 1 root root    257004 Oct 21 10:38 config-5.13.0-7620-generic
-rw-r--r-- 1 root root    259191 Nov 29 10:59 config-5.15.5-76051505-generic
-rw-r--r-- 1 root root    259480 Dec 14 13:07 config-5.15.8-76051508-generic
drwxr-xr-x 4 root root      4096 Dec 18 11:30 grub
lrwxrwxrwx 1 root root        34 Dec 28 07:24 initrd.img -> initrd.img-5.15.8-76051508-generic
-rw-r--r-- 1 root root 110596979 Dec 18 11:28 initrd.img-5.13.0-7614-generic
-rw-r--r-- 1 root root 110620282 Dec 18 11:29 initrd.img-5.13.0-7620-generic
-rw-r--r-- 1 root root 113376177 Dec 18 11:30 initrd.img-5.15.5-76051505-generic
lrwxrwxrwx 1 root root        34 Dec 28 07:24 initrd.img.old -> initrd.img-5.15.5-76051505-generic
drwx------ 2 root root     16384 Sep 21 07:57 lost+found
-rw------- 1 root root   5947253 Sep 14 15:19 System.map-5.13.0-7614-generic
-rw------- 1 root root   5949831 Oct 21 10:38 System.map-5.13.0-7620-generic
-rw------- 1 root root   6013722 Nov 29 10:59 System.map-5.15.5-76051505-generic
-rw------- 1 root root   6204418 Dec 14 13:07 System.map-5.15.8-76051508-generic
lrwxrwxrwx 1 root root        31 Dec 28 07:24 vmlinuz -> vmlinuz-5.15.8-76051508-generic
-rw------- 1 root root  10150592 Sep 14 15:19 vmlinuz-5.13.0-7614-generic
-rw------- 1 root root  10151296 Oct 21 10:38 vmlinuz-5.13.0-7620-generic
-rw------- 1 root root  10292192 Nov 29 10:59 vmlinuz-5.15.5-76051505-generic
-rw------- 1 root root  10992640 Dec 14 13:07 vmlinuz-5.15.8-76051508-generic
lrwxrwxrwx 1 root root        31 Dec 28 07:24 vmlinuz.old -> vmlinuz-5.15.5-76051505-generic
kloudkoder@cloud:/boot$ sudo autoremove --purge
[sudo] password for kloudkoder: 
sudo: autoremove: command not found
kloudkoder@cloud:/boot$ sudo apt autoremove
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following packages will be REMOVED:
  libllvm11 linux-headers-5.11.0-7633 linux-headers-5.11.0-7633-generic linux-image-5.11.0-7633-generic linux-modules-5.11.0-7633-generic linux-modules-extra-5.11.0-7633-generic
  python3-distro-info python3-distupgrade python3-update-manager
0 upgraded, 0 newly installed, 9 to remove and 0 not upgraded.
2 not fully installed or removed.
After this operation, 501 MB disk space will be freed.
Do you want to continue? [Y/n] y
Setting up initramfs-tools (0.136ubuntu6.6) ...
update-initramfs: deferring update (trigger activated)
Setting up linux-image-5.15.8-76051508-generic (5.15.8-76051508.202112141040~1639505278~20.04~0ede46a) ...
(Reading database ... 355006 files and directories currently installed.)
Removing libllvm11:amd64 (1:11.0.0-2~ubuntu20.04.1) ...
Removing linux-headers-5.11.0-7633-generic (5.11.0-7633.35~1630100930~20.04~ae2753e) ...
Removing linux-headers-5.11.0-7633 (5.11.0-7633.35~1630100930~20.04~ae2753e) ...
Removing linux-modules-extra-5.11.0-7633-generic (5.11.0-7633.35~1630100930~20.04~ae2753e) ...
Removing python3-distupgrade (1.0.0~1639771550~20.04~d87beb5) ...
Removing python3-update-manager (1.0.0~1639771550~20.04~d87beb5) ...
Removing python3-distro-info (0.23ubuntu1) ...
Removing linux-modules-5.11.0-7633-generic (5.11.0-7633.35~1630100930~20.04~ae2753e) ...
Removing linux-image-5.11.0-7633-generic (5.11.0-7633.35~1630100930~20.04~ae2753e) ...
/etc/kernel/prerm.d/dkms:
dkms: removing: system76 1.0.13~1630342318~20.04~68bd479 (5.11.0-7633-generic) (x86_64)

-------- Uninstall Beginning --------
Module:  system76
Version: 1.0.13~1630342318~20.04~68bd479
Kernel:  5.11.0-7633-generic (x86_64)
-------------------------------------

Status: Before uninstall, this module version was ACTIVE on this kernel.

system76.ko:
 - Uninstallation
   - Deleting from: /lib/modules/5.11.0-7633-generic/updates/dkms/
 - Original module
   - No original module was found for this module on this kernel.
   - Use the dkms install command to reinstall any previous module version.

depmod...

DKMS: uninstall completed.
dkms: removing: system76-io 1.0.1~1616424744~20.04~57c15ff (5.11.0-7633-generic) (x86_64)

-------- Uninstall Beginning --------
Module:  system76-io
Version: 1.0.1~1616424744~20.04~57c15ff
Kernel:  5.11.0-7633-generic (x86_64)
-------------------------------------

Status: Before uninstall, this module version was ACTIVE on this kernel.

system76-io.ko:
 - Uninstallation
   - Deleting from: /lib/modules/5.11.0-7633-generic/updates/dkms/
 - Original module
   - No original module was found for this module on this kernel.
   - Use the dkms install command to reinstall any previous module version.

depmod...

DKMS: uninstall completed.
/etc/kernel/postrm.d/initramfs-tools:
update-initramfs: Deleting /boot/initrd.img-5.11.0-7633-generic
/etc/kernel/postrm.d/zz-update-grub:
Sourcing file `/etc/default/grub'
Sourcing file `/etc/default/grub.d/init-select.cfg'
Generating grub configuration file ...
Found linux image: /boot/vmlinuz-5.15.8-76051508-generic
Found linux image: /boot/vmlinuz-5.15.5-76051505-generic
Found initrd image: /boot/initrd.img-5.15.5-76051505-generic
Found linux image: /boot/vmlinuz-5.13.0-7620-generic
Found initrd image: /boot/initrd.img-5.13.0-7620-generic
Found linux image: /boot/vmlinuz-5.13.0-7614-generic
Found initrd image: /boot/initrd.img-5.13.0-7614-generic
done
Processing triggers for libc-bin (2.31-0ubuntu9.2) ...
Processing triggers for linux-image-5.15.8-76051508-generic (5.15.8-76051508.202112141040~1639505278~20.04~0ede46a) ...
/etc/kernel/postinst.d/dkms:
 * dkms: running auto installation service for kernel 5.15.8-76051508-generic
   ...done.
/etc/kernel/postinst.d/initramfs-tools:
update-initramfs: Generating /boot/initrd.img-5.15.8-76051508-generic
cryptsetup: WARNING: Resume target cryptswap uses a key file
Error 24 : Write error : cannot write compressed block 
E: mkinitramfs failure cpio 141 lz4 -9 -l 24
update-initramfs: failed for /boot/initrd.img-5.15.8-76051508-generic with 1.
run-parts: /etc/kernel/postinst.d/initramfs-tools exited with return code 1
dpkg: error processing package linux-image-5.15.8-76051508-generic (--configure):
 installed linux-image-5.15.8-76051508-generic package post-installation script subprocess returned error exit status 1
Processing triggers for initramfs-tools (0.136ubuntu6.6) ...
update-initramfs: Generating /boot/initrd.img-5.15.5-76051505-generic
cryptsetup: WARNING: Resume target cryptswap uses a key file
Error 24 : Write error : cannot write compressed block 
E: mkinitramfs failure cpio 141 lz4 -9 -l 24
update-initramfs: failed for /boot/initrd.img-5.15.5-76051505-generic with 1.
dpkg: error processing package initramfs-tools (--configure):
 installed initramfs-tools package post-installation script subprocess returned error exit status 1
Errors were encountered while processing:
 linux-image-5.15.8-76051508-generic
 initramfs-tools
E: Sub-process /usr/bin/dpkg returned an error code (1)

@KitFristo
Copy link

There is no /boot/efi. Does that mean that I'm somehow not using EFI?

Probably. You can confirm with ls /sys/firmware/efi/, the directory will exist if you're booted in UEFI mode and it will not exist if you're booted in legacy BIOS mode.

One partitiion, totally reformatted during the install process.

My question now would be how your boot partition could be filling up if you don't have a boot partition. You should not be receiving notifications for a "folder" filling up. Can you please provide a screenshot of the warning you're referring to?

I'm not really comfortable dumping fingerprinting info on Github,

I will explain why I need each output that I asked for:

  • lsblk: to confirm your partition configuration.
  • df -h: to confirm the low free space.
  • sudo tree /boot: to show what files are present that would be taking up the free space.
  • apt list --installed: to show which packages you have installed that would place files in your boot partition (typically kernel packages.)

Depending on the output of the final command, I would recommend trying a sudo apt autoremove --purge to clear out older kernel versions.

@jacobgkau FYI I was facing this issue and running sudo apt autoremove freed up space by removing the old linux images in the boot partition. The "Low Disk Space on Boot" menu is no longer showing up.

@KloudKoder
Copy link
Author

@KitFristo Thanks for looking into this. sys/firmware/efi is populated. Therefore I'm booting in EFI mode. Nothing has changed which would lead me to believe that I haven't been doing so all along. (I just didn't know about this particular evidence.)

"how your boot partition could be filling up if you don't have a boot partition" Apparently /boot/efi is in its own partition, while /boot isn't. But if that's the reason, then it's really weird that the problem goes away if I delete stale kernels from /boot. (Or at least, it did so the last time I had this problem, which was months ago. Perhaps it wouldn't do so any longer.) Based on your df command, /boot/efi is roughly half full. It currently contains just a single kernel's worth of images. I don't customize my partition configuration at all, leaving just whatever the installer default is.

The screenshot you're seeking has already been posted upthread.

Sorry I can't be more specific but I will continue to try to provide you with helpful information if you have questions which can be answered approximately.

1 similar comment
@KloudKoder
Copy link
Author

@KitFristo Thanks for looking into this. sys/firmware/efi is populated. Therefore I'm booting in EFI mode. Nothing has changed which would lead me to believe that I haven't been doing so all along. (I just didn't know about this particular evidence.)

"how your boot partition could be filling up if you don't have a boot partition" Apparently /boot/efi is in its own partition, while /boot isn't. But if that's the reason, then it's really weird that the problem goes away if I delete stale kernels from /boot. (Or at least, it did so the last time I had this problem, which was months ago. Perhaps it wouldn't do so any longer.) Based on your df command, /boot/efi is roughly half full. It currently contains just a single kernel's worth of images. I don't customize my partition configuration at all, leaving just whatever the installer default is.

The screenshot you're seeking has already been posted upthread.

Sorry I can't be more specific but I will continue to try to provide you with helpful information if you have questions which can be answered approximately.

@pktiuk
Copy link

pktiuk commented Aug 22, 2022

I have also problem with it.
It happened after the latest update (which bumped kernel version from 5.18 to 5.19 in pop 22.04)

I found also a thread on reddit about it https://www.reddit.com/r/pop_os/comments/l39uu6/low_disk_space_on_efi/

@pktiuk
Copy link

pktiuk commented Aug 22, 2022

root@pop-os:/boot/efi# tree -h -D
[4.0K Jan  1  1970]  .
├── [4.0K Mar 12  2021]  350be8550e496ec16e6397d8604bb1cd
├── [4.0K Mar 12  2021]  4b3f4897d51832ccb80f7835604bbafc
├── [4.0K Aug  3 21:58]  63fc31dab8bec0f04ddf74a162ead368
├── [4.0K Aug  3 21:58]  EFI
│   ├── [4.0K Aug  3 21:58]  BOOT
│   │   └── [ 96K Apr 18 22:12]  BOOTX64.EFI
│   ├── [4.0K Mar 12  2021]  Linux
│   ├── [4.0K Aug 22 22:17]  Pop_OS-beb88c4e-8d1b-4275-a60c-afc3facbdd2e
│   │   ├── [ 167 Aug 22 22:17]  cmdline
│   │   ├── [113M Aug 22 22:17]  initrd.img
│   │   ├── [ 34M Aug 22 22:17]  initrd.img-previous
│   │   ├── [ 11M Aug 22 22:17]  vmlinuz.efi
│   │   └── [ 11M Aug 22 22:17]  vmlinuz-previous.efi
│   ├── [4.0K Apr  3  2021]  Pop_OS-f6180e55-14bc-4775-aa1b-e90140bbb2dd
│   │   ├── [ 167 Jun 17 16:09]  cmdline
│   │   ├── [129M Jun 17 16:09]  initrd.img
│   │   ├── [126M Jun 17 16:09]  initrd.img-previous
│   │   ├── [ 11M Jun 17 16:09]  vmlinuz.efi
│   │   └── [ 11M Jun 17 16:09]  vmlinuz-previous.efi
│   ├── [4.0K Mar 12  2021]  Pop_OS-fd37f077-642d-4bfd-a0f8-0040342dbb1c
│   │   ├── [ 202 Mar 12  2021]  cmdline
│   │   ├── [ 85M Mar 12  2021]  initrd.img
│   │   └── [ 13M Mar 12  2021]  vmlinuz.efi
│   └── [4.0K Aug  3 21:58]  systemd
│       └── [ 96K Apr 18 22:12]  systemd-bootx64.efi
├── [4.0K Aug 22 23:09]  loader
│   ├── [4.0K Apr  3  2021]  entries
│   │   ├── [ 256 Aug 22 22:17]  Pop_OS-current.conf
│   │   └── [ 274 Jun 17 16:09]  Pop_OS-oldkern.conf
│   ├── [  23 Aug  3 21:58]  loader.conf
│   └── [ 512 Aug 22 23:09]  random-seed
└── [4.0K Mar 13  2021]  System Volume Information

@pktiuk
Copy link

pktiuk commented Sep 10, 2022

For now as a temp fix removing the oldest directory (in my case Pop_OS-fd37f077-642d-4bfd-a0f8-0040342dbb1c helped)

@KloudKoder
Copy link
Author

@pktiuk If you look in /boot, you'll also see a bunch of System.map, config, initrd.img, and vmlinuz files. You only need the most recent 2 versions, but the system update process seems to keep older ones as well, taking up space and ultimately resulting in a "partition full" condition. A quicker easier fix would just be to change the installer in order to expand the partition size. Even 10X wouldn't make it very big by modern standards.

@pktiuk
Copy link

pktiuk commented Sep 13, 2022

@KloudKoder
TBH I don't want to mess up with partitions after system installation.
During installation, I used default values, so I would rather expect that they will be sufficient for the regular desktop user.

@leviport
Copy link
Member

They usually are. Also, we just made the default ESP version larger, so this should no longer be a problem once we release our latest ISO.

@smpoptut
Copy link

smpoptut commented Nov 3, 2022

Hi Guys, I also have this problem, the second time now... Two days ago I kind of solved it with sudo apt autoremove

simspopos@pop-os:~$ sudo apt autoremove
[sudo] password for simspopos: 
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following packages will be REMOVED:
  linux-headers-5.19.0-76051900 linux-headers-5.19.0-76051900-generic
  linux-image-5.19.0-76051900-generic linux-modules-5.19.0-76051900-generic
0 upgraded, 0 newly installed, 4 to remove and 0 not upgraded.
After this operation, 729 MB disk space will be freed.
Do you want to continue? [Y/n] y
(Reading database ... 275675 files and directories currently installed.)
Removing linux-headers-5.19.0-76051900-generic (5.19.0-76051900.202207312230~1663791054~22.04~28340d4) ...
Removing linux-headers-5.19.0-76051900 (5.19.0-76051900.202207312230~1663791054~22.04~28340d4) ...
Removing linux-modules-5.19.0-76051900-generic (5.19.0-76051900.202207312230~1663791054~22.04~28340d4) ...
Removing linux-image-5.19.0-76051900-generic (5.19.0-76051900.202207312230~1663791054~22.04~28340d4) ...
/etc/kernel/prerm.d/dkms:
dkms: removing: system76 1.0.14~1643391291~22.04~78ede46 (5.19.0-76051900-generic) (x86_64)
Module system76-1.0.14~1643391291~22.04~78ede46 for kernel 5.19.0-76051900-gener
ic (x86_64).
Before uninstall, this module version was ACTIVE on this kernel.

system76.ko:
 - Uninstallation
   - Deleting from: /lib/modules/5.19.0-76051900-generic/updates/dkms/
 - Original module
   - No original module was found for this module on this kernel.
   - Use the dkms install command to reinstall any previous module version.

depmod...
dkms: removing: system76_acpi 1.0.2~1659568831~22.04~a88dcce (5.19.0-76051900-ge
neric) (x86_64)
Module system76_acpi-1.0.2~1659568831~22.04~a88dcce for kernel 5.19.0-76051900-g
eneric (x86_64).
Before uninstall, this module version was ACTIVE on this kernel.

system76_acpi.ko:
 - Uninstallation
   - Deleting from: /lib/modules/5.19.0-76051900-generic/updates/dkms/
 - Original module
   - Archived original module found in the DKMS tree
   - Moving it to: /lib/modules/5.19.0-76051900-generic/updates/dkms/
depmod...

Removing original_module from DKMS tree for kernel 5.19.0-76051900-generic (x86_
64)
dkms: removing: system76-io 1.0.2~1655490480~22.04~0217576 (5.19.0-76051900-gene
ric) (x86_64)
Module system76-io-1.0.2~1655490480~22.04~0217576 for kernel 5.19.0-76051900-gen
eric (x86_64).
Before uninstall, this module version was ACTIVE on this kernel.

system76-io.ko:
 - Uninstallation
   - Deleting from: /lib/modules/5.19.0-76051900-generic/updates/dkms/
 - Original module
   - No original module was found for this module on this kernel.
   - Use the dkms install command to reinstall any previous module version.

depmod...
/etc/kernel/postrm.d/initramfs-tools:
update-initramfs: Deleting /boot/initrd.img-5.19.0-76051900-generic
/etc/kernel/postrm.d/zz-update-grub:
Sourcing file `/etc/default/grub'
Sourcing file `/etc/default/grub.d/init-select.cfg'
Generating grub configuration file ...
Found linux image: /boot/vmlinuz-6.0.2-76060002-generic
Found initrd image: /boot/initrd.img-6.0.2-76060002-generic
Found linux image: /boot/vmlinuz-5.19.16-76051916-generic
Found initrd image: /boot/initrd.img-5.19.16-76051916-generic
Warning: os-prober will not be executed to detect other bootable partitions.
Systems on them will not be added to the GRUB boot configuration.
Check GRUB_DISABLE_OS_PROBER documentation entry.
done
simspopos@pop-os:~$ 

but today I'm getting errors and the same solution doesn't work...

Screenshot from 2022-11-03 09-15-25

Screenshot from 2022-11-03 09-16-18
Screenshot from 2022-11-03 09-16-29
Screenshot from 2022-11-03 09-17-35

simspopos@pop-os:~$ sudo apt autoremove
[sudo] password for simspopos: 
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
1 not fully installed or removed.
After this operation, 0 B of additional disk space will be used.
Setting up initramfs-tools (0.140ubuntu13) ...
update-initramfs: deferring update (trigger activated)
Processing triggers for initramfs-tools (0.140ubuntu13) ...
update-initramfs: Generating /boot/initrd.img-6.0.3-76060003-generic
cryptsetup: WARNING: Resume target cryptswap uses a key file
zstd: error 25 : Write error : No space left on device (cannot write compressed block) 
E: mkinitramfs failure zstd -q -1 -T0 25
update-initramfs: failed for /boot/initrd.img-6.0.3-76060003-generic with 1.
dpkg: error processing package initramfs-tools (--configure):
 installed initramfs-tools package post-installation script subprocess returned error exit status 1
Errors were encountered while processing:
 initramfs-tools
E: Sub-process /usr/bin/dpkg returned an error code (1)
simspopos@pop-os:~$ 

Also when I tried to provide the input mentioned above

lsblk
df -h
sudo tree /boot
apt list --installed

I got error on installing the tree

simspopos@pop-os:~$ sudo apt install tree
[sudo] password for simspopos: 
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following NEW packages will be installed:
  tree
0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
1 not fully installed or removed.
Need to get 47.9 kB of archives.
After this operation, 116 kB of additional disk space will be used.
Get:1 http://apt.pop-os.org/ubuntu jammy/universe amd64 tree amd64 2.0.2-1 [47.9 kB]
Fetched 47.9 kB in 0s (143 kB/s)
Selecting previously unselected package tree.
(Reading database ... 276950 files and directories currently installed.)
Preparing to unpack .../tree_2.0.2-1_amd64.deb ...
Unpacking tree (2.0.2-1) ...
Setting up initramfs-tools (0.140ubuntu13) ...
update-initramfs: deferring update (trigger activated)
Setting up tree (2.0.2-1) ...
Processing triggers for man-db (2.10.2-1) ...
Processing triggers for initramfs-tools (0.140ubuntu13) ...
update-initramfs: Generating /boot/initrd.img-6.0.3-76060003-generic
cryptsetup: WARNING: Resume target cryptswap uses a key file
zstd: error 25 : Write error : No space left on device (cannot write compressed block) 
E: mkinitramfs failure zstd -q -1 -T0 25
update-initramfs: failed for /boot/initrd.img-6.0.3-76060003-generic with 1.
dpkg: error processing package initramfs-tools (--configure):
 installed initramfs-tools package post-installation script subprocess returned error exit status 1
Errors were encountered while processing:
 initramfs-tools
E: Sub-process /usr/bin/dpkg returned an error code (1)
simspopos@pop-os:~$ 

I am beginner in the Linux world thru PopOs so please keep that in mind if you decide to help me with some info...thanks in advance :)

@smpoptut
Copy link

smpoptut commented Nov 3, 2022

I also tried sudo apt autoremove --purge but it doesn't help..

simspopos@pop-os:~$ sudo apt autoremove --purge 
[sudo] password for simspopos: 
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
1 not fully installed or removed.
After this operation, 0 B of additional disk space will be used.
Setting up initramfs-tools (0.140ubuntu13) ...
update-initramfs: deferring update (trigger activated)
Processing triggers for initramfs-tools (0.140ubuntu13) ...
update-initramfs: Generating /boot/initrd.img-6.0.3-76060003-generic
cryptsetup: WARNING: Resume target cryptswap uses a key file
zstd: error 25 : Write error : No space left on device (cannot write compressed block) 
E: mkinitramfs failure zstd -q -1 -T0 25
update-initramfs: failed for /boot/initrd.img-6.0.3-76060003-generic with 1.
dpkg: error processing package initramfs-tools (--configure):
 installed initramfs-tools package post-installation script subprocess returned error exit status 1
Errors were encountered while processing:
 initramfs-tools
E: Sub-process /usr/bin/dpkg returned an error code (1)
simspopos@pop-os:~$ 

@smpoptut
Copy link

smpoptut commented Nov 3, 2022

Me again :)
I managed to fix the situation by manually removing the oldest kernel, since from what I've researched you should leave intact the last two. So in my case I had 3 kernels in /boot and the oldest was initrd.img-5.19.16-76051916-generic
than I sudo rm deleted that one and 3 more files asociated with that same 5.19.16-76051916 number in their name. After that I updated the OS from the Pop!_Shop without problem.

simspopos@pop-os:~$ uname -a
Linux pop-os 6.0.3-76060003-generic #202210211149~1666452039~22.04~1891946 SMP PREEMPT_DYNAMIC Sat O x86_64 x86_64 x86_64 GNU/Linux
simspopos@pop-os:~$ sudo rm /boot/initrd.img-5.19.16-76051916-generic
[sudo] password for simspopos: 
simspopos@pop-os:~$ sudo rm /boot/System.map-5.19.16-76051916-generic
simspopos@pop-os:~$ sudo rm /boot/vmlinuz-5.19.16-76051916-generic
simspopos@pop-os:~$ sudo rm /boot/config-5.19.16-76051916-generic
simspopos@pop-os:~$ 

@davidskeck
Copy link

Having this issue myself, but I don't see any older kernels. I only have the two latest ones.

@walkrr
Copy link

walkrr commented Mar 8, 2023

also having this issue

@davidskeck
Copy link

davidskeck commented Mar 8, 2023

This fix worked for me. I contacted HP Dev One support.

This looks like the firmware update left files in the EFI partition. Fixing this can be done with:
sudo rm /boot/efi/EFI/HP -rf

@walkrr
Copy link

walkrr commented Mar 8, 2023

I don't have an HP dev one, thanks though

@agaillar
Copy link

agaillar commented Mar 23, 2023

Hello,

I have the same problem.
The 512Mb partition is 99% full.
I tried the command apt autoremove but it did not change anything.

Do you have any idea what I could delete to free up space ?

❯ df -h
Filesystem             Size  Used Avail Use% Mounted on
tmpfs                  1.6G  2.5M  1.6G   1% /run
/dev/mapper/data-root  461G  126G  312G  29% /
tmpfs                  7.7G     0  7.7G   0% /dev/shm
tmpfs                  5.0M     0  5.0M   0% /run/lock
/dev/nvme0n1p2         4.0G  3.0G  1.1G  73% /recovery
/dev/nvme0n1p1         497M  489M  8.5M  99% /boot/efi
tmpfs                  1.6G  160K  1.6G   1% /run/user/1000
❯ sudo apt autoremove
[sudo] password for agaillar: 
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
0 upgraded, 0 newly installed, 0 to remove and 11 not upgraded.
❯ sudo tree /boot/efi/
/boot/efi/
├── EFI
│   ├── BOOT
│   │   └── BOOTX64.EFI
│   ├── dell
│   │   ├── bios
│   │   │   └── recovery
│   │   │       ├── BIOS_BIR.rcv
│   │   │       ├── BIOS_CUR.rcv
│   │   │       └── BIOS_PRE.rcv
│   │   └── logs
│   │       ├── diags_current.xml
│   │       └── diags_previous.xml
│   ├── Linux
│   ├── pop
│   │   ├── fw
│   │   │   └── fwupd-ecfae66b-98c3-443c-9b48-4627200c75b7.cap
│   │   └── fwupdx64.efi
│   ├── Pop_OS-f3df13f8-1e5e-41a5-9713-7763dc7eca34
│   │   ├── cmdline
│   │   ├── initrd.img
│   │   ├── initrd.img-previous
│   │   ├── vmlinuz.efi
│   │   └── vmlinuz-previous.efi
│   ├── Recovery-24A3-F5F7
│   │   ├── initrd.gz
│   │   └── vmlinuz.efi
│   └── systemd
│       └── systemd-bootx64.efi
└── loader
    ├── entries
    │   ├── Pop_OS-current.conf
    │   ├── Pop_OS-oldkern.conf
    │   └── Recovery-24A3-F5F7.conf
    ├── loader.conf
    └── random-seed
❯ lsb_release -a
No LSB modules are available.
Distributor ID:	Pop
Description:	Pop!_OS 22.04 LTS
Release:	22.04
Codename:	jammy

image

@leviport
Copy link
Member

It helps to change the initramfs compression method from zstd to xz. This can be changed in /etc/initramfs-tools/initramfs.conf at line 36. Just change COMPRESS=zstd to COMPRESS=xz.

Then regenerate initramfs with this command:

update-initramfs -c -k all

@agaillar
Copy link

agaillar commented Mar 24, 2023

Hello @leviport ,

Thank you for your help. 🙏
It's perfect the compression allowed to recover disk space !

/dev/nvme0n1p1         497M  389M  109M  79% /boot/efi

@eccles
Copy link

eccles commented Apr 8, 2023

I have the same error without doing anything out of the ordinary so I suggest that this is a definite problem ...
I only have the standard last 2 kernels and some small firmware files for my dell laptop.
Changing compression to xz as suggested by @leviport above works
but is slightly dangerous perhaps if the update-initramfs command is interrupted?

The solution appears to be to use xz compression or to increase the size of /boot/efi. Changing to xz compression recovered 100Mb in my case..

@cryptoarashi
Copy link

Same problem. Very sad to see this has been an issue for such a long time.

@leviport
Copy link
Member

but is slightly dangerous perhaps if the update-initramfs command is interrupted?

No, it is not dangerous. It runs when many packages, especially the kernel, are updated with apt or Pop Shop.

@cryptoarashi did you try the xz compression workaround I mentioned?

@cryptoarashi
Copy link

@cryptoarashi did you try the xz compression workaround I mentioned?

Yes, thank you, it should alleviate the problem for quite a while, hopefully until the issue is resolved :)

@rjt
Copy link

rjt commented Jun 12, 2023

If autoremove is somehow causing problems, then a better way than directly deleting files under /boot is to remove the package responsible for the old files:

# df -h
# dpkg-query --search /boot/vmlinuz-5.11.0-7620-generic 
# dpkg-query --search /boot/vmlinuz-5.11.0-7620-generic | cut -d':' -f1
# # arrow up and modify the line like so:
# apt remove $(dpkg-query --search /boot/vmlinuz-5.11.0-7620-generic | cut -d':' -f1 )
# df -h

Frustrating this happens on a barely used 1TB. I have made /boot/ 1GB for a decade or so. Even bigger to store bootable.ISOs under /boot/

@MarcGodard
Copy link

Having the same issue...

@ImUnlikely
Copy link

I'm also having this issue. Using xz compression got me ~49MB of space.

@sheldonth
Copy link

I have a 4tb hard drive on a brand new system76 lemur which has run out of space with almost nothing installed on it. Can anyone advise what I should do? Even LibreOffice can't make routine writes to disk so I can't use that. Output of df -h is

Filesystem             Size  Used Avail Use% Mounted on
tmpfs                  2.4G  2.0M  2.4G   1% /run
efivarfs                64K   28K   32K  46% /sys/firmware/efi/efivars
/dev/mapper/data-root  3.6T  3.4T     0 100% /
tmpfs                   12G  305M   12G   3% /dev/shm
tmpfs                  5.0M     0  5.0M   0% /run/lock
/dev/nvme0n1p2         4.0G  2.9G  1.2G  72% /recovery
/dev/nvme0n1p1         511M  399M  113M  79% /boot/efi
tmpfs                  2.4G   37M  2.3G   2% /run/user/1000

@MarcGodard
Copy link

I reformatted. That solved it. For now at least. 😵‍💫

@leviport
Copy link
Member

@sheldonth Try switching to xz compression, like I explained above. To clarify, the /boot/efi partition is filling up, not the rest of your 4TB drive. If you need further assistance, please reach out to our support team here: https://support.system76.com/. Since you are an owner of System76 hardware, they will be happy to help you out.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests