Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

R4.2: debian-11 fails to boot with in-vm kernel #8277

Closed
marmarek opened this issue Jun 19, 2023 · 18 comments
Closed

R4.2: debian-11 fails to boot with in-vm kernel #8277

marmarek opened this issue Jun 19, 2023 · 18 comments
Labels
affects-4.1 This issue affects Qubes OS 4.1. affects-4.2 This issue affects Qubes OS 4.2. backport pending On closed issues, indicates fix released for newer version will be backported to older version. C: Debian/Ubuntu C: kernel diagnosed Technical diagnosis has been performed (see issue comments). P: default Priority: default. Default priority for new issues, to be replaced given sufficient information. r4.2-host-stable r4.2-vm-bookworm-stable r4.2-vm-bullseye-stable r4.2-vm-centos-stream8-stable r4.2-vm-fc36-cur-test r4.2-vm-fc37-stable r4.2-vm-fc38-stable T: bug Type: bug report. A problem or defect resulting in unintended behavior in something that exists.
Milestone
Release 4.2

Comments

@marmarek
Copy link
Member

Observation

openQA test in scenario qubesos-4.2-update-x86_64-system_tests_pvgrub_salt_storage@64bit fails in
TC_41_HVMGrub_debian-11

All three modes (PV, PVH, HVM) fail to boot with a timeout:

  • PV: panic Kernel panic - not syncing: System is deadlocked on memory
  • HVM: empty log
  • PVH: some kernel errors are visible, but looks unrelated to boot failure

If fails with both StandaloneVM, and AppVM.

Test suite description

Try booting debian-11 based VM with an in-vm kernel.

Reproducible

Fails since (at least) Build 2023060606-4.2

Expected result

Last good: 2023053020-4.2 (or more recent)

Further details

Always latest result in this scenario: latest
@marmarek marmarek added T: bug Type: bug report. A problem or defect resulting in unintended behavior in something that exists. C: kernel C: Debian/Ubuntu labels Jun 19, 2023
@marmarek marmarek added this to the Release 4.2 milestone Jun 19, 2023
@andrewdavidwong andrewdavidwong added P: default Priority: default. Default priority for new issues, to be replaced given sufficient information. needs diagnosis Requires technical diagnosis from developer. Replace with "diagnosed" or remove if otherwise closed. labels Jun 20, 2023
@marmarek
Copy link
Member Author

This seems to be caused by initramfs having ~70MB compressed (~230MB uncompressed). Default initial 400MB of RAM is not enough to handle that.

128MB is used for modules (it isn't all of them, the whole modules dir in rootfs takes over 300MB), and then 57MB is used for firmware. Both could be much more limited, but I'm not sure how to do that in an elegant way (to not list all the modules verbatim as that could change between kernel versions and also to leave an option to override by the user).

@marmarek
Copy link
Member Author

The same issue applies to debian-12 too.

@DemiMarie
Copy link

What about keeping only the modules required to mount /? That should mostly be various Xen and block device stuff. Debian generates a huge initramfs by default because they have to support bare hardware, but templates know they are running in a VM.

@brendanhoar
Copy link

This seems to be caused by initramfs having ~70MB compressed (~230MB uncompressed). Default initial 400MB of RAM is not enough to handle that.

Is the issue directly the amount of RAM used by the uncompressed initramfs, or is it that plus the amount of additional RAM used during the decompression process?

I seem to recall reading the latter being an issue for raspberry pi units (some < 1GB RAM) in some distro's recent move to different/newer kernel compression algs, particularly at the higher compression settings...which turned out to need an additional 200MB+ in memory dictionary space at those high compression levels.

B

@andrewdavidwong andrewdavidwong added diagnosed Technical diagnosis has been performed (see issue comments). and removed needs diagnosis Requires technical diagnosis from developer. Replace with "diagnosed" or remove if otherwise closed. labels Jun 23, 2023
@marmarek
Copy link
Member Author

Hmm, I just found out technically this is a duplicate of #8235, and I guess "not our bug" resolution applies here too.

But at the some time, since I've identified it's about initramfs, actually there may be a solution: initramfs.conf has MODULES=dep option that should reduce number of included modules (the default is MODULES=most). I'll try this approach.

@marmarek marmarek mentioned this issue Jun 26, 2023
Merged
@qubesos-bot qubesos-bot mentioned this issue Jun 30, 2023
Closed
@qubesos-bot
Copy link

Automated announcement from builder-github

The package linux-utils has been pushed to the r4.2 testing repository for the Debian template.
To test this update, first enable the testing repository in /etc/apt/sources.list.d/qubes-*.list by uncommenting the line containing bullseye-testing (or appropriate equivalent for your template version), then use the standard update command:

sudo apt-get update && sudo apt-get dist-upgrade

Changes included in this update

@qubesos-bot
Copy link

Automated announcement from builder-github

The package linux-utils has been pushed to the r4.2 testing repository for the Debian template.
To test this update, first enable the testing repository in /etc/apt/sources.list.d/qubes-*.list by uncommenting the line containing bookworm-testing (or appropriate equivalent for your template version), then use the standard update command:

sudo apt-get update && sudo apt-get dist-upgrade

Changes included in this update

@qubesos-bot
Copy link

Automated announcement from builder-github

The package linux-utils has been pushed to the r4.2 testing repository for the CentOS centos-stream8 template.
To test this update, please install it with the following command:

sudo yum update --enablerepo=qubes-vm-r4.2-current-testing

Changes included in this update

@qubesos-bot
Copy link

Automated announcement from builder-github

The component linux-utils (including package linux-utils) has been pushed to the r4.2 testing repository for the Fedora template.
To test this update, please install it with the following command:

sudo dnf update --enablerepo=qubes-vm-r4.2-current-testing

Changes included in this update

@qubesos-bot
Copy link

Automated announcement from builder-github

The component linux-utils (including package linux-utils) has been pushed to the r4.2 testing repository for the Fedora template.
To test this update, please install it with the following command:

sudo dnf update --enablerepo=qubes-vm-r4.2-current-testing

Changes included in this update

@qubesos-bot
Copy link

Automated announcement from builder-github

The component linux-utils (including package linux-utils) has been pushed to the r4.2 testing repository for the Fedora template.
To test this update, please install it with the following command:

sudo dnf update --enablerepo=qubes-vm-r4.2-current-testing

Changes included in this update

@qubesos-bot
Copy link

Automated announcement from builder-github

The package linux-utils has been pushed to the r4.2 stable repository for the Debian template.
To install this update, please use the standard update command:

sudo apt-get update && sudo apt-get dist-upgrade

Changes included in this update

@qubesos-bot
Copy link

Automated announcement from builder-github

The package linux-utils has been pushed to the r4.2 stable repository for the CentOS centos-stream8 template.
To install this update, please use the standard update command:

sudo yum update

Changes included in this update

@qubesos-bot
Copy link

Automated announcement from builder-github

The package linux-utils has been pushed to the r4.2 stable repository for the Debian template.
To install this update, please use the standard update command:

sudo apt-get update && sudo apt-get dist-upgrade

Changes included in this update

@qubesos-bot
Copy link

Automated announcement from builder-github

The component linux-utils (including package linux-utils) has been pushed to the r4.2 stable repository for the Fedora template.
To install this update, please use the standard update command:

sudo dnf update

Changes included in this update

@qubesos-bot
Copy link

Automated announcement from builder-github

The component linux-utils (including package linux-utils) has been pushed to the r4.2 stable repository for the Fedora template.
To install this update, please use the standard update command:

sudo dnf update

Changes included in this update

@andrewdavidwong andrewdavidwong added the affects-4.2 This issue affects Qubes OS 4.2. label Aug 8, 2023
@marmarek marmarek mentioned this issue Sep 21, 2023
Closed
@3hhh 3hhh mentioned this issue Dec 25, 2023
Closed
@3hhh
Copy link

3hhh commented Dec 25, 2023 •
edited
Loading

This also affects debian-12 on 4.1, not just on 4.2.

@andrewdavidwong Please re-open.

@andrewdavidwong andrewdavidwong added the affects-4.1 This issue affects Qubes OS 4.1. label Dec 25, 2023
@andrewdavidwong andrewdavidwong removed this from the Release 4.2 milestone Dec 25, 2023
@andrewdavidwong andrewdavidwong added the backport pending On closed issues, indicates fix released for newer version will be backported to older version. label Dec 25, 2023
@andrewdavidwong
Copy link
Member

@3hhh: Per @marmarek, backport pending On closed issues, indicates fix released for newer version will be backported to older version. is for closed issues. In other words, the correct procedure is to add backport pending On closed issues, indicates fix released for newer version will be backported to older version. to a closed issue, then the backport to 4.1 will occur. Therefore, I am re-closing this issue.

@andrewdavidwong andrewdavidwong added this to the Release 4.2 milestone Dec 25, 2023
marmarek added a commit to QubesOS/qubes-linux-utils that referenced this issue Jan 27, 2024
@marmarek
debian: reduce initramfs size
18f8058 
Build initramfs with only required modules, not all of them. Otherwise
it's too big to boot with default 400M.
Additionally disable looking for suspend-to-disk signature, this feature
is never used on Qubes VM (swap device is not persistent).

Fixes QubesOS/qubes-issues#8277

(cherry picked from commit 9efc9e3)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
affects-4.1 This issue affects Qubes OS 4.1. affects-4.2 This issue affects Qubes OS 4.2. backport pending On closed issues, indicates fix released for newer version will be backported to older version. C: Debian/Ubuntu C: kernel diagnosed Technical diagnosis has been performed (see issue comments). P: default Priority: default. Default priority for new issues, to be replaced given sufficient information. r4.2-host-stable r4.2-vm-bookworm-stable r4.2-vm-bullseye-stable r4.2-vm-centos-stream8-stable r4.2-vm-fc36-cur-test r4.2-vm-fc37-stable r4.2-vm-fc38-stable T: bug Type: bug report. A problem or defect resulting in unintended behavior in something that exists.
Projects
None yet
Milestone
Release 4.2
Development

No branches or pull requests
6 participants
@marmarek @DemiMarie @brendanhoar @andrewdavidwong @3hhh @qubesos-bot