Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Simplify and promote using in-vm kernel #5212

Open
marmarek opened this issue Aug 1, 2019 · 33 comments
Open

Simplify and promote using in-vm kernel #5212

marmarek opened this issue Aug 1, 2019 · 33 comments

Comments

@marmarek
Copy link
Member

@marmarek marmarek commented Aug 1, 2019

The problem you're addressing (if any)
Currently by default all qubes use kernel provided by dom0. This has multiple issues:

  • kernel build for dom0 environment has incompatible kernel-devel package (especially painful for Debian templates, but also an issue for different Fedora versions): #2844 #3835
  • template doesn't control kernel options: #4088
  • kernel config not necessary matching template expectation (QubesOS/qubes-linux-kernel@96b8fba)

Using in-vm kernel isn't enabled by default and it's not straightforward:

  • for HVM, one needs to choose "(none)" / "" kernel
  • for PV, one needs to choose "pvgrub2" kernel
  • for PVH, one needs to choose "pvgrub2-pvh" kernel (only recently available)

Describe the solution you'd like
Unify setting in-vm kernel - possibly translate "(none)" to "pvgrub2" if virt_mode is PV and to "pvgrub2-pvh" if virt_mode is PVH. Note that "(none)" normally is an invalid choice for PV/PVH.
This would require adjusting "(none)" label at UI level, to be less confusing/magic. Something like "(use kernel from within the qube)".
Set is as default value.

Where is the value to a user, and who might that user be?
Less deviation from template's system, according to https://www.qubes-os.org/faq/#what-is-qubes-attitude-toward-changing-guest-distros

Possible drawbacks
This change may lead also to some issues:

  • less control over qube kernel means we won't be able to quickly apply qubes-specific fixes there - we'll need to wait until relevant distribution pick up updated kernel
  • less control over qube kernel config - for example if some kernel feature is disabled, we may have a problem
  • harder to provide extra kernel modules (fortunately we don't need u2mfn in R4.1 anymore)
  • different grub2 version - for example the one in Fedora is heavily patched and grub.cfg may rely on it (for example support for "Boot Loader Specification")

Describe alternatives you've considered
Implement automatic pvgrub choice for kernel "(none)", but don't set it as default.

Relevant documentation you've consulted
https://www.qubes-os.org/doc/managing-vm-kernel/

@brendanhoar

This comment has been minimized.

Copy link

@brendanhoar brendanhoar commented Aug 1, 2019

Peanut gallery has arrived. :)

How would Qubes dom0 be safely informed what bootloader/kernel/initrd combo is installed/available inside PV/PVH VMs? Or is there an agreed upon naming standard that is assumed?

So...Qubes already uses a notation of :object elsewhere in Qubes (e.g. devices).

Therefore, perhaps consider something along the lines of:
dom0:(default)
...
dom0:4.19.56-1.pvops
dom0:5.1.13-1.pvops
dom0:5.1.15-1.pvops
dom0:5.1.17-1.pvops
vmname:(default) [or vmname:(not specified)]
vmname:(default_for_Linux_HVM)
vmname:(default_for_Linux_PV)
vmname:(default_for_Linux_PVH)
vmname:<custom_item1_from_new_scheme>
vmname:<custom_item2_from_new_scheme>

Perhaps greying out items that aren't applicable to the current VM type?

Naming strategy allows future possibility of attempting to boot VM1 off the bootloader/kernel/initrd locally stored with VM2, if that would ever be useful (possibly just a crazy thought).

For kernel options, store per-VM changes from the default per selectable kernel, nothing if stored if the default value is left as is?

Brendan

@marmarek

This comment has been minimized.

Copy link
Member Author

@marmarek marmarek commented Aug 1, 2019

How would Qubes dom0 be safely informed what bootloader/kernel/initrd combo is installed/available inside PV/PVH VMs?

The whole idea is it wont. It will leave it to a bootloader within the VM itself, using predetermined protocol. In case of HVM, it's loading it from MBR (in the future there may be an option for UEFI). In case of PV/PVH, it's launching grub2, which will load a config from /boot within a VM.

@rustybird

This comment has been minimized.

Copy link

@rustybird rustybird commented Aug 2, 2019

Will there still be a (non-default) Qubes provided VM kernel? I guess I should look into upstreaming 0006-block-add-no_part_scan-module-parameter.patch (used by Split dm-crypt)...

@marmarek

This comment has been minimized.

Copy link
Member Author

@marmarek marmarek commented Aug 2, 2019

Will there still be a (non-default) Qubes provided VM kernel?

I don't plan to. But there will be still (non-default) dom0-provided kernel.

@rustybird

This comment has been minimized.

Copy link

@rustybird rustybird commented Aug 2, 2019

Will there still be a (non-default) Qubes provided VM kernel?

I don't plan to. But there will be still (non-default) dom0-provided kernel.

You mean, built by Fedora?

@marmarek

This comment has been minimized.

Copy link
Member Author

@marmarek marmarek commented Aug 2, 2019

No, same as now.

@rustybird

This comment has been minimized.

Copy link

@rustybird rustybird commented Aug 2, 2019

Oh, I see. By "VM kernel", I meant a kernel used by the VM but located in dom0. Sorry for the confusion.

@adrelanos

This comment has been minimized.

Copy link
Member

@adrelanos adrelanos commented Aug 3, 2019

In-VM kernel all the way. Seems much more expected from all parties. (If Qubes didn't introduce dom0 kernel, I'd never expect it.) (Parties include users, developers, applications by distribution, distribution maintainers, grub.d.)

I'd go as far as making dom0 kernel user opt-in only.
If not totally deprecate dom0 kernel long term.

less control over qube kernel means we won't be able to quickly apply qubes-specific fixes there - we'll need to wait until relevant distribution pick up updated kernel

This is one of the strongest points for dom0 I have to acknowledge even though I strongly favor In-VM kernel.

For any supported template, this would be better handled by distribution specific kernel fixes inside the VM by Qubes repository?

You might say this is hard? Might be good point. But... Well, if you promote in-vm kernel, then "half" users will have one thing, and half the other thing. What you tell them then? "We have a security issue, therefore go back to dom0 provided kernel." Double maintenance effort.

@marmarek

This comment has been minimized.

Copy link
Member Author

@marmarek marmarek commented Aug 4, 2019

It's already possible to test and use in-vm kernel, even for PVH. If you want to help make it default @adrelanos , here are testing steps for PVH:

  • install grub2-xen-pvh package in dom0 (from current-testing repo)
  • switch kernel to pvgrub2-pvh

It should "just work" for Debian 10 and result in a working AppArmor and other features. But require more testing. In case of Whonix, which is based on Debian 10 minimal, it might require installing qubes-kernel-vm-support if not already there. And building u2mfn module (DKMS package shipped with qubes-kernel-vm-support package.

Note it won't work out of the box on Fedora 30, because of BLS mentioned in issue description. It might be enough to set GRUB_ENABLE_BLSCFG=false in /etc/default/grub and regenerate grub config.

@adrelanos

This comment has been minimized.

Copy link
Member

@adrelanos adrelanos commented Aug 6, 2019

@adrelanos

This comment has been minimized.

Copy link
Member

@adrelanos adrelanos commented Aug 6, 2019

@adrelanos

This comment has been minimized.

Copy link
Member

@adrelanos adrelanos commented Aug 6, 2019

@adrelanos

This comment has been minimized.

Copy link
Member

@adrelanos adrelanos commented Aug 7, 2019

This happened in debian-10 template after sudo apt install anbox --no-install-recommends.

Aug 07 14:55:51 debian-10 kernel: ------------[ cut here ]------------
Aug 07 14:55:51 debian-10 kernel: kernel BUG at mm/memory.c:2201!
Aug 07 14:55:51 debian-10 kernel: invalid opcode: 0000 [#1] SMP PTI
Aug 07 14:55:51 debian-10 kernel: CPU: 1 PID: 621 Comm: Xorg Tainted: G           OE     4.19.0-5-amd64 #1 Debian 4.19.37-5+deb10u1
Aug 07 14:55:51 debian-10 kernel: RIP: 0010:apply_to_page_range+0x396/0x440
Aug 07 14:55:51 debian-10 kernel: Code: 40 00 48 bb 00 00 e0 ff ff ff 0f 00 81 e7 80 00 00 00 48 0f 44 1c 24 48 03 15 0e 35 cc 00 48 21 d8 48 8d 1c 02 e9 c5 fe ff ff <0f> 0b 48 8b 7c 24 08 4c 89 fa 4c 89 ee e8 78 bb ff ff 85 c0 75 18
Aug 07 14:55:51 debian-10 kernel: RSP: 0018:ffffb31440aebdb0 EFLAGS: 00010202
Aug 07 14:55:51 debian-10 kernel: RAX: 0000000000000001 RBX: ffff90fdd8200078 RCX: 000ffffffffff000
Aug 07 14:55:51 debian-10 kernel: RDX: 0000000000000001 RSI: ffffe76400608028 RDI: 80000000182008e7
Aug 07 14:55:51 debian-10 kernel: RBP: 000055ab4e810000 R08: ffffb31440aebe50 R09: ffff90fe7f5639c0
Aug 07 14:55:51 debian-10 kernel: R10: 0000000000000018 R11: 0000000000000000 R12: 000055ab4e80f000
Aug 07 14:55:51 debian-10 kernel: R13: ffff90fe6800a3a0 R14: 000055ab4e810000 R15: 000055ab4e80f000
Aug 07 14:55:51 debian-10 kernel: FS:  00007f345d225f00(0000) GS:ffff90feb5b00000(0000) knlGS:0000000000000000
Aug 07 14:55:51 debian-10 kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Aug 07 14:55:51 debian-10 kernel: CR2: 00007f345cc1e214 CR3: 00000000ac654005 CR4: 00000000001606e0
Aug 07 14:55:51 debian-10 kernel: Call Trace:
Aug 07 14:55:51 debian-10 kernel:  ? 0xffffffffc0366000
Aug 07 14:55:51 debian-10 kernel:  u2mfn_ioctl+0xd3/0x150 [u2mfn]
Aug 07 14:55:51 debian-10 kernel:  proc_reg_unlocked_ioctl+0x37/0x60
Aug 07 14:55:51 debian-10 kernel:  do_vfs_ioctl+0xa4/0x630
Aug 07 14:55:51 debian-10 kernel:  ? handle_mm_fault+0xda/0x200
Aug 07 14:55:51 debian-10 kernel:  ksys_ioctl+0x60/0x90
Aug 07 14:55:51 debian-10 kernel:  __x64_sys_ioctl+0x16/0x20
Aug 07 14:55:51 debian-10 kernel:  do_syscall_64+0x53/0x110
Aug 07 14:55:51 debian-10 kernel:  entry_SYSCALL_64_after_hwframe+0x44/0xa9
Aug 07 14:55:51 debian-10 kernel: RIP: 0033:0x7f345d955427
Aug 07 14:55:51 debian-10 kernel: Code: 00 00 90 48 8b 05 69 aa 0c 00 64 c7 00 26 00 00 00 48 c7 c0 ff ff ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 b8 10 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 39 aa 0c 00 f7 d8 64 89 01 48
Aug 07 14:55:51 debian-10 kernel: RSP: 002b:00007ffe4b0ce778 EFLAGS: 00000202 ORIG_RAX: 0000000000000010
Aug 07 14:55:51 debian-10 kernel: RAX: ffffffffffffffda RBX: 000055ab4e80f000 RCX: 00007f345d955427
Aug 07 14:55:51 debian-10 kernel: RDX: 000055ab4e80f000 RSI: 000000004004f501 RDI: 0000000000000018
Aug 07 14:55:51 debian-10 kernel: RBP: 00007ffe4b0ce7a0 R08: 0000000000000000 R09: 00000000000003fc
Aug 07 14:55:51 debian-10 kernel: R10: 0005d5ed6a000000 R11: 0000000000000202 R12: 000000000000000f
Aug 07 14:55:51 debian-10 kernel: R13: 000055ab4ef88000 R14: 000000000000000b R15: 00007f345d225b78
Aug 07 14:55:51 debian-10 kernel: Modules linked in: ipt_REJECT nf_reject_ipv4 xt_conntrack xt_tcpudp nft_counter nft_chain_nat_ipv4 ipt_MASQUERADE nf_nat_ipv4 nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c nft_compat nf_tables nfnetlink intel_rapl crct10dif_pclmul crc32_pclmul evdev ghash_clmulni_intel intel_rapl_perf pcspkr xen_netback u2mfn(OE) xen_gntdev xen_gntalloc xen_blkback xen_evtchn parport_pc ppdev lp parport xenfs xen_privcmd ip_tables x_tables autofs4 ext4 crc16 mbcache jbd2 crc32c_generic fscrypto ecb dm_snapshot dm_bufio dm_mod crc32c_intel xen_blkfront aesni_intel aes_x86_64 crypto_simd cryptd glue_helper
Aug 07 14:55:51 debian-10 kernel: ---[ end trace 5ddb4171addc1ec7 ]---
Aug 07 14:55:51 debian-10 kernel: RIP: 0010:apply_to_page_range+0x396/0x440
Aug 07 14:55:51 debian-10 kernel: Code: 40 00 48 bb 00 00 e0 ff ff ff 0f 00 81 e7 80 00 00 00 48 0f 44 1c 24 48 03 15 0e 35 cc 00 48 21 d8 48 8d 1c 02 e9 c5 fe ff ff <0f> 0b 48 8b 7c 24 08 4c 89 fa 4c 89 ee e8 78 bb ff ff 85 c0 75 18
Aug 07 14:55:51 debian-10 kernel: RSP: 0018:ffffb31440aebdb0 EFLAGS: 00010202
Aug 07 14:55:51 debian-10 kernel: RAX: 0000000000000001 RBX: ffff90fdd8200078 RCX: 000ffffffffff000
Aug 07 14:55:51 debian-10 kernel: RDX: 0000000000000001 RSI: ffffe76400608028 RDI: 80000000182008e7
Aug 07 14:55:51 debian-10 kernel: RBP: 000055ab4e810000 R08: ffffb31440aebe50 R09: ffff90fe7f5639c0
Aug 07 14:55:51 debian-10 kernel: R10: 0000000000000018 R11: 0000000000000000 R12: 000055ab4e80f000
Aug 07 14:55:51 debian-10 kernel: R13: ffff90fe6800a3a0 R14: 000055ab4e810000 R15: 000055ab4e80f000
Aug 07 14:55:51 debian-10 kernel: FS:  00007f345d225f00(0000) GS:ffff90feb5b00000(0000) knlGS:0000000000000000
Aug 07 14:55:51 debian-10 kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Aug 07 14:55:51 debian-10 kernel: CR2: 00007f345cc1e214 CR3: 00000000ac654005 CR4: 00000000001606e0
@adrelanos

This comment has been minimized.

Copy link
Member

@adrelanos adrelanos commented Aug 7, 2019

This messed up graphics inside the VM. After VM reboot, maximizing terminal emulator window triggered this error again. Installation of anbox may or may not be the actual cause. Didn't test VM kernel much before.

@marmarek

This comment has been minimized.

Copy link
Member Author

@marmarek marmarek commented Aug 7, 2019

https://www.qubes-os.org/doc/managing-vm-kernel/ does not mention installing grub inside VMs.

grub itself isn't necessary. Just grub config. Which means a tool to generate it.

Whonix VMs has qubes-kernel-vm-support package installed but lacks /usr/sbin/update-grub2. So the "right" grub package to be installed.

Oh, so qubes-kernel-vm-support should depend on grub2-common.

user@host:~$ mount | grep boot systemd-1 on /boot type autofs (rw,relatime,fd=42,pgrp=1,timeout=120,minproto=5,maxproto=5,direct,pipe_ino=11581)

what? I don't have anything like this. Do you have /boot in /etc/fstab? It shouldn't be there, /boot shouldn't be a separate partition in a template.

@marmarek

This comment has been minimized.

Copy link
Member Author

@marmarek marmarek commented Aug 7, 2019

And do we need "sudo apt install --no-install-recommends os-prober"? (grub2 recommended package.)

I recommend against.

@marmarek

This comment has been minimized.

Copy link
Member Author

@marmarek marmarek commented Aug 7, 2019

Aug 07 14:55:51 debian-10 kernel: kernel BUG at mm/memory.c:2201!

This is weird. It's this one: https://github.com/torvalds/linux/blob/33920f1ec5bf47c5c0a1d2113989bdd9dfb3fae9/mm/memory.c#L2013 (page passed to u2mfn is a "huge page")
I don't see why it would crash this way when using Debian kernel, but not our Qubes one. Maybe something about kernel config?

@adrelanos

This comment has been minimized.

Copy link
Member

@adrelanos adrelanos commented Aug 8, 2019

Again (previously reported in #5212 (comment)) debian-buster (manually updated debian stretch to debain buster) (not debian-10) TemplateVM

cat /etc/fstab
# Accessible filesystems, by reference, are maintained under '/dev/disk'
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info
#
/dev/mapper/dmroot /                       ext4 defaults,discard,noatime        1 1
/dev/xvdb		/rw			auto	noauto,defaults,discard	1 2
/rw/home        /home       none    noauto,bind,defaults 0 0
/rw/usrlocal        /usr/local       none    noauto,bind,defaults 0 0
/dev/xvdc1      swap                    swap    defaults        0 0
tmpfs                   /dev/shm                tmpfs   defaults,size=1G        0 0
devpts                  /dev/pts                devpts  gid=5,mode=620  0 0
sysfs                   /sys                    sysfs   defaults        0 0
proc                    /proc                   proc    defaults        0 0
xen                     /proc/xen               xenfs   defaults        0 0
/dev/xvdi	/mnt/removable	auto noauto,user,rw 0 0
mount | grep boot

So cannot reproduce boot being mounted in unexpected ways.

user@debian-buster:~$ mount | grep boot
user@debian-buster:~$ sudo mkdir /boot
mkdir: cannot create directory ‘/boot’: File exists
user@debian-buster:~$ mount | grep boot
user@debian-buster:~$ sudo ls -la /boot
total 78748
drwxr-xr-x  3 root root     4096 Aug  6 08:51 .
drwxr-xr-x 25 root root     4096 Jun 30 06:45 ..
-rw-r--r--  1 root root   206118 Mar 14 22:16 config-4.19.0-4-amd64
-rw-r--r--  1 root root   206212 Jul 19 04:45 config-4.19.0-5-amd64
drwxr-xr-x  5 root root     4096 Aug  8 03:29 grub
-rw-r--r--  1 root root 34793573 Apr 13 05:30 initrd.img-4.19.0-4-amd64
-rw-r--r--  1 root root 34970199 Aug  6 08:51 initrd.img-4.19.0-5-amd64
-rw-r--r--  1 root root  5213424 Mar 14 22:16 vmlinuz-4.19.0-4-amd64
-rw-r--r--  1 root root  5225712 Jul 19 04:45 vmlinuz-4.19.0-5-amd64
user@debian-buster:~$ sudo mkdir /boot/grub
mkdir: cannot create directory ‘/boot/grub’: File exists
user@debian-buster:~$ ls -la /boot/grub/
total 2384
drwxr-xr-x 5 root root    4096 Aug  8 03:29 .
drwxr-xr-x 3 root root    4096 Aug  6 08:51 ..
drwxr-xr-x 2 root root    4096 Mar 26  2018 fonts
-r--r--r-- 1 root root    8416 Aug  8 03:29 grub.cfg
-rw-r--r-- 1 root root    1024 Mar 26  2018 grubenv
drwxr-xr-x 2 root root   12288 Mar 26  2018 i386-pc
drwxr-xr-x 2 root root    4096 Mar 26  2018 locale
-rw-r--r-- 1 root root 2395475 Jun 29 08:33 unicode.pf2
cat /boot/grub/grub.cfg 
#
# DO NOT EDIT THIS FILE
#
# It is automatically generated by grub-mkconfig using templates
# from /etc/grub.d and settings from /etc/default/grub
#

### BEGIN /etc/grub.d/00_header ###
if [ -s $prefix/grubenv ]; then
  set have_grubenv=true
  load_env
fi
if [ "${next_entry}" ] ; then
   set default="${next_entry}"
   set next_entry=
   save_env next_entry
   set boot_once=true
else
   set default="0"
fi

if [ x"${feature_menuentry_id}" = xy ]; then
  menuentry_id_option="--id"
else
  menuentry_id_option=""
fi

export menuentry_id_option

if [ "${prev_saved_entry}" ]; then
  set saved_entry="${prev_saved_entry}"
  save_env saved_entry
  set prev_saved_entry=
  save_env prev_saved_entry
  set boot_once=true
fi

function savedefault {
  if [ -z "${boot_once}" ]; then
    saved_entry="${chosen}"
    save_env saved_entry
  fi
}
function load_video {
  if [ x$feature_all_video_module = xy ]; then
    insmod all_video
  else
    insmod efi_gop
    insmod efi_uga
    insmod ieee1275_fb
    insmod vbe
    insmod vga
    insmod video_bochs
    insmod video_cirrus
  fi
}

if [ x$feature_default_font_path = xy ] ; then
   font=unicode
else
insmod part_gpt
insmod ext2
if [ x$feature_platform_search_hint = xy ]; then
  search --no-floppy --fs-uuid --set=root  573ceeab-c2f3-4a00-bddf-de440ac1da9d
else
  search --no-floppy --fs-uuid --set=root 573ceeab-c2f3-4a00-bddf-de440ac1da9d
fi
    font="/usr/share/grub/unicode.pf2"
fi

if loadfont $font ; then
  set gfxmode=auto
  load_video
  insmod gfxterm
  set locale_dir=$prefix/locale
  set lang=en_US
  insmod gettext
fi
terminal_output gfxterm
if [ "${recordfail}" = 1 ] ; then
  set timeout=30
else
  if [ x$feature_timeout_style = xy ] ; then
    set timeout_style=menu
    set timeout=0
  # Fallback normal timeout code in case the timeout_style feature is
  # unavailable.
  else
    set timeout=0
  fi
fi
### END /etc/grub.d/00_header ###

### BEGIN /etc/grub.d/05_debian_theme ###
set menu_color_normal=cyan/blue
set menu_color_highlight=white/blue
### END /etc/grub.d/05_debian_theme ###

### BEGIN /etc/grub.d/10_linux ###
function gfxmode {
	set gfxpayload="${1}"
}
set linux_gfx_mode=
export linux_gfx_mode
menuentry 'Debian GNU/Linux' --class debian --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-simple-573ceeab-c2f3-4a00-bddf-de440ac1da9d' {
	load_video
	insmod gzio
	if [ x$grub_platform = xxen ]; then insmod xzio; insmod lzopio; fi
	insmod part_gpt
	insmod ext2
	if [ x$feature_platform_search_hint = xy ]; then
	  search --no-floppy --fs-uuid --set=root  573ceeab-c2f3-4a00-bddf-de440ac1da9d
	else
	  search --no-floppy --fs-uuid --set=root 573ceeab-c2f3-4a00-bddf-de440ac1da9d
	fi
	echo	'Loading Linux 4.19.0-5-amd64 ...'
	linux	/boot/vmlinuz-4.19.0-5-amd64 root=/dev/xvda3 ro root=/dev/mapper/dmroot console=hvc0 console=tty0 swiotlb=8192 root=/dev/mapper/dmroot console=hvc0 console=tty0 swiotlb=8192 noresume xen_scrub_pages=0 root=/dev/mapper/dmroot console=hvc0 console=tty0 swiotlb=8192 noresume intel_iommu=on amd_iommu=on slab_nomerge slab_debug=FZP page_poison=1 mce=0 pti=on mds=full,nosmt quiet
	echo	'Loading initial ramdisk ...'
	initrd	/boot/initrd.img-4.19.0-5-amd64
}
submenu 'Advanced options for Debian GNU/Linux' $menuentry_id_option 'gnulinux-advanced-573ceeab-c2f3-4a00-bddf-de440ac1da9d' {
	menuentry 'Debian GNU/Linux, with Linux 4.19.0-5-amd64' --class debian --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-4.19.0-5-amd64-advanced-573ceeab-c2f3-4a00-bddf-de440ac1da9d' {
		load_video
		insmod gzio
		if [ x$grub_platform = xxen ]; then insmod xzio; insmod lzopio; fi
		insmod part_gpt
		insmod ext2
		if [ x$feature_platform_search_hint = xy ]; then
		  search --no-floppy --fs-uuid --set=root  573ceeab-c2f3-4a00-bddf-de440ac1da9d
		else
		  search --no-floppy --fs-uuid --set=root 573ceeab-c2f3-4a00-bddf-de440ac1da9d
		fi
		echo	'Loading Linux 4.19.0-5-amd64 ...'
		linux	/boot/vmlinuz-4.19.0-5-amd64 root=/dev/xvda3 ro root=/dev/mapper/dmroot console=hvc0 console=tty0 swiotlb=8192 root=/dev/mapper/dmroot console=hvc0 console=tty0 swiotlb=8192 noresume xen_scrub_pages=0 root=/dev/mapper/dmroot console=hvc0 console=tty0 swiotlb=8192 noresume intel_iommu=on amd_iommu=on slab_nomerge slab_debug=FZP page_poison=1 mce=0 pti=on mds=full,nosmt quiet
		echo	'Loading initial ramdisk ...'
		initrd	/boot/initrd.img-4.19.0-5-amd64
	}
	menuentry 'Debian GNU/Linux, with Linux 4.19.0-5-amd64 (recovery mode)' --class debian --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-4.19.0-5-amd64-recovery-573ceeab-c2f3-4a00-bddf-de440ac1da9d' {
		load_video
		insmod gzio
		if [ x$grub_platform = xxen ]; then insmod xzio; insmod lzopio; fi
		insmod part_gpt
		insmod ext2
		if [ x$feature_platform_search_hint = xy ]; then
		  search --no-floppy --fs-uuid --set=root  573ceeab-c2f3-4a00-bddf-de440ac1da9d
		else
		  search --no-floppy --fs-uuid --set=root 573ceeab-c2f3-4a00-bddf-de440ac1da9d
		fi
		echo	'Loading Linux 4.19.0-5-amd64 ...'
		linux	/boot/vmlinuz-4.19.0-5-amd64 root=/dev/xvda3 ro single root=/dev/mapper/dmroot console=hvc0 console=tty0 swiotlb=8192 root=/dev/mapper/dmroot console=hvc0 console=tty0 swiotlb=8192 noresume xen_scrub_pages=0 root=/dev/mapper/dmroot console=hvc0 console=tty0 swiotlb=8192 noresume intel_iommu=on amd_iommu=on slab_nomerge slab_debug=FZP page_poison=1 mce=0 pti=on mds=full,nosmt
		echo	'Loading initial ramdisk ...'
		initrd	/boot/initrd.img-4.19.0-5-amd64
	}
	menuentry 'Debian GNU/Linux, with Linux 4.19.0-4-amd64' --class debian --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-4.19.0-4-amd64-advanced-573ceeab-c2f3-4a00-bddf-de440ac1da9d' {
		load_video
		insmod gzio
		if [ x$grub_platform = xxen ]; then insmod xzio; insmod lzopio; fi
		insmod part_gpt
		insmod ext2
		if [ x$feature_platform_search_hint = xy ]; then
		  search --no-floppy --fs-uuid --set=root  573ceeab-c2f3-4a00-bddf-de440ac1da9d
		else
		  search --no-floppy --fs-uuid --set=root 573ceeab-c2f3-4a00-bddf-de440ac1da9d
		fi
		echo	'Loading Linux 4.19.0-4-amd64 ...'
		linux	/boot/vmlinuz-4.19.0-4-amd64 root=/dev/xvda3 ro root=/dev/mapper/dmroot console=hvc0 console=tty0 swiotlb=8192 root=/dev/mapper/dmroot console=hvc0 console=tty0 swiotlb=8192 noresume xen_scrub_pages=0 root=/dev/mapper/dmroot console=hvc0 console=tty0 swiotlb=8192 noresume intel_iommu=on amd_iommu=on slab_nomerge slab_debug=FZP page_poison=1 mce=0 pti=on mds=full,nosmt quiet
		echo	'Loading initial ramdisk ...'
		initrd	/boot/initrd.img-4.19.0-4-amd64
	}
	menuentry 'Debian GNU/Linux, with Linux 4.19.0-4-amd64 (recovery mode)' --class debian --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-4.19.0-4-amd64-recovery-573ceeab-c2f3-4a00-bddf-de440ac1da9d' {
		load_video
		insmod gzio
		if [ x$grub_platform = xxen ]; then insmod xzio; insmod lzopio; fi
		insmod part_gpt
		insmod ext2
		if [ x$feature_platform_search_hint = xy ]; then
		  search --no-floppy --fs-uuid --set=root  573ceeab-c2f3-4a00-bddf-de440ac1da9d
		else
		  search --no-floppy --fs-uuid --set=root 573ceeab-c2f3-4a00-bddf-de440ac1da9d
		fi
		echo	'Loading Linux 4.19.0-4-amd64 ...'
		linux	/boot/vmlinuz-4.19.0-4-amd64 root=/dev/xvda3 ro single root=/dev/mapper/dmroot console=hvc0 console=tty0 swiotlb=8192 root=/dev/mapper/dmroot console=hvc0 console=tty0 swiotlb=8192 noresume xen_scrub_pages=0 root=/dev/mapper/dmroot console=hvc0 console=tty0 swiotlb=8192 noresume intel_iommu=on amd_iommu=on slab_nomerge slab_debug=FZP page_poison=1 mce=0 pti=on mds=full,nosmt
		echo	'Loading initial ramdisk ...'
		initrd	/boot/initrd.img-4.19.0-4-amd64
	}
}

### END /etc/grub.d/10_linux ###

### BEGIN /etc/grub.d/20_linux_xen ###

### END /etc/grub.d/20_linux_xen ###

### BEGIN /etc/grub.d/30_os-prober ###
### END /etc/grub.d/30_os-prober ###

### BEGIN /etc/grub.d/30_uefi-firmware ###
### END /etc/grub.d/30_uefi-firmware ###

### BEGIN /etc/grub.d/40_custom ###
# This file provides an easy way to add custom menu entries.  Simply type the
# menu entries you want to add after this comment.  Be careful not to change
# the 'exec tail' line above.
### END /etc/grub.d/40_custom ###

### BEGIN /etc/grub.d/41_custom ###
if [ -f  ${config_directory}/custom.cfg ]; then
  source ${config_directory}/custom.cfg
elif [ -z "${config_directory}" -a -f  $prefix/custom.cfg ]; then
  source $prefix/custom.cfg;
fi
### END /etc/grub.d/41_custom ###
user@debian-buster:~$

Does not boot with dom0 setting pvgrub2-phv.

.[30m.[47mWelcome to GRUB!

.[37m.[40m.[37m.[40m.[37m.[40m.[3;35H      [ grub-xen.cfg  424B  100%  1.50KiB/s ].[3;1Herror: no such device: /boot/xen/pvboot-x86_64.elf.
Reading (xen/xvda,gpt3/boot/grub/grub.cfg
.[H.[J.[1;1Herror: file `/boot/grub/fonts/unicode.pf2' not found.
error: no suitable video mode found.
.[3;36H      [ grub.cfg  8.22KiB  100%  9.01KiB/s ].[3;1H.[H.[J.[1;1H  Booting `Debian GNU/Linux'

Loading Linux 4.19.0-5-amd64 ...
.[4;24H      [ vmlinuz-4.19.0-5-amd  4.98MiB  100%  3.78MiB/s ].[4;1HLoading initial ramdisk ...
.[5;22H      [ initrd.img-4.19.0-5-  33.35MiB  100%  22.24MiB/s ].[5;1H

Is there any debug information that I could provide?

@adrelanos

This comment has been minimized.

Copy link
Member

@adrelanos adrelanos commented Aug 8, 2019

@marmarek

This comment has been minimized.

Copy link
Member Author

@marmarek marmarek commented Aug 8, 2019

Is there any debug information that I could provide?

Try removing quiet option from kernel cmdline to see some more details.

Also, something went wrong with kernel options, as some are duplicated. But I don't think that cause the problem.

@adrelanos

This comment has been minimized.

Copy link
Member

@adrelanos adrelanos commented Aug 9, 2019

Same output without quiet. debian-buster TemplateVM:

.[30m.[47mWelcome to GRUB!

.[37m.[40m.[37m.[40m.[37m.[40m.[3;35H      [ grub-xen.cfg  424B  100%  2.03KiB/s ].[3;1Herror: no such device: /boot/xen/pvboot-x86_64.elf.
Reading (xen/xvda,gpt3/boot/grub/grub.cfg
.[H.[J.[1;1Herror: file `/boot/grub/fonts/unicode.pf2' not found.
error: no suitable video mode found.
.[3;35H      [ grub.cfg  8.20KiB  100%  12.74KiB/s ].[3;1H.[H.[J.[1;1H  Booting `Debian GNU/Linux'

Loading Linux 4.19.0-5-amd64 ...
.[4;24H      [ vmlinuz-4.19.0-5-amd  4.98MiB  100%  5.09MiB/s ].[4;1HLoading initial ramdisk ...
.[5;22H      [ initrd.img-4.19.0-5-  33.35MiB  100%  22.90MiB/s ].[5;1H[user@dom0 ~]$ 

For comparison debian-10 TemplateVM.

.[30m.[47mWelcome to GRUB!

.[37m.[40m.[37m.[40m.[37m.[40m.[3;35H      [ grub-xen.cfg  424B  100%  2.03KiB/s ].[3;1Herror: no such device: /boot/xen/pvboot-x86_64.elf.
Reading (xen/xvda,gpt3/boot/grub/grub.cfg
.[H.[J.[1;1Herror: file `/boot/grub/fonts/unicode.pf2' not found.
error: no suitable video mode found.
.[3;35H      [ grub.cfg  8.20KiB  100%  12.74KiB/s ].[3;1H.[H.[J.[1;1H  Booting `Debian GNU/Linux'

Loading Linux 4.19.0-5-amd64 ...
.[4;24H      [ vmlinuz-4.19.0-5-amd  4.98MiB  100%  5.09MiB/s ].[4;1HLoading initial ramdisk ...
.[5;22H      [ initrd.img-4.19.0-5-  33.35MiB  100%  22.90MiB/s ].[5;1H[user@dom0 ~]$ 

And debian-10 TemplateVM without quiet:

[user@dom0 ~]$ sudo xl console debian-10
.[30m.[47mWelcome to GRUB!

.[37m.[40m.[37m.[40m.[37m.[40m.[3;35H      [ grub-xen.cfg  424B  100%  1.73KiB/s ].[3;1Herror: no such device: /boot/xen/pvboot-x86_64.elf.
Reading (xen/xvda,gpt3/boot/grub/grub.cfg
.[H.[J.[1;1H.[1;36H      [ grub.cfg  5.42KiB  100%  8.64KiB/s ].[1;1Herror: file `/boot/grub/fonts/unicode.pf2' not found.
error: no suitable video mode found.
.[H.[J.[1;1H  Booting `Debian GNU/Linux'

Loading Linux 4.19.0-5-amd64 ...
.[4;24H      [ vmlinuz-4.19.0-5-amd  4.98MiB  100%  5.23MiB/s ].[4;1HLoading initial ramdisk ...
.[5;22H      [ initrd.img-4.19.0-5-  31.95MiB  100%  21.94MiB/s ].[5;1H[    0.000000] Linux version 4.19.0-5-amd64 (debian-kernel@lists.debian.org) (gcc version 8.3.0 (Debian 8.3.0-6)) #1 SMP Debian 4.19.37-5+deb10u1 (2019-07-19)
[    0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-4.19.0-5-amd64 root=/dev/xvda3 ro xen_scrub_pages=0 root=/dev/mapper/dmroot console=hvc0 console=tty0 swiotlb=8192 noresume
[    0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers'
[    0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers'
[    0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers'
[    0.000000] x86/fpu: xstate_offset[2]:  576, xstate_sizes[2]:  256
[    0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format.
[    0.000000] BIOS-provided physical RAM map:
[    0.000000] BIOS-e820: [mem 0x0000000000000000-0x00000000000fefff] usable
[    0.000000] BIOS-e820: [mem 0x00000000000ff000-0x00000000000fffff] ACPI data
[    0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000f9ffffff] usable
[    0.000000] BIOS-e820: [mem 0x00000000fc000000-0x00000000fc007fff] ACPI data
[    0.000000] NX (Execute Disable) protection: active
[    0.000000] DMI not present or invalid.
[    0.000000] Hypervisor detected: Xen HVM
[    0.000000] Xen version 4.8.
[    0.000000] Xen Platform PCI: unrecognised magic value
[    0.149070] tsc: Fast TSC calibration failed
@adrelanos

This comment has been minimized.

Copy link
Member

@adrelanos adrelanos commented Aug 9, 2019

Some boot parameter by package https://github.com/Whonix/security-misc is causing this.

intel_iommu=on amd_iommu=on slab_nomerge slab_debug=FZP page_poison=1 mce=0 pti=on mds=full,nosmt

After removing all of these, VM boots.

Any idea which one, and why?

The other bug (kernel crash, making VM unusable) kernel BUG at mm/memory.c:2201! upon maximizing xfce4-terminal (as reported earlier in this ticket in #5212 (comment)) still happening.

@adrelanos

This comment has been minimized.

Copy link
Member

@adrelanos adrelanos commented Sep 21, 2019

marmarek added a commit to marmarek/qubes-gui-agent-linux that referenced this issue Sep 21, 2019
Kernel in Debian has this set to "always" by default, which breaks u2mfn
module. Specifically, it tries to get address of an arbitrary memory
page mapped by X server, and it fails if it's merged into huge page.
u2mfn module is gone in the next Qubes release, so as a temporary
measure revert transparent hugepage to "madvise".

Do that only if it was set to "always". This will prevent overriding
other user choices.

QubesOS/qubes-issues#5212
marmarek added a commit to marmarek/qubes-gui-agent-linux that referenced this issue Sep 21, 2019
Kernel in Debian has this set to "always" by default, which breaks u2mfn
module. Specifically, it tries to get address of an arbitrary memory
page mapped by X server, and it fails if it's merged into huge page.
u2mfn module is gone in the next Qubes release, so as a temporary
measure revert transparent hugepage to "madvise".

Do that only if it was set to "always". This will prevent overriding
other user choices.

QubesOS/qubes-issues#5212
marmarek added a commit to marmarek/qubes-linux-utils that referenced this issue Sep 22, 2019
Fedora use "Boot Loader Specification" config style by default. This is
available only in Fedora-patched grub, so it's incompatible with vanilla
grub shipped by dom0.
Disable it, to generate normal boot entries.

QubesOS/qubes-issues#5212
@lead4good

This comment has been minimized.

Copy link

@lead4good lead4good commented Sep 22, 2019

@adrelanos
I will have a look at it

@marmarek
I just tried to attach my usb webcam to a Fedora-30 AppVM which was running with the pvgrub2 kernel. This failed with an unspecific error message. Running the sys-usb VM with the pvgrub2 kernel makes attaching the usb webcam fail no matter what kind of kernel the destination VM is running. I wasn't able to find the error messages in /var/log inside dom0, where can I find the usb logs? I forgot to mention: Using debian busters kernel via pvgrub2 worked, both attaching the webcam and running as sys-usb.

@lead4good

This comment has been minimized.

Copy link

@lead4good lead4good commented Sep 22, 2019

@adrelanos
If I remove page_poison=1 from the kernel parameters the whonix-ws-15 templateVM boots with the pvgrub2 kernel.

@adrelanos

This comment has been minimized.

Copy link
Member

@adrelanos adrelanos commented Sep 23, 2019

@marmarek

Also, something went wrong with kernel options, as some are duplicated. But I don't think that cause the problem.

Created #5340 for it.

@marmarek

Try adding earlyprintk=xen console=hvc0.

That does not produce different kernel messages either.

@lead4good

If I remove page_poison=1 from the kernel parameters the whonix-ws-15 templateVM boots with the pvgrub2 kernel.

Yay!

Any idea why page_poison=1 breaks Qubes or Xen? Something that can be fixed? @marmarek

@marmarek

This comment has been minimized.

Copy link
Member Author

@marmarek marmarek commented Sep 23, 2019

Any idea why page_poison=1 breaks Qubes or Xen? Something that can be fixed? @marmarek

Xen says: p2m_pod_demand_populate: Dom11 out of PoD memory! (tot=102416 ents=921600 dom11)
So, it looks like kernel is trying to use/fill all the memory pages, including those to be ballooned out (balloon driver is not running yet at this stage). I'm not sure what is the best way to fix it, some (unverified) ideas:

  • modify kernel to poison unused pages later
  • enable page poisoning later (after balloon driver is started), like from initramfs or even startup script
  • set memory=maxmem initially and expand later using memory hotplug (it was unstable few years ago, but maybe now is better; not sure how it will affect kernel metadata overhead)
marmarek added a commit to QubesOS/qubes-linux-utils that referenced this issue Sep 28, 2019
Fedora use "Boot Loader Specification" config style by default. This is
available only in Fedora-patched grub, so it's incompatible with vanilla
grub shipped by dom0.
Disable it, to generate normal boot entries.

QubesOS/qubes-issues#5212

(cherry picked from commit bd55c1e)
marmarek added a commit to QubesOS/qubes-gui-agent-linux that referenced this issue Oct 7, 2019
According to systemd documentation, environment variables in form of
$foo are substituted by systemd itself before calling the command. To
avoid this, $$foo should be used instead. In practice, on my system
both versions works, but lets be on the safe side, compliant with the
documentation. Specifically, I've noticed unintended behavior here:
QubesOS/qubes-issues#5375 (comment)

QubesOS/qubes-issues#5212
adrelanos added a commit to Whonix/security-misc that referenced this issue Nov 5, 2019
adrelanos added a commit to adrelanos/qubes-linux-utils-1 that referenced this issue Dec 5, 2019
adrelanos added a commit to adrelanos/qubes-linux-utils-1 that referenced this issue Dec 5, 2019
marmarek added a commit to QubesOS/qubes-linux-utils that referenced this issue Dec 28, 2019
QubesOS/qubes-issues#5212
(cherry picked from commit 34c8212)
marmarek added a commit to QubesOS/qubes-linux-utils that referenced this issue Dec 28, 2019
QubesOS/qubes-issues#5212
(cherry picked from commit 8821906)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
7 participants
You can’t perform that action at this time.