PV Grub

[marmarek]

Implement ability to boot VM from the kernel provided by the VM (template) itself. This would greatly ease kernel package management:

• no need to "blacklist" kernel packages
• easier to compile kernel modules (use standard instructions, instead of some Qubes specific steps)
• easier to install non-default modules - possibility to install and use upstream packages directly
• no more problems with read-only /lib/modules

Required steps:

• package pvgrub for dom0 (fc20 grub package doesn't contain it)
• make sure that VM have the right initramfs additions (/dev/mapper/dmroot assembling code) - theoretically dracut module is already available in qubes-kernel-vm-support Fedora package.
• make sure that the above is actually used on VM kernel package upgrades
• generate grub configuration in the VM (no need to have grub installed in boot sector, but shouldn't harm)
• do not mount /lib/modules when the VM kernel is used
• decide what to do with kernelopts VM setting (ignore in this case? somehow pass to the VM kernel?)
• write documentation, Qubes Manager support, add to release notes

[marmarek]

General idea is to provide PVGRUB as one of "VM kernel" in /var/lib/qubes/vm-kernels. It would require placing pvgrub binary as vmlinuz (handle it somehow better?) and some dummy initramfs and modules.img.

[marmarek]

General idea is to provide PVGRUB as one of "VM kernel" in /var/lib/qubes/vm-kernels. It would require placing pvgrub binary as vmlinuz (handle it somehow better?) and some dummy initramfs and modules.img.

[qubesuser]

Another advantage is that PV VMs can then be booted as an HVM with no changes other than changing the VM type in the libvirt config (as long as normal GRUB is installed).

This is useful for things that Xen only supports on HVM guests like using hardware nested paging for better performance on some workloads, CPUID hiding for anonymity or development purposes and nested virtualization.

decide what to do with kernelopts VM setting (ignore in this case? somehow pass to the VM kernel?)

Could possibly point pvgrub to a config file generated by dom0 and placed in the modules or volatile disk, which would define a GRUB variable with the kernelopts and source the main config file in /boot, which would apply the options as long as the VM has a Qubes package installed that adds a reference to the GRUB variable in the commandline defined in /etc/default/grub. Not sure if it's that useful though.

[qubesuser]

Another advantage is that PV VMs can then be booted as an HVM with no changes other than changing the VM type in the libvirt config (as long as normal GRUB is installed).

This is useful for things that Xen only supports on HVM guests like using hardware nested paging for better performance on some workloads, CPUID hiding for anonymity or development purposes and nested virtualization.

decide what to do with kernelopts VM setting (ignore in this case? somehow pass to the VM kernel?)

Could possibly point pvgrub to a config file generated by dom0 and placed in the modules or volatile disk, which would define a GRUB variable with the kernelopts and source the main config file in /boot, which would apply the options as long as the VM has a Qubes package installed that adds a reference to the GRUB variable in the commandline defined in /etc/default/grub. Not sure if it's that useful though.

[marmarek]

Another advantage is that PV VMs can then be booted as an HVM with no changes other than changing the VM type in the libvirt config (as long as normal GRUB is installed).

Yes, this is also an important step. Also because we're going to integrate underlying mechanisms for PV/HVM and for example be able to use HVM NetVM.

BTW We're also considering usage of PVH. Do you know if CPUID hiding is available there?

Could possibly point pvgrub to a config file generated by dom0 and placed in the modules or volatile disk, which would define a GRUB variable with the kernelopts, source the main config file, which would apply the options as long as the VM has a Qubes package installed that adds a reference to the GRUB variable in the commandline defined in /etc/default/grub. Not sure if it's that useful though.

If it is possible to do, yes, this may be an option. Haven't got that far in PV Grub configuration yet. All I know is possibility (requirement?) for config file bundled to PV Grub binary, visible as memdisk, which is (I think) responsible for loading the real config from VM disk. If you can provide some examples, that would be great :)

[marmarek]

Another advantage is that PV VMs can then be booted as an HVM with no changes other than changing the VM type in the libvirt config (as long as normal GRUB is installed).

Yes, this is also an important step. Also because we're going to integrate underlying mechanisms for PV/HVM and for example be able to use HVM NetVM.

BTW We're also considering usage of PVH. Do you know if CPUID hiding is available there?

Could possibly point pvgrub to a config file generated by dom0 and placed in the modules or volatile disk, which would define a GRUB variable with the kernelopts, source the main config file, which would apply the options as long as the VM has a Qubes package installed that adds a reference to the GRUB variable in the commandline defined in /etc/default/grub. Not sure if it's that useful though.

If it is possible to do, yes, this may be an option. Haven't got that far in PV Grub configuration yet. All I know is possibility (requirement?) for config file bundled to PV Grub binary, visible as memdisk, which is (I think) responsible for loading the real config from VM disk. If you can provide some examples, that would be great :)

[qubesuser]

Unfortunately PVH doesn't currently support CPUID hiding and nested virtualization, and in general it seems like it was more of a "quick hack" than a well-designed mechanism.

It seems that to fix that the Xen team has deprecated it favor of "HVMlite", which seems to be a modification of the HVM code to run without a device model but which is not yet ready.

[qubesuser]

Unfortunately PVH doesn't currently support CPUID hiding and nested virtualization, and in general it seems like it was more of a "quick hack" than a well-designed mechanism.

It seems that to fix that the Xen team has deprecated it favor of "HVMlite", which seems to be a modification of the HVM code to run without a device model but which is not yet ready.

[northox]

Hum, not sure about that last part.

As we are planning to rename HVMlite to PVH, I would avoid the use of HVMlite in committed code.

See http://lists.xenproject.org/archives/html/xen-devel/2015-08/msg01927.html

[northox]

Hum, not sure about that last part.

As we are planning to rename HVMlite to PVH, I would avoid the use of HVMlite in committed code.

See http://lists.xenproject.org/archives/html/xen-devel/2015-08/msg01927.html

[marmarek]

Having VM-provided kernel means that it also must obtain (compile) u2mfn kernel module. Currently it is done using dkms. This means that dkms with all its dependencies (gcc, kernel headers etc) needs to be installed in the VM.
A question: should it be available in minimal template by default? Or another way: should it be required package? @axon-qubes

[marmarek]

Having VM-provided kernel means that it also must obtain (compile) u2mfn kernel module. Currently it is done using dkms. This means that dkms with all its dependencies (gcc, kernel headers etc) needs to be installed in the VM.
A question: should it be available in minimal template by default? Or another way: should it be required package? @axon-qubes

[andrewdavidwong]

Did you mention me by mistake, @marmarek?

[andrewdavidwong]

Did you mention me by mistake, @marmarek?

[marmarek]

No mistake, you are using minimal templates a lot ;)

[marmarek]

No mistake, you are using minimal templates a lot ;)

[andrewdavidwong]

IIUC, the downsides would be increasing the size of the minimal template (by how much approximately?) and arguably increasing the attack surface(?).

What would the benefit be in the case of the minimal template? Are there many common use-cases where someone would need this in a minimal template?

IMHO, we should err on the side of keeping the minimal template truly "minimal," and let the user install any additional software at her discretion. This seems preferable to requiring users to try to further strip down the minimal template when they need an actually-minimal template.

[andrewdavidwong]

IIUC, the downsides would be increasing the size of the minimal template (by how much approximately?) and arguably increasing the attack surface(?).

What would the benefit be in the case of the minimal template? Are there many common use-cases where someone would need this in a minimal template?

IMHO, we should err on the side of keeping the minimal template truly "minimal," and let the user install any additional software at her discretion. This seems preferable to requiring users to try to further strip down the minimal template when they need an actually-minimal template.

[marmarek]

Ouch, a lot: 167MB (https://gist.github.com/marmarek/d79382f300331a8f5db2)
Generally it is about ability to use kernel installed in the VM, which makes it easier to update, install additional modules etc.
Anyway I think you're right - minimal should stay minimal, especially given above size.

[marmarek]

Ouch, a lot: 167MB (https://gist.github.com/marmarek/d79382f300331a8f5db2)
Generally it is about ability to use kernel installed in the VM, which makes it easier to update, install additional modules etc.
Anyway I think you're right - minimal should stay minimal, especially given above size.

[marmarek]

Done.

[marmarek]

Done.