Xen not showing up as boot entry in grub in Ubuntu Server 18.04 LTS

[d0mnik]

Describe the bug

When installing the drakvuf bundle on Ubuntu Server 18.04 LTS, xen is not showing up in the grub bootloader, and it does not boot into xen by default.

How to reproduce

Steps to reproduce the behavior:

1. Install drakvuf bundle
2. Execute sudo reboot
3. xen-detect always returns not running on xen

[icedevml]

Hello, could you post the output of sudo update-grub command?

[d0mnik]

Hello, could you post the output of sudo update-grub command?

Sourcing file `/etc/default/grub'
Sourcing file `/etc/default/grub.d/50-cloudimg-settings.cfg'
Sourcing file `/etc/default/grub.d/xen.cfg'
Including Xen overrides from /etc/default/grub.d/xen.cfg
Using DRAKVUF-optimized settings for Xen
Not running on Xen.
------------------------------------------

Resources dedicated to Dom0 (host system):
-> CPU core(s): 1
-> Memory: 3967 MB

------------------------------------------
You can edit these settings in /etc/default/grub.d/xen.cfg
WARNING: GRUB_DEFAULT changed to boot into Xen by default!
         Edit /etc/default/grub.d/xen.cfg to avoid this warning.
Generating grub configuration file ...
Found linux image: /boot/vmlinuz-5.3.0-1020-azure
Found initrd image: /boot/initrd.img-5.3.0-1020-azure
done

[icedevml]

Could you also post the list of grub menu entries?

awk -F\' '/menuentry / {print $2}' /boot/grub/grub.cfg

And check package installation status:

dpkg -s drakvuf-bundle

[d0mnik]

awk -F' '/menuentry / {print $2}' /boot/grub/grub.cfg

Ubuntu
Ubuntu, with Linux 5.3.0-1020-azure
Ubuntu, with Linux 5.3.0-1020-azure (recovery mode)

dpkg -s drakvuf-bundle

Package: drakvuf-bundle
Status: install ok installed
Priority: optional
Section: admin
Installed-Size: 280152
Maintainer: Unmaintained snapshot
Architecture: amd64
Source: drakvuf-bundle
Version: 0.7-git20200502053527+bfe6985-1
Depends: libpixman-1-0, libpng16-16, libnettle6 | libnettle7, libgnutls30, libfdt1, libglib2.0-0, libglib2.0-dev, libjson-c3 | libjson-c4, libyajl2, libaio1
Conflicts: xen-hypervisor-4.6-amd64, xen-hypervisor-4.7-amd64, xen-hypervisor-4.8-amd64, xen-hypervisor-4.9-amd64, xen-hypervisor-4.10-amd64, xen-hypervisor-4.11-amd64, xen-hypervisor-4.12-amd64
Conffiles:
 /etc/default/xencommons 2f3ed0018b9fe9d9793a84dfa7b9d06b
 /etc/default/grub.d/xen.cfg 921913beeee1b548dd9b5422b6f038b5
 /etc/default/xendomains 9b9ad1a743bea91aaf3d2ac086fc5e12
 /etc/modules-load.d/xen.conf 56ccc513f18c4e1526c5811a4ed3dcc4
 /etc/bash_completion.d/xl.sh f38a5edea4b7a35ce89b9fd6f27628d6
 /etc/init.d/xencommons 3c7b6bd42a768d1c84b6673fdaeb3f21
 /etc/init.d/xen-watchdog dad8aa834cfd161897fd96c5f9105e44
 /etc/init.d/xendriverdomain 00fb257cc678906ea24ecb3504fae3ab
 /etc/init.d/xendomains 6d47ee36d9a82bb9fa690729dafe80b8
 /etc/xen/scripts/block-nbd 701f8d04194c77ccae7ef275ee9ac2a0
 /etc/xen/scripts/block-common.sh 79fbb3d0c5c4d6a724d1729433065b5f
 /etc/xen/scripts/vscsi ed8f791bb1b2ca48b5c4aa1bd49b672d
 /etc/xen/scripts/logging.sh 0a6fbb675e185f2133f111f9d3f1c0d5
 /etc/xen/scripts/external-device-migrate e4eaca6ac72e1510bcc31bf5d8a69858
 /etc/xen/scripts/colo-proxy-setup bbcdab929b9abeb50bf29a4dee691f49
 /etc/xen/scripts/block-tap d05c0eaad540b45c52adefa8f30ae9f1
 /etc/xen/scripts/vif-nat e3037907eb54d912ead456487a5367ac
 /etc/xen/scripts/vif-openvswitch 7bcf63a7789b3299a5c439c97ec76642
 /etc/xen/scripts/hotplugpath.sh 18b2056ec0a113e9c3bc5f0bce1b6137
 /etc/xen/scripts/xen-script-common.sh 9fb804d7c1bed7ed0e9dbbc1bb3e6bdb
 /etc/xen/scripts/launch-xenstore 63e1aa024a34b2428f07ee3d010fa8d7
 /etc/xen/scripts/vif-bridge c60a79fefea0f9333077ecef79284fac
 /etc/xen/scripts/vif-setup 96d12dbf85c3823b2a644e58bf3fbc73
 /etc/xen/scripts/block-drbd-probe 23301fb2ef205e3242c3dd32861dc2a8
 /etc/xen/scripts/block 80a7a4696c50bc72f5df81e63e7cc873
 /etc/xen/scripts/xen-network-common.sh e717d4e2498055008720535bbc8d1670
 /etc/xen/scripts/vif2 99d0e975ba05ef155c185fe1df81a543
 /etc/xen/scripts/block-dummy 868084672960cab1e88c3ce8e3a965a8
 /etc/xen/scripts/vif-route 64ffa60dc66938b0706b2e7e072c884c
 /etc/xen/scripts/vif-common.sh f342592bab30b4f946e0fc599da0aded
 /etc/xen/scripts/qemu-ifup ef82ee54ab1e5c48f3d2ecedfe8d74e4
 /etc/xen/scripts/block-enbd c4f8e069e14476111529d6ee764f318f
 /etc/xen/scripts/xen-hotplug-common.sh 7a5fe62640ae9a3984794a6a49c660af
 /etc/xen/scripts/locking.sh 71b2021d5e8d2045be8fe418c1773d35
 /etc/xen/scripts/block-iscsi 89fb7466c9fc4a228dcbf3287132d871
 /etc/xen/xlexample.pvlinux 6c53b42c3d57a86494a5120a76a2580d
 /etc/xen/xlexample.hvm bde1739358ac7e39ac34527de4e24732
 /etc/xen/README.incompatibilities 242ffb8ed39a5027e0d4eac3a587bf8f
 /etc/xen/README 1fc159cda280a6d69226a04f6cecfe20
 /etc/xen/cpupool f71474b881c216f62a30921eae035379
 /etc/xen/xl.conf 1d5a78fe8166f93b7526c5d114f72501
Description: Testing build of DRAKVUF, LibVMI and Xen. Not supported. No guarantee.

[icedevml]

Looks like Xen hypervisor binary is not being detected properly on Ubuntu instances which were set up using cloud-init. The issue may be related to this bug: https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/1379080

We will do some further checks to debug it.

[d0mnik]

Is there anyway for me to bypass this in the meanwhile?

[icedevml]

In general, using a cloud system may not work anyway, because the required nested virtualization extensions may be unavailable.

The best option is to use dedicated hardware for DRAKVUF Sandbox (must be an Intel processor with Extended Page Tables feature support). It is also confirmed to work well on VMware Workstation Player (you need to check Virtualize EPT option for a VM; Intel processor with EPT still required).

[d0mnik]

I checked the cpuinfo and the cloud system seems to support the necessary extensions. Nonetheless, looking forward to the release where this bug is fixed.

[icedevml]

Related: https://wiki.xenproject.org/wiki/Compiling_Xen_From_Source#Updated_.2Fsbin.2Finstallkernel_on_Linux

Linux distributions shipping with grub2 will need to ensure that their /sbin/installkernel script, which has to be provided by each Linux distribution, copies the the kernel configuration upon a custom kernel install time. The requirement for the config file comes from upstream grub2 /etc/grub.d/20_linux_xen which will only add xen as an instance to your grub.cfg if and only if it finds in your config file either of:

CONFIG_XEN_DOM0=y
CONFIG_XEN_PRIVILEGED_GUEST=y

Without this a user compiling and installing their own kernel with proper support for xen and with the xen hypervisor present will not get their respective grub2 update script to pick up the xen hypervisor.

and as far as we've been debugging for now, the Azure VMs are shipped with a custom -azure kernel which doesn't meet the above requirements, i.e. there are no CONFIG_XEN_* entries in /boot/config-5.3.0-1020-azure. Seems like their kernel doesn't declare explicit Xen support and that's why the Xen entry doesn't appear in grub.

[d0mnik]

hmmm... that's unfortunate. Thank you so much for your time. I will be closing this issue now. Cheers

[icedevml]

I suppose that changing the kernel to stock generic one may help with this issue.