Some images fail to find a rootfs

[mrc0mmand]

Hello!

I have a strange issue with the some boxes under libvirt provider. For some reason the box can't find the rootfs after booting up. After spending countless hours rummaging through config files and ending up with reinstalling vagrant, libvirt, and qemu, I still haven't managed to find the root cause.

Current behavior:

$ vagrant init generic/arch
$ vagrant up --provider=libvirt
Bringing machine 'default' up with 'libvirt' provider...
==> default: Box 'generic/arch' could not be found. Attempting to find and install...
    default: Box Provider: libvirt
    default: Box Version: >= 0
==> default: Loading metadata for box 'generic/arch'
    default: URL: https://vagrantcloud.com/generic/arch
==> default: Adding box 'generic/arch' (v1.9.6) for provider: libvirt
    default: Downloading: https://vagrantcloud.com/generic/boxes/arch/versions/1.9.6/providers/libvirt.box
==> default: Successfully added box 'generic/arch' (v1.9.6) for 'libvirt'!
==> default: Uploading base box image as volume into libvirt storage...
==> default: Creating image (snapshot of base box volume).
==> default: Creating domain with the following settings...
==> default:  -- Name:              vagrant_default
==> default:  -- Domain type:       kvm
==> default:  -- Cpus:              2
==> default:  -- Feature:           acpi
==> default:  -- Feature:           apic
==> default:  -- Feature:           pae
==> default:  -- Memory:            2048M
==> default:  -- Management MAC:    
==> default:  -- Loader:            
==> default:  -- Base box:          generic/arch
==> default:  -- Storage pool:      default
==> default:  -- Image:             /var/lib/libvirt/images/vagrant_default.img (32G)
==> default:  -- Volume Cache:      default
==> default:  -- Kernel:            
==> default:  -- Initrd:            
==> default:  -- Graphics Type:     vnc
==> default:  -- Graphics Port:     5900
==> default:  -- Graphics IP:       127.0.0.1
==> default:  -- Graphics Password: Not defined
==> default:  -- Video Type:        cirrus
==> default:  -- Video VRAM:        256
==> default:  -- Sound Type:	
==> default:  -- Keymap:            en-us
==> default:  -- TPM Path:          
==> default:  -- INPUT:             type=mouse, bus=ps2
==> default: Creating shared folders metadata...
==> default: Starting domain.
==> default: Waiting for domain to get an IP address...

This phase never finishes. However, on the serial console initramfs complains about missing rootfs:

dmesg tail:

I have this issue only on Fedora, the generic/arch box works fine on CentOS 7.

$ cat /etc/fedora-release 
Fedora release 29 (Twenty Nine)
$ rpm -q libvirt vagrant vagrant-libvirt 
libvirt-4.7.0-1.fc29.x86_64
vagrant-2.1.2-3.fc29.noarch
vagrant-libvirt-0.0.40-5.fc29.noarch

I'm under impression I keep missing something pretty obvious here, as some boxes work (like generic/debian9) and some don't (like generic/arch or generic/centos7).

[mrc0mmand]

Okay, thanks to the rubber duck debugging I just did, I realized I haven't tried other disk_bus methods. And, it, of course, works.

The generic/centos7 works with the sata bus, i.e.:

  config.vm.provider "libvirt" do |libvirt|
    libvirt.disk_bus="sata"
  end

Similarly, the generic/arch box successfully boots up with the ide bus:

  config.vm.provider "libvirt" do |libvirt|
    libvirt.disk_bus="ide"
  end

I'm not sure what's different on my system that the default disk bus doesn't work, but at least there's some easy solution.

[jerrac]

Thanks @mrc0mmand , I just ran into this (or something very similar) issue with generic/centos7 and generic/oraclelinux7. Setting it to use sata made it work.

Interestingly, generic/ubuntu1604 and generic/debian9 worked just fine without me having to make any changes.

I'm on Ubuntu 18.04, a 2016 Dell XPS 13 laptop.

Just in case it helps someone down the road, here's my Vagrantfile.

Vagrant.configure("2") do |config|

  config.vm.define :xenial do |xenial|
    xenial.vm.box = "generic/ubuntu1604"
    xenial.vm.network :private_network, ip: "192.168.222.2"
    xenial.vm.hostname = "aspectsxenial"
  end

  config.vm.define :centos7 do |centos7|
    centos7.vm.box = "generic/centos7"
    centos7.vm.network :private_network, ip: "192.168.222.4"
    centos7.vm.hostname = "aspectscentos7"
    config.vm.provider "libvirt" do |libvirt|
      libvirt.disk_bus="sata"
    end
  end


  config.vm.define :stretch do |stretch|
    stretch.vm.box = "generic/debian9"
    stretch.vm.network :private_network, ip: "192.168.222.5"
    stretch.vm.hostname = "aspectsstretch"
  end

  config.vm.define :oraclelinux7 do |oraclelinux7|
    oraclelinux7.vm.box = "generic/oracle7"
    oraclelinux7.vm.network :private_network, ip: "192.168.222.7"
    oraclelinux7.vm.hostname = "aspectsoraclelinux7"
    config.vm.provider "libvirt" do |libvirt|
      libvirt.disk_bus="sata"
    end
  end
  config.vm.provider "virtualbox" do | oraclelinux7|
    oraclelinux7.customize ["modifyvm", :id, "--memory", "2048"]
  end

  config.vm.define :bionic do |bionic|
    bionic.vm.box = "generic/ubuntu1804"
    bionic.vm.network :private_network, ip: "192.168.222.9"
    bionic.vm.hostname = "aspectsbionic"
    bionic.vm.provision "shell", inline: "apt-get -y install python-apt"
  end

end

[gorshunovr]

Got the same issue, solved. Thank you!

[ladar]

I've run into this issue before, I tried to add mitigations when I hit it, but sadly, I don't have the resources to test every image on every possible host.

It seems the images failing to find the rootfs aren't finding the right driver in the kernel image. And that all depends on what's bundled by default with the kernel for that distro.

If we want to add support for alt controllers to the RHEL/CentOS 7 images, we'd need to update the dracut config, and rebuild the kernel image. I'd accept a patch, if someone wanted to put the time into making that happen.

With Arch, it would involve updating the /etc/mkinitcpio.conf config file, and rebuilding the kernel image.

That being said, I've added default directives to the Vagrantfile bundled with the some of the boxes, to help mitigate this issue. For example CentOS 7 has disk_bus = "scsi" by default, inside the libvirt config block. But I suspect the problem is that the scsi disk bus is actually a different emulation on your Fedora system, than the default scsi disk bus on older libvirt/qemu systems like CentOS 7.

On my CentOS 7 host I see:

[ladar@host ~]# /usr/libexec/qemu-kvm -device help 2>&1 | grep scsi
name "scsi-block", bus SCSI, desc "SCSI block device passthrough"
name "scsi-cd", bus SCSI, desc "virtual SCSI CD-ROM"
name "scsi-disk", bus SCSI, desc "virtual SCSI disk or CD-ROM (legacy)"
name "scsi-generic", bus SCSI, desc "pass through generic scsi device (/dev/sg*)"
name "scsi-hd", bus SCSI, desc "virtual SCSI disk"
name "virtio-scsi-device", bus virtio-bus
name "virtio-scsi-pci", bus PCI, alias "virtio-scsi"

[jerrac]

Just FYI, with version 1.9.16 of the CentOS7 box, I had to change 'sata' to 'ide' for it to work. I also tried just commenting the entire centos7.vm.provider "libvirt" do |domain| section out, but it didn't help.