multipass runtime dependencies & nested virtualization with VMWare crash

[makspiechota]

Environment: VM Ubuntu 18.04 LTS virtualized by VMWare Fusion 8 on OSX with option "Enable hypervisor applications in this virtual machine"

1. I run multipass downloaded by PPA:

multipass --version

multipass  0.8.1
multipassd 0.8.1

multipass launch

Starting reverent-adder
Launched: reverent-adder

multipass list

Name                    State             IPv4             Image
lettered-hyrax          Running           10.62.123.117    Ubuntu 18.04 LTS

2. Run local build (commit ccf8c4c)
   ./multipass --version

multipass  0.9.0-dev.236+gccf8c4c
multipassd 0.9.0-dev.236+gccf8c4c

sudo ./multipassd &
sudo ./multipass launch

[2019-09-24T19:08:49.166] [error] [dnsmasq] Process operation timed out

launch failed: Multipass dnsmasq failed to start: Process operation timed out  

Following #990
sudo ./multipass set local.driver=libvirt
sudo ./multipassd &

Configuring assisting-akita \libvirt: QEMU Driver error : Domain not found: no domain with matching name 'assisting-akita'
Starting assisting-akita -libvirt: Storage Driver error : Cannot access storage file '/home/maks-piechota/.local/share/multipassd/vault/instances/assisting-akita/ubuntu-18.04-server-cloudimg-amd64.img' (as uid:64055, gid:127): Permission denied
launch failed: Cannot access storage file '/home/maks-piechota/.local/share/multipassd/vault/instances/assisting-akita/ubuntu-18.04-server-cloudimg-amd64.img' (as uid:64055, gid:127): Permission denied

[townsend2010]

Hi @makspiechota,

When you say:

I run multipass downloaded by PPA

Do you mean you're running the Multipass Snap or is there some PPA somewhere that we don't know about?

Regarding running a "from build" version of Multipass, I have better luck running multipassd in a complete root environment like the following:

$ su -
Password:
$ /path/to/multipass/build/multipassd &

That said, the issue is that dnsmasq is not happy for some reason. To help figure out why, you'll need to look in journalctl right after this happens and look for dnsmasq entries.

And the libvirt issue is because libvirt is quite pedantic when the default image location is changed like what we do in Multipass. You can get around this by modifying /etc/libvirt/qemu.conf' and uncommenting the user = "root"line (or thegroup = "root") and then restart libvirtd`.

I hope this helps!

[makspiechota]

Thanks for your reply! I've started with libvirt for now and uncommented mentioned line. multipass daemon started without complaining, but when I do
multipass launch
I get

Configuring deep-spoonbill \libvirt: QEMU Driver error : Domain not found: no domain with matching name 'deep-spoonbill'
Starting deep-spoonbill -libvirt: QEMU Driver error : internal error: qemu unexpectedly closed the monitor: 2019-09-24T19:13:11.454677Z qemu-system-x86_64: error: failed to set MSR 0x38d to 0x0
qemu-system-x86_64: /build/qemu-iYRv0n/qemu-2.11+dfsg/target/i386/kvm.c:1906: kvm_put_msrs: Assertion `ret == cpu->kvm_msr_buf->nmsrs' failed.
launch failed: internal error: qemu unexpectedly closed the monitor: 2019-09-24T19:13:11.454677Z qemu-system-x86_64: error: failed to set MSR 0x38d to 0x0
qemu-system-x86_64: /build/qemu-iYRv0n/qemu-2.11+dfsg/target/i386/kvm.c:1906: kvm_put_msrs: Assertion `ret == cpu->kvm_msr_buf->nmsrs' failed.

[townsend2010]

Hi @makspiechota,

Doing a search of qemu-system-x86_64: error: failed to set MSR 0x38d to 0x0, it appears VMWare is not properly set up for nested virtualization. I suggest digging more into if/how to get your VMWare install to support that properly.

[townsend2010]

Also, I'm curious as to why you are not running Multipass natively on your MacOS machine 😁

[makspiechota]

I would like to build multipass and contribute, but I can't manage to build it on OSX?

[makspiechota]

If VMWare cant support it properly why multipass downloaded (yes, you're right - its Multipass Snap) works well?

[townsend2010]

@makspiechota,

Oh, I see, that's totally cool!

Regarding why the snap works vs. libvirt, by default, the snap uses what we call the qemu driver in which multipass itself is responsible for starting and managing the qemu process. We use many less options than what libvirt uses when it starts qemu and it seems an option libvirt uses is incompatible with VMWare.

When you build and run multipass, are you disabling the snap? If not, the dnsmasq the snap runs will conflict with the version of dnsmasq the built version tries to start. Be sure to snap disable multipass before running the built version. If you have already done that, then I think we need to figure out why dnsmasq is not working on the built version.

[makspiechota]

Hm, running snap multipass with local.driver=qemu seems to work. I've disabled snap now and tried to run built multipass (as root as you suggested):
./multipass set local.driver=qemu
./multipassd &
multipass launch
And i get again:

launch failed: The following errors occurred:                                   
qemu-system-x86_64: error: failed to set MSR 0x38d to 0x0
qemu-system-x86_64: /build/qemu-iYRv0n/qemu-2.11+dfsg/target/i386/kvm.c:1906: kvm_put_msrs: Assertion `ret == cpu->kvm_msr_buf->nmsrs' failed.

I think I should start with fresh environment and then disable snap. Playing with libvirt, because of non disabled snap, could mess something.

[townsend2010]

Ok, sounds like a plan. Let me know if you are successful or if you still have issues.

Thanks for your patience!

[makspiechota]

Thank you, for your help.
I've set fresh environment. Snap multipass works totally well, and built one (as a root, after disabling snap) gives me following error:
launch failed: Cannot read image format: qemu-img failed (execve: No such file or directory) with output

EDIT: Seems from the code that image path (that I assume was just created) passed to qemu does not exists?

[townsend2010]

Hi @makspiechota,

Sounds like you're making progress 😁 So the problem here is that you're missing the qemu-img runtime dependency. We're pretty bad about not documenting the runtime dependencies for developers ☹️ We need to fix that.

At any rate, you need to install qemu-img in your environment via apt install qemu-utils. If you do already have that installed, then we need to dig further why multipassd can't find qemu-img in the PATH.

[makspiechota]

I have installed it and now I'm getting:
launch failed: failed to start qemu instance: execve: No such file or directory
However I'm able to run
qemu-img --help
from terminal

[townsend2010]

Ugh, another missing runtime dependency.

Try apt install qemu-system-x86.

[makspiechota]

It would be useful to have dependencies documented. Maybe this my fresh environment is a good case study for it 😁
Maybe there should be some configure script with it?

However installing it lead me to this point again:

launch failed: The following errors occurred:                                   
qemu-system-x86_64: error: failed to set MSR 0x38d to 0x0
qemu-system-x86_64: /build/qemu-iYRv0n/qemu-2.11+dfsg/target/i386/kvm.c:1906: kvm_put_msrs: Assertion `ret == cpu->kvm_msr_buf->nmsrs' failed.

So again we have the question why snap version works and built one not?

[townsend2010]

Yes, we're going to have to at least document it in a HACKING.md or something.

Regarding the error, I'm afraid I'm at my limit on how to help. I don't have a VMWare Fusion 8 on OSX environment to help debug this and it is really unclear to me why the snap works, but the built version doesn't. I really wish I could help more and I'm sorry I can't.

[makspiechota]

I understand. Is it possible then to build multipass on other platform? I've failed on OSX and Ubuntu 16.04 and for now these are the only native options for me.

[makspiechota]

Good news! After quick research (I became inspired with your second post, now, little bit too late, I could have do it faster) I've found that enabling counters in VMWare helps. And indeed it works now!

[townsend2010]

You had issues building on a Ubuntu 16.04 machine? What were the issues? Also, I assume this is 16.04 installed directly on the machine itself and not in a hypervisor, correct?

If it is a 16.04 bare metal machine, then I think there are few options we can do to get you going. If you need a 18.04 VM so Multipass will build and run, you could use virt-manager to create a 18.04 VM and then do work in that VM. KVM supports nested virtualization quite well, so inside that VM you should be able to run and build Multipass.

Another option would be just do your work on your 16.04 and then build the snap via snapcraft and test it out that way.

Lastly, if you're comfortable using IRC, we have a #multipass channel on freenode where you can drop in and chat with us. I'm in the eastern US and a couple of other Multipass folks are in Europe and we could help you in a more "realtime" way 😁

[townsend2010]

Oh, I just saw your post after I posted my last response. That's great \o/

The invitation stands to join us on IRC if you'd like to chat.

[townsend2010]

I'm going to close this issue now, but we'll use #612 to track that we need to be better about the runtime dependencies.

[makspiechota]

1. I had some major issues with Qt compatibility. Ubuntu 16 has Qt v5.5.1 and 18 has v5.9.5. I don't know if it is the reason, however the qt headers couldn't be built.

2. I haven't actually thought that other hypervisor could resolve my issues. Good point.

3. I've tried to ask my questions on IRC first, but I couldn't get any response there 😁

[townsend2010]

Oh, right the Qt version, yeah, that will cause problems. We should probably see if there is some way to have CMake gate on the Qt version.

[makspiechota]

Alright! However many thanks for your help! I think I could start with some first easy issue :)

[townsend2010]

I've tried to ask my questions on IRC first, but I couldn't get any response there

Sorry about that. I'm not sure when you asked, but we were all in Paris last week at an engineering sprint, so our communication would definitely be lacking then. But don't hesitate to come back again and try 😀

[townsend2010]

And good luck on your contribution(s)! Just let us know in the issue that you are going to work on it so we don't accidentally duplicate any work.

[duwudi]

Good news! After quick research (I became inspired with your second post, now, little bit too late, I could have do it faster) I've found that enabling counters in VMWare helps. And indeed it works now!

Thanks a lot, I spent a long time trying to debug this and missed this comment the first time I looked in this thread - all working now!