VM doesn't run automatically on boot on FreeNAS #150

jonblack · 2016-04-27T11:04:11Z

This is related to comments I made in #114.

When I reboot my FreeNAS box running version 9.10 the vmm.ko and nmnd.ko modules aren't automatically loaded.

I have the following lines at the end of /boot/loader.conf:

# iohyve
vmm_load="YES"
nmdm_load="YES"
if_bridge_load="YES"
if_tap_load="YES"

I tried adding the following lines to the end of /etc/rc.conf:

iohyve_enable="YES"
iohyve_flags="kmod=1 net=eno"

But whenever I reboot, they are gone.

This means every time I restart (which is daily because it shuts down at night to save electricity), I have to do the following:

kldload vmm
kldload nmdm
iohyve start debian

The text was updated successfully, but these errors were encountered:

pr1ntf · 2016-04-27T12:02:30Z

I think that is going to be the case for /etc/rc.conf and /boot/loader.conf according to their handbook.

I did find this in their handbook. You may want to take a look at it.

I'll ping the FreeNAS folks and see if they mention that in their iohyve portion of their handbook.

I'm going to leave this issue open to remind me to add this mention to the readme file.

EpiJunkie · 2016-04-27T17:08:03Z

@jonblack even with the changes to /boot/loader.conf they are not registered until you run a command to rebuild the grub.cfg file. When I did this I rendered my test system non-bootable. This requirement is unique to FreeNAS because it uses grub for multi-boot compatibility reasons. PC-BSD might also have this requirement. Adding these changes via the web interface in the tunables section as @pr1ntf pointed out seems to be the proper way to do it.

jonblack · 2016-04-28T06:27:21Z

Thanks for the tips and your patience. I'm really inexperienced with BSD so this is very new to me.

I've tried using the tunables in the web interface but I still see VMM set to NO when I run iohyve list after booting.

pr1ntf · 2016-04-28T06:49:31Z

When running iohyve list, the VMM label tells you whether or not the guest has been loaded into the VMM kernel module.

To see if the actual vmm.ko and nmdm.ko kernel modules are loaded, simply run kldstat. (It's kind of like lsmod for Linux distros)

If you see vmm.ko and nmdm.ko in your output, you should be good to go.

jonblack · 2016-04-29T16:15:05Z

I see. You're right, they are in the kldstat list. I still need to run iohyve start debian to start the VM. That bit still isn't automatic.

pr1ntf · 2016-04-30T01:14:12Z

If you have version 0.7.5 you should be able to run something like iohyve set debian boot=1 to get it to start at boot time.

Note that it still may be a bit buggy.

pr1ntf · 2016-04-30T03:33:03Z

"Tunables" handbook link has been added to the README.

Leaving this issue open until we hear back from @jonblack on his debian guest starting at boot time.

jonblack · 2016-04-30T07:07:42Z

We're almost there. I'm learning a lot about BSD which is a good thing. iohyve set debian boot=1 does the trick - the virtual machine starts at boot.

I've noticed another issue in the meantime (sorry). I've lost network access to the VM. In FreeNAS I can connect with iohyve console debian and log in fine, but trying to connect via ssh fails. This also used to work.

The output of ifconfig is below:

re0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=82099<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC,LINKSTATE>
    ether 50:46:5d:51:db:a5
    inet6 fe80::5246:5dff:fe51:dba5%re0 prefixlen 64 tentative scopeid 0x1 
    inet 192.168.2.46 netmask 0xffffff00 broadcast 192.168.2.255 
    nd6 options=2b<PERFORMNUD,ACCEPT_RTADV,IFDISABLED,AUTO_LINKLOCAL>
    media: Ethernet autoselect (1000baseT <full-duplex>)
    status: active
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
    options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
    inet6 ::1 prefixlen 128 
    inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2 
    inet 127.0.0.1 netmask 0xff000000 
    nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
tap0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: iohyve-debian
    options=80000<LINKSTATE>
    ether 00:bd:05:18:01:00
    inet 0.0.0.0 netmask 0xff000000 broadcast 255.255.255.255 
    nd6 options=9<PERFORMNUD,IFDISABLED>
    media: Ethernet autoselect
    status: active
    Opened by PID 4137
bridge1: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
    ether 02:d2:e4:fb:79:01
    nd6 options=9<PERFORMNUD,IFDISABLED>
    id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
    maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
    root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
    member: epair3a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 9 priority 128 path cost 2000
    member: epair2a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 8 priority 128 path cost 2000
    member: epair1a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 7 priority 128 path cost 2000
    member: epair0a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 6 priority 128 path cost 2000
    member: re0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 1 priority 128 path cost 20000
epair0a: flags=8942<BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=8<VLAN_MTU>
    ether 02:ff:20:00:06:0a
    nd6 options=9<PERFORMNUD,IFDISABLED>
    media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
    status: active
epair1a: flags=8942<BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=8<VLAN_MTU>
    ether 02:ff:20:00:07:0a
    nd6 options=9<PERFORMNUD,IFDISABLED>
    media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
    status: active
epair2a: flags=8942<BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=8<VLAN_MTU>
    ether 02:ff:20:00:08:0a
    nd6 options=9<PERFORMNUD,IFDISABLED>
    media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
    status: active
epair3a: flags=8942<BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=8<VLAN_MTU>
    ether 02:ff:20:00:09:0a
    nd6 options=9<PERFORMNUD,IFDISABLED>
    media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
    status: active

I understand that a virtual machine connects via a bridge and a tap (not sure how exactly). The Debian virtual machine is using tap0. The primary interface re0 and the tap interface tap0 are both members of bridge1, but I have no idea if iohyve/the vm are setup to use that bridge. There was a bridge0 which only had tap0 as a member so I removed it hoping that would fix it. Perhaps that was a mistake. sysctl net.link.tap.up_on_open returns net.link.tap.up_on_open: 1 so that seems good.

What's really odd is that I'm pretty sure that when I set this up I used em0 as the interface and that worked. Now I don't even see an em0 interface.

I'm tempted to destroy the VM and recreate it because iohyve can set this up for me, but I also want to understand what the problem might be in case it happens again.

Thanks again for being patient with me and helping through this.

pr1ntf · 2016-04-30T07:16:27Z

Everything is hardcoded to a bridge0 right now. This might be your issue.
The iohyve flags="kmod=1 net=re0" tunable you set should setup the iohyve networking so you don't have to touch that stuff.

jonblack · 2016-04-30T07:23:25Z

You're right, rebooting recreates the bridges and taps, but I'm still without network access, which is odd.

pr1ntf · 2016-04-30T07:26:31Z

Is the output of sysctl net.link.tap.up_on_open still 1?
Can you post your ifconfig output after the reboot?

jonblack · 2016-04-30T07:33:57Z

[root@mnemosyne] ~# sysctl net.link.tap.up_on_open
net.link.tap.up_on_open: 1
[root@mnemosyne] ~# ifconfig
re0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=82099<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC,LINKSTATE>
    ether 50:46:5d:51:db:a5
    inet 192.168.2.46 netmask 0xffffff00 broadcast 192.168.2.255 
    nd6 options=9<PERFORMNUD,IFDISABLED>
    media: Ethernet autoselect (1000baseT <full-duplex>)
    status: active
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
    options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
    inet6 ::1 prefixlen 128 
    inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2 
    inet 127.0.0.1 netmask 0xff000000 
    nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: iohyve-bridge
    ether 02:d2:e4:fb:79:00
    nd6 options=1<PERFORMNUD>
    id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
    maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
    root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
    member: tap0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 4 priority 128 path cost 2000000
tap0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: iohyve-debian
    options=80000<LINKSTATE>
    ether 00:bd:15:0f:01:00
    inet 0.0.0.0 netmask 0xff000000 broadcast 255.255.255.255 
    nd6 options=9<PERFORMNUD,IFDISABLED>
    media: Ethernet autoselect
    status: active
    Opened by PID 4138
bridge1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    ether 02:d2:e4:fb:79:01
    nd6 options=1<PERFORMNUD>
    id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
    maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
    root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
    member: epair3a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 9 priority 128 path cost 2000
    member: epair2a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 8 priority 128 path cost 2000
    member: epair1a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 7 priority 128 path cost 2000
    member: epair0a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 6 priority 128 path cost 2000
    member: re0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 1 priority 128 path cost 20000
epair0a: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=8<VLAN_MTU>
    ether 02:ff:20:00:06:0a
    nd6 options=1<PERFORMNUD>
    media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
    status: active
epair1a: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=8<VLAN_MTU>
    ether 02:ff:20:00:07:0a
    nd6 options=1<PERFORMNUD>
    media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
    status: active
epair2a: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=8<VLAN_MTU>
    ether 02:ff:20:00:08:0a
    nd6 options=1<PERFORMNUD>
    media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
    status: active
epair3a: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=8<VLAN_MTU>
    ether 02:ff:20:00:09:0a
    nd6 options=1<PERFORMNUD>
    media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
    status: active

pr1ntf · 2016-04-30T07:35:33Z

Your adapter re0 is not a member of bridge0.

How do you let iohyve set up networking?

pr1ntf · 2016-04-30T07:35:56Z

Thanks for the spam, @skarekrow.

skarekrow · 2016-04-30T07:38:02Z

Meh. Email right? You said what I did anyways.

pr1ntf · 2016-04-30T07:42:45Z

Yeah, you said the same thing I did, but like 4 seconds before I did.

As before, re0 needs to be a member of bridge0.

iohyve can do this for you with the FreeNAS "tunables" iohyve_enable="YES" and iohyve_flags="kmod=1 net=re0"

Otherwise, re0 needs to be added to bridge0 manually.

jonblack · 2016-04-30T07:49:56Z

My tunable had the wrong interface. em0 instead of re0. So now there is only a bridge0 and a re0 and tap0 are members. Still not networking though 😟

[root@mnemosyne] ~# ifconfig
re0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=82099<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC,LINKSTATE>
    ether 50:46:5d:51:db:a5
    inet 192.168.2.46 netmask 0xffffff00 broadcast 192.168.2.255 
    nd6 options=9<PERFORMNUD,IFDISABLED>
    media: Ethernet autoselect (1000baseT <full-duplex>)
    status: active
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
    options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
    inet6 ::1 prefixlen 128 
    inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2 
    inet 127.0.0.1 netmask 0xff000000 
    nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: iohyve-bridge
    ether 02:d2:e4:fb:79:00
    nd6 options=1<PERFORMNUD>
    id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
    maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
    root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
    member: epair1a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 6 priority 128 path cost 2000
    member: epair0a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 5 priority 128 path cost 2000
    member: tap0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 4 priority 128 path cost 2000000
    member: re0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 1 priority 128 path cost 20000
tap0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: iohyve-debian
    options=80000<LINKSTATE>
    ether 00:bd:44:18:01:00
    inet 0.0.0.0 netmask 0xff000000 broadcast 255.255.255.255 
    nd6 options=9<PERFORMNUD,IFDISABLED>
    media: Ethernet autoselect
    status: active
    Opened by PID 4166
epair0a: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=8<VLAN_MTU>
    ether 02:ff:20:00:05:0a
    nd6 options=1<PERFORMNUD>
    media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
    status: active
epair1a: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=8<VLAN_MTU>
    ether 02:ff:20:00:06:0a
    nd6 options=1<PERFORMNUD>
    media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
    status: active

This is confirmed when I using iohyve console debian and log in to the machine and try to ping google.

pr1ntf · 2016-04-30T07:55:57Z

I'll try to reproduce, but I suspect the guest is started before iohyve does the networking stuff.

pr1ntf · 2016-04-30T07:59:10Z

Can I get the output of your dmesg -a right after boot time when you get a chance?

jonblack · 2016-04-30T08:13:23Z

The dmesg output is here: http://pastebin.com/FyTSeckc

pr1ntf · 2016-04-30T08:17:55Z

Awesome, I'll do some digging and see what I can come up with.

Thanks for all your work, by the way. Some would have given up by now. 😎

jonblack · 2016-04-30T08:20:53Z

No worries. If you need anything else, let me know.

pr1ntf · 2016-05-06T07:40:36Z

I have been unable to reproduce this issue on "vanilla" FreeBSD, not FreeNAS. I'm unable to ascertain what's going on, but I do find that your tap0 interface having a 0.0.0.0 IP Address to be a bit strange.

Cloudxtreme · 2016-05-13T12:45:13Z

Set the IP on the bridge0 interface not re0. You should not have addresses on individual interfaces within a bridge.

Then ensure your /etc/resolv.conf is setup correctly.

Then most likely you will need to fix your routes unless you reboot.

Example:
route del default
route add 192.168.2.0 default

(I believe that's the command - not looking at it right this second)

pr1ntf · 2016-05-15T06:05:34Z

@jonblack are you still seeing this problem? I still cannot reproduce this issue in the latest version.

If you are still seeing an issue I will reopen the issue and see what we can do.

jonblack · 2016-05-15T10:07:45Z

I tired to route fix above but without success so I deleted the guest and re-created everything and now it's working fine. Perhaps something went wrong when I deleted the tap and bridge interfaces...who knows? In any case, with the tunables shown above and setting boot on the guest, everything starts up as it should and I can ssh into the guest now.

pr1ntf · 2016-05-16T04:00:02Z

Glad to hear you got it sorted out. Let us know if you have any further issues.
👍

pr1ntf added the question label Apr 27, 2016

pr1ntf closed this as completed May 15, 2016

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

VM doesn't run automatically on boot on FreeNAS #150

VM doesn't run automatically on boot on FreeNAS #150

jonblack commented Apr 27, 2016

pr1ntf commented Apr 27, 2016

EpiJunkie commented Apr 27, 2016

jonblack commented Apr 28, 2016

pr1ntf commented Apr 28, 2016

jonblack commented Apr 29, 2016

pr1ntf commented Apr 30, 2016

pr1ntf commented Apr 30, 2016

jonblack commented Apr 30, 2016

pr1ntf commented Apr 30, 2016

jonblack commented Apr 30, 2016

pr1ntf commented Apr 30, 2016

jonblack commented Apr 30, 2016

pr1ntf commented Apr 30, 2016

pr1ntf commented Apr 30, 2016

skarekrow commented Apr 30, 2016

pr1ntf commented Apr 30, 2016

jonblack commented Apr 30, 2016

pr1ntf commented Apr 30, 2016

pr1ntf commented Apr 30, 2016

jonblack commented Apr 30, 2016

pr1ntf commented Apr 30, 2016

jonblack commented Apr 30, 2016

pr1ntf commented May 6, 2016

Cloudxtreme commented May 13, 2016 •

edited

pr1ntf commented May 15, 2016

jonblack commented May 15, 2016

pr1ntf commented May 16, 2016

VM doesn't run automatically on boot on FreeNAS #150

VM doesn't run automatically on boot on FreeNAS #150

Comments

jonblack commented Apr 27, 2016

pr1ntf commented Apr 27, 2016

EpiJunkie commented Apr 27, 2016

jonblack commented Apr 28, 2016

pr1ntf commented Apr 28, 2016

jonblack commented Apr 29, 2016

pr1ntf commented Apr 30, 2016

pr1ntf commented Apr 30, 2016

jonblack commented Apr 30, 2016

pr1ntf commented Apr 30, 2016

jonblack commented Apr 30, 2016

pr1ntf commented Apr 30, 2016

jonblack commented Apr 30, 2016

pr1ntf commented Apr 30, 2016

pr1ntf commented Apr 30, 2016

skarekrow commented Apr 30, 2016

pr1ntf commented Apr 30, 2016

jonblack commented Apr 30, 2016

pr1ntf commented Apr 30, 2016

pr1ntf commented Apr 30, 2016

jonblack commented Apr 30, 2016

pr1ntf commented Apr 30, 2016

jonblack commented Apr 30, 2016

pr1ntf commented May 6, 2016

Cloudxtreme commented May 13, 2016 • edited

pr1ntf commented May 15, 2016

jonblack commented May 15, 2016

pr1ntf commented May 16, 2016

Cloudxtreme commented May 13, 2016 •

edited