kvm2: minikube start fails after host reboot: "network 'minikube-net' is not active'

[abonas]

Is this a BUG REPORT or FEATURE REQUEST? (choose one):

bug

Environment:

Minikube version (use minikube version):

0.25

• OS (e.g. from /etc/os-release):
  Fedora 26
• VM Driver (e.g. cat ~/.minikube/machines/minikube/config.json | grep DriverName):

kvm2

• ISO version (e.g. cat ~/.minikube/machines/minikube/config.json | grep -i ISO or minikube ssh cat /etc/VERSION):
• Install tools:
• Others:
  The above can be generated in one go with the following commands (can be copied and pasted directly into your terminal):

minikube version
echo "";
echo "OS:";
cat /etc/os-release
echo "";
echo "VM driver": 
grep DriverName ~/.minikube/machines/minikube/config.json
echo "";
echo "ISO version";
grep -i ISO ~/.minikube/machines/minikube/config.json

What happened:

minikube start --vm-driver=kvm2

Starting local Kubernetes v1.9.0 cluster...
Starting VM...
E0204 22:35:52.820568 30508 start.go:159] Error starting host: Error starting stopped host: Error creating VM: virError(Code=55, Domain=19, Message='Requested operation is not valid: network 'minikube-net' is not active').

How to reproduce it (as minimally and precisely as possible):
minikube start --vm-driver=kvm2

[codmajik]

if you have virt-manager package you can do: net-start minikube-net on the virsh shell or just use the virt-manager UI that comes with the package.

on the UI, go to menu Edit > Connection Details and on the resulting dialog box select the Virtual Network Tab.
select minikube-net from the list and click the start button (the button with the play symbol at the bottom of the list) then click apply

I use archlinux so package name might be different. to start

[abonas]

@codmajik thanks for your response.
the minikube-net does not exist.
I verified both in the virsh shell and in UI.
in virsh shell the output was:
error: Network not found: no network with matching name 'minikube-net'

[gabx]

On a fresh Fedora 27 install, I run into the same problem about domain not found

[ghost]

To resolve this:

Assuming you can access virsh do the following:

1. Type virsh
2. Type net-list --all - should see that minikube-net is inactive
3. Type net-start minikube-net - should get an error message about "Network is already in use by interface virbr1" or similar
4. Quit virsh
5. Type sudo ifconfig virbr1 down
6. Type sudo brctl delbr virbr1
7. Type virsh
8. Type net-start minikube-net - should now start-up
9. Quit virsh
10. Type minikube start

[andrewbenton]

I was able to resolve this on Fedora 27.

1. run minikube start --vm-driver=kvm2
2. let it fail
3. switch to root
4. start virsh
5. under virsh, run net-edit default
6. change network>bridge.name in the xml to virbr1 (or some other unoccupied address)
7. save and quit
8. under virsh, run net-start minikube-net
9. exit root
10. run minikube start --vm-driver=kvm2, this time it should succeed

The issue for me seems to stem from the minikube-net trying to also occupy virbr0, which is what default does naturally. Ideally, minikube-net should pick another default.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[bperel]

/remove-lifecycle stale

[bperel]

The fix provided by @andrewbenton works for me but I have to apply it each time my machine restarts, otherwise I keep having this error

[zikes]

On Arch Linux I had to fix this with:

1. sudo virsh
2. net-start minikube-net
3. quit
4. minikube start --vm-driver=kvm2

The sudo was necessary for virsh to see the available networks via net-list --all, whereas other examples here seem to omit it.

[michaelfung]

On Debian stretch I have similar problem after machine restarted normally.

minikube version: v0.28.2
driver: kvm2

After following @zikes instructions, minikube can start, but one of the deployments (http-echo) cannot start anymore.

get pods shows:

NAME                             READY     STATUS             RESTARTS   AGE
hello-minikube-6c47c66d8-vdr9w   1/1       Running            2          4h
http-echo-7ff6d594c-6s59n        0/1       CrashLoopBackOff   13         4h

syslog has a bunch of:

Jul 28 06:02:03 debian libvirtd[481]: 2018-07-28 10:02:03.912+0000: 481: error : virNetSocketReadWire:1801 : End of file while reading data: Input/output error
Jul 28 06:02:03 debian libvirtd[481]: 2018-07-28 10:02:03.912+0000: 481: error : virNetSocketReadWire:1801 : End of file while reading data: Input/output error
Jul 28 06:02:03 debian libvirtd[481]: 2018-07-28 10:02:03.912+0000: 481: error : virNetSocketReadWire:1801 : End of file while reading data: Input/output error

[nirmoy]

also virsh net-autostart minikube-net --> to have a persistent minikube-net over restarts

[michaelfung]

Thanks @nirmoy , that works for me.

Also, for my deployment not running anymore problem, it is due to I used environment variable that clashed with the one generated by kubernetes using the deployment name as prefix. Perhaps it is good to advice users of this behavior in the doc.

[mheese]

There seem to be two problems discussed here in the ticket:

1. minikube start fails with the error message as described in the subject

and I can actually see the bug in the source code, and I'm in the process of writing a fix and making a PR for it: it's coming from here:

minikube/pkg/drivers/kvm/network.go

Line 47 in 616c3e5

n, err := conn.LookupNetworkByName(defaultNetworkName)

The problem is that because of the wrong variable being used, the private network that is created does not get the autostart feature, and when minikube gets stopped the network is inactive. It is not being recreated on another minikube start and that leads to the error above. I believe (not 100% on this yet) that on the first create the network is also started which is the reason why there is no problem on the first create.

2. the defined XML leaves a virbrN behind which might be in use already which leads to the second error for some people when they try to start the minikube-net network using virsh.

I'm unclear as to why people run into this second issue. And I can't reproduce this here either so far because the used template correctly omits to specify a bridge in the XML which means according to the libvirt docs that it creates the next free virbrN device. So unless you have created other bridges in the meantime while this one was inactive (and even then I believe one should get a different virbrN from libvirt, there is no way to run into this. That said, it might also be a libvirt bug in some version?

If people could give me a bit more insight for the second problem, I hope that I'm able to address both in one PR.

[mheese]

My fix seems to work, will need to check in with my company to see how exactly I am allowed to create the PR (organization forked repo vs my private github forked repo for the PR).

[mheese]

The branch is up for anybody who already wants to take a peak: https://github.com/etherparty/minikube/tree/fix-issue-2513

We are currently working on signing of the CNCF CLA as an organization, then I can do the PR.

[tstromberg]

@mheese - Great news. Feel free to send the PR to myself.

In the mean time, I'm using this workaround the first time I run into it:

sudo virsh net-autostart minikube-net                                                                                                                                                                            
sudo virsh net-start minikube-net   

[mheese]

@tstromberg created the PR, it's linked above. Feel free to ping me on the Kubernetes slack. I'm mheese there as well.

[tstromberg]

Surprisingly, I ran into this today with v0.30.0: worked around it by using minikube delete && minikube start. This appears to have been due to minikube expecting the previously configured VM to have a network.

[mheese]

@tstromberg that actually would exactly mean that this has not been included in 0.30.0 ... because that is kind of what it fixes. However, was that a minikube VM that already existed prior to 0.30.0?

[tstromberg]

@mheese - Good point, I'm not sure in this case. I'll let you know the next time I see breakage (along with logs), since it would definitely be with a post-0.30.0 VM.

[tstromberg]

I think this may still be an issue, at least on the version of Debian I have. To replicate: minikube start && reboot && minikube start

[mheese]

@tstromberg which Debian version are you running? I'll try to reproduce it because I really can't :) ... the problem is totally gone for me. The driver is now creating the network with autostart, plus it is actually starting the network if it is not active. So it really should be covered.

[tstromberg]

I tried to duplicate this in v0.33.1, but I couldn't across several reboots. I know there is a latent condition here somewhere, but I'm going to close this for now until we see it again and can root cause it appropriately.

Thank you @mheese!