(meta) Make Windows/Hyper-V a first-class solution

[tstromberg]

Here are the issues to be resolved:

• Hyper-V: Use the "Default Switch" (NAT based) by default - Hyper-V: Use the "Default Switch" (NAT based) by default #4079
• Hyper-V: Stable integration testing Hyper-V: Stable integration testing #3591
• Hyper-V: minikube mount fails with windows 10 default switch - Hyper-V: Stable integration testing #3591
• [NON_C_DRIVE] when run from C:\ but user config is on different drive - Hyper-V: Use the "Default Switch" (NAT based) by default #4079

Please feel free to comment with any others that cause a large user experience gap for users of Windows + Hyper-V.

[orthoxerox]

Being able to run minikube as a non-admin user on Windows is pretty important if you value enterprise adoption of minikube.

Docker Desktop, for example, achieves this by running a privileged service that creates and destroys VMs when commanded by a non-privileged application launched by the user.

[tstromberg]

@orthoxerox - Understood. I personally am not a fan of that model, as it subverts the more typical Hyper-V Administrators ACL: https://blogs.msdn.microsoft.com/virtual_pc_guy/2014/06/11/allowing-non-administrators-to-control-hyper-vupdated/

[orthoxerox]

@tstromberg I think the idea was to allow the user to create and run just one VM, the only with MobyLinux on it. Making a user a Hyper-V admin allows them to create and run all possible VMs.

[brainfull]

I spent the last few days on automating the install and start/stop/delete of minikube on Windows 10 Pro with Hyper-V and went through a lot of problems. So here are the problems I faced and how I solved. I hope it will be useful.

1. External VM Switch : While it is easy to setup an external VM switch, it goes through your network DHCP server to get an address and you cannot put your laptop to sleep and open it under another Wi-Fi network and hope that everything runs fine, it won't. You would have to restart Minikube everytime to get a new valid IP address and that means you need to be online to get an IP address from the network DHCP server. So it doesn't allow to work offline once your docker images are built. Moreover it is insecure because your minikube VM get an IP address available on the network you are connected to. So it's better to go with internal VM Switch in my opinion.

2. Internal VM Switch : There are many advantages to use internal VM Switch but there's a lack of understanding overall on the web about it.
   2.1 First, it is necessary to setup an old Windows technology called ICS (Internet Connection Sharing) which deploys its own internal ICS DHCP server. This allows to connect to another Wi-Fi network without having trouble to restart minikube because the minikube VM IP address is given by the internal ICS DHCP server. I tried for hours to set a static address within minikube VM and got some partial success but it was never stable from one run to the next. And by the way I don't understand why minikube is built on such a custom version of linux, it's very difficult to find how to do anything in minikube that you can do with a popular distritbution of linux....
   2.2 Second, it comes with a neat feature of assigning an hostname to your VM. For example, if your VM is called 'minikube' you will be able to connect to it through 'minikube.mshome.net' which is defined under C:\Windows\System32\drivers\etc\hosts.ics automatically. Every times you restart minikube the hostname will be updated with the latest IP address assigned by the internal ICS DHCP server. So there's no need to try to setup a static IP address or setup a hostname in the hosts file by yourself. As an example, I actually use this internal hostname in Pycharm to connect with SSH to a Nodeport service that expose a container running Python within my cluster, allowing me to debug within Linux directly even if I'm running on Windows. Having a static hostname allows to keep stable settings in Pycharm like root@minikube.mshome.net:30022 which is actually a python container with Open-SSH listening on port 22. See more about that here access external ip from outside network #5812 .
   2.3 However, everytime you reboot your computer, even if the ICS settings survived, somehow minikube is not able to start correctly and it will fail when launching kubernetes (hyperv: cannot use "fe80::215:5dff:fe45:3110" as the bind address for the API Server (IPv6) #3963 ). So it is necessary to disable ICS and re-enable it again between your Wi-FI and the internal VM switch after each reboot. That is why I came with my powershell script that does it for me. I copied it in the comment for your pleasure. I call the powershell script automatically just before 'minikube start'. Essentially, it creates the internal VM switch if it doesn't exist, it disable whatever ICS settings that pre-existed, it setups ICS between the internet connection and the internal VM switch. It would be great to have such a feature in minikube, maybe a command line 'minikube start ... --hyperv-virtual-switch minikube --auto-enable-ics'
   Set-ICS.ps1.txt

3. Kubectl : Unfortunately the kubernetes client that get installed on your machine keeps a cache that may become a problem when your minikube VM IP address changes. So it is better to delete C:/Users//.kube (same as %userprofile%/.kube) before every 'minikube start'. This way you ensure that the old kubernetes cache will not interfere with your new minikube state when you call minikube through the local installation of kubectl.

4. Docker Build : For some reason when you restart minikube many times you end up with a very slow docker build. Somehow, pulling images like ubuntu or simply calling apt-get within ubuntu will get very slow. And I finally found out why by looking at the network usage on my internal VM Switch. I installed only docker-cli on my laptop and use docker engine pre-installed within minikube by invoking 'minikube docker-env'. While it works very well, sometimes the network bridge that is created between docker-cli and minikube container during the build process is very slow and the pull image will literally get stuck for many minutes. To avoid this problem always build images with 'docker build .... --network host'. This way you make sure the docker image that is currently building is using the network within minikube VM, not using a bridge between your machine and the container within minikube VM, which can become very slow.
   4.1 It would be great to have docker-cli installed automatically with minikube. At least when I install minikube with chocolatey it depends on kubernetes-cli. Would be great that it depends on docker-cli too. Why would you want to have a complex system with Docker on Windows and minikube altogether.
   4.2 It would be great to have 'minikube docker-env' called automatically by minikube. Maybe a command like 'minikube start ... --auto-docker-env' could setup the environment variables as system environment variables.
   4.3 It would be great to find a way to make docker use '--network host' by default during docker build and during docker run. I don't know if there is a configuration that could be setup when docker is launched in minikube. I definitely found many post all over the internet about slow pull image and slow apt-get and there was rarely a clear answer. Actually no post pointed to the default use of bridge network during docker build.

So that's it for today. I'll post other findings if I still get problems. I definitely find there is a lack of documentation on minikube on Windows.

But for now I am pretty happy with the dev environment on my laptop. It allows to automate the deployment, the debug and the test and all the team will have the same dev environment.

Oh! I almost forgot about a bug in https://github.com/kubernetes/minikube/blob/master/cmd/minikube/cmd/start.go .

The code below will delete your minikube VM because of any reason. That means that if for some reason you didn't have enough memory to start minikube VM, IT WILL GET DELETED 5 TIMES. Not only you wait 5 useless retry but also you lose everything you did in your minikube VM. I don't think the code below make any sense. 'minikube start' should never delete the VM. We should explicitly use 'minikube delete' if we ever think the solution is to delete the minikube VM.

start := func() (err error) {
	host, err = cluster.StartHost(api, mc)
	if err != nil {
		out.T(out.Resetting, "Retriable failure: {{.error}}", out.V{"error": err})
		if derr := cluster.DeleteHost(api); derr != nil {
			glog.Warningf("DeleteHost: %v", derr)
		}
	}
	return err
}

if err = retry.Expo(start, 5*time.Second, 3*time.Minute, 3); err != nil {
	exit.WithError("Unable to start VM", err)
}

Also see other issues like #5812, #5627, #5941 ...

[kfox1111]

Defaulting to internal switch like the other platforms would be very good. Some sites, like my org's capture unknown mac address dhcp requests and assign them temporary ip's and isolate traffic until they agree to some terms of service / register some additional information. Making it unsuitable for minikube. Internal switch with nat works fine though as the host is already registered.

[brainfull]

Defaulting to internal switch like the other platforms would be very good. Some sites, like my org's capture unknown mac address dhcp requests and assign them temporary ip's and isolate traffic until they agree to some terms of service / register some additional information. Making it unsuitable for minikube. Internal switch with nat works fine though as the host is already registered.

FYI If you would need to set a pre-defined mac address on the minikube VM this is the script to do it. It could be nice to have a feature in minikube to set that mac address automatically on 'minikube start ... --mac-address ....'.

The following in Powershell works fine but is cumbersome because you need to start and stop and start again. And you need to redo it everytime the VM is deleted.

minikube start
minikube stop
Set-VMNetworkAdapter -VMName minikube -StaticMacAddress 00-15-5D-2B-D9-26
minikube start

[kfox1111]

yes, but you still need a mechanism to get that mac address either preregistered with the network, or get a web browser in the vm long enough to do the registration, before minikube starts trying to fetch artefacts. Either way, not a great user experience. Defaulting to internal and making nat work out of the box like the other platforms would be seamless/much preferred.

[medyagh]

this is still on our roadmap !

[dsebastien]

Really interested in seeing progress for this.

I've spent the whole day trying to make things work for my use case.

I had trouble getting the Minikube VM to access internet (I suppose because I didn't do anything for ICS with the default switch?). I hoped everything would workd OOB but it didn't :)

I ended up using an external vSwitch in Hyper-V which fixed that issue, but now my problem is that I can only access the VM ports when using minikube tunnel, through the unstable IP.

The problem is that our setup within K8S uses an Nginx ingress which maps the "localhost" host to some web service, which of course does not work when the host header is set to... something else.. :)

On Linux this setup works fine because I can setup port forwarding to the host using commands such as:
vboxmanage controlvm "minikube" natpf1 "blabla,tcp,127.0.0.1,4200,,30000"

I could work around this by forwarding the ports exposed by the Hyper-V VM using commands such as: netsh interface portproxy add v4tov4 listenaddress=127.0.0.1 listenport=4200 connectaddress="$(minikube ip)" connectport=30000.

This seems to work but of course remains fragile as the Minikube VM IP on my network is not stable and will be different for each developer (assuming an heterogeneous environment). For the moment I'm just re-executing those commands in my start script (right after minikube start), but this ain't great as it requires admin rights if I'm not mistaken (same for minikube start though).

I suppose I could indeed define a static Mac after the VM's creation and register that on my network but this feels flaky and tiresome, plus it puts too many expectations on things outside of my scripts control.

I'm really hoping for something better/simpler :)

[tstromberg]

Related: #6264

[tstromberg]

I think we should be able to close this once v1.9 is released, given the improvements to hyper-v deletion. The rest of the major issues seem sorted out.

That is not to say that Windows support is completely bug-free or as well tested as other platforms, but that we should track those issues with a finer granularity.

[tstromberg]

Closing because we've reached enough parity that we can now track this via individual issues. Thank you for your patience!