Network \ Startup issues on new installation

[mrcsdf]

Steps to reproduce the issue:

1. Run the following commands

minikube start --vm-driver="hyperv" --hyperv-virtual-switch="minikube"
kubectl create -f .\helloworld.yaml
kubectl expose deployment helloworld --type=NodePort
minikube service helloworld

Here is the helloworld.yaml file for reference

apiVersion: apps/v1
kind: Deployment
metadata:
  name: helloworld
spec:
  selector:
    matchLabels:
      app: helloworld
  replicas: 1 # tells deployment to run 1 pods matching the template
  template: # create pods using pod definition in this template
    metadata:
      labels:
        app: helloworld
    spec:
      containers:
      - name: helloworld
        image: karthequian/helloworld:latest
        ports:
        - containerPort: 80

2. Page fails to load, so I check network configuration and modify the Hyper-v connection from internal to external
3. Restart minikube vm

Full output of minikube logs command:

==> Audit <==
|---------|----------------------------------|----------|----------------------|---------|-------------------------------|-------------------------------|
| Command | Args | Profile | User | Version | Start Time | End Time |
|---------|----------------------------------|----------|----------------------|---------|-------------------------------|-------------------------------|
| start | --vm-driver=hyperv | minikube | NORTHAMERICA\marcusf | v1.20.0 | Thu, 10 Jun 2021 10:23:28 EDT | Thu, 10 Jun 2021 10:25:45 EDT |
| | --hyperv-virtual-switch=minikube | | | |
| |
| start | | minikube | NORTHAMERICA\marcusf | v1.20.0 | Thu, 10 Jun 2021 10:32:20 EDT | Thu, 10 Jun 2021 10:32:51 EDT |
| service | helloworld | minikube | NORTHAMERICA\marcusf | v1.20.0 | Thu, 10 Jun 2021 10:36:22 EDT | Thu, 10 Jun 2021 10:36:26 EDT |
| config | | minikube | NORTHAMERICA\marcusf | v1.20.0 | Thu, 10 Jun 2021 10:46:47 EDT | Thu, 10 Jun 2021 10:46:47 EDT |
|---------|----------------------------------|----------|----------------------|---------|-------------------------------|-------------------------------|

==> Last Start <==
Log file created at: 2021/06/10 10:49:50
Running on machine: marcusf-sfc3
Binary: Built with gc go1.16.1 for windows/amd64
Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
I0610 10:49:50.789657 19260 out.go:291] Setting OutFile to fd 92 ...
I0610 10:49:50.790657 19260 out.go:304] Setting ErrFile to fd 96...
W0610 10:49:50.804544 19260 root.go:291] Error reading config file at C:\Users\marcusf.minikube\config\config.json: open C:\Users\marcusf.minikube\config\config.json: The system cannot find the file specified.
I0610 10:49:50.805086 19260 out.go:298] Setting JSON to false
I0610 10:49:50.813725 19260 start.go:108] hostinfo: {"hostname":"marcusf-sfc3","uptime":92613,"bootTime":1623243977,"procs":272,"os":"windows","platform":"Microsoft Windows 10 Enterprise","platformFamily":"Standalone Workstation","platformVersion":"10.0.19043 Build 19043","kernelVersion":"10.0.19043 Build 19043","kernelArch":"x86_64","virtualizationSystem":"","virtualizationRole":"","hostId":"e584d59e-2cf5-4627-b929-c7871dca71b7"}
W0610 10:49:50.813725 19260 start.go:116] gopshost.Virtualization returned error: not implemented yet
I0610 10:49:50.816909 19260 out.go:170] 😄 minikube v1.20.0 on Microsoft Windows 10 Enterprise 10.0.19043 Build 19043
I0610 10:49:50.817055 19260 notify.go:169] Checking for updates...
I0610 10:49:50.818802 19260 driver.go:322] Setting default libvirt URI to qemu:///system
I0610 10:49:51.711422 19260 out.go:170] ✨ Using the hyperv driver based on existing profile
I0610 10:49:51.711422 19260 start.go:276] selected driver: hyperv
I0610 10:49:51.711422 19260 start.go:718] validating driver "hyperv" against &{Name:minikube KeepContext:false EmbedCerts:false MinikubeISO:https://storage.googleapis.com/minikube/iso/minikube-v1.20.0.iso KicBaseImage:gcr.io/k8s-minikube/kicbase:v0.0.22@sha256:7cc3a3cb6e51c628d8ede157ad9e1f797e8d22a1b3cedc12d3f1999cb52f962e Memory:6000 CPUs:2 DiskSize:20000 VMDriver: Driver:hyperv HyperkitVpnKitSock: HyperkitVSockPorts:[] DockerEnv:[] ContainerVolumeMounts:[] InsecureRegistry:[] RegistryMirror:[] HostOnlyCIDR:192.168.99.1/24 HypervVirtualSwitch:minikube HypervUseExternalSwitch:false HypervExternalAdapter: KVMNetwork:default KVMQemuURI:qemu:///system KVMGPU:false KVMHidden:false KVMNUMACount:1 DockerOpt:[] DisableDriverMounts:false NFSShare:[] NFSSharesRoot:/nfsshares UUID: NoVTXCheck:false DNSProxy:false HostDNSResolver:true HostOnlyNicType:virtio NatNicType:virtio SSHIPAddress: SSHUser:root SSHKey: SSHPort:22 KubernetesConfig:{KubernetesVersion:v1.20.2 ClusterName:minikube Namespace:default APIServerName:minikubeCA APIServerNames:[] APIServerIPs:[] DNSDomain:cluster.local ContainerRuntime:docker CRISocket: NetworkPlugin: FeatureGates: ServiceCIDR:10.96.0.0/12 ImageRepository: LoadBalancerStartIP: LoadBalancerEndIP: CustomIngressCert: ExtraOptions:[] ShouldLoadCachedImages:true EnableDefaultCNI:false CNI: NodeIP: NodePort:8443 NodeName:} Nodes:[{Name: IP:192.168.137.34 Port:8443 KubernetesVersion:v1.20.2 ControlPlane:true Worker:true}] Addons:map[ambassador:false auto-pause:false csi-hostpath-driver:false dashboard:false default-storageclass:true efk:false freshpod:false gcp-auth:false gvisor:false helm-tiller:false ingress:false ingress-dns:false istio:false istio-provisioner:false kubevirt:false logviewer:false metallb:false metrics-server:false nvidia-driver-installer:false nvidia-gpu-device-plugin:false olm:false pod-security-policy:false registry:false registry-aliases:false registry-creds:false storage-provisioner:true storage-provisioner-gluster:false volumesnapshots:false] VerifyComponents:map[apiserver:true system_pods:true] StartHostTimeout:6m0s ScheduledStop: ExposedPorts:[] ListenAddress: Network: MultiNodeRequested:false}
I0610 10:49:51.711422 19260 start.go:729] status for hyperv: {Installed:true Healthy:true Running:false NeedsImprovement:false Error: Reason: Fix: Doc:}
I0610 10:49:55.722807 19260 cni.go:93] Creating CNI manager for ""
I0610 10:49:55.722807 19260 cni.go:167] CNI unnecessary in this configuration, recommending no CNI
I0610 10:49:55.722843 19260 start_flags.go:273] config:
{Name:minikube KeepContext:false EmbedCerts:false MinikubeISO:https://storage.googleapis.com/minikube/iso/minikube-v1.20.0.iso KicBaseImage:gcr.io/k8s-minikube/kicbase:v0.0.22@sha256:7cc3a3cb6e51c628d8ede157ad9e1f797e8d22a1b3cedc12d3f1999cb52f962e Memory:6000 CPUs:2 DiskSize:20000 VMDriver: Driver:hyperv HyperkitVpnKitSock: HyperkitVSockPorts:[] DockerEnv:[] ContainerVolumeMounts:[] InsecureRegistry:[] RegistryMirror:[] HostOnlyCIDR:192.168.99.1/24 HypervVirtualSwitch:minikube HypervUseExternalSwitch:false HypervExternalAdapter: KVMNetwork:default KVMQemuURI:qemu:///system KVMGPU:false KVMHidden:false KVMNUMACount:1 DockerOpt:[] DisableDriverMounts:false NFSShare:[] NFSSharesRoot:/nfsshares UUID: NoVTXCheck:false DNSProxy:false HostDNSResolver:true HostOnlyNicType:virtio NatNicType:virtio SSHIPAddress: SSHUser:root SSHKey: SSHPort:22 KubernetesConfig:{KubernetesVersion:v1.20.2 ClusterName:minikube Namespace:default APIServerName:minikubeCA APIServerNames:[] APIServerIPs:[] DNSDomain:cluster.local ContainerRuntime:docker CRISocket: NetworkPlugin: FeatureGates: ServiceCIDR:10.96.0.0/12 ImageRepository: LoadBalancerStartIP: LoadBalancerEndIP: CustomIngressCert: ExtraOptions:[] ShouldLoadCachedImages:true EnableDefaultCNI:false CNI: NodeIP: NodePort:8443 NodeName:} Nodes:[{Name: IP:192.168.137.34 Port:8443 KubernetesVersion:v1.20.2 ControlPlane:true Worker:true}] Addons:map[ambassador:false auto-pause:false csi-hostpath-driver:false dashboard:false default-storageclass:true efk:false freshpod:false gcp-auth:false gvisor:false helm-tiller:false ingress:false ingress-dns:false istio:false istio-provisioner:false kubevirt:false logviewer:false metallb:false metrics-server:false nvidia-driver-installer:false nvidia-gpu-device-plugin:false olm:false pod-security-policy:false registry:false registry-aliases:false registry-creds:false storage-provisioner:true storage-provisioner-gluster:false volumesnapshots:false] VerifyComponents:map[apiserver:true system_pods:true] StartHostTimeout:6m0s ScheduledStop: ExposedPorts:[] ListenAddress: Network: MultiNodeRequested:false}
I0610 10:49:55.722843 19260 iso.go:123] acquiring lock: {Name:mkfa2fe4faee0c68900d0f52acbccd60dba2b5a0 Clock:{} Delay:500ms Timeout:10m0s Cancel:}
I0610 10:49:55.725588 19260 out.go:170] 👍 Starting control plane node minikube in cluster minikube
I0610 10:49:55.725598 19260 preload.go:98] Checking if preload exists for k8s version v1.20.2 and runtime docker
I0610 10:49:55.725598 19260 preload.go:106] Found local preload: C:\Users\marcusf.minikube\cache\preloaded-tarball\preloaded-images-k8s-v10-v1.20.2-docker-overlay2-amd64.tar.lz4
I0610 10:49:55.725598 19260 cache.go:54] Caching tarball of preloaded images
I0610 10:49:55.725598 19260 preload.go:132] Found C:\Users\marcusf.minikube\cache\preloaded-tarball\preloaded-images-k8s-v10-v1.20.2-docker-overlay2-amd64.tar.lz4 in cache, skipping download
I0610 10:49:55.725598 19260 cache.go:57] Finished verifying existence of preloaded tar for v1.20.2 on docker
I0610 10:49:55.726137 19260 profile.go:148] Saving config to C:\Users\marcusf.minikube\profiles\minikube\config.json ...
I0610 10:49:55.728119 19260 cache.go:194] Successfully downloaded all kic artifacts
I0610 10:49:55.728119 19260 start.go:313] acquiring machines lock for minikube: {Name:mk0c24b2cc23b94afecccaea90da889eafa73b04 Clock:{} Delay:500ms Timeout:13m0s Cancel:}
I0610 10:49:55.728119 19260 start.go:317] acquired machines lock for "minikube" in 0s
I0610 10:49:55.728626 19260 start.go:93] Skipping create...Using existing machine configuration
I0610 10:49:55.728645 19260 fix.go:55] fixHost starting:
I0610 10:49:55.728645 19260 main.go:128] libmachine: [executing ==>] : C:\WINDOWS\System32\WindowsPowerShell\v1.0\powershell.exe -NoProfile -NonInteractive ( Hyper-V\Get-VM minikube ).state
I0610 10:49:56.409267 19260 main.go:128] libmachine: [stdout =====>] : Running

I0610 10:49:56.409267 19260 main.go:128] libmachine: [stderr =====>] :
I0610 10:49:56.409267 19260 fix.go:108] recreateIfNeeded on minikube: state=Running err=
W0610 10:49:56.409267 19260 fix.go:134] unexpected machine state, will restart:
I0610 10:49:56.413155 19260 out.go:170] 🏃 Updating the running hyperv "minikube" VM ...
I0610 10:49:56.413330 19260 machine.go:88] provisioning docker machine ...
I0610 10:49:56.413330 19260 buildroot.go:166] provisioning hostname "minikube"
I0610 10:49:56.413330 19260 main.go:128] libmachine: [executing ==>] : C:\WINDOWS\System32\WindowsPowerShell\v1.0\powershell.exe -NoProfile -NonInteractive ( Hyper-V\Get-VM minikube ).state
I0610 10:49:56.995772 19260 main.go:128] libmachine: [stdout =====>] : Running

I0610 10:49:56.995811 19260 main.go:128] libmachine: [stderr =====>] :
I0610 10:49:56.995811 19260 main.go:128] libmachine: [executing ==>] : C:\WINDOWS\System32\WindowsPowerShell\v1.0\powershell.exe -NoProfile -NonInteractive (( Hyper-V\Get-VM minikube ).networkadapters[0]).ipaddresses[0]
I0610 10:49:57.755792 19260 main.go:128] libmachine: [stdout =====>] :
I0610 10:49:57.755792 19260 main.go:128] libmachine: [stderr =====>] :
I0610 10:49:57.755792 19260 machine.go:91] provisioned docker machine in 1.3424627s
I0610 10:49:57.755792 19260 fix.go:57] fixHost completed within 2.0271663s
I0610 10:49:57.755792 19260 start.go:80] releasing machines lock for "minikube", held for 2.0276735s
W0610 10:49:57.755792 19260 start.go:505] error starting host: provision: IP not found
W0610 10:49:57.756314 19260 out.go:235] 🤦 StartHost failed, but will try again: provision: IP not found
I0610 10:49:57.756382 19260 start.go:520] Will try again in 5 seconds ...
I0610 10:50:02.762723 19260 start.go:313] acquiring machines lock for minikube: {Name:mk0c24b2cc23b94afecccaea90da889eafa73b04 Clock:{} Delay:500ms Timeout:13m0s Cancel:}
I0610 10:50:02.763295 19260 start.go:317] acquired machines lock for "minikube" in 572.3µs
I0610 10:50:02.763603 19260 start.go:93] Skipping create...Using existing machine configuration
I0610 10:50:02.763603 19260 fix.go:55] fixHost starting:
I0610 10:50:02.763906 19260 main.go:128] libmachine: [executing ==>] : C:\WINDOWS\System32\WindowsPowerShell\v1.0\powershell.exe -NoProfile -NonInteractive ( Hyper-V\Get-VM minikube ).state
I0610 10:50:03.379071 19260 main.go:128] libmachine: [stdout =====>] : Running

I0610 10:50:03.379071 19260 main.go:128] libmachine: [stderr =====>] :
I0610 10:50:03.379071 19260 fix.go:108] recreateIfNeeded on minikube: state=Running err=
W0610 10:50:03.379071 19260 fix.go:134] unexpected machine state, will restart:
I0610 10:50:03.381880 19260 out.go:170] 🏃 Updating the running hyperv "minikube" VM ...
I0610 10:50:03.381901 19260 machine.go:88] provisioning docker machine ...
I0610 10:50:03.381901 19260 buildroot.go:166] provisioning hostname "minikube"
I0610 10:50:03.381901 19260 main.go:128] libmachine: [executing ==>] : C:\WINDOWS\System32\WindowsPowerShell\v1.0\powershell.exe -NoProfile -NonInteractive ( Hyper-V\Get-VM minikube ).state
I0610 10:50:03.950272 19260 main.go:128] libmachine: [stdout =====>] : Running

I0610 10:50:03.950272 19260 main.go:128] libmachine: [stderr =====>] :
I0610 10:50:03.950272 19260 main.go:128] libmachine: [executing ==>] : C:\WINDOWS\System32\WindowsPowerShell\v1.0\powershell.exe -NoProfile -NonInteractive (( Hyper-V\Get-VM minikube ).networkadapters[0]).ipaddresses[0]
I0610 10:50:04.862438 19260 main.go:128] libmachine: [stdout =====>] :
I0610 10:50:04.862438 19260 main.go:128] libmachine: [stderr =====>] :
I0610 10:50:04.862438 19260 machine.go:91] provisioned docker machine in 1.4805378s
I0610 10:50:04.862438 19260 fix.go:57] fixHost completed within 2.098835s
I0610 10:50:04.862438 19260 start.go:80] releasing machines lock for "minikube", held for 2.0991433s
W0610 10:50:04.862438 19260 out.go:235] 😿 Failed to start hyperv VM. Running "minikube delete" may fix it: provision: IP not found
I0610 10:50:04.864927 19260 out.go:170]
W0610 10:50:04.865455 19260 out.go:235] ❌ Exiting due to GUEST_PROVISION: Failed to start host: provision: IP not found
W0610 10:50:04.865455 19260 out.go:424] no arguments passed for "\n" - returning raw string
W0610 10:50:04.865455 19260 out.go:235]
W0610 10:50:04.865960 19260 out.go:424] no arguments passed for "😿 If the above advice does not help, please let us know:\n" - returning raw string
W0610 10:50:04.865970 19260 out.go:424] no arguments passed for "👉 https://github.com/kubernetes/minikube/issues/new/choose\n\n" - returning raw string
W0610 10:50:04.865970 19260 out.go:424] no arguments passed for "Please attach the following file to the GitHub issue:\n" - returning raw string
W0610 10:50:04.865979 19260 out.go:424] no arguments passed for "😿 If the above advice does not help, please let us know:\n👉 https://github.com/kubernetes/minikube/issues/new/choose\n\nPlease attach the following file to the GitHub issue:\n- C:\Users\marcusf\.minikube\logs\lastStart.txt\n\n" - returning raw string
W0610 10:50:04.866510 19260 out.go:235] ╭────────────────────────────────────────────────────────────────────╮
W0610 10:50:04.866651 19260 out.go:235] │
│
W0610 10:50:04.866673 19260 out.go:235] │ 😿 If the above advice does not help, please let us know:
│
W0610 10:50:04.866673 19260 out.go:235] │ 👉 https://github.com/kubernetes/minikube/issues/new/choose │
W0610 10:50:04.866673 19260 out.go:235] │
│
W0610 10:50:04.866673 19260 out.go:235] │ Please attach the following file to the GitHub issue:
│
W0610 10:50:04.866673 19260 out.go:235] │ - C:\Users\marcusf.minikube\logs\lastStart.txt
│
W0610 10:50:04.867197 19260 out.go:235] │
│
W0610 10:50:04.867197 19260 out.go:235] ╰────────────────────────────────────────────────────────────────────╯
W0610 10:50:04.867197 19260 out.go:235]

E0610 10:53:44.474615 18772 logs.go:267] Failed to list containers for "kube-apiserver": docker: NewSession: new client: new client: Error creating new ssh host from driver: Error getting ssh host name for driver: IP not found
E0610 10:53:47.727321 18772 logs.go:267] Failed to list containers for "etcd": docker: NewSession: new client: new client: Error creating new ssh host from driver: Error getting ssh host name for driver: IP not found
E0610 10:53:50.806208 18772 logs.go:267] Failed to list containers for "coredns": docker: NewSession: new client: new client: Error creating new ssh host from driver: Error getting ssh host name for driver: IP not found

Full output of failed command:

😄 minikube v1.20.0 on Microsoft Windows 10 Enterprise 10.0.19043 Build 19043 ✨ Using the hyperv driver based on existing profile 👍 Starting control plane node minikube in cluster minikube 🏃 Updating the running hyperv "minikube" VM ... 🤦 StartHost failed, but will try again: provision: IP not found 🏃 Updating the running hyperv "minikube" VM ... 😿 Failed to start hyperv VM. Running "minikube delete" may fix it: provision: IP not found

❌ Exiting due to GUEST_PROVISION: Failed to start host: provision: IP not found

╭────────────────────────────────────────────────────────────────────╮
│ │
│ 😿 If the above advice does not help, please let us know: │
│ 👉 https://github.com/kubernetes/minikube/issues/new/choose │
│ │
│ Please attach the following file to the GitHub issue: │
│ - C:\Users\marcusf.minikube\logs\lastStart.txt │
│ │
╰────────────────────────────────────────────────────────────────────╯

[spowelljr]

Hi @smtrock, thanks for reporting your issue!

As initial debugging steps, have you tried the advice of the output?

minikube delete --all

Based on the logs, you're restarting an existing minikube instance (Using the hyperv driver based on existing profile). Starting with a fresh cluster may resolve your issue, if not, let us know and we can debug further, thanks!

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[spowelljr]

Hi @smtrock, we haven't heard back from you, do you still have this issue?
There isn't enough information in this issue to make it actionable, and a long enough duration has passed, so this issue is likely difficult to replicate.

I will close this issue for now but feel free to reopen when you feel ready to provide more information.