Using Multi-Node Clusters - tutorial

[pguerin3]

What Happened?

I'm following this tutorial but can't get the last part to work:

https://minikube.sigs.k8s.io/docs/tutorials/multi_node/

At the final stage I enter :

minikube service list -p multinode-demo

then the output is missing the URL:

|-------------|------------|--------------|-----|
|  NAMESPACE  |    NAME    | TARGET PORT  | URL |
|-------------|------------|--------------|-----|
| default     | hello      |           80 |     |
| default     | kubernetes | No node port |     |
| kube-system | kube-dns   | No node port |     |
|-------------|------------|--------------|-----|

From previous issues there is a suggestion that the problem has something to do with the lack of CNI.
But I have tried the following suggestion: https://docs.tigera.io/calico/latest/getting-started/kubernetes/minikube
So the tutorial start would look like this:

minikube start --nodes 2 -p multinode-demo --network-plugin=cni --cni=calico

But this does not solve the issue.

Note: I'm using rootless with a driver=docker and container-runtime=containerd in Fedora 36.

Attach the log file

file:///home/me/me/log.txt

Operating System

Redhat/Fedora

Driver

Docker

[termdew]

Could you please try it again without Calico and provide the logs from CoreDNS pod? 🙂

There might be an issue, where CoreDNS cannot find some plugins for containerd.
It expects them in /opt/cni/bin , but they are actually stored in /usr/libexec/cni.
See this issue here: https://bugzilla.redhat.com/show_bug.cgi?id=1731597
(Maybe the workaround in the last comment fixes your issue 🙂 )

[pguerin3]

file:///home/me/log.txt

[pguerin3]

My Fedora 36 doesn't have an /opt/cni/bin nor an /usr/libexec/cni

[pguerin3]

─>$ kubectl -n kube-system describe pod coredns

Name:                 coredns-787d4945fb-qlqx7
Namespace:            kube-system
Priority:             2000000000
Priority Class Name:  system-cluster-critical
Service Account:      coredns
Node:                 multinode-demo/192.168.49.2
Start Time:           Sun, 09 Apr 2023 23:27:29 +1000
Labels:               k8s-app=kube-dns
                      pod-template-hash=787d4945fb
Annotations:          <none>
Status:               Running
IP:                   10.244.0.2
IPs:
  IP:           10.244.0.2
Controlled By:  ReplicaSet/coredns-787d4945fb
Containers:
  coredns:
    Container ID:  containerd://07f32d630ac96388bd46c11710bb312d797d1812de5915750330f33b4ab67394
    Image:         registry.k8s.io/coredns/coredns:v1.9.3
    Image ID:      sha256:5185b96f0becf59032b8e3646e99f84d9655dff3ac9e2605e0dc77f9c441ae4a
    Ports:         53/UDP, 53/TCP, 9153/TCP
    Host Ports:    0/UDP, 0/TCP, 0/TCP
    Args:
      -conf
      /etc/coredns/Corefile
    State:          Running
      Started:      Sun, 09 Apr 2023 23:28:09 +1000
    Ready:          True
    Restart Count:  0
    Limits:
      memory:  170Mi
    Requests:
      cpu:        100m
      memory:     70Mi
    Liveness:     http-get http://:8080/health delay=60s timeout=5s period=10s #success=1 #failure=5
    Readiness:    http-get http://:8181/ready delay=0s timeout=1s period=10s #success=1 #failure=3
    Environment:  <none>
    Mounts:
      /etc/coredns from config-volume (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-np8zk (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             True 
  ContainersReady   True 
  PodScheduled      True 
Volumes:
  config-volume:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      coredns
    Optional:  false
  kube-api-access-np8zk:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              kubernetes.io/os=linux
Tolerations:                 CriticalAddonsOnly op=Exists
                             node-role.kubernetes.io/control-plane:NoSchedule
                             node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason                  Age                From               Message
  ----     ------                  ----               ----               -------
  Normal   Scheduled               20m                default-scheduler  Successfully assigned kube-system/coredns-787d4945fb-qlqx7 to multinode-demo
  Warning  FailedCreatePodSandBox  20m                kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "84cb1eafad2077028149726a38e7780f71c5709014afeec8e5d730cee41eb17f": failed to find network info for sandbox "84cb1eafad2077028149726a38e7780f71c5709014afeec8e5d730cee41eb17f"
  Warning  FailedCreatePodSandBox  19m                kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "259344ae0ff6eae0a110365aa20b90edf3b4b5138a2d1e93eb36a8946d5464ae": failed to find network info for sandbox "259344ae0ff6eae0a110365aa20b90edf3b4b5138a2d1e93eb36a8946d5464ae"
  Warning  FailedCreatePodSandBox  19m                kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "882f8ad0291807a788d3c5f85200528a9281e92ce1a9c827314f8089be7ac89a": failed to find network info for sandbox "882f8ad0291807a788d3c5f85200528a9281e92ce1a9c827314f8089be7ac89a"
  Normal   Pulled                  19m                kubelet            Container image "registry.k8s.io/coredns/coredns:v1.9.3" already present on machine
  Normal   Created                 19m                kubelet            Created container coredns
  Normal   Started                 19m                kubelet            Started container coredns
  Warning  Unhealthy               19m (x3 over 19m)  kubelet            Readiness probe failed: Get "http://10.244.0.2:8181/ready": dial tcp 10.244.0.2:8181: connect: connection refused

[pguerin3]

Also note that I installed Minikube from the website, and not from the Fedora 36 repository (via DNF).

[termdew]

Good morning @pguerin3,

I set up a test machine with Fedora 36 and installed everything like it is in your setup.
Somehow for my machine it works.

The only difference I can see in your logs is the line: Readiness probe failed: Get "http://10.244.0.2:8181/ready": dial tcp 10.244.0.2:8181: connect: connection refused

Unfortunately, I cannot open the logs that you provided. I only can see file:///home/me/log.txt.

Could you maybe please copy and paste the content of the log file in the comment box here? 🙂

Thank you in advance 🙂

[pguerin3]

log.txt

[termdew]

I tried it again, this time with actual rootless docker context. 😉

Turns out that the minikube service list command is somehow misbehaving.
If you run minikube service --all -p multinode-demo, it shows the URL and creates a tunnel accordingly.

I'll debug it later today 🙂

[pguerin3]

ok - I can see the URL now.....

╰─>$ minikube service --all -p multinode-demo
|-----------|-------|-------------|---------------------------|
| NAMESPACE | NAME  | TARGET PORT |            URL            |
|-----------|-------|-------------|---------------------------|
| default   | hello |          80 | http://192.168.49.2:31000 |
|-----------|-------|-------------|---------------------------|
|-----------|------------|-------------|--------------|
| NAMESPACE |    NAME    | TARGET PORT |     URL      |
|-----------|------------|-------------|--------------|
| default   | kubernetes |             | No node port |
|-----------|------------|-------------|--------------|
😿  service default/kubernetes has no node port
🏃  Starting tunnel for service hello.
🏃  Starting tunnel for service kubernetes.
|-----------|------------|-------------|------------------------|
| NAMESPACE |    NAME    | TARGET PORT |          URL           |
|-----------|------------|-------------|------------------------|
| default   | hello      |             | http://127.0.0.1:34231 |
| default   | kubernetes |             | http://127.0.0.1:43891 |
|-----------|------------|-------------|------------------------|
🎉  Opening service default/hello in default browser...

[termdew]

HI @pguerin3,

I finally had some time to look a bit further into it.

TL;DR: The last step from the multi-node tutorial doesn't work for rootless containers, because the networking is different.

Let's begin with the normal docker mode:
When you create the Minikube cluster with non-rootless containers, you'll get a bridge network interface. (You can verify this with docker network ls. You should then see a network with the Minikube profile name.)
As it's a bridge network, you can actually access the Service URL from your machine.

Now for rootless docker:
When you are using the Minikube cluster with rootless containers, it has no permissions to create such a bridge network.
Therefore the serviceURL cannot be accessed from your machine and you would need a port forward in order to access it.

When you run minikube service list, it collects all service URLs. But then it removes out every inaccessible URL from the table. That's why you can't see it. (https://github.com/kubernetes/minikube/blob/master/cmd/minikube/cmd/service_list.go#L54)

When you run minikube service --all, it displays the inaccessible URL AND creates a port forward to 127.0.0.1.

I hope I could explain it in an understandable way. 🙂

Maybe we could give the user a hint that there might be a port forward needed instead of having no URL at all.

[termdew]

ping @pguerin3 🙂

I just wanted to ask if you have some additional questions or if the explanation was clear 🙂

[pguerin3]

Your explanation needs to be clear in the Using Multi-Node Clusters tutorial.
That way users of rootless Docker will know up front why the tutorial isn't getting the expected results.

The lack of a bridged network could also be made clearer in the minikube notes on rootless Docker.

If there are other unexpected behaviours from rootless Docker then it should be documented somewhere in the minikube doco.

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale