No network access from second node (multiple nodes cluster)

[mrbenosborne]

Steps to reproduce the issue:

Start Minikube with 2 nodes

# Start minikube
minikube start --driver=docker --nodes 2 -p minikube
kubectl label nodes minikube nodeType=standard
kubectl label nodes minikube-m02 nodeType=high-mem

Apply a deployment and service

kubectl apply -f ./echo-server.yaml

echo-server.yaml

The deployment echoserver is set to be placed on the node that has a matching label of nodeType=standard, the deployment debug is set to be placed on the label matching nodeType=high-mem.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: echoserver
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: echoserver
    spec:
      containers:
      - image: k8s.gcr.io/echoserver:1.4
        imagePullPolicy: IfNotPresent
        name: echoserver
        ports:
         - containerPort: 8080
      nodeSelector:
        nodeType: standard
  selector:
    matchLabels:
      app: echoserver
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: debug
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: debug
    spec:
      containers:
      - image: k8s.gcr.io/echoserver:1.4
        imagePullPolicy: IfNotPresent
        name: echoserver
        ports:
         - containerPort: 8080
      nodeSelector:
        nodeType: high-mem
  selector:
    matchLabels:
      app: debug

Get pods

kubectl get pods

NAME                          READY   STATUS    RESTARTS   AGE
debug-86cf8b5b6b-qtlzm        1/1     Running   0          10m
echoserver-6fb448987b-jpcwx   1/1     Running   0          21m

Issue

When I exec into the echoserver pod and run:

apt-get update

the command runs successfully, however if I run the same command on debug pod which is on node 2 then it fails to resolve.

Is there something I need to enable on minikube or the node 2 to enable outbound network access?

[priyawadhwa]

Hey @mrbenosborne thanks for opening this issue. I'm not very familiar with mulitnode, but I think we have merged some network related fixes in the past few weeks. Would you mind upgrading to minikube v1.13.0 and see if that fixes the issue?

[mrbenosborne]

Hi @priyawadhwa .

I updated minikube but I am still hitting the same issue:

[medyagh]

we will need to investigate this. CC: @sharifelgamal

[rejohnst]

FWIW, I still see this issue on Minikube 1.16.0 (Linux, kvm2 driver). Please let me know if there's any information I can provide to help debug this.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle stale

[rejohnst]

I retested this on Minikube 1.21.0 (Linux, kvm2 driver) and the issue seems to be fixed. I created a 4 node cluster and all nodes had network access.

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle rotten

[spowelljr]

Like @rejohnst said, this seems to resolved. I tried the listed steps on v1.22.0 and the debug pod had network access, closing issue.