New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We鈥檒l occasionally send you account related emails.

Already on GitHub? Sign in to your account

v1.10: make root mounted as rshared to fix kube-addon and kube-dns issues #8

Merged
merged 1 commit into from Jul 2, 2018

Conversation

Projects
None yet
2 participants
@xmudrii
Collaborator

xmudrii commented Jul 2, 2018

Hi there! 馃憢

This PR fixes the problems with kube-addon-manager and kube-dns by mounting the root partition as rshared. This issues is present with Kubernetes v1.10, while for older versions everything works flawlessly.

The problem I had is that all basic Kubernetes functionalities worked perfectly (such as pod creating and scheduling, etc), but I was not able to resolve Services via URLs. For example, if you would try to fetch something from http://service-name.namespace.svc, it would return error. Also, nslookup would fail for the Service URLs.

Executing kubectl get all --all-namespaces returns:

NAMESPACE   NAME                    DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
default     deploy/travis-example   1         1         1            1           1m

NAMESPACE   NAME                           DESIRED   CURRENT   READY     AGE
default     rs/travis-example-7769d67684   1         1         1         1m

NAMESPACE     NAME                                                                    READY     STATUS             RESTARTS   AGE
default       po/travis-example-7769d67684-rbwtt                                      1/1       Running            0          1m
kube-system   po/kube-addon-manager-travis-job-fab2cc62-8ddb-45e1-82b4-17837946f72c   0/1       CrashLoopBackOff   3          1m

NAMESPACE   NAME             TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
default     svc/kubernetes   ClusterIP   10.96.0.1    <none>        443/TCP   1m

For some reason, kube-addon-manage is looping in CrashLoopBackOff and there're no kube-dns pods. Describing the kube-addon-manager pod mentions the mount issue:

Containers:
  kube-addon-manager:
    Container ID:   docker://6328fc593a1184d31fc5d3cf71342232a4019de1cb6ad9c045c4ee20daa93b37
    Image:          k8s.gcr.io/kube-addon-manager:v8.6
    Image ID:       docker-pullable://k8s.gcr.io/kube-addon-manager@sha256:3519273916ba45cfc9b318448d4629819cb5fbccbb0822cce054dd8c1f68cb60
    Port:           <none>
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       ContainerCannotRun
      Message:      linux mounts: path /etc/kubernetes is mounted on / but it is not a shared or slave mount
      Exit Code:    128
      Started:      Mon, 02 Jul 2018 12:58:20 +0000
      Finished:     Mon, 02 Jul 2018 12:58:20 +0000
    Ready:          False
    Restart Count:  4
    Requests:
      cpu:     5m
      memory:  50Mi
    Environment:
      KUBECONFIG:  /var/lib/localkube/kubeconfig
    Mounts:
      /etc/kubernetes/ from addons (ro)
      /var/lib/localkube/ from kubeconfig (ro)

I think this could be related to kubernetes/kubernetes#61058. The local-up-cluster.sh script mounts root as rshared when running in CI, in order to fix some DIND problems.

I've tried to use that for Minikube setup, and that seems to fix the described problems. I also tried to test this with my E2E tests, and everything works perfectly and Services are resolvable.

@LiliC

LiliC approved these changes Jul 2, 2018

Thanks for the fix!

I should really add a test in the future to test that all the k8s components work correctly, to prevent these problems in the future.

@LiliC LiliC merged commit 79b8f3c into LiliC:minikube-26-kube-1.10 Jul 2, 2018

1 check passed

continuous-integration/travis-ci/pr The Travis CI build passed
Details
@xmudrii

This comment has been minimized.

Collaborator

xmudrii commented Jul 2, 2018

@LiliC No problem! If you want I could give it a try to add few more tests cases. Could be a nice Kube practice. 馃槃

@LiliC

This comment has been minimized.

Owner

LiliC commented Jul 2, 2018

@xmudrii Hvala! :)

@xmudrii xmudrii deleted the xmudrii:1.10-rshared-root branch Jul 3, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment