DNS resolve issue

[vincent-pli]

I don't know, seems I hit a long existing issue of alpine:
kubernetes/kubernetes#30215

I installed microk8s from snap:
microk8s v1.14.1 522 stable canonical✓ classic

I checked the pod of microk8s' dns, it's:

NAME="Alpine Linux"
ID=alpine
VERSION_ID=3.6.2
PRETTY_NAME="Alpine Linux v3.6"
HOME_URL="http://alpinelinux.org"
BUG_REPORT_URL="http://bugs.alpinelinux.org"

Also, I have another cluster not mircok8s, it's kube-dns is also based on alpine, but:
VERSION_ID=3.8.2
No such issue.
So could anyone give suggestion, thanks a a lot

[ktsakalozos]

@vincent-pli CoreDNS will soon replace kubedns as the default dns. This is coming in the next 1.15 release (there is already a PR up). In the meantime can you try the fix suggested on this comment: #271 (comment) ?

[vincent-pli]

@ktsakalozos
Thanks very much, It works.
I struggled for the issue for 3 days 😭

[vincent-pli]

@ktsakalozos
Seems my pod terminating:

coredns-55f46dd959-mpqmq    0/1     Running       0          14m
coredns-55f46dd959-pfz7l    0/1     Running       0          14m
kube-dns-6bfbdd666c-4cm8j   3/3     Terminating   0          14m

When I describe kube-dns-6bfbdd666c-4cm8j I get this:

  Warning  FailedKillPod  8m4s  kubelet, monacid1.fyre.ibm.com  error killing pod: [failed to "KillContainer" for "sidecar" with KillContainerError: "rpc error: code = Unknown desc = failed to stop contai
ner \"99dcde7d66ef313013523bcfbdf96258517f150066d82f06f28cfa65d36d5f34\": unknown error after kill: runc did not terminate sucessfully: container_linux.go:388: signaling init process caused \"permission d
enied\"\n: unknown"
, failed to "KillContainer" for "dnsmasq" with KillContainerError: "rpc error: code = Unknown desc = failed to stop container \"99892aa10e3a3fc936f6afeacba29f6a89807f01925a517861f0ee7334e181ef\": unknown
error after kill: runc did not terminate sucessfully: container_linux.go:388: signaling init process caused \"permission denied\"\n: unknown"
, failed to "KillContainer" for "kubedns" with KillContainerError: "rpc error: code = Unknown desc = failed to stop container \"b7ee15c42c231c22d6b107be96804c4f025f6c8efbc35a68af95fec78b895a27\": unknown
error after kill: runc did not terminate sucessfully: container_linux.go:388: signaling init process caused \"permission denied\"\n: unknown"
, failed to "KillPodSandbox" for "ecc995c7-85cd-11e9-a9d7-00163e01e6ab" with KillPodSandboxError: "rpc error: code = Unknown desc = failed to stop container \"b7ee15c42c231c22d6b107be96804c4f025f6c8efbc35
a68af95fec78b895a27\": failed to kill container \"b7ee15c42c231c22d6b107be96804c4f025f6c8efbc35a68af95fec78b895a27\": unknown error after kill: runc did not terminate sucessfully: container_linux.go:388:
signaling init process caused \"permission denied\"\n: unknown"
]
  Warning  FailedKillPod  7m50s  kubelet, monacid1.fyre.ibm.com  error killing pod: [failed to "KillContainer" for "dnsmasq" with KillContainerError: "rpc error: code = Unknown desc = failed to stop conta
iner \"99892aa10e3a3fc936f6afeacba29f6a89807f01925a517861f0ee7334e181ef\": unknown error after kill: runc did not terminate sucessfully: container_linux.go:388: signaling init process caused \"permission
denied\"\n: unknown"
, failed to "KillContainer" for "sidecar" with KillContainerError: "rpc error: code = Unknown desc = failed to stop container \"99dcde7d66ef313013523bcfbdf96258517f150066d82f06f28cfa65d36d5f34\": unknown
error after kill: runc did not terminate sucessfully: container_linux.go:388: signaling init process caused \"permission denied\"\n: unknown"
, failed to "KillContainer" for "kubedns" with KillContainerError: "rpc error: code = Unknown desc = failed to stop container \"b7ee15c42c231c22d6b107be96804c4f025f6c8efbc35a68af95fec78b895a27\": unknown
error after kill: runc did not terminate sucessfully: container_linux.go:388: signaling init process caused \"permission denied\"\n: unknown"
, failed to "KillPodSandbox" for "ecc995c7-85cd-11e9-a9d7-00163e01e6ab" with KillPodSandboxError: "rpc error: code = Unknown desc = failed to stop container \"b7ee15c42c231c22d6b107be96804c4f025f6c8efbc35
a68af95fec78b895a27\": failed to kill container \"b7ee15c42c231c22d6b107be96804c4f025f6c8efbc35a68af95fec78b895a27\": unknown error after kill: runc did not terminate sucessfully: container_linux.go:388:
signaling init process caused \"permission denied\"\n: unknown"
]
  Warning  FailedKillPod  7m36s  kubelet, monacid1.fyre.ibm.com  error killing pod: [failed to "KillContainer" for "sidecar" with KillContainerError: "rpc error: code = Unknown desc = failed to stop conta
iner \"99dcde7d66ef313013523bcfbdf96258517f150066d82f06f28cfa65d36d5f34\": unknown error after kill: runc did not terminate sucessfully: container_linux.go:388: signaling init process caused \"permission
denied\"\n: unknown"

And, it's not work again 😢

[ktsakalozos]

What distribution are you on?

Can you attach the tarball produced by microk8s.inspect?

Can you describe in detail how I can reproduce the error you are seeing in a clean system?

It is strange that the containers of the coredns pods are not running either. Do you see and errors/denials in dmesg. Anything interesting in the containerd logs? journalctl -u snap.microk8s.daemon-containerd.service

[vincent-pli]

inspection-report-20190603_003619.tar.gz

My cluster is almost a clean one, only installed tekton, I follow this:

microk8s.disable dns
git clone https://github.com/coredns/deployment.git
cd deployment/kubernetes
./deploy.sh | kubectl apply -f -

it tells me the dns service is not existed, so I use -i to set a IP
it works at the very first, then failed.

[ktsakalozos]

If you do a dmesg you will see a number of AppArmor denials. We apply the right aa profile in https://github.com/ubuntu/microk8s/blob/master/microk8s-resources/wrappers/run-containerd-with-args . Could try microk8s.stop and microk8s.start so the profile gets reapplied?

Can you think of any reason why this profile misalignment would happen? Anything special about that machine?

[vincent-pli]

Stop - Start resolve the issue, thanks.

[ktsakalozos]

Good to hear. I would very much be interested to understand why this happened so we can guard against such cases.