Kubernetes system pods are not showing up but containers are in health state

[iponnam]

Is this a BUG REPORT or FEATURE REQUEST?:

Uncomment only one, leave it on its own line:

/kind support

/kind feature

What happened:
Enabled --client-cert-auth=true in etcd yaml to communicate over TLS and restarted the kubelet for the changes to effect. kube-apiserver, controller, scheduler and etd containers along with pause container came up. however when we do kubectl -n kube-system get pods or anything else for that matter results are coming up empty and worker nodes that already joined no longer show up.

kubectl get pods -n kube-system
No resources found.

When reverted etcd yaml without tls settings, everything back online.

kubelet keeps on throwing errors for all 4 components:

pods "kube-controller-manager-azwushubqaadmmaster01" is forbidden: no providers available to validate pod request

What you expected to happen:
The nodes registered before enabling TLS on ETCD should show.

How to reproduce it (as minimally and precisely as possible):
Kubeadm init on master and nodes.
create certs for etcd server and client. Followed document
Apply the changes are mentioned in the yaml files attached.
Attached etcd.yaml & kube-apiserver.yaml
tls.zip

Anything else we need to know?:

Environment:

• Kubernetes version (use kubectl version): 1.9.5

• Cloud provider or hardware configuration:

• OS (e.g. from /etc/os-release): Ubuntu

• Kernel (e.g. uname -a): 4.13.0-1012-azure Port forwarding should be through iptables #15-Ubuntu SMP Thu Mar 8 10:47:27 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

• Install tools:

• Others:
  @kubernetes/sig-auth-bug

[iponnam]

@kubernetes/sig-auth-bugs

[k8s-ci-robot]

@iponnam: Reiterating the mentions to trigger a notification:
@kubernetes/sig-auth-bugs

Details

In response to this:

@kubernetes/sig-auth-bugs

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[liggitt]

it sounds like the etcd client credentials given to the apiserver are not correct

can you capture the apiserver logs (from the very beginning of startup with etcd client credentials) and the output of these URLs:

https://<apiserver>:6443/healthz
https://<apiserver>:6443/healthz/etcd

[iponnam]

Good Morning @liggitt

Thank you for quick reply.
Attached logs from api-server & etcd logs.
out put of healthz & healthz/etcd are also included in the zip.
k8s-logs.zip

[iponnam]

When passed the certs to curl, healthz passed

curl -v -k --cacert ca.crt --key etcd-client.key --cert etcd-client.crt https://127.0.0.1:6443/healthz  *   Trying 127.0.0.1...
* Connected to 127.0.0.1 (127.0.0.1) port 6443 (#0)
* found 1 certificates in ca.crt
* found 594 certificates in /etc/ssl/certs
* ALPN, offering http/1.1
* SSL connection using TLS1.2 / ECDHE_RSA_AES_128_GCM_SHA256
*        server certificate verification SKIPPED
*        server certificate status verification SKIPPED
*        common name: kube-apiserver (does not match '127.0.0.1')
*        server certificate expiration date OK
*        server certificate activation date OK
*        certificate public key: RSA
*        certificate version: #3
*        subject: CN=kube-apiserver
*        start date: Fri, 30 Mar 2018 19:33:05 GMT
*        expire date: Sat, 30 Mar 2019 19:33:06 GMT
*        issuer: CN=kubernetes
*        compression: NULL
* ALPN, server accepted to use http/1.1
> GET /healthz HTTP/1.1
> Host: 127.0.0.1:6443
> User-Agent: curl/7.47.0
> Accept: */*
>
< HTTP/1.1 200 OK
< Date: Sat, 14 Apr 2018 19:38:21 GMT
< Content-Length: 2
< Content-Type: text/plain; charset=utf-8
<
* Connection #0 to host 127.0.0.1 left intact

healthz/etcd - output

 curl -v -k --cacert ca.crt --key etcd-client.key --cert etcd-client.crt https://127.0.0.1:6443/healthz/etcd
*   Trying 127.0.0.1...
* Connected to 127.0.0.1 (127.0.0.1) port 6443 (#0)
* found 1 certificates in ca.crt
* found 594 certificates in /etc/ssl/certs
* ALPN, offering http/1.1
* SSL connection using TLS1.2 / ECDHE_RSA_AES_128_GCM_SHA256
*        server certificate verification SKIPPED
*        server certificate status verification SKIPPED
*        common name: kube-apiserver (does not match '127.0.0.1')
*        server certificate expiration date OK
*        server certificate activation date OK
*        certificate public key: RSA
*        certificate version: #3
*        subject: CN=kube-apiserver
*        start date: Fri, 30 Mar 2018 19:33:05 GMT
*        expire date: Sat, 30 Mar 2019 19:33:06 GMT
*        issuer: CN=kubernetes
*        compression: NULL
* ALPN, server accepted to use http/1.1
> GET /healthzz//etcd HTTP/1.1
> Host: 127.0.0.1:6443
> User-Agent: curl/7.47.0
> Accept: */*
>
< HTTP/1.1 403 Forbidden
< Content-Type: application/json
< X-Content-Type-Options: nosniff
< Date: Sat, 14 Apr 2018 19:38:43 GMT
< Content-Length: 247
<
{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {

  },
  "status": "Failure",
  "message": "forbidden: User \"system:anonymous\" cannot get path \"/healthzz//etcd\"",
  "reason": "Forbidden",
  "details": {

  },
  "code": 403
* Connection #0 to host 127.0.0.1 left intact

[liggitt]

When reverted etcd yaml without tls settings, everything back online.

Do you have the API server yaml for the reverted mode that is working? Was the etcd TLS settings the only difference between the two setups? Based on the logs, it looks like you enabled the PodSecurityPolicy admission plugin without creating a privileged policy and granting access to the kubelet (which it needs to report on the static apiserver pod)

As an example, this is the policy GCE sets up to give the kubelet access:

• Policy
• Role
• Binding

[iponnam]

Yes, etcd TLS is the only setting different.
etcd-yaml-without-tls.txt

kube-apiserver-yaml-without-tls.txt