Getting "cluster doesn't provide client-ca-file"

[day0ops]

I attempted at deploying cert-manager 0.5.2 with Helm but running into the following error and the pod for the webhook is in a crashloop.

I get the following error message,

ERROR: logging before flag.Parse: I0116 10:26:12.457905       1 round_trippers.go:386] curl -k -v -XGET  -H "User-Agent: webhook/v0.0.0 (linux/amd64) kubernetes/$Format" -H "Accept: application/json, */*" -H "Authorization: Bearer <not given for security reasons>" https://10.100.0.1:443/api/v1/namespaces/kube-system/configmaps/extension-apiserver-authentication
ERROR: logging before flag.Parse: I0116 10:26:12.672853       1 round_trippers.go:405] GET https://10.100.0.1:443/api/v1/namespaces/kube-system/configmaps/extension-apiserver-authentication 200 OK in 214 milliseconds
ERROR: logging before flag.Parse: I0116 10:26:12.672880       1 round_trippers.go:411] Response Headers:
ERROR: logging before flag.Parse: I0116 10:26:12.672885       1 round_trippers.go:414]     Audit-Id: 90541976-528d-4d36-a88a-aa2d50630c9d
ERROR: logging before flag.Parse: I0116 10:26:12.672891       1 round_trippers.go:414]     Content-Type: application/json
ERROR: logging before flag.Parse: I0116 10:26:12.672896       1 round_trippers.go:414]     Content-Length: 1634
ERROR: logging before flag.Parse: I0116 10:26:12.672900       1 round_trippers.go:414]     Date: Wed, 16 Jan 2019 10:26:12 GMT
ERROR: logging before flag.Parse: I0116 10:26:12.672937       1 request.go:874] Response Body: {"kind":"ConfigMap","apiVersion":"v1","metadata":{"name":"extension-apiserver-authentication","namespace":"kube-system","selfLink":"/api/v1/namespaces/kube-system/configmaps/extension-apiserver-authentication","uid":"ce2b6f64-17b8-11e9-a6dd-021a269d3ce8","resourceVersion":"39","creationTimestamp":"2019-01-14T04:56:51Z"},"data":{"requestheader-allowed-names":"[\"front-proxy-client\"]","requestheader-client-ca-file":"<not given for security reasons>","requestheader-extra-headers-prefix":"[\"X-Remote-Extra-\"]","requestheader-group-headers":"[\"X-Remote-Group\"]","requestheader-username-headers":"[\"X-Remote-User\"]"}}
Error: cluster doesn't provide client-ca-file
Usage:
   [flags]

Flags:
      --admission-control-config-file string                    File with admission control configuration.
      --alsologtostderr                                         log to standard error as well as files
      --audit-log-batch-buffer-size int                         The size of the buffer to store events before batching and writing. Only used in batch mode. (default 10000)
      --audit-log-batch-max-size int                            The maximum size of a batch. Only used in batch mode. (default 400)
      --audit-log-batch-max-wait duration                       The amount of time to wait before force writing the batch that hadn't reached the max size. Only used in batch mode. (default 30s)
      --audit-log-batch-throttle-burst int                      Maximum number of requests sent at the same moment if ThrottleQPS was not utilized before. Only used in batch mode. (default 15)
      --audit-log-batch-throttle-enable                         Whether batching throttling is enabled. Only used in batch mode.
      --audit-log-batch-throttle-qps float32                    Maximum average number of batches per second. Only used in batch mode. (default 10)
      --audit-log-format string                                 Format of saved audits. "legacy" indicates 1-line text format for each event. "json" indicates structured json format. Requires the 'AdvancedAuditing' feature gate. Known formats are legacy,json. (default "json")
      --audit-log-maxage int                                    The maximum number of days to retain old audit log files based on the timestamp encoded in their filename.
      --audit-log-maxbackup int                                 The maximum number of old audit log files to retain.
      --audit-log-maxsize int                                   The maximum size in megabytes of the audit log file before it gets rotated.
      --audit-log-mode string                                   Strategy for sending audit events. Blocking indicates sending events should block server responses. Batch causes the backend to buffer and write events asynchronously. Known modes are batch,blocking. (default "blocking")
      --audit-log-path string                                   If set, all requests coming to the apiserver will be logged to this file.  '-' means standard out.
      --audit-policy-file string                                Path to the file that defines the audit policy configuration. Requires the 'AdvancedAuditing' feature gate. With AdvancedAuditing, a profile is required to enable auditing.
      --audit-webhook-batch-buffer-size int                     The size of the buffer to store events before batching and writing. Only used in batch mode. (default 10000)
      --audit-webhook-batch-max-size int                        The maximum size of a batch. Only used in batch mode. (default 400)
      --audit-webhook-batch-max-wait duration                   The amount of time to wait before force writing the batch that hadn't reached the max size. Only used in batch mode. (default 30s)
      --audit-webhook-batch-throttle-burst int                  Maximum number of requests sent at the same moment if ThrottleQPS was not utilized before. Only used in batch mode. (default 15)
      --audit-webhook-batch-throttle-enable                     Whether batching throttling is enabled. Only used in batch mode. (default true)
      --audit-webhook-batch-throttle-qps float32                Maximum average number of batches per second. Only used in batch mode. (default 10)
      --audit-webhook-config-file string                        Path to a kubeconfig formatted file that defines the audit webhook configuration. Requires the 'AdvancedAuditing' feature gate.
      --audit-webhook-initial-backoff duration                  The amount of time to wait before retrying the first failed request. (default 10s)
      --audit-webhook-mode string                               Strategy for sending audit events. Blocking indicates sending events should block server responses. Batch causes the backend to buffer and write events asynchronously. Known modes are batch,blocking. (default "batch")
      --authentication-kubeconfig string                        kubeconfig file pointing at the 'core' kubernetes server with enough rights to create tokenaccessreviews.authentication.k8s.io.
      --authentication-skip-lookup                              If false, the authentication-kubeconfig will be used to lookup missing authentication configuration from the cluster.
      --authentication-token-webhook-cache-ttl duration         The duration to cache responses from the webhook token authenticator. (default 10s)
      --authorization-kubeconfig string                         kubeconfig file pointing at the 'core' kubernetes server with enough rights to create  subjectaccessreviews.authorization.k8s.io.
      --authorization-webhook-cache-authorized-ttl duration     The duration to cache 'authorized' responses from the webhook authorizer. (default 10s)
      --authorization-webhook-cache-unauthorized-ttl duration   The duration to cache 'unauthorized' responses from the webhook authorizer. (default 10s)
      --bind-address ip                                         The IP address on which to listen for the --secure-port port. The associated interface(s) must be reachable by the rest of the cluster, and by CLI/web clients. If blank, all interfaces will be used (0.0.0.0 for all IPv4 interfaces and :: for all IPv6 interfaces). (default 0.0.0.0)
      --cert-dir string                                         The directory where the TLS certs are located. If --tls-cert-file and --tls-private-key-file are provided, this flag will be ignored. (default "apiserver.local.config/certificates")
      --client-ca-file string                                   If set, any request presenting a client certificate signed by one of the authorities in the client-ca-file is authenticated with an identity corresponding to the CommonName of the client certificate.
      --contention-profiling                                    Enable lock contention profiling, if profiling is enabled
      --disable-admission-plugins stringSlice                   admission plugins that should be disabled although they are in the default enabled plugins list. Comma-delimited list of admission plugins: Initializers, MutatingAdmissionWebhook, NamespaceLifecycle, ValidatingAdmissionWebhook. The order of plugins in this flag does not matter.
      --enable-admission-plugins stringSlice                    admission plugins that should be enabled in addition to default enabled ones. Comma-delimited list of admission plugins: Initializers, MutatingAdmissionWebhook, NamespaceLifecycle, ValidatingAdmissionWebhook. The order of plugins in this flag does not matter.
      --enable-swagger-ui                                       Enables swagger ui on the apiserver at /swagger-ui
  -h, --help                                                    help for this command
      --http2-max-streams-per-connection int                    The limit that the server gives to clients for the maximum number of streams in an HTTP/2 connection. Zero means to use golang's default. (default 1000)
      --kubeconfig string                                       kubeconfig file pointing at the 'core' kubernetes server.
      --log-flush-frequency duration                            Maximum number of seconds between log flushes (default 5s)
      --log_backtrace_at traceLocation                          when logging hits line file:N, emit a stack trace (default :0)
      --log_dir string                                          If non-empty, write log files in this directory
      --logtostderr                                             log to standard error instead of files (default true)
      --profiling                                               Enable profiling via web interface host:port/debug/pprof/ (default true)
      --requestheader-allowed-names stringSlice                 List of client certificate common names to allow to provide usernames in headers specified by --requestheader-username-headers. If empty, any client certificate validated by the authorities in --requestheader-client-ca-file is allowed.
      --requestheader-client-ca-file string                     Root certificate bundle to use to verify client certificates on incoming requests before trusting usernames in headers specified by --requestheader-username-headers
      --requestheader-extra-headers-prefix stringSlice          List of request header prefixes to inspect. X-Remote-Extra- is suggested. (default [x-remote-extra-])
      --requestheader-group-headers stringSlice                 List of request headers to inspect for groups. X-Remote-Group is suggested. (default [x-remote-group])
      --requestheader-username-headers stringSlice              List of request headers to inspect for usernames. X-Remote-User is common. (default [x-remote-user])
      --secure-port int                                         The port on which to serve HTTPS with authentication and authorization. If 0, don't serve HTTPS at all. (default 443)
      --stderrthreshold severity                                logs at or above this threshold go to stderr (default 2)
      --tls-cert-file string                                    File containing the default x509 Certificate for HTTPS. (CA cert, if any, concatenated after server cert). If HTTPS serving is enabled, and --tls-cert-file and --tls-private-key-file are not provided, a self-signed certificate and key are generated for the public address and saved to the directory specified by --cert-dir.
      --tls-cipher-suites stringSlice                           Comma-separated list of cipher suites for the server. Values are from tls package constants (https://golang.org/pkg/crypto/tls/#pkg-constants). If omitted, the default Go cipher suites will be used
      --tls-min-version string                                  Minimum TLS version supported. Value must match version names from https://golang.org/pkg/crypto/tls/#pkg-constants.
      --tls-private-key-file string                             File containing the default x509 private key matching --tls-cert-file.
      --tls-sni-cert-key namedCertKey                           A pair of x509 certificate and private key file paths, optionally suffixed with a list of domain patterns which are fully qualified domain names, possibly with prefixed wildcard segments. If no domain patterns are provided, the names of the certificate are extracted. Non-wildcard matches trump over wildcard matches, explicit domain patterns trump over extracted names. For multiple key/certificate pairs, use the --tls-sni-cert-key multiple times. Examples: "example.crt,example.key" or "foo.crt,foo.key:*.foo.com,foo.com". (default [])
  -v, --v Level                                                 log level for V logs (default 0)
      --vmodule moduleSpec                                      comma-separated list of pattern=N settings for file-filtered logging

ERROR: logging before flag.Parse: F0116 10:26:12.756308       1 cmd.go:42] cluster doesn't provide client-ca-file

Looking at extension-apiserver-authentication I can see that client-ca-file is missing.

kubectl get configmap extension-apiserver-authentication -n kube-system -o yaml

apiVersion: v1
data:
  requestheader-allowed-names: '["front-proxy-client"]'
  requestheader-client-ca-file: |
    <not given for security reasons>
  requestheader-extra-headers-prefix: '["X-Remote-Extra-"]'
  requestheader-group-headers: '["X-Remote-Group"]'
  requestheader-username-headers: '["X-Remote-User"]'
kind: ConfigMap
metadata:
  creationTimestamp: 2019-01-14T04:56:51Z
  name: extension-apiserver-authentication
  namespace: kube-system
  resourceVersion: "39"
  selfLink: /api/v1/namespaces/kube-system/configmaps/extension-apiserver-authentication
  uid: ce2b6f64-17b8-11e9-a6dd-021a269d3ce8

Any idea what could be causing this ? I have not tried reaching out to AWS yet.

Environment details::

• Kubernetes version (e.g. v1.10.2): 1.11.5
• Cloud-provider/provisioner (e.g. GKE, kops AWS, etc): AWS EKS
• cert-manager version (e.g. v0.4.0): 0.5.2
• Install method (e.g. helm or static manifests): Helm

/kind bug

[munnerz]

This sounds like EKS is not configuring one of the core Kubernetes CAs properly - I've heard reports of this in the past with EKS, but I also thought they'd been resolved in newer versions.

Are there any upgrades available for EKS that you've not applied yet?

[day0ops]

@munnerz Nope no upgrades what so ever.
Id like to understand how cert-manager uses the k8 client authentication and in tern client-ca-file. Is there any documentation around this ?

[munnerz]

You can see more info here: https://cert-manager.readthedocs.io/en/master/admin/resource-validation-webhook.html

Basically, we install a ValidatingWebhookConfiguration resource in order to provide resource validation for your API types. This requires your Kubernetes cluster to have properly configure API aggregation certificates (which as I understand it is now part of the conformance test suite as well)

[munnerz]

You can see some more info on configuring this here: https://kubernetes.io/docs/tasks/access-kubernetes-api/configure-aggregation-layer/

[day0ops]

Spoke to AWS customer support about this. This isnt currently possible on the platform. They said theyl take it up as a feature request. So i guess on EKS, validation isnt possible.

[gordcorp]

I am getting the same error with EKS with k8s 1.11, and I believe EKS has supported this since October 2018 - and it says it supports ValidatingWebhookConfiguration at
https://docs.aws.amazon.com/eks/latest/userguide/platform-versions.html

[gordcorp]

I did some digging, and noticed there was a feature request in apiserver to address this issue: kubernetes/kubernetes#65724
cert-manager is using apiserver 1.10.0, and the PR which added the feature was cherry-picked into 1.10.7, so I tried it by bumping cert- apiserver to 1.10.7 and it worked for me on EKS.
I was going to put in a PR, but I saw there is already one to bump all the k8s dependencies to 1.13 which would include the fix for this issue. Since that's already well in progress, I wont bother with my PR, but let me know if it's still needed.

[munnerz]

@gordcorp nice find! Thanks for digging into this for us.

Hopefully the 1.13 apimachinery PR will land soon, but if that takes a while it may be worth us putting in a patch to bump within the 1.10 series and cherry picking this into v0.6 😄

/reopen
/priority important-soon
/help

[jetstack-bot]

@munnerz: Reopened this issue.

In response to this:

@gordcorp nice find! Thanks for digging into this for us.

Hopefully the 1.13 apimachinery PR will land soon, but if that takes a while it may be worth us putting in a patch to bump within the 1.10 series and cherry picking this into v0.6 😄

/reopen
/priority important-soon
/help

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[munnerz]

I've opened #1344 to bump the 1.10 dependencies 😄

[MilanDasek]

@gordcorp

Can I ask you for more details how you "bumped cert-apiserver to 1.10.7" ?
Is there a way how to do this in helm values?

Thank you

M

[gordcorp]

@MilanDasek No not just with helm values alone - I bumped apiserver in Gopkg.toml, rebuilt the binary, published new docker images, and then set the helm values to use the new image:

...
  --set image.repository=docker.io/govau/cert-manager-controller \	
  --set image.tag=v0.6.0 \	
  --set webhook.image.repository=docker.io/govau/cert-manager-webhook \	
  --set webhook.image.tag=v0.6.0	\
...

[day0ops]

@MilanDasek In case you are wondering version 0.6.2 has already been released with v1.10.12 k8 deps so you no longer need to manually build it.

[MilanDasek]

@nixgadget

but helm installs still 0.6.0

and --version 0.6.2 gives me No chart version found for cert-manager-0.6.2

[gordcorp]

@MilanDasek try chart version v0.6.6 - gives you cert-manager 0.6.2
https://github.com/helm/charts/blob/master/stable/cert-manager/Chart.yaml