[EKS] [Aggregation Layer]: Configure Aggregation Layer ('client-ca-file' ConfigMap missing)

[whereisaaron]

Tell us about your request
Support webhooks for the API Aggregation Layer by issue a creating the client-ca-file ConfigMap.

How to configure the Aggregation Layer:
https://kubernetes.io/docs/tasks/access-kubernetes-api/configure-aggregation-layer/

See --requestheader-client-ca-file in https://kubernetes.io/docs/reference/command-line-tools-reference/kube-apiserver/ (this is current version reference but this functionality has been available since at least v1.8.x)

Which service(s) is this request for?
EKS (k8s v1.11.5)

Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?

Trying to install cert-manager with webhook CRD validation. Installation fails because EKS appears not to be k8s conformant in this regard.

Trying to install Prometheus on EKS but again Aggregation Layer has not been properly configured in the EKS control plane.

• AWS EKS: cluster doesn't provide client-ca-file kubernetes-sigs/prometheus-adapter#119
• AWS EKS - Error: cluster doesn't provide client-ca-file kubernetes-sigs/prometheus-adapter#108

Are you currently working around this issue?
Disabling Aggregation Layer functionality where possible.

Additional context
The Aggregation Layer with webhooks has been available since k8s v1.8.x or earlier.
This works out of the box on kube-aws and GKE clusters.

[christopherhein]

@whereisaaron support for Aggregation and Admission controllers was added in September - https://aws.amazon.com/about-aws/whats-new/2018/10/amazon-eks-enables-support-for-kubernetes-dynamic-admission-cont/

The caveat is projects need to be running at least the 1.10+ client-go libraries which added support for the exec credential authentication flow.

Looks like k8s prometheus-adapter should have added support with - kubernetes-sigs/prometheus-adapter#110

Looks like cert-manager was updated by @munnerz cert-manager/cert-manager#1344

Is this still an issue?

[whereisaaron]

Excellent. Thanks for the update and explanation @christopherhein I'll retest with these new versions.

[whereisaaron]

There is no release of cert-manager with this patch at the moment (last release 10 days ago, this patch 4 days ago) though you can use the 'canary' tag to get images with the patch.

[christopherhein]

Awesome, we’re you able to test with the canary release and validate? If so we should close this.

@munnerz do you have an expected stable release date with that patch?

[whereisaaron]

I tested with the canary cert-manager release (ed0c86e), but I still getting the webhook errors on EKS. Though it strangely works fine on my GKE k8s v1.11 clusters. I guess GKE supports both the old and new flow? It looks like cert-manager needs other changes for the new flow.

I0217 03:59:47.536290       1 controller.go:142] issuers controller: syncing item 'cert-manager/cert-manager-webhook-ca'
I0217 03:59:47.536377       1 setup.go:45] Error getting keypair for CA issuer: secret "cert-manager-webhook-ca" not found
I0217 03:59:47.536408       1 sync.go:72] Error initializing issuer: secret "cert-manager-webhook-ca" not found
E0217 03:59:47.536432       1 controller.go:144] issuers controller: Re-queuing item "cert-manager/cert-manager-webhook-ca" due to error processing: secret "cert-manager-webhook-ca" not found

Error from server (InternalError): error when creating "cluster-issuer.yaml": Internal error occurred: failed calling admission webhook "clusterissuers.admission.certmanager.k8s.io": the server is currently unable to handle the request

container "webhook" in pod "cert-manager-webhook-6756b488bb-g65t2" is waiting to start: ContainerCreating
Warning  FailedMount  75s (x11 over 7m27s)  kubelet, ip-172-21-49-10.ap-southeast-2.compute.internal  MountVolume.SetUp failed for volume "certs" : secrets "cert-manager-webhook-webhook-tls" not found

[munnerz]

Hey Aaron - that looks like an error unrelated to API aggregation - we've got a troubleshooting guide that should help walk you through resolving it 😄 https://cert-manager.readthedocs.io/en/latest/getting-started/troubleshooting.html I'll be cutting a new release early next week (hopefully Monday). My vacation aligned badly with that PR being merged, hence the delay 🙈

…

On Sun, 17 Feb 2019 at 04:12, Aaron Roydhouse ***@***.***> wrote: I tested with the canary cert-manager release (ed0c86e <https://quay.io/repository/jetstack/cert-manager-controller?tag=latest&tab=tags>), but I still getting the webhook errors on EKS. Though it strangely works fine on my GKE k8s v1.11 clusters. I guess GKE supports both the old and new flow? It looks like cert-manager needs other changes for the new flow. I0217 03:59:47.536290 1 controller.go:142] issuers controller: syncing item 'cert-manager/cert-manager-webhook-ca' I0217 03:59:47.536377 1 setup.go:45] Error getting keypair for CA issuer: secret "cert-manager-webhook-ca" not found I0217 03:59:47.536408 1 sync.go:72] Error initializing issuer: secret "cert-manager-webhook-ca" not found E0217 03:59:47.536432 1 controller.go:144] issuers controller: Re-queuing item "cert-manager/cert-manager-webhook-ca" due to error processing: secret "cert-manager-webhook-ca" not found Error from server (InternalError): error when creating "cluster-issuer.yaml": Internal error occurred: failed calling admission webhook "clusterissuers.admission.certmanager.k8s.io": the server is currently unable to handle the request container "webhook" in pod "cert-manager-webhook-6756b488bb-g65t2" is waiting to start: ContainerCreating Warning FailedMount 75s (x11 over 7m27s) kubelet, ip-172-21-49-10.ap-southeast-2.compute.internal MountVolume.SetUp failed for volume "certs" : secrets "cert-manager-webhook-webhook-tls" not found — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#152 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAMbP4X8IVngwKHiTI2Dm1J4bGRdONKHks5vONargaJpZM4akkJw> .

[stormmore]

I came across this issue while running Sonobuoy E2E tests as well. Here are the log details:

Apr 29 21:23:48.097: INFO: logs of sample-apiserver-deployment-6846bb97fc-9xg62/sample-apiserver (error: <nil>): Error: cluster doesn't provide client-ca-file
Usage:
   [flags]

Flags:
      --admission-control-config-file string                    File with admission control configuration.
      --alsologtostderr                                         log to standard error as well as files
      --audit-log-batch-buffer-size int                         The size of the buffer to store events before batching and writing. Only used in batch mode. (default 10000)
      --audit-log-batch-max-size int                            The maximum size of a batch. Only used in batch mode. (default 400)
      --audit-log-batch-max-wait duration                       The amount of time to wait before force writing the batch that hadn't reached the max size. Only used in batch mode
      --audit-log-batch-throttle-burst int                      Maximum number of requests sent at the same moment if ThrottleQPS was not utilized before. Only used in batch mode.
      --audit-log-batch-throttle-enable                         Whether batching throttling is enabled. Only used in batch mode.
      --audit-log-batch-throttle-qps float32                    Maximum average number of batches per second. Only used in batch mode. (default 10)
      --audit-log-format string                                 Format of saved audits. "legacy" indicates 1-line text format for each event. "json" indicates structured json form
      --audit-log-maxage int                                    The maximum number of days to retain old audit log files based on the timestamp encoded in their filename.
      --audit-log-maxbackup int                                 The maximum number of old audit log files to retain.
      --audit-log-maxsize int                                   The maximum size in megabytes of the audit log file before it gets rotated.
      --audit-log-mode string                                   Strategy for sending audit events. Blocking indicates sending events should block server responses. Batch causes th
      --audit-log-path string                                   If set, all requests coming to the apiserver will be logged to this file.  '-' means standard out.
      --audit-log-truncate-enabled                              Whether event and batch truncating is enabled.
      --audit-log-truncate-max-batch-size int                   Maximum size of the batch sent to the underlying backend. Actual serialized size can be several hundreds of bytes g
      --audit-log-truncate-max-event-size int                   Maximum size of the audit event sent to the underlying backend. If the size of an event is greater than this number
      --audit-policy-file string                                Path to the file that defines the audit policy configuration. Requires the 'AdvancedAuditing' feature gate. With Ad
      --audit-webhook-batch-buffer-size int                     The size of the buffer to store events before batching and writing. Only used in batch mode. (default 10000)
      --audit-webhook-batch-max-size int                        The maximum size of a batch. Only used in batch mode. (default 400)
      --audit-webhook-batch-max-wait duration                   The amount of time to wait before force writing the batch that hadn't reached the max size. Only used in batch mode
      --audit-webhook-batch-throttle-burst int                  Maximum number of requests sent at the same moment if ThrottleQPS was not utilized before. Only used in batch mode.
      --audit-webhook-batch-throttle-enable                     Whether batching throttling is enabled. Only used in batch mode. (default true)
      --audit-webhook-batch-throttle-qps float32                Maximum average number of batches per second. Only used in batch mode. (default 10)
      --audit-webhook-config-file string                        Path to a kubeconfig formatted file that defines the audit webhook configuration. Requires the 'AdvancedAuditing' f
      --audit-webhook-initial-backoff duration                  The amount of time to wait before retrying the first failed request. (default 10s)
      --audit-webhook-mode string                               Strategy for sending audit events. Blocking indicates sending events should block server responses. Batch causes th
      --audit-webhook-truncate-enabled                          Whether event and batch truncating is enabled.
      --audit-webhook-truncate-max-batch-size int               Maximum size of the batch sent to the underlying backend. Actual serialized size can be several hundreds of bytes g
      --audit-webhook-truncate-max-event-size int               Maximum size of the audit event sent to the underlying backend. If the size of an event is greater than this number
      --authentication-kubeconfig string                        kubeconfig file pointing at the 'core' kubernetes server with enough rights to create tokenaccessreviews.authentica
      --authentication-skip-lookup                              If false, the authentication-kubeconfig will be used to lookup missing authentication configuration from the cluste
      --authentication-token-webhook-cache-ttl duration         The duration to cache responses from the webhook token authenticator. (default 10s)
      --authorization-kubeconfig string                         kubeconfig file pointing at the 'core' kubernetes server with enough rights to create  subjectaccessreviews.authori
      --authorization-webhook-cache-authorized-ttl duration     The duration to cache 'authorized' responses from the webhook authorizer. (default 10s)
      --authorization-webhook-cache-unauthorized-ttl duration   The duration to cache 'unauthorized' responses from the webhook authorizer. (default 10s)
      --bind-address ip                                         The IP address on which to listen for the --secure-port port. The associated interface(s) must be reachable by the
      --cert-dir string                                         The directory where the TLS certs are located. If --tls-cert-file and --tls-private-key-file are provided, this fla
      --client-ca-file string                                   If set, any request presenting a client certificate signed by one of the authorities in the client-ca-file is authe
      --contention-profiling                                    Enable lock contention profiling, if profiling is enabled
      --default-watch-cache-size int                            Default watch cache size. If zero, watch cache will be disabled for resources that do not have a default watch size
      --delete-collection-workers int                           Number of workers spawned for DeleteCollection call. These are used to speed up namespace cleanup. (default 1)
      --deserialization-cache-size int                          Number of deserialized json objects to cache in memory.
      --disable-admission-plugins strings                       admission plugins that should be disabled although they are in the default enabled plugins list. Comma-delimited li
      --enable-admission-plugins strings                        admission plugins that should be enabled in addition to default enabled ones. Comma-delimited list of admission plu
      --enable-garbage-collector                                Enables the generic garbage collector. MUST be synced with the corresponding flag of the kube-controller-manager. (
      --enable-swagger-ui                                       Enables swagger ui on the apiserver at /swagger-ui
      --etcd-cafile string                                      SSL Certificate Authority file used to secure etcd communication.
      --etcd-certfile string                                    SSL certification file used to secure etcd communication.
      --etcd-compaction-interval duration                       The interval of compaction requests. If 0, the compaction request from apiserver is disabled. (default 5m0s)
      --etcd-count-metric-poll-period duration                  Frequency of polling etcd for number of resources per type. 0 disables the metric collection. (default 1m0s)
      --etcd-keyfile string                                     SSL key file used to secure etcd communication.
      --etcd-prefix string                                      The prefix to prepend to all resource paths in etcd. (default "/registry/wardle.kubernetes.io")
      --etcd-servers strings                                    List of etcd servers to connect with (scheme://ip:port), comma separated.
      --etcd-servers-overrides strings                          Per-resource etcd servers overrides, comma separated. The individual override format: group/resource#servers, where
      --experimental-encryption-provider-config string          The file containing configuration for encryption providers to be used for storing secrets in etcd
  -h, --help                                                    help for this command
      --http2-max-streams-per-connection int                    The limit that the server gives to clients for the maximum number of streams in an HTTP/2 connection. Zero means to
      --kubeconfig string                                       kubeconfig file pointing at the 'core' kubernetes server.
      --log-flush-frequency duration                            Maximum number of seconds between log flushes (default 5s)
      --log_backtrace_at traceLocation                          when logging hits line file:N, emit a stack trace (default :0)
      --log_dir string                                          If non-empty, write log files in this directory
      --logtostderr                                             log to standard error instead of files (default true)
      --profiling                                               Enable profiling via web interface host:port/debug/pprof/ (default true)
      --requestheader-allowed-names strings                     List of client certificate common names to allow to provide usernames in headers specified by --requestheader-usern
      --requestheader-client-ca-file string                     Root certificate bundle to use to verify client certificates on incoming requests before trusting usernames in head
      --requestheader-extra-headers-prefix strings              List of request header prefixes to inspect. X-Remote-Extra- is suggested. (default [x-remote-extra-])
      --requestheader-group-headers strings                     List of request headers to inspect for groups. X-Remote-Group is suggested. (default [x-remote-group])
      --requestheader-username-headers strings                  List of request headers to inspect for usernames. X-Remote-User is common. (default [x-remote-user])
      --secure-port int                                         The port on which to serve HTTPS with authentication and authorization. If 0, don't serve HTTPS at all. (default 44
      --stderrthreshold severity                                logs at or above this threshold go to stderr (default 2)
      --storage-backend string                                  The storage backend for persistence. Options: 'etcd3' (default), 'etcd2'.
      --storage-media-type string                               The media type to use to store objects in storage. Some resources or storage backends may only support a specific m
      --tls-cert-file string                                    File containing the default x509 Certificate for HTTPS. (CA cert, if any, concatenated after server cert). If HTTPS
      --tls-cipher-suites strings                               Comma-separated list of cipher suites for the server. Values are from tls package constants (https://golang.org/pkg
      --tls-min-version string                                  Minimum TLS version supported. Value must match version names from https://golang.org/pkg/crypto/tls/#pkg-constants
      --tls-private-key-file string                             File containing the default x509 private key matching --tls-cert-file.
      --tls-sni-cert-key namedCertKey                           A pair of x509 certificate and private key file paths, optionally suffixed with a list of domain patterns which are
  -v, --v Level                                                 log level for V logs
      --vmodule moduleSpec                                      comma-separated list of pattern=N settings for file-filtered logging
      --watch-cache                                             Enable watch caching in the apiserver (default true)
      --watch-cache-sizes strings                               List of watch cache sizes for every resource (pods, nodes, etc.), comma separated. The individual override format:

F0429 21:23:33.886693       1 main.go:44] cluster doesn't provide client-ca-file

Other useful info:

kubectl version
Client Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.11", GitCommit:"637c7e288581ee40ab4ca210618a89a555b6e7e9", GitTreeState:"clean", BuildDate:"2018-11-26T14:38:32Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"11+", GitVersion:"v1.11.8-eks-7c34c0", GitCommit:"7c34c0d2f2d0f11f397d55a46945193a0e22d8f3", GitTreeState:"clean", BuildDate:"2019-03-01T22:49:39Z", GoVersion:"go1.10.8", Compiler:"gc", Platform:"linux/amd64"}

Does Amazon not run the E2E Conformance tests?