Catalog APIService Object Failed

[RamanPndy]

Bug Report

What happened:
Catalog APIService object failed (FailedDiscoveryCheck)

What you expected to happen:
Catalog APIService object should not fail.

How to reproduce it (as minimally and precisely as possible):
cd charts/catalog
helm install . --name catalog --namespace catalog
kubectl describe apiservice v1beta1.servicecatalog.k8s.io

Anything else we need to know?:
Name: v1beta1.servicecatalog.k8s.io
Namespace:
Labels:
Annotations:
API Version: apiregistration.k8s.io/v1
Kind: APIService
Metadata:
Creation Timestamp: 2019-06-03T09:51:47Z
Resource Version: 354707
Self Link: /apis/apiregistration.k8s.io/v1/apiservices/v1beta1.servicecatalog.k8s.io
UID: 33aa59bb-85e5-11e9-8570-005056a591de
Spec:
Ca Bundle: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUM5akNDQWQ2Z0F3SUJBZ0lRWnVOejlKSDV2SWxuVC9rNm9wT1JWVEFOQmdrcWhraUc5dzBCQVFzRkFEQVYKTVJNd0VRWURWUVFERXdwemRtTXRZMkYwTFdOaE1CNFhEVEU1TURZd016QTVOVE0wTkZvWERUSTVNRFV6TVRBNQpOVE0wTkZvd0ZURVRNQkVHQTFVRUF4TUtjM1pqTFdOaGRDMWpZVENDQVNJd0RRWUpLb1pJaHZjTkFRRUJCUUFECmdnRVBBRENDQVFvQ2dnRUJBTEJ0MXZCWkxxaE1aYWFMRWRZNmVjWGpwY2JXVmNOOWduZEc5L1FwZFVGVXM1eUEKR0lrbi95MTcvaWRmbHBqS2pYZDkrYzI2NmVmTTN1R0ZnNVRjczVETXFvWi81S3NLNVorVHVFd1cxeWdxWW1oagplRGV1WlhZWWxSNEFjbkpkQnpFV3RwcnY0eHI1V1hrNUdWTHMvVXRaOWpNSUFoY09uVU1aVnNEa0crUFk1ZU9hCmsydTdIYUs4clBKRldOYTYxemd3YS94c3N0S0xSbkFENlRFbnJWOEVuT3czMUFhWFZqaWN3WS95VGJqUzNyQUQKdy8vM096VG5scEhlaUFLYnM4MUZKdjRvbHdTM2Mrb1l5bkNpQm1DNmZzQmV0aHVHYkRzaDhqa2s0MXhJeUhNMwptNkVDanpxNmJXZFFqb3owaGh3WU5IZmtCbXAvTFVaRER3RkJYUE1DQXdFQUFhTkNNRUF3RGdZRFZSMFBBUUgvCkJBUURBZ0trTUIwR0ExVWRKUVFXTUJRR0NDc0dBUVVGQndNQkJnZ3JCZ0VGQlFjREFqQVBCZ05WSFJNQkFmOEUKQlRBREFRSC9NQTBHQ1NxR1NJYjNEUUVCQ3dVQUE0SUJBUUIzMHhBbjdrQ25ydE52T0xDV0J4MS9YRkN3NFF5UgplWGxQc3BDSzY0Q3Uyd1RtYzhMc29vMVFoNXVMb3VFamcyRnhBc0NTSzdZTnJjckJ3QzhhQkVIdXkxUE5qT081CmVVRHBaUzNrNXNCZUxCQ0Q3WnBHWjRhb3owc3BqdWVPR2pqcFlIa1R4clNMK3FMTXNLTGY3UncwcnRVeEwyc1UKZjFhdzlFY3BldnJsUS9xVUkvSC9VMVNRN1FSaDMwV3NMQzdxYU83ZEhIdDF2ZHZsVDVoYXVtYzloZUZSdXV1UgpwS25BeGo3VUIwaFl2Y2NYcnR2eHVJT1VTN04wWS9XV0pWSmpPL2RhNk5tQnlIZzlDdkljcFR4VEs1NktUaU5XCi9WK3ZrSW5qVWdyZnUwVFU2WDZpbktjRzk5VGRiVnBldzQyTUc4Z0RpU0s3WFVmLy82MDYxUGl2Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K
Group: servicecatalog.k8s.io
Group Priority Minimum: 10000
Service:
Name: catalog-catalog-apiserver
Namespace: catalog
Version: v1beta1
Version Priority: 20
Status:
Conditions:
Last Transition Time: 2019-06-03T09:51:47Z
Message: no response from https://10.233.11.240:443: Get https://10.233.11.240:443: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Reason: FailedDiscoveryCheck
Status: False
Type: Available
Events:

Note:- I alreay have setup http_proxy and https_proxy in /etc/environment. also catalog-catalog-apiserver service and relater pods with this service are running fine.

Environment:

• Kubernetes version (use kubectl version):
  Client Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.5", GitCommit:"2166946f41b36dea2c4626f90a77706f426cdea2", GitTreeState:"clean", BuildDate:"2019-03-25T15:19:22Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"linux/amd64"}
  Server Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.5", GitCommit:"2166946f41b36dea2c4626f90a77706f426cdea2", GitTreeState:"clean", BuildDate:"2019-03-25T15:19:22Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"linux/amd64"}
• service-catalog version:
• Cloud provider or hardware configuration: Bare Metal
• Do you have api aggregation enabled? Yes
  • Do you see the configmap in kube-system? Yes
  • Does it have all the necessary fields? Yes
    • kubectl get cm -n kube-system extension-apiserver-authentication -o yaml and look for requestheader-XXX fields
• Install tools:
  • Did you use helm? Yes
    What were the helm arguments? Did you --set any extra values? No
• Are you trying to use ALPHA features? Did you enable them?

[mszostok]

The detailed discussion was on our Service Catalog channel on k8s slack workspace, see: https://kubernetes.slack.com/archives/C232SF3TK/p1559642796001100

TL;DR;

The problem is the same as already reported some time on the k8s repo: kubernetes/kubernetes#66231

This issue is not related directly to the Service Catalog, as you also confirmed that with the k8s metrics-server you also ran in that issue.

The root cause could be a misconfigured network layer.

[RamanPndy]

this issues is fixed now. issue was with the kube-apiserver was running with http proxies. removing http proxies and no proxy from env of kube-apiserver fixed this issue.
but still catalog controller manager is failing with these errors:
1 round_trippers.go:438] GET https://10.233.0.1:443/apis/servicecatalog.k8s.io/v1beta1?timeout=32s in 32000 milliseconds
I0618 09:08:35.426432 1 round_trippers.go:444] Response Headers:
E0618 09:08:35.426462 1 controller_manager.go:311] Failed to get supported resources from server: Get https://10.233.0.1:443/apis/servicecatalog.k8s.io/v1beta1?timeout=32s: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
I0618 09:08:35.426615 1 controller_manager.go:306] Created client for API discovery
I0618 09:08:35.426716 1 round_trippers.go:419] curl -k -v -XGET -H "Accept: application/json, /" -H "User-Agent: service-catalog/v0.2.1 (linux/amd64) kubernetes/6c6a49d/service-catalog-controller-discovery" -H "Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IiJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJrdWJlLXN5c3RlbSIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJzZXJ2aWNlLWNhdGFsb2ctY29udHJvbGxlci1tYW5hZ2VyLXRva2VuLXpmaGNiIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQubmFtZSI6InNlcnZpY2UtY2F0YWxvZy1jb250cm9sbGVyLW1hbmFnZXIiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC51aWQiOiJhNTdmMTgwZC05MWE1LTExZTktYmE1YS0wMDUwNTZhNWVmYjQiLCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291bnQ6a3ViZS1zeXN0ZW06c2VydmljZS1jYXRhbG9nLWNvbnRyb2xsZXItbWFuYWdlciJ9.HNTDRl9Nw2UgKfp6nZzCafO_I4NDGRb1uCjNIUEQFr_KtMxdE53EAj9JKoYZEQwNqDCH8StJrnSVvKdmp0bXsYHoxnPWvX7l9k12l82XnhE29wijIcpcE8rCX_KE9mtZoDlpKyG2ngeDNOmv520Rsw87HO8ytMDGlATScVhkY5n9VpzcKw2AIe24UbFSeziXnhp5sd1SMvBpIbya2z9CQ4LuhWAfzx6RbgLiKvn-vsPrJnX200voVDEs0vsGJ0nQTXmcP5_croawu51QHmr4S0-TSGQjW6gCGOl9MMkBI6Ur7gAaxLnv_Ao6Pk7Z14iLye6Ub4USf04wI45Ft3Es5g" 'https://10.233.0.1:443/apis/servicecatalog.k8s.io/v1beta1?timeout=32s'
I0618 09:09:07.426865 1 round_trippers.go:438] GET https://10.233.0.1:443/apis/servicecatalog.k8s.io/v1beta1?timeout=32s in 32000 milliseconds
I0618 09:09:07.426899 1 round_trippers.go:444] Response Headers:
E0618 09:09:07.427120 1 controller_manager.go:311] Failed to get supported resources from server: Get https://10.233.0.1:443/apis/servicecatalog.k8s.io/v1beta1?timeout=32s: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
F0618 09:09:07.427222 1 controller_manager.go:236] error running controllers: failed to get api versions from server: "timed out waiting for the condition", "Get https://10.233.0.1:443/apis/servicecatalog.k8s.io/v1beta1?timeout=32s: net/http: request canceled (Client.Timeout exceeded while awaiting headers)"

where 10.233.0.1 is the CLUSTERIP of the kubernetes service.

also vm running with these pods gone too slow. is there any performance impact with service catalog with existing K8S cluster ??

These are the logs of kube-apiserver:
Trace[2007787961]: [23m59.116296457s] [23m59.114330668s] Transformed response object
E0618 09:05:13.750206 1 controller.go:111] loading OpenAPI spec for "v1beta1.servicecatalog.k8s.io" failed with: failed to retrieve openAPI spec, http error: ResponseCode: 503, Body: request timed out, Header: map[]
I0618 09:05:13.750230 1 controller.go:119] OpenAPI AggregationController: action for item v1beta1.servicecatalog.k8s.io: Rate Limited Requeue.
E0618 09:05:32.967096 1 memcache.go:135] couldn't get resource list for servicecatalog.k8s.io/v1beta1: Get https://[::1]:6443/apis/servicecatalog.k8s.io/v1beta1?timeout=32s: context deadline exceeded (Client.Timeout exceeded while awaiting headers)
E0618 09:06:34.979867 1 memcache.go:135] couldn't get resource list for servicecatalog.k8s.io/v1beta1: Get https://[::1]:6443/apis/servicecatalog.k8s.io/v1beta1?timeout=32s: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
I0618 09:07:13.750376 1 controller.go:105] OpenAPI AggregationController: Processing item v1beta1.servicecatalog.k8s.io
E0618 09:07:36.991287 1 memcache.go:135] couldn't get resource list for servicecatalog.k8s.io/v1beta1: Get https://[::1]:6443/apis/servicecatalog.k8s.io/v1beta1?timeout=32s: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
I0618 09:08:09.770424 1 trace.go:76] Trace[111176043]: "Get /api/v1/namespaces/kube-system/pods/catalog-catalog-controller-manager-5c5fb5d8f6-8f8jc/log" (started: 2019-06-18 09:06:06.927204232 +0000 UTC m=+11941.486482605) (total time: 2m2.843179696s):
Trace[111176043]: [2m2.843177011s] [2m2.841889191s] Transformed response object
E0618 09:08:13.750551 1 controller.go:111] loading OpenAPI spec for "v1beta1.servicecatalog.k8s.io" failed with: failed to retrieve openAPI spec, http error: ResponseCode: 503, Body: request timed out, Header: map[]
I0618 09:08:13.750582 1 controller.go:119] OpenAPI AggregationController: action for item v1beta1.servicecatalog.k8s.io: Rate Limited Requeue.
E0618 09:08:39.003511 1 memcache.go:135] couldn't get resource list for servicecatalog.k8s.io/v1beta1: Get https://[::1]:6443/apis/servicecatalog.k8s.io/v1beta1?timeout=32s: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
I0618 09:09:13.750724 1 controller.go:105] OpenAPI AggregationController: Processing item v1beta1.servicecatalog.k8s.io
E0618 09:09:41.017192 1 memcache.go:135] couldn't get resource list for servicecatalog.k8s.io/v1beta1: Get https://[::1]:6443/apis/servicecatalog.k8s.io/v1beta1?timeout=32s: context deadline exceeded
E0618 09:10:13.750889 1 controller.go:111] loading OpenAPI spec for "v1beta1.servicecatalog.k8s.io" failed with: failed to retrieve openAPI spec, http error: ResponseCode: 503, Body: request timed out, Header: map[]
I0618 09:10:13.750916 1 controller.go:119] OpenAPI AggregationController: action for item v1beta1.servicecatalog.k8s.io: Rate Limited Requeue.
E0618 09:10:43.028793 1 memcache.go:135] couldn't get resource list for servicecatalog.k8s.io/v1beta1: Get https://[::1]:6443/apis/servicecatalog.k8s.io/v1beta1?timeout=32s: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
E0618 09:11:45.041314 1 memcache.go:135] couldn't get resource list for servicecatalog.k8s.io/v1beta1: Get https://[::1]:6443/apis/servicecatalog.k8s.io/v1beta1?timeout=32s: net/http: request canceled (Client.Timeout exceeded while awaiting headers)

but catalog apiserver is running fine.

Any thoughts and suggestions on this would be much appreciated.

[mszostok]

Ok, so it looks like the same issue that you had for api-server. So right probably the API Server can reach the Pod but Pods are not able to reach the API Server.

Quick illustration:

so the question is if you API Server is able to reach the Pod from the nodes?

[RamanPndy]

Ok, so it looks like the same issue that you had for api-server. So right probably the API Server can reach the Pod but Pods are not able to reach the API Server.

Quick illustration:

so the question is if you API Server is able to reach the Pod from the nodes?

but catalog apiserver pod is running fine only controller manager is failing. I'm not able to understand why kube-api server is not accessible from catalog controller manager pod. is there any way to debug this issue ?? also master node became too slow. is it impacting performance somewhere ??

[mszostok]

In Kyma we are doing Service Management using the Service Catalog. We do not encounter any performance issue with it.

In our case, we just create clusters using the GKE or AKS. In such setup, everything is working. Previously we also used the acs-enging to create clusters that were managed by us. And in such scenario, it also worked.

IMO it's still sth with the network between master and worker nodes.

And about the performence, it could be because you still have some errors with registering the service catalog api-server into k8s.

You can check if you are able to deploy the sample-api-server and use registered kind by generated client set.

[RamanPndy]

In Kyma we are doing Service Management using the Service Catalog. We do not encounter any performance issue with it.

In our case, we just create clusters using the GKE or AKS. In such setup, everything is working. Previously we also used the acs-enging to create clusters that were managed by us. And in such scenario, it also worked.

IMO it's still sth with the network between master and worker nodes.

And about the performence, it could be because you still have some errors with registering the service catalog api-server into k8s.

You can check if you are able to deploy the sample-api-server and use registered kind by generated client set.

can you please suggest is there any way that i can make sure catalog apiserver is registered with kube-apiserver ??

[RamanPndy]

#kubernetes/kubernetes#56430

[RamanPndy]

This issue is fixed now by enabling hostNetwork to true in apiserver-deployment.yaml and controller-manager-deployment.yaml i.e. by adding hostNetwork: true on containers level and adding insecureSkipTLSVerify: true in APIService Object i.e. apiregistration.yaml and removing caBundle. after this kube apiserver lagging and catalog controller manager pod crashing both are resolved. also for successful catalog api server registration with kube-apiserver, i suspect that kube-apiserver must be run without any http proxies set as env on container level.

Thanks @mszostok for all suggestions and advises.