apiservice v1alpha1.tenancy.kiosk.sh is unready even though kiosk is running

[iliyahoo]

I'm getting an error on the kiosk pod:

controller.go:498] unable to retrieve the complete list of server APIs: tenancy.kiosk.sh/v1alpha1: the server is currently unable to handle the request

Helm chart: kiosk-0.1.20
Kubernetes: v1.16.13-gke.1 (GKE)

Please have a look on the attached kiosk pod's log.
kiosk.log

[FabianKramm]

@iliyahoo thanks for reporting this issue! Does kiosk crash because of this error or is it only displayed in the logs?

[iliyahoo]

Pod is running without restarts.
Only this error is displayed in logs.

$ kc get po
NAME READY STATUS RESTARTS AGE
kiosk-746bbcb4b4-mw79n 1/1 Running 0 15h

$ kc get crd | grep kiosk
accountquotas.config.kiosk.sh 2020-09-16T19:40:28Z
accounts.config.kiosk.sh 2020-09-16T19:40:28Z
templateinstances.config.kiosk.sh 2020-09-16T19:40:28Z
templates.config.kiosk.sh 2020-09-16T19:40:28Z

[FabianKramm]

@iliyahoo ah okay I see! Yes this can occur and shouldn't have any effect on kiosk itself, the problem is that the apiservice v1alpha1.tenancy.kiosk.sh is not ready yet (because kiosk is just starting up), but already registered at the kubernetes api server, it will be ready after kiosk started.

[iliyahoo]

$ cat <<EOF | kc apply -f -

---

apiVersion: tenancy.kiosk.sh/v1alpha1
kind: Account
metadata:
name: johns-account
spec:
space:
limit: 2
subjects:

• kind: User
  name: john
  apiGroup: rbac.authorization.k8s.io
  EOF
  error: unable to recognize "STDIN": no matches for kind "Account" in version "tenancy.kiosk.sh/v1alpha1"

[iliyahoo]

$ kc api-resources | grep kiosk
accountquotas config.kiosk.sh false AccountQuota
accounts config.kiosk.sh false Account
templateinstances config.kiosk.sh true TemplateInstance
templates config.kiosk.sh false Template
error: unable to retrieve the complete list of server APIs: tenancy.kiosk.sh/v1alpha1: the server is currently unable to handle the request

[FabianKramm]

@iliyahoo thanks for the information. Does this still occur after:

kubectl delete apiservice v1alpha1.tenancy.kiosk.sh
kubectl delete pods --selector app=kiosk --namespace kiosk

I will add this as a bug, since kiosk could sort this out during startup. The problem is that the apiservice object is not that often updated by kuberentes and could stay in a non ready state.

[iliyahoo]

Yes, it still persists after deleting apiservice and pod.

$ kubectl describe apiservice v1alpha1.tenancy.kiosk.sh
...
Message: failing or missing response from https://10.204.3.20:8443/apis/tenancy.kiosk.sh/v1alpha1: Get https://10.204.3.20:8443/apis/tenancy.kiosk.sh/v1alpha1: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Reason: FailedDiscoveryCheck
Status: False
Type: Available
...

[FabianKramm]

@iliyahoo okay that is really strange, it should definitely work after deleting the apiservice and pod. I just tested it in gke with 1.16.13 and it works for me, even after deleting the deployment and recreating it. Maybe the service for the apiservice is somehow broken, can you do a kubectl get apiservices | grep kiosk.sh? Did you maybe install kiosk in multiple namespaces or did you install loft as well?

[iliyahoo]

This is a GKE cluster.
Server Version: version.Info{Major:"1", Minor:"16+", GitVersion:"v1.16.13-gke.1", GitCommit:"688c6543aa4b285355723f100302d80431e411cc", GitTreeState:"clean", BuildDate:"2020-07-21T02:37:26Z", GoVersion:"go1.13.9b4", Compiler:"gc", Platform:"linux/amd64"}

[iliyahoo]

$ kubectl get apiservices | grep kiosk.sh
v1alpha1.config.kiosk.sh Local True 4h17m
v1alpha1.tenancy.kiosk.sh kiosk/kiosk-apiservice False (FailedDiscoveryCheck) 38m

$ kubectl get po --all-namespaces | grep kiosk
kiosk kiosk-746bbcb4b4-5dw4d 1/1 Running 0 40m
]$

[FabianKramm]

@iliyahoo thanks for the information! Do you have a NetworkPolicy configured or is there any other special configuration in the GKE cluster?

[iliyahoo]

I'll install a GKE cluster from scratch and retest it again.

[iliyahoo]

Thank you very much, Fabian !
I've just deployed it on a fresh GKE cluster and there are not any errors.
The issue is definitely on my side.
Thank you again and I'm closing the issue.

[iliyahoo]

You can reproduce the issue if you create a private GKE cluster.
It works on a public one.

[FabianKramm]

@iliyahoo thanks for the information! Ah okay, yes that makes sense, the master API server cannot access kiosk, because the api server and cluster are isolated. You'll have to allow this in GKE and create a firewall rule, check elastic/cloud-on-k8s#1437 for a solution. (also see https://cloud.google.com/run/docs/gke/troubleshooting#deployment_to_private_cluster_failure_failed_calling_webhook_error)

[FabianKramm]

I'm closing this since it is mostly a GKE configuration issue and there is not much we can do in kiosk itself