Applying resources to kubevirt namespace fails with "caches not synchronized"

[booxter]

When applying the kubevirt manifests from manifests/ directory, oc fails
to post the resources with the following errors:

[root@zeus07 manifests]# oc --config ../ocp/auth/kubeconfig apply -f 110_cnv_kubevirt_op.yaml 
Warning: oc apply should be used on resource created by either oc create --save-config or oc apply
Warning: oc apply should be used on resource created by either oc create --save-config or oc apply
customresourcedefinition.apiextensions.k8s.io/kubevirts.kubevirt.io configured
Warning: oc apply should be used on resource created by either oc create --save-config or oc apply
clusterrole.rbac.authorization.k8s.io/kubevirt.io:operator configured
Warning: oc apply should be used on resource created by either oc create --save-config or oc apply
clusterrolebinding.rbac.authorization.k8s.io/kubevirt-operator configured
Error from server (Forbidden): error when applying patch:
{"metadata":{"annotations":{"kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"v1\",\"kind\":\"Namespace\",\"metadata\":{\"annotations\":{},\"labels\":{\"kubevirt.io\":\"\"},\"name\":\"kubevirt\",\"namespace\":\"\"}}\n"},"namespace":""}}
to:
Resource: "/v1, Resource=namespaces", GroupVersionKind: "/v1, Kind=Namespace"
Name: "kubevirt", Namespace: ""
Object: &{map["kind":"Namespace" "apiVersion":"v1" "metadata":map["name":"kubevirt" "selfLink":"/api/v1/namespaces/kubevirt" "uid":"cfc4fdce-41bb-11e9-a9c0-52fdfc072182" "resourceVersion":"904" "creationTimestamp":"2019-03-08T16:04:11Z" "labels":map["kubevirt.io":""]] "spec":map["finalizers":["kubernetes"]] "status":map["phase":"Active"]]}
for: "110_cnv_kubevirt_op.yaml": namespaces "kubevirt" is forbidden: caches not synchronized
Error from server (Forbidden): error when creating "110_cnv_kubevirt_op.yaml": serviceaccounts "kubevirt-operator" is forbidden: caches not synchronized
Error from server (Forbidden): error when creating "110_cnv_kubevirt_op.yaml": deployments.apps "virt-operator" is forbidden: caches not synchronized

When we switch to kube-system namespace then it works fine.

The same error happens when manifests are applied by openshift-install
itself, as in #127.

I plan to post a PR that switches to kube-system to make progress on
deployment. This issue will be used to track investigation on reasons behind
the failure and to revert back to kubevirt namespace once we have the root
cause.

[hardys]

Some discussion with @vrutkovs on slack indicates this may be due to cert rotation being broken, workaround is oc get csr -o name | xargs -n 1 oc adm certificate approve

[dhellmann]

Do we feel like that's a short-term state, or should we add a script to make it easier to remember that cantrip?

[abhinavdahiya]

Error from server (Forbidden): error when creating "110_cnv_kubevirt_op.yaml": serviceaccounts "kubevirt-operator" is forbidden: caches not synchronized
Error from server (Forbidden): error when creating "110_cnv_kubevirt_op.yaml": deployments.apps "virt-operator" is forbidden: caches not synchronized

you are seeing these errors because openshift-apiserver is not running. Openshift-apiserver provides some of the api required to run admission controllers in the kube-apiserver, therefore kube-apiserver rejects objects in all namespaces until openshift-apiserver is running.

Now because there are components that need to be running in the cluster before openshift-apiserver, those Cluster Operators make use of run-level 0/1 to skip the admission controllers that depend on openshit-apiserver.
eg: https://github.com/openshift/machine-api-operator/blob/master/install/0000_30_machine-api-operator_00_namespace.yaml#L7

There might be conditions that prevent only some namespaces to have run-levels and master team can guide you on that a bit more.

[russellb]

Since it's not clear if the originally reported issues is caused by the cert approval issue, i'm going to open a new issue to track that.

[russellb]

cert approval issue is now tracked in #260

[hardys]

I think with the csr approval workarounds landed this can be closed, and we'll track remaining issues around that via #260?

[russellb]

I'm closing this as stale, please file a new issue if you have more problems.