WIP Add extra manifests during 06_deploy_bootstrap_vm.sh

[hardys]

We add a create manifests step before generating the ignition
configs, because the assets in kni-installer will read all files
in the location, so we can easily patch in the extra manifests
needed.

In future it may be cleaner to add the manifest assets directly
to kni-installer, but for now this should be a simple way to
iterate on testing the manifests.

Note that the index prefix was changed to align with the existing
manifests generated by kni_installer, I'm assuming using a larger
index here will result in the desired serialization so these are
applied post-openshift things, testing needed to confirm this.

[hardys]

typo s/not/note

[derekhiggins]

Build SUCCESS, see build http://10.8.144.11:8080/job/dev-tools/133/

[derekhiggins]

Build SUCCESS, see build http://10.8.144.11:8080/job/dev-tools/135/

[hardys]

Ok so this appears to apply the manifests (I see the kubevirt CRD) but kubevirt doesn't yet come up - from talking with @fabiand it sounds like we're blocked on some testing of kubevirt on OCP4 on other platforms, so we should probably revisit debugging when that is completed, and metalkube can deploy workers.

Probably no harm in deploying with these manifests though, provided the deploy still works and it doesn't add much time to the walltime.

[booxter]

@hardys I don't think it actually works. Yes I see kubevirt CRD but then it fails as follows:

[root@zeus07 manifests]# oc --config ../ocp/auth/kubeconfig apply -f 110_cnv_kubevirt_op.yaml 
Warning: oc apply should be used on resource created by either oc create --save-config or oc apply
Warning: oc apply should be used on resource created by either oc create --save-config or oc apply
customresourcedefinition.apiextensions.k8s.io/kubevirts.kubevirt.io configured
Warning: oc apply should be used on resource created by either oc create --save-config or oc apply
clusterrole.rbac.authorization.k8s.io/kubevirt.io:operator configured
Warning: oc apply should be used on resource created by either oc create --save-config or oc apply
clusterrolebinding.rbac.authorization.k8s.io/kubevirt-operator configured
Error from server (Forbidden): error when applying patch:
{"metadata":{"annotations":{"kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"v1\",\"kind\":\"Namespace\",\"metadata\":{\"annotations\":{},\"labels\":{\"kubevirt.io\":\"\"},\"name\":\"kubevirt\",\"namespace\":\"\"}}\n"},"namespace":""}}
to:
Resource: "/v1, Resource=namespaces", GroupVersionKind: "/v1, Kind=Namespace"
Name: "kubevirt", Namespace: ""
Object: &{map["kind":"Namespace" "apiVersion":"v1" "metadata":map["name":"kubevirt" "selfLink":"/api/v1/namespaces/kubevirt" "uid":"cfc4fdce-41bb-11e9-a9c0-52fdfc072182" "resourceVersion":"904" "creationTimestamp":"2019-03-08T16:04:11Z" "labels":map["kubevirt.io":""]] "spec":map["finalizers":["kubernetes"]] "status":map["phase":"Active"]]}
for: "110_cnv_kubevirt_op.yaml": namespaces "kubevirt" is forbidden: caches not synchronized
Error from server (Forbidden): error when creating "110_cnv_kubevirt_op.yaml": serviceaccounts "kubevirt-operator" is forbidden: caches not synchronized
Error from server (Forbidden): error when creating "110_cnv_kubevirt_op.yaml": deployments.apps "virt-operator" is forbidden: caches not synchronized

This is weird because I was able to bring up kubevirt several days ago.

[booxter]

@hardys I see the following errors in bootkube.service logs too:

Mar 08 19:46:25 localhost.localdomain openshift.sh[3479]: Error from server (Forbidden): error when creating "./111_cnv_kubevirt_cr.yaml": kubevirts.kubevirt.io "kubevirt" is forbidden: caches not synchronized
Mar 08 19:46:25 localhost.localdomain openshift.sh[3479]: kubectl create --filename ./111_cnv_kubevirt_cr.yaml failed. Retrying in 5 seconds...

Which is the same error as when trying to apply manifests manually.

[booxter]

I think there is some issue with kubevirt namespace used for KubeVirt manifests. When the manifests use the namespace then I get the error above. When they use kube-system instead, they apply just fine. (Automatically.) Perhaps we should temporarily revert to using kube-system for CNV / KubeVirt services and create an issue to investigate why there is an issue with caches that forbid creation for legit resources in a separate namespace.

[fabiand]

Hm, let's not move kubevirt to kube-system, but let's understand why we have the kubevirt anemspace issues.

@slintes you are an expert in this area. Thoughts?

[hardys]

@vrutkovs perhaps you can help outline the plan as discussed on slack here, my understanding is short-term we need to replace this approach with some script that will run after the masters are deployed (with the wait stuff from #130), but that long term we should be able to correctly serialize the manifests but I'm still not completely clear on what is needed to make that work?

[vrutkovs]

long term we should be able to correctly serialize the manifests but I'm still not completely clear on what is needed to make that work?

Yes, according to https://github.com/openshift/installer/blob/master/docs/user/customization.md#kubernetes-customization-unvalidated the installer can tell CVO to apply additional manifests during/after install, similar to what #137 is doing. In the end it would become a part of kni-install

[hardys]

Ok closing this for now and we'll apply the manifests with a script, then revisit applying them via kni-installer later