CI is broken tracker

[cgwalters]

Splitting this from #502

So far what appears to have happened is a config change broke the steering of jobs to build02 where we have nested virt enabled.

But now, AFAICS our jobs are running on build02 again, and:

kvm-device-plugin is healthy:

$ oc get ds
NAME                DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
kvm-device-plugin   42        42        42      42           42          <none>          39d
$ 

Looking at the pods from one CI run:

$ oc project ci-op-d7w80vyk
Now using project "ci-op-d7w80vyk" on server "https://api.build02.gcp.ci.openshift.org:6443".
$ oc get pods
NAME        READY   STATUS      RESTARTS   AGE
bin-build   0/1     Error       0          33m
src-build   0/1     Completed   0          35m
$

That's on node/build0-gstfj-w-c-gttfw.c.openshift-ci-build-farm.internal.

Looking at that node, I see:

Capacity:
  attachable-volumes-gce-pd:  127
  cpu:                        16
  devices.kubevirt.io/kvm:    2
  ephemeral-storage:          1257739244Ki
  hugepages-1Gi:              0
  hugepages-2Mi:              0
  memory:                     61832252Ki
  pods:                       250
Allocatable:
  attachable-volumes-gce-pd:  127
  cpu:                        15
  devices.kubevirt.io/kvm:    2
  ephemeral-storage:          1158058743528
  hugepages-1Gi:              0
  hugepages-2Mi:              0
  memory:                     58584124Ki
  pods:                       250

So we have devices.kubevirt.io/kvm on that node.

And that node is backed by machine/build0-gstfj-w-c-gttfw which was booted with Image: projects/rhcos-cloud/global/images/rhcos-46-82-202011260640-0-gcp-x86-64 which looks right.

[cgwalters]

I came across this helpful doc https://cloud.google.com/compute/docs/troubleshooting/troubleshooting-nested-vms

And got the output of that, seems we have /dev/kvm just fine and have vmx:

$ oc --context build02 --as system:admin debug node/build0-gstfj-w-c-gttfw.c.openshift-ci-build-farm.internal -- grep -c vmx /proc/cpuinfo
...
16
$ oc --as system:admin --context build02 debug node/build0-gstfj-w-c-gttfw.c.openshift-ci-build-farm.internal -- ls -al /dev/kvm
...
crw-rw-rw-. 1 root kvm 10, 232 Feb 24 15:23 /dev/kvm
$

So that all looks fine...something going wrong in kvm-device-plugin providing the device to the pod via kubelet? Going to add some debug stuff there.

[cgwalters]

One theory I had was that it might relate to an upgrade of build02 to 4.7 - perhaps new kubelet or so? But no, I just stood up a fresh 4.7.0 cluster in GCP with the same kvm-device-plugin container and that all worked fine.

[cgwalters]

Hmm wait, something else odd is going on because e.g. this rpm-ostree CI job which uses nested virt worked.

[cgwalters]

I think the current summary of this is that:

• DPTP says: "We have a periodic that automatically balances load among buildclusters by reshuffeling jobs based on their resource usage in the past"
• That manifested as e.g. this commit which moved the openshift-os jobs
• They are going to implement some labeling to ensure this doesn't happen for our nested-virt jobs, i.e. anything that requests kvm should be steered to a cluster supporting it

[cgwalters]

Man I am at such a loss on this one.

AFAICS, the CI jobs here should be getting /dev/kvm. I can see on the build pod the annotations. I added some debug logging, deployed that on build02 and I see the hook allocating /dev/kvm multiple times.

The way we're doing the build here is very similar (although not exactly identical) to the rpm-ostree job - they both appear as docker-style build pods with the kvm annotation.

Hmmm...I think I'm going to try adding more fcos-e2e jobs on other repos to see whether they hit this too.