[poc] kubevirt: Add hypershift requirements to primary interface

[qinqon]

- What this PR does and why is it needed
This is a draft PR to implement features needed for hypershift hosted workers:

• seamless kubevirt live migration
  • stable IPs
  • stable connections
• east/west and north/south communication
• access to services

TODO

• Don't set ip config at veth if kubevirt
• Configure DHCP options at LSP
• Use proxy_arp option from router LSP ovn-org/ovn@8087cbc
  • northd, lsp: Arp proxy additional features ovn#176
• Add /32 routes ovn_cluster_router
• Use route instead of policy for inbound traffic to specify the port.
• Sometimes new connections are not possible after live migration looks like endpoint slices are not correctly updated
  • svc: Support pods with same address kubernetes/kubernetes#115907
  • backport: Automated cherry pick of #115907: svc: Support pods with same address kubernetes/kubernetes#116084
• Investigate net.netfilter.nf_conntrack_tcp_loose=1 for snat + live migration
• Check that forcing the mac to the VM ends at ovn-k
• Cleanup
  • policy
  • routes
  • dhcp-options
• Test services
• Test connectivity + live migration
  • with proxy arp + LRP shared mac address we have < 1 second disruption
• Test postcopy migraiton
  • Ther is a 10s downtime, point to point route is correctly updated when annotation appears but traffic is not working.
  • Ednpoint slice for the new virt-launcher are not ready and not serving until src virt-launcher is not Completed
  • Traffic goes back between route change and virt-launcher src completed, but not at route change.
  • When annotation appears the src virt-launcher is not able to manage traffic
  • Maybe traffic is still going to the src virt-launcher when it already does not accept it.
  • Adding unpaused condition to target virt-launcher decreased 3-4 seconds the downtime
• Test network policies
• Test hypershift
• Use kubevirt PRs:
  • Sync VMI dynamic labels to pod after live migration kubevirt/kubevirt#9144
  • virt-handler: Add live-migration bridge pod net annotation kubevirt/kubevirt#9193
  • Indication of when target pod hosts active qemu instance during live migration kubevirt/kubevirt#9290
  • virtctl, expose: Ignore migration target kubevirt/kubevirt#9330
• Use hypershift PRs:
  • nodepool: Add live migration allowance annotation openshift/hypershift#2244
• Investigate multi requested-chassis feature for live migration
  • It only works when LSP is not moved, meaning it only work with distributed switch.
• Reverse on to new ovnkube-master controllers, cluster-manager and network-controller-manager

openshift issues

"kube-dns" not found`

problem getting dns server addLogicalPort failed for cluster1/virt-launcher-worker1-chdqx: failed composing DHCP options: services "kube-dns" not found

For the poc we just fallback to openshift one, but we shoiuld "mimic" what kubevirt dhcp server is doing and read for virt-launcher pod resolv.conf, this can be done by the CNI annotating the pod then ovnkube-master will use those to generate DHCP options.

node-valid-hostname.service

Failed Units: 1
  node-valid-hostname.service

cannot migrate VMI: PVC test1-k7m4b-rhcos is not shared

You are using a client virtctl version that is different from the KubeVirt version running in the cluster
Client Version: v0.58.0
Server Version: v0.59.0-rc.1-19-ga5e3a1b9b
Error migrating VirtualMachine Internal error occurred: admission webhook "migration-create-validator.kubevirt.io" denied the request: Cannot migrate VMI, Reason: DisksNotLiveMigratable, Message: cannot migrate VMI: PVC test1-k7m4b-rhcos is not shared, live migration requires that all PVCs must be shared (using ReadWriteMany access mode)

We have to use the patch from other hypershift pocs to change the mode

We have to pass "hostname" same as kubevirt DHCP server and set it with the vm name labels from virt-launcher pod.

Using "hostname" at DHCP options is causing problem with DHCP and the VM end up without IP address

hostname option has to be quoted

Cannot connect after live-migration

Looks like the endpoint is referencing the old virt-launcher pod wich has the same ip address, removing old virt-launchers fix
the issue

apiVersion: v1
kind: Endpoints
metadata:
  annotations:
    endpoints.kubernetes.io/last-change-trigger-time: "2023-02-16T14:24:00Z"
  creationTimestamp: "2023-02-16T14:17:08Z"
  name: ssh-test1-2ghnh
  namespace: clusters-test1
  resourceVersion: "220258"
  uid: c7d4a953-f876-40b9-bfa7-a1ce5d3e43d0
subsets:
- addresses:
  - ip: 10.129.2.25
    nodeName: worker-1
    targetRef:
      kind: Pod
      name: virt-launcher-test1-2ghnh-tkhsb
      namespace: clusters-test1
      uid: d779f8a5-ba16-4146-bb4d-0b489ffee47b
  notReadyAddresses:
  - ip: 10.129.2.25
    nodeName: worker-2
    targetRef:
      kind: Pod
      name: virt-launcher-test1-2ghnh-t2x28
      namespace: clusters-test1
      uid: 650dfac8-00af-4f3f-8c14-c40cc3feb778
  ports:
  - port: 22
    protocol: TCP

Sometime the notReadyAddress is not there, looks like ovn-k Recocnile bug that appears when both pods has the same IP

Nice to have

• Investigate solutions for VM restart

Capk demo

https://asciinema.org/a/hEqzoFveibsuHKynQxZ3RxCHZ

- How to verify it
This is verify with the umbrella project https://github.com/qinqon/ovn-kubevirt

Docs:

• ovn + dhcp
• Live migration
• How to create an Open Virtual Network distributed gateway router
• http://www.openvswitch.org/support/boston2017/1150-ovn-gateways-ipv6.pdf
• localnet
• Load Balancer https://www.cnblogs.com/yaodd/p/7477848.html
• ovn-kuberntes lb: -

  ovn-kubernetes/go-controller/pkg/ovn/loadbalancer/loadbalancer.go

  Line 42 in 90d77b7

  func EnsureLBs(nbClient libovsdbclient.Client, service *corev1.Service, LBs []LB) error {

[davidvossel]

nice! I made a few comments in line.

[qinqon]

/hold

[qinqon]

/retest-requiered

[qinqon]

/retest

[ovn-robot]

Oops, something went wrong:

This workflow is already running

[coveralls]

Coverage: 51.713% (-0.3%) from 51.984% when pulling 174c0b7 on qinqon:spike-kubevirt-migration into b1ded23 on ovn-org:master.