Enhance support for real time workloads

[jordigilh]

What this PR does / why we need it:

This PR aims to improve the support for real-time workloads and tune the libvirt's XML for low CPU latency. The motivation for this change is to make kubevirt aware of realtime workloads and tune libvirt's XML to reduce the CPU latency, following libvirt's recommendation. It archieves this goal by implementing the following changes:

• Extends the VMI schema to add a new knob structure inside spec.domain.cpu named realtime. This new knob is on itself a new structure that so far only contains the field mask. The mask field allows the user to define the range of vcpus to pin in the vcpusched xml field for scheduling type as fifo priority 1. If this field is left undefined, the logic will pin all allocated vcpus for real-time workload.

Example when enabling the realtime knob without defining the mask:

    spec:
      domain:
        cpu:
          realtime: {}

• When the realtime knob is enabled, the virt-launcher will add the following libvirt XML elements:
• When huge pages is supported
  • <nosharepages/>
  • virt-handler will set the memory lock limits for the qemu-kvm process.
• Disable the Performance Monitoring Unit in the Features section:
  • <pmu state="off"/>
• Add the vcpusched element in the CPUTune section:
  • <vcpusched scheduler="fifo" priority="1" vcpus="<pinned vcpus>">
• Nodes that support realtime workloads (kernel setting kernel.sched_rt_runtime_us=-1 to allow unlimited runtime of processes running with realtime scheduling) will be labeled with kubevirt.io/realtime.
• When deploying a VMI that has the realtime knob enabled in its manifest, the generated pod manifest will be mutated to include the node label selector kubevirt.io/realtime, so that the pod is scheduled to run in a node that supports realtime workloads.
• In short, a VMI with the realtime knob enabled will require a node that is capable of running realtime workloads ( kubevirt.io/realtime label), is able to isolate CPUs (cpumanager=true label) and has been configured with hugepages.
• It also includes a commit for a test that will run a realtime cpu latency tool (cyclictest) to measure the CPU latency in the VM. There is another PR in the project-infra that will trigger the job periodically.

Example of a complete VM manifest:

---
apiVersion: kubevirt.io/v1
kind: VirtualMachine
metadata:
  labels:
    kubevirt.io/vm: vm-realtime
  name: vm-realtime
  namespace: poc
spec:
  running: true
  template:
    metadata:
      labels:
        kubevirt.io/vm: vm-realtime
    spec:
      domain:
        devices:
          autoattachSerialConsole: true
          autoattachMemBalloon: false
          autoattachGraphicsDevice: false
          disks:
          - disk:
              bus: virtio
            name: containerdisk      
        machine:
          type: ""
        resources:
          requests:
            memory: 4Gi
            cpu: 2
          limits:
            memory: 4Gi
            cpu: 2
        cpu:
          model: host-passthrough
          cores: 2
          sockets: 1
          threads: 1
          dedicatedCpuPlacement: true
          isolateEmulatorThread: true
          ioThreadsPolicy: auto
          features:
            - name: tsc-deadline
              policy: require
          numa:
            guestMappingPassthrough: {}
          realtime: {}
        memory:
          hugepages:
            pageSize: 1Gi
          guest: 3Gi
      terminationGracePeriodSeconds: 0
      volumes:
      - containerDisk:
          image: quay.io/jordigilh/centos8-realtime:latest
        name: containerdisk

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):

TBD

Special notes for your reviewer:

/cc @rmohr @vladikr @davidvossel
/cc @pkliczewski @fabiand

Release note:

adds support for real time workloads

[jordigilh]

@vladikr @davidvossel @rmohr I've finished implementing the changes for the real-time enhancements. Can you take a look whenever you have time?

Thanks!

/Jordi

[davidvossel]

any thoughts on how to approach functional testing? I don't think it's realistic to actually test realtime consistency or anything with the way our CI is set up. However, there should be some basic validations we can test to ensure realtime guests bootup properly.

[jordigilh]

/test pull-kubevirt-e2e-k8s-1.22-sig-compute-realtime

[jordigilh]

/test pull-kubevirt-e2e-kind-1.19-sriov

[jordigilh]

@davidvossel @rmohr @vladikr @fabiand Development for this PR is completed. When you have time, can you take a look and let me know what you think? Thanks!

[jordigilh]

/test pull-kubevirt-e2e-k8s-1.22-sig-compute-realtime

[jordigilh]

/retest

[jordigilh]

/retest

[jordigilh]

/test pull-kubevirt-e2e-k8s-1.22-sig-compute-realtime

[vladikr]

/approve
This looks very good to me. Thank you @jordigilh for this effort!

[kubevirt-bot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: vladikr

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [vladikr]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[vladikr]

/lgtm

[jordigilh]

/test pull-kubevirt-e2e-k8s-1.20-sig-compute-nonroot

[vladikr]

@jordigilh perhaps you could also add a user-guide section about this.

[jordigilh]

/retest

[jordigilh]

/retest

[fabiand]

I can just second Vladik - Thank you very much for this contribution! And all the work that specifically went into testing it.

[fabiand]

@jordigilh did you also add a user-guide section for this new feature?
And a blog might also be a good thing to close this topic a little more

[jordigilh]

I can just second Vladik - Thank you very much for this contribution! And all the work that specifically went into testing it.

Thanks! it was a pleasure working in this project 😄

[jordigilh]

@jordigilh did you also add a user-guide section for this new feature? And a blog might also be a good thing to close this topic a little more

I plan to write a user-guide. I can also do the blog post as well 😄 . I'll PM you about it.

[jordigilh]

@jordigilh perhaps you could also add a user-guide section about this.

Yep, I'll work on that as soon as the other related PRs in project-infra and kubevirtci are merged 😄