v0.15.0

This is a new major release of NRI Reference Plugins. It includes major new features, bugfixes, and improvements to the testing infrastructure and coverage.

What's New:

Topology-Aware Policy

• DRA Devices, Claim Based Allocation:
  The topology-aware policy can allocate CPUs using Dynamic Resource Allocation Claims. With DRA enabled (config.dra.enabled: true in the Helm chart), the plugin registers the DRA driver topology-aware.nri.io, and the chart adds a DeviceClass of the same name. A pod claims CPUs of a NUMA node with a ResourceClaim. The scheduler picks a node with enough capacity, and the policy runs the containers of the pod on the claimed CPUs. Here is an example claim picking 3 CPUs from a single NUMA node:

apiVersion: resource.k8s.io/v1
kind: ResourceClaimTemplate
metadata:
  name: three-cpus
spec:
  spec:
    devices:
      requests:
      - name: cpus
        exactly:
          deviceClassName: topology-aware.nri.io
          selectors:
          - cel:
              expression: device.attributes["resource.kubernetes.io"].numaNode == 1
          capacity:
            requests:
              dra.cpu/cpu: "3"

A container which holds a claim runs on the CPUs of the claim. A container with a CPU request of its own runs on both its own CPUs and the claimed ones. A container which names only some requests of a claim runs on the CPUs of those requests.

Additionally, a claim, or its device class, can set the Linux memory policy of the containers holding the claim. The policy is given in a TopologyAwareClaimConfig in the opaque configuration of the claim. For instance, this claim takes one CPU of NUMA node 0 and one of NUMA node 1, and interleaves the memory of its containers over the two nodes:

apiVersion: resource.k8s.io/v1
kind: ResourceClaimTemplate
metadata:
  name: interleaved
spec:
  spec:
    devices:
      requests:
      - name: first
        exactly:
          deviceClassName: topology-aware.nri.io
          selectors:
          - cel:
              expression: device.attributes["resource.kubernetes.io"].numaNode == 0
      - name: second
        exactly:
          deviceClassName: topology-aware.nri.io
          selectors:
          - cel:
              expression: device.attributes["resource.kubernetes.io"].numaNode == 1
      config:
      - opaque:
          driver: topology-aware.nri.io
          parameters:
            apiVersion: config.nri/v1alpha1
            kind: TopologyAwareClaimConfig
            memoryPolicy:
              mode: interleave

For more information see the Topology-Aware policy documentation. For more information about DRA in general, see the Kubernetes DRA documentation

• topology-hinted IRQ claims: Topology hints can provide now IRQ affinity info for devices. An annotated irq-affinity allows now claiming IRQs of topology-hinted devices matched by a pattern.

Template Policy

• Sample DRA Devices, Claim Based Allocation:
  Although the template policy is a wireframe implementation which normally does not assign resources to containers, it can now allocate CPUs to DRA claims, so it can now be used as a starting point for creating new DRA-enabled policies. Here is a sample claim asking for a number of CPUs from the template policy:

apiVersion: resource.k8s.io/v1
kind: ResourceClaimTemplate
metadata:
  name: two-cpus
spec:
  spec:
    devices:
      requests:
      - name: cpus
        exactly:
          deviceClassName: template.nri.io
          capacity:
            requests:
              dra.cpu/cpu: "2"

For more information, see the template policy documentation

What's Changed

• DRA: template policy by @bart0sh in #817
• DRA: refresh node resources after claim changes by @bart0sh in #829
• DRA: publish topology-aware devices by @bart0sh in #820
• DRA: publish PCIe roots on topology-aware devices by @bart0sh in #831
• DRA: policy boundary by @bart0sh in #803
• DRA: pin topology-aware containers to their claims by @bart0sh in #823
• DRA: pin containers to the claim requests they hold by @bart0sh in #835
• DRA: pass memory policy settings in claim configs by @bart0sh in #857
• DRA: export claimed CPUs to topology-aware containers by @bart0sh in #834
• DRA: driver skeleton by @bart0sh in #788
• DRA: claim handling by @bart0sh in #806
• DRA: allocate topology-aware claims by @bart0sh in #821
• DRA: Preparing the DRA support for the release by @bart0sh in #858
• balloons: don't discover the system topology twice. by @klihub in #779
• topology-aware: share one test sysfs helper by @bart0sh in #827
• topology-aware: restore the config on a refused reconfiguration. by @bart0sh in #822
• topology-aware: fix TestAllocationMarshalling. by @klihub in #777
• topology-aware: eliminate dead code. by @klihub in #837
• topology-aware: add support for claiming device-hinted IRQs. by @klihub in #767
• topology: add PCIe chain and IRQ info to device hints by @kad in #761
• resctrl-mon: OTel telemetry via goresctrl pkg/monitor by @cmcantalupo in #757
• memory-policy: fix policies with no nodes by @bart0sh in #832
• resmgr: retain pending updates on nri failure by @bart0sh in #809
• resmgr: remove the CPU controller. by @klihub in #780
• resmgr: let policies reset their own cache entries. by @bart0sh in #818
• resmgr: stop the resource manager on shutdown by @bart0sh in #799
• config: fix scheduling flags enumeration. by @klihub in #850
• sysfs: return untyped nil for hardware which is not there. by @klihub in #778
• refactor: kubernetes client by @bart0sh in #782
• policy: drop SendEvent and policy events by @bart0sh in #824
• helm/topology-aware: add OpenShift SCC support by @klihub in #766
• helm, e2e: fix gating of the plugin test APIs. by @klihub in #784
• e2e: render the reports live by @klihub in #811
• e2e: publish the results of a run packed, and serve them by @klihub in #795
• e2e: increase minimum memory limit for crio by @askervin in #805
• e2e: improve nightly runner. by @klihub in #793
• e2e: generate the e2e reports in go by @klihub in #794
• e2e: fix verification of named pods. by @klihub in #851
• e2e: fix topology-aware test31-disable-disambiguation on k8scri=crio by @askervin in #808
• e2e: fix pod resolution, NUMA kernel restore and swap detection by @klihub in #791
• e2e: fix loaded image tagging on crio by @askervin in #804
• e2e: fix incorrect annotated topology level 'socket'. by @klihub in #815
• e2e: fix accidental addition of stray line breaking scheduling classes test. by @klihub in #862
• e2e: fast VM bootstrap using cached preprovisioned VM images ('boxes'). by @klihub in #769
• e2e: distill common test functionality to lib/test.bash. by @klihub in #768
• e2e: disambiguate reimported image on the host. by @klihub in #836
• e2e: containerd requires kernel CONFIG_FS_VERITY=y by @askervin in #798
• e2e: collect coverage data from e2e test runs by @klihub in #792
• e2e: build template plugin for e2e tests. by @klihub in #825
• e2e: allow quasi-polling from a cron job. by @klihub in #812
• e2e: allow batched runs of multiple e2e tests. by @klihub in #856
• e2e: add topology-aware coldstart test. by @klihub in #826
• e2e: add more topology-aware end-to-end tests. by @klihub in #833
• e2e: add an option to test a branch with our own tooling. by @klihub in #813
• e2e: add e2e-ish/replay support. by @klihub in #840
• e2e-ish: TestE2eishReplay for topology-aware and balloons. by @klihub in #841
• e2e-ish: restrict the cache directory of a launch. by @klihub in #866
• e2e-ish: add cpuclass infra dry-run support. by @klihub in #839
• e2e-runner: tear down and pack up a run reliably. by @klihub in #810
• cache: don't hand-quote cpuset.CPUSet.String()'s. by @klihub in #775
• build: extract licenses faster and cache them across builds. by @klihub in #854
• build: drop deprecated golint from the Makefile by @kad in #763
• fixes: apply a bunch of go fixes. by @klihub in #765
• scripts: kube-cgroups improvements by @askervin in #797
• skills: use cavemen as reference baseline target lengh for AI-generated commit messages. by @klihub in #861
• deps: bump deps with known issues. by @klihub in #845
• deps: bump Kubernetes to v0.34 and refresh module dependencies by @kad in #762
• .gitignore: ignore *.pyc, too. by @klihub in #772
• .gitignore: ignore pycache directories. by @klihub in #770
• .gitignore: ignore extracted testdata directories. by @klihub in #783
  Full Changelog: v0.14.0...v0.15.0