This repository is an automation/CI framework to test NVIDIA operators, the GPU Operator and Network Operator. This project is based on golang + ginkgo framework.

Golang and ginkgo versions based on versions specified in go.mod file.

The framework in this repository is designed to test NVIDIA's operators on a pre-installed OpenShift Container Platform (OCP) cluster which meets the following requirements:

• OCP cluster installed with version >=4.12

• Regular cluster 3 master nodes (VMs or BMs) and minimum of 2 workers (VMs or BMs)
• Single Node Cluster (VM or BM)
• Public Clouds Cluster (AWS, GCP and Azure) - For GPU Operator Only
• On Premise Cluster

• KUBECONFIG - Path to kubeconfig file.

• Logging with glog

We use glog library for logging. In order to enable verbose logging the following needs to be done:

1. Make sure to import inittool package in your go script, per this example:

2. Need to export the following SHELL variable:

export VERBOSE_LEVEL=100

1. The value for the variable has to be >= 100.
2. The variable can simply be exported in the shell where you run your automation.
3. The go file you work on has to be in a directory under github.com/rh-ecosystem-edge/nvidia-ci/tests/ directory for being able to import inittools.
4. Importing inittool also initializes the api client and it's available via "APIClient" variable.

• Collect logs from cluster with reporter

We use k8reporter library for collecting resource from cluster in case of test failure. In order to enable k8reporter the following needs to be done:

1. Export DUMP_FAILED_TESTS and set it to true. Use example below

export DUMP_FAILED_TESTS=true

2. Specify absolute path for logs directory like it appears below. By default /tmp/reports directory is used.

export REPORTS_DUMP_DIR=/tmp/logs_directory

The test-runner script is the recommended way for executing tests.

General Parameters for the script are controlled by the following environment variables:

• TEST_FEATURES: list of features to be tested. Subdirectories under tests dir that match a feature will be included (internal directories are excluded). When we have more than one subdirectory ot tests, they can be listed comma-separated.- required
• TEST_LABELS: ginkgo query passed to the label-filter option for including/excluding tests. Supports comma-separated labels (AND logic) and || operator (OR logic). Examples: 'nvidia-ci,gpu', 'nvidia-ci,mps', 'nvidia-ci,mig', 'deploy || rdma-legacy-sriov' - optional
• TEST_VERBOSE: executes ginkgo with verbose test output - optional
• TEST_TRACE: includes full stack trace from ginkgo tests when a failure occurs - optional
• VERBOSE_SCRIPT: prints verbose script information when executing the script - optional
• NO_COLOR: {true|anything else} when used, omits the coloring of logs that appear on beginning of the functions. However it does not affect on the coloring of the logs that ginkgo framework generates. - optional

NVIDIA GPU Operator-specific parameters for the script are controlled by the following environment variables:

• NVIDIAGPU_GPU_MACHINESET_INSTANCE_TYPE: Use only when OCP is on a public cloud, and when you need to scale the cluster to add a GPU-enabled compute node. If cluster already has a GPU enabled worker node, this variable should be unset.
  • Example instance type: "g4dn.xlarge" in AWS, or "a2-highgpu-1g" in GCP, or "Standard_NC4as_T4_v3" in Azure - required when need to scale cluster to add GPU node
• NVIDIAGPU_CATALOGSOURCE: custom catalogsource to be used. If not specified, the default "certified-operators" catalog is used - optional
• NVIDIAGPU_SUBSCRIPTION_CHANNEL: specific subscription channel to be used. If not specified, the latest channel is used - optional
• NVIDIAGPU_BUNDLE_IMAGE: GPU Operator bundle image to deploy with operator-sdk if NVIDIAGPU_DEPLOY_FROM_BUNDLE variable is set to true. Default value for bundle image if not set: ghcr.io/nvidia/gpu-operator/gpu-operator-bundle:main-latest - optional when deploying from bundlle
• NVIDIAGPU_DEPLOY_FROM_BUNDLE: boolean flag to deploy GPU operator from bundle image with operator-sdk - Default value is false - required when deploying from bundle
• NVIDIAGPU_SUBSCRIPTION_UPGRADE_TO_CHANNEL: specific subscription channel to upgrade to from previous version. required when running operator-upgrade testcase
• NVIDIAGPU_CLEANUP: boolean flag to cleanup up resources created by testcase after testcase execution - Default value is true - required only when cleanup is not needed. See the known issue note in Cleaning up leftover resources about automatic cleanup occasionally reporting a spurious NFD-related failure.
• NVIDIAGPU_GPU_FALLBACK_CATALOGSOURCE_INDEX_IMAGE: custom certified-operators catalogsource index image for GPU package - required when deploying fallback custom GPU catalogsource
• NVIDIAGPU_GPU_CLUSTER_POLICY_PATCH: a JSON patch to apply to a default cluster policy from ALM examples, written according to RFC 6902 (also see kubectl patch) - optional
• NVIDIAGPU_USE_PRECOMPILED_DRIVER: boolean flag to enable precompiled/signed driver testing. When set to true, the test discovers the latest precompiled driver version from registry.redhat.io/nvidia/gpu-driver-rhel9 and patches the ClusterPolicy to use it - Default value is false - optional
• NVIDIAGPU_PRECOMPILED_DRIVER_BRANCH: controls which precompiled driver branch(es) to test (only applies when NVIDIAGPU_USE_PRECOMPILED_DRIVER=true). When empty (default), tests the first discovered branch. Set to all to test all discovered branches (deduplicated by major version). Set to a comma-separated list (e.g. 580.178.04,595.91.07) to test specific branches. Each branch runs a full GPU burn validation - optional
• NFD_FALLBACK_CATALOGSOURCE_INDEX_IMAGE: custom redhat-operators catalogsource index image for NFD package - required when deploying fallback custom NFD catalogsource

See the Testing native DRA with GPU Operator section below for the "native-dra" testcase, which is part of the nvidiagpu feature/suite and reuses the NVIDIAGPU_* variables above (no separate variables of its own).

NVIDIA Network Operator-specific (NNO) parameters for the script are controlled by the following environment variables:

• NVIDIANETWORK_CATALOGSOURCE: custom catalogsource to be used. If not specified, the default "certified-operators" catalog is used - optional
• NVIDIANETWORK_SUBSCRIPTION_CHANNEL: specific subscription channel to be used. If not specified, the latest channel is used - optional
• NVIDIANETWORK_BUNDLE_IMAGE: Network Operator bundle image to deploy with operator-sdk if NVIDIANETWORK_DEPLOY_FROM_BUNDLE variable is set to true. Default value for bundle image if not set: TBD - optional when deploying from bundlle
• NVIDIANETWORK_DEPLOY_FROM_BUNDLE: boolean flag to deploy Network Operator from bundle image with operator-sdk - Default value is false - required when deploying from bundle
• NVIDIANETWORK_SUBSCRIPTION_UPGRADE_TO_CHANNEL: specific subscription channel to upgrade to from previous version. required when running operator-upgrade testcase
• NVIDIANETWORK_CLEANUP: boolean flag to cleanup up resources created by testcase after testcase execution - Default value is true - required only when cleanup is not needed
• NVIDIANETWORK_NNO_FALLBACK_CATALOGSOURCE_INDEX_IMAGE: custom certified-operators catalogsource index image for GPU package - required when deploying fallback custom NNO catalogsource
• NFD_FALLBACK_CATALOGSOURCE_INDEX_IMAGE: custom redhat-operators catalogsource index image for NFD package - required when deploying fallback custom NFD catalogsource
• NVIDIANETWORK_OFED_DRIVER_VERSION: OFED Driver Version. If not specified, the default driver version is used - optional
• NVIDIANETWORK_OFED_REPOSITORY: OFED Driver Repository. If not specified, the default repository is used - optional
• NVIDIANETWORK_RDMA_WORKLOAD_NAMESPACE: RDMA workload pod namespace - required
• NVIDIANETWORK_RDMA_LINK_TYPE Layer 2 link type, Infinband or Ethernet - required
• NVIDIANETWORK_RDMA_MLX_DEVICE: mlx5 device ID corresponding to the interface port connected to Spectrum or Infiniband switch - required
• NVIDIANETWORK_RDMA_CLIENT_HOSTNAME: RDMA Client hostname of first worker node for ib_write_bw test - required when running the RDMA testcase
• NVIDIANETWORK_RDMA_SERVER_HOSTNAME: RDMA Server hostname of second worker node for ib_write_bw test - required when running the RDMA testcase
• NVIDIANETWORK_RDMA_NETWORK_TYPE: RDMA network type, e.g. sriov, shared-device. Defaults to shared-device if not specified - required when running the RDMA testcase
• NVIDIANETWORK_RDMA_TEST_IMAGE: RDMA Test Container Image that runs the entrypoint.sh script with optional arguments specified in the pod spec. This container will clone the "https://github.com/linux-rdma/perftest" repo and builds the ib_write_bw binaries with or without cuda headers. It will also run the ib_write_bw command with arguments either in CLient or Server mode. Defaults to "quay.io/wabouham/ecosys-nvidia/rdma-tools:0.0.3" - optional
• NVIDIANETWORK_RDMA_SRIOV_NETWORK_NAME: sriovnetwork resource name - required when running the Legacy SRIOV RDMA testcase
• NVIDIANETWORK_MELLANOX_ETH_INTERFACE_NAME: Mellanox Ethernet Interface Name - Defaults to "ens8f0np0" if not specified - optional
• NVIDIANETWORK_MELLANOX_IB_INTERFACE_NAME: Mellanox Infiniband Interface Name - Defaults to "ens8f0np0" if not specified - optional
• NVIDIANETWORK_MACVLANNETWORK_NAME: MacvlanNetwork Custom Resource instance name - Defaults to name from Cluster Service Version alm-examples section if not specified - optional
• NVIDIANETWORK_MACVLANNETWORK_IPAM_RANGE: MacvlanNetwork Custom Resource instance IPAM or IP Address/Subnet mask range for Eth or IB interface - required
• NVIDIANETWORK_MACVLANNETWORK_IPAM_GATEWAY: MacvlanNetwork Custom Resource instance IPAM Default Gateway for specified ip address range - required
• NVIDIANETWORK_RDMA_GPUDIRECT: Boolean flag to run RDMA workload with 1 nvidia.com/gpu resource - optional

NVIDIA MIG parameters for the script are controlled by the following ginkgo parameters which are delivered as ARGS="-- [{parameter}...]" for the make run-tests (check the examples):

• --single.mig-profile=n, where n is typically a value of int type between 0-5. The parameter is used to choose the MIG profile from list of available MIG profiles (e.g. 1g.5gb is usually referenced with index 0). If not specified, a valid random number is used. Typically values 0-5. - optional
• --mixed.mig.instances=xxx, where xxx is a comma-separated string inside quotation marks (e.g. "2,0,1,1,0,0") The list of numbers represent how many instances are to be used for each profile when creating a pod. The first number indicates how many instances are to be used for the first profile etc. The instances of different profiles consume GPU slices in a different way. The name of the profile (e.g. 2g.10gb) describes the consumption of each instance (each instance would consume 2 slices and 10gb of memory). optional
• --mixed.mig.pod-delay=n, where n is a number in range 0 - 315 (seconds). In mixed MIG testcase there are usually more than 1 pod launched (depends on available GPU and mixed.mig.instances parameter). Since GPU workload is 300 seconds, this parameter can be used to control the delay between the pod launches so that the pods are running completely simultaneously, mostly overlapping (e.g. 15-80), slightly overlapping (e.g. 200-280 seconds), or non-overlapping (over 300 seconds). Values outside valid range are reset to closest limit (either 0 or 315). optional

To test the Multi-Process Service (MPS) functionality, you need to first deploy the GPU Operator and then run the MPS tests without cleaning up the GPU Operator deployment between test suites.

It is recommended to execute the runner script through the make run-tests make target.

1. First, deploy the GPU Operator with cleanup disabled for example:

$ export KUBECONFIG=/path/to/kubeconfig
$ export DUMP_FAILED_TESTS=true
$ export REPORTS_DUMP_DIR=/tmp/nvidia-ci-gpu-logs-dir
$ export TEST_FEATURES="nvidiagpu"
$ export TEST_LABELS='nvidia-ci,gpu'
$ export TEST_TRACE=true
$ export VERBOSE_LEVEL=100
$ export NVIDIAGPU_GPU_MACHINESET_INSTANCE_TYPE="g4dn.xlarge"
$ export NVIDIAGPU_CATALOGSOURCE="certified-operators"
$ export NVIDIAGPU_SUBSCRIPTION_CHANNEL="v23.9"
$ export NVIDIAGPU_CLEANUP=false  # Important: don't clean up after deployment
$ make run-tests

2. After the GPU Operator deployment completes successfully, run the MPS tests:

$ export TEST_FEATURES="mps"
$ export TEST_LABELS='nvidia-ci,mps'  # Run MPS-specific tests
$ make run-tests

The MPS tests will use the existing GPU Operator deployment that was left in place from the previous test run. This ensures that the MPS tests can properly validate MPS functionality on an already configured GPU environment.

The test framework ensures that the GPU Operator deployment tests run before MPS tests through Ginkgo's ordering mechanisms. If you need to add new MPS tests, make sure they are organized to run after the GPU Operator deployment by using proper labeling and ordering in your test files.

After completing the MPS tests, you may want to clean up all resources. Re-run the base nvidiagpu deploy testcase with NVIDIAGPU_CLEANUP=true:

$ export TEST_FEATURES="nvidiagpu"
$ export TEST_LABELS='nvidia-ci,gpu'
$ export NVIDIAGPU_CLEANUP=true
$ make run-tests

This removes NFD, the GPU Operator (namespace, Subscription, OperatorGroup, CSV, ClusterPolicy), and any leftover gpu-burn resources.

To test the Multi-Instance GPU (MIG) functionality, you need to first deploy the GPU Operator and then run the MIG tests without cleaning up the GPU Operator deployment between test suites.

It is recommended to execute the runner script through the make run-tests make target.

1. Run mig testcases (single-mig and mixed-mig) after nvidia-ci on any cluster, while selecting single.mig.profile=1

$ export KUBECONFIG=/path/to/kubeconfig
$ export DUMP_FAILED_TESTS=true
$ export REPORTS_DUMP_DIR=/tmp/nvidia-ci-gpu-logs-dir
$ export TEST_FEATURES="nvidiagpu"
$ export TEST_LABELS='nvidia-ci,gpu,single-mig,mixed-mig'
$ export TEST_TRACE=true
$ export VERBOSE_LEVEL=100
$ export NVIDIAGPU_CLEANUP=false
$ make run-tests ARGS="-- --single.mig.profile=1"

2. Running only MIG testcases on an existing cluster which has GPU operator installed, e.g. after executing step 1. MIG testcase(s) can be used from either nvidiagpu or mig package. MIG is used in this example. In the other case, use TEST_FEATURES="nvidiagpu" to execute the testcase from nvidiagpu package. With these MIG parameters single-mig testcase would choose a random MIG profile (as the single.mig.profile is not included) , mixed-mig testcase would use 1 instance amount for A100 GPU (1x 1g.5gb, 1x 2g.10gb and 1x 3g.20gb, leaving the second profile 1g.10gb unused). mixed-mig testcase would wait 35 seconds between the pods launching with mixed.mig.pod-delay parameter You can deliver the ginkgo cli parameter using ARGS after the "make run-tests"

$ export KUBECONFIG=/path/to/kubeconfig
$ export DUMP_FAILED_TESTS=true
$ export REPORTS_DUMP_DIR=/tmp/nvidia-ci-gpu-logs-dir
$ export TEST_FEATURES="mig"
$ export TEST_LABELS='single-mig,mixed-mig'
$ export TEST_TRACE=true
$ export VERBOSE_LEVEL=100
$ export NVIDIAGPU_CLEANUP=false
$ make run-mig-tests ARGS="-- --mixed.mig.instances='1,0,1,1' --mixed.mig.pod-delay=35"

If the GPU operator and gpu burn pod need to be cleaned up, just set NVIDIAGPU_CLEANUP=true in the last execution of either steps 1 or 2.

To test GPU time-slicing, deploy the GPU Operator first, then run the time-slicing tests. The test creates a device plugin ConfigMap with time-slicing configuration, deploys a ClusterPolicy referencing it, and validates that multiple CUDA workloads run concurrently on a single time-sliced GPU.

1. Deploy the GPU Operator with cleanup disabled (same as MPS step 1 above)
2. Run the time-slicing tests:

$ export TEST_FEATURES="timeslicing"
$ export TEST_LABELS='nvidia-ci,timeslicing'
$ make run-tests

Starting with GPU Operator 26.7.0, DRA (Dynamic Resource Allocation) enablement can be managed natively by the operator through two new custom resources, NVIDIADriver and GPUCluster, instead of ClusterPolicy. The native-dra testcase (in the nvidiagpu feature/suite, tests/nvidiagpu) deploys the GPU Operator and validates this stack.

It is a self-contained alternative to the base "Deploy NVIDIA GPU Operator with DTK" testcase above — the two are mutually exclusive ways of installing/configuring the same GPU Operator (ClusterPolicy is never created by native-dra, and vice versa) — so run it with only the native-dra label selected, not together with the gpu label. Unlike the pre-release DRA suites (tests/dra/gpuallocation, tests/dra/computedomain), which assume a ClusterPolicy is already deployed and install the DRA driver separately via a Helm chart, native-dra lets the GPU Operator manage the DRA driver itself.

1. Installs NFD, the same way the base testcase does.
2. Installs the GPU Operator via OLM (namespace, OperatorGroup, Subscription, waits for the operator Deployment and its CSV to succeed) — reusing the same NVIDIAGPU_CATALOGSOURCE/ NVIDIAGPU_SUBSCRIPTION_CHANNEL variables as the base testcase.
3. Checks whether the installed GPU Operator version serves the GPUCluster CRD. If it doesn't (i.e. the installed version is < 26.7.0), the test is skipped.
4. Creates a minimal NVIDIADriver and a minimal GPUCluster custom resource, built directly from the CSV's alm-examples (the operator's own suggested minimal samples), and waits for both to reach the ready state.
5. Validates GPU functionality by executing nvidia-smi inside the driver pod(s) rendered by NVIDIADriver.

$ export KUBECONFIG=/path/to/kubeconfig
$ export DUMP_FAILED_TESTS=true
$ export REPORTS_DUMP_DIR=/tmp/nvidia-ci-gpu-logs-dir
$ export TEST_FEATURES="nvidiagpu"
$ export TEST_LABELS='native-dra'
$ export TEST_TRACE=true
$ export VERBOSE_LEVEL=100
$ export NVIDIAGPU_CATALOGSOURCE="certified-operators"
$ export NVIDIAGPU_SUBSCRIPTION_CHANNEL="v26.7"  # must resolve to GPU Operator >= 26.7.0
$ export NVIDIAGPU_CLEANUP=false
$ make run-tests

By default (NVIDIAGPU_CLEANUP=true), the GPUCluster, NVIDIADriver, CSV, Subscription, OperatorGroup, GPU Operator namespace and NFD are all removed at the end of the run. To leave them in place for inspection (e.g. to manually inspect the rendered DRA driver pods, or debug a failure), set NVIDIAGPU_CLEANUP=false (same variable used by the base testcase). See Cleaning up leftover resources below.

If you ran any nvidiagpu testcase with NVIDIAGPU_CLEANUP=false (e.g. to chain into MPS/MIG/ time-slicing, or to leave resources in place for debugging) and now want to remove everything, re-run the same testcase with NVIDIAGPU_CLEANUP=true. If the GPU Operator/NFD are already installed and ready, this completes quickly (no full redeployment) and then runs the same AfterAll cleanup a normal run performs, removing NFD, the GPU Operator (namespace, Subscription, OperatorGroup, CSV, ClusterPolicy and/or NVIDIADriver/GPUCluster, whichever was deployed), and any leftover gpu-burn resources.

Known issue: AfterAll's automatic cleanup (triggered when NVIDIAGPU_CLEANUP=true, the default) can occasionally report a spurious failure like Error cleaning up NFD resources: failed to delete NFD CR: NodeFeatureDiscovery object nfd-instance doesn't exist in namespace openshift-nfd — this is a timing race in NFD's own delete-and-wait cleanup logic (pkg/nfd/deploynfd.go), not an actual test failure; the resource in question was in fact successfully deleted (tracked for a separate PR). If you see this, simply re-run the same command again to confirm cleanup completed.

Example running the end-to-end GPU Operator test case:

$ export KUBECONFIG=/path/to/kubeconfig
$ export DUMP_FAILED_TESTS=true
$ export REPORTS_DUMP_DIR=/tmp/nvidia-ci-gpu-logs-dir
$ export TEST_FEATURES="nvidiagpu"
$ export TEST_LABELS='nvidia-ci,gpu,single-mig,mixed-mig'
$ export TEST_TRACE=true
$ export VERBOSE_LEVEL=100
$ export NVIDIAGPU_GPU_MACHINESET_INSTANCE_TYPE="g4dn.xlarge"
$ export NVIDIAGPU_CATALOGSOURCE="certified-operators"
$ export NVIDIAGPU_SUBSCRIPTION_CHANNEL="v23.9"
$ make run-tests
Executing nvidiagpu test-runner script
scripts/test-runner.sh
ginkgo -timeout=24h --keep-going --require-suite -r -vv --trace --label-filter="nvidia-ci,gpu,single-mig,mixed-mig" ./tests/nvidiagpu

Example running the GPU Operator upgrade testcase (from v23.6 to v24.3) after the end-end testcase. Note: you must run the end-to-end testcase first to deploy a previous version, set NVIDIAGPU_CLEANUP=false, and specify the channel to upgrade to NVIDIAGPU_SUBSCRIPTION_UPGRADE_TO_CHANNEL=v24.3, along with the label 'operator-upgrade' in TEST_LABELS. Otherwise, the upgrade testcase will not be executed:

$ export KUBECONFIG=/path/to/kubeconfig
$ export DUMP_FAILED_TESTS=true
$ export REPORTS_DUMP_DIR=/tmp/nvidia-ci-gpu-logs-dir
$ export TEST_FEATURES="nvidiagpu"
$ export TEST_LABELS='nvidia-ci,gpu,operator-upgrade'
$ export TEST_TRACE=true
$ export VERBOSE_LEVEL=100
$ export NVIDIAGPU_GPU_MACHINESET_INSTANCE_TYPE="g4dn.xlarge"
$ export NVIDIAGPU_CATALOGSOURCE="certified-operators"
$ export NVIDIAGPU_SUBSCRIPTION_CHANNEL="v23.9"
$ export NVIDIAGPU_SUBSCRIPTION_UPGRADE_TO_CHANNEL=v24.3
$ export NVIDIAGPU_CLEANUP=false
$ make run-tests
Executing nvidiagpu test-runner script
scripts/test-runner.sh
ginkgo -timeout=24h --keep-going --require-suite -r -vv --trace --label-filter="nvidia-ci,gpu,operator-upgrade" ./tests/nvidiagpu

Example running the end-to-end test case and creating custom catalogsources for NFD and GPU Operator packagmanifests when missing from their default catalogsources.

$ export KUBECONFIG=/path/to/kubeconfig
$ export DUMP_FAILED_TESTS=true
$ export REPORTS_DUMP_DIR=/tmp/nvidia-gpu-ci-logs-dir
$ export TEST_FEATURES="nvidiagpu"
$ export TEST_LABELS='nvidia-ci,gpu'
$ export TEST_TRACE=true
$ export VERBOSE_LEVEL=100
$ export NVIDIAGPU_GPU_MACHINESET_INSTANCE_TYPE="g4dn.xlarge"
$ export NVIDIAGPU_GPU_FALLBACK_CATALOGSOURCE_INDEX_IMAGE="registry.redhat.io/redhat/certified-operator-index:v4.16"
$ export NFD_FALLBACK_CATALOGSOURCE_INDEX_IMAGE="registry.redhat.io/redhat/redhat-operator-index:v4.17"
$ make run-tests

Example running the end-to-end Network Operator test case, with the Legacy SRIOV RDMA testcase.
Note: both TEST_LABELS "deploy || rdma-legacy-sriov' are specified in examples below:

$ export KUBECONFIG=/path/to/kubeconfig
$ export DUMP_FAILED_TESTS=true
$ export REPORTS_DUMP_DIR=/tmp/nvidia-nno-ci-logs-dir
$ export TEST_FEATURES="nvidianetwork"
# To run the NNO deploy testcase followed by RDMA Shared Device testcase,
# set: export TEST_LABELS="deploy || rdma-shared-dev"
$ export TEST_LABELS="deploy || rdma-legacy-sriov"
$ export TEST_TRACE=true
$ export VERBOSE_LEVEL=100
$ export NVIDIANETWORK_CATALOGSOURCE="certified-operators"
$ export NVIDIANETWORK_SUBSCRIPTION_CHANNEL="v24.7"
$ export NVIDIANETWORK_NNO_FALLBACK_CATALOGSOURCE_INDEX_IMAGE="registry.redhat.io/redhat/certified-operator-index:v4.17"
$ export NFD_FALLBACK_CATALOGSOURCE_INDEX_IMAGE="registry.redhat.io/redhat/redhat-operator-index:v4.17"
$ export NVIDIANETWORK_OFED_DRIVER_VERSION="25.01-0.6.0.0-0"
$ export NVIDIANETWORK_OFED_REPOSITORY="quay.io/wabouham/ecosys-nvidia"
$ export NVIDIANETWORK_RDMA_CLIENT_HOSTNAME=nvd-srv-3.nvidia.eng.redhat.com
$ export NVIDIANETWORK_RDMA_SERVER_HOSTNAME=nvd-srv-2.nvidia.eng.redhat.com
$ export NVIDIANETWORK_MACVLANNETWORK_IPAM_RANGE=192.168.2.0/24
$ export NVIDIANETWORK_MACVLANNETWORK_IPAM_GATEWAY=192.168.2.1
$ export NVIDIANETWORK_DEPLOY_FROM_BUNDLE=true
$ export NVIDIANETWORK_BUNDLE_IMAGE="nvcr.io/.../network-operator-bundle:v25.1.0-rc.2"
$ export NVIDIANETWORK_MELLANOX_ETH_INTERFACE_NAME="ens8f0np0"
$ export NVIDIANETWORK_MELLANOX_IB_INTERFACE_NAME="ibs2f0"
$ export NVIDIANETWORK_MACVLANNETWORK_NAME="rdmashared-net"
$ export NVIDIANETWORK_RDMA_WORKLOAD_NAMESPACE="default"
$ export NVIDIANETWORK_RDMA_LINK_TYPE="ethernet"
$ export NVIDIANETWORK_RDMA_MLX_DEVICE="mlx5_2"
$ export NVIDIANETWORK_RDMA_GPUDIRECT=true
# NVIDIANETWORK_RDMA_NETWORK_TYPE supported values are: "sriov", "shared-device"
$ export NVIDIANETWORK_RDMA_NETWORK_TYPE=sriov


$ make run-tests
Executing nvidiagpu test-runner script
scripts/test-runner.sh
ginkgo -timeout=24h --keep-going --require-suite -r -vv --trace --label-filter="deploy || rdma-legacy-sriov" ./tests/nvidianetwork

Separate from the Ginkgo test suites above, scripts/apply-manifests.sh applies (or deletes) a directory of plain YAML manifests against an already-installed OpenShift cluster, via the make apply-manifests / make delete-manifests targets. This is intended for one-off, day-2 cluster configuration (e.g. worker MachineConfigs, ServiceAccounts) that isn't part of a Ginkgo test flow.

• KUBECONFIG - Path to kubeconfig file - required (same variable used everywhere else in this repo)
• MANIFEST_DIR - Directory containing the *.yaml/*.yml files to apply/delete - Defaults to manifests/nno - optional
• MANIFEST_VARS - Space-separated list of $VAR tokens (envsubst's own filter syntax) restricting variable substitution to only those variables. If unset, every variable present in the environment is eligible for substitution in every manifest - optional
• WAIT_FOR_WORKER_MCP - {true|false} - When true and the action is apply, waits for the worker MachineConfigPool to finish rolling out after each manifest is applied, before moving on to the next one. make apply-manifests defaults this to true (since the default MANIFEST_DIR=manifests/nno contains MachineConfigs that reboot workers); the underlying script itself defaults to false - optional
• WORKER_MCP_NAME - Name of the MachineConfigPool to wait on when WAIT_FOR_WORKER_MCP=true - Defaults to worker - optional
• WORKER_MCP_ROLLOUT_START_TIMEOUT - How long to wait for the pool to start updating before giving up and moving on (tolerant: a manifest that doesn't touch worker MachineConfigs, like a ServiceAccount, never makes the pool start updating, so this timeout is expected to elapse harmlessly for those files) - Defaults to 2m - optional
• WORKER_MCP_ROLLOUT_TIMEOUT - How long to wait for the pool to finish updating once it starts (fatal: the whole run aborts if this elapses) - Defaults to 40m - optional

• Ordering: files are applied in lexical filename order (reverse order on delete), so prefix files with numbers (00-, 10-, 20-, ...) to control ordering when one manifest depends on another (e.g. a namespace before objects created in it).
• Templating: manifests may contain ${VAR} / $VAR placeholders, substituted from the current environment via envsubst before being applied.
• Required variables (<name>.env-required): if a sibling file with the same base name and a .env-required extension exists next to a manifest (e.g. 10-foo.yaml -> 10-foo.env-required, one variable name per line, blank lines and #-comments ignored), the script verifies every listed variable is set and non-empty before rendering that manifest, failing fast instead of letting envsubst silently substitute an empty string for a variable nobody remembered to export.
• Post-apply hooks (<name>.sh): if a sibling file with the same base name and a .sh extension exists next to a manifest (e.g. 20-foo.yaml -> 20-foo.sh), it is executed immediately after that manifest is applied (and after any WAIT_FOR_WORKER_MCP wait completes). This is how manifest-specific follow-up commands are attached without hardcoding them into the generic script.

• 00-doca1-99-machine-config-blacklist-irdma.yaml - MachineConfig (role worker) that blacklists the irdma kernel module.
• 10-doca1-99-machine-config-udev-network.yaml - MachineConfig (role worker) that installs a udev rules file mapping specific NIC MAC addresses to interface names (ib_rdma0/ eth_rdma0). The rules content is parameterized as base64 in contents.source, so DOCA1_UDEV_NETWORK_RULES_BASE64 must be exported before running make apply-manifests (enforced by the sibling 10-doca1-99-machine-config-udev-network.env-required file) - required
• 20-doca1-rdma-service-account-default-namespace.yaml - ServiceAccount named rdma in the default namespace.
• 21-doca1-rdma-privileged-scc.yaml - RoleBinding in the default namespace that lets that ServiceAccount use the privileged SCC. It references ClusterRole system:openshift:scc:privileged and is applied after the ServiceAccount. It is a separate binding from the shared system:openshift:scc:privileged RoleBinding, so deleting it removes only this grant.

$ export KUBECONFIG=/path/to/kubeconfig
$ export DOCA1_UDEV_NETWORK_RULES_BASE64="$(base64 <<'EOF' | tr -d '\n'
SUBSYSTEM=="net",ACTION=="add",ATTR{address}=="<mac>",ATTR{type}=="1",NAME="ib_rdma0"
SUBSYSTEM=="net",ACTION=="add",ATTR{address}=="<mac>",ATTR{type}=="1",NAME="eth_rdma0"
EOF
)"
$ make apply-manifests

The two MachineConfig files each trigger a worker MachineConfigPool rollout (cordon/drain/ reboot/uncordon per node); with the default WAIT_FOR_WORKER_MCP=true, make apply-manifests waits for that to finish after each one before moving on. The rdma ServiceAccount is applied next, then 21-doca1-rdma-privileged-scc.yaml grants it the privileged SCC.

To remove everything again:

$ export KUBECONFIG=/path/to/kubeconfig
$ make delete-manifests

delete-manifests does not wait on the MachineConfigPool or run any hooks. It removes the manifests/nno files themselves, in reverse order, so the privileged SCC RoleBinding is deleted before the rdma ServiceAccount.