diff --git a/_topic_maps/_topic_map.yml b/_topic_maps/_topic_map.yml index a48553599ff9..b84de807a9b8 100644 --- a/_topic_maps/_topic_map.yml +++ b/_topic_maps/_topic_map.yml @@ -3392,6 +3392,8 @@ Topics: File: cnf-understanding-low-latency - Name: Tuning nodes for low latency with the performance profile File: cnf-tuning-low-latency-nodes-with-perf-profile +- Name: Tuning hosted control planes for low latency with the performance profile + File: cnf-tuning-low-latency-hosted-cp-nodes-with-perf-profile - Name: Provisioning real-time and low latency workloads File: cnf-provisioning-low-latency-workloads - Name: Debugging low latency tuning diff --git a/modules/apply-performance-profile-hosted-cluster.adoc b/modules/apply-performance-profile-hosted-cluster.adoc new file mode 100644 index 000000000000..ef61cc0d1a69 --- /dev/null +++ b/modules/apply-performance-profile-hosted-cluster.adoc @@ -0,0 +1,184 @@ +// Module included in the following assemblies: +// +// * scalability_and_performance/cnf-tuning-low-latency-hosted-cp-nodes-with-perf-profile.adoc + +:_mod-docs-content-type: PROCEDURE +[id="apply-performance-profile-hosted-cluster_{context}"] += Configuring low-latency tuning in a hosted cluster + +To set low latency with the performance profile on the nodes in your hosted cluster, you can use the Node Tuning Operator. In {hcp}, you can configure low-latency tuning by creating config maps that contain `Tuned` objects and referencing those config maps in your node pools. The tuned object in this case is a `PerformanceProfile` object that defines the performance profile you want to apply to the nodes in a node pool. + +.Procedure + +. Export the management cluster `kubeconfig` file by running the following command: ++ +[source,terminal] +---- +$ export MGMT_KUBECONFIG= +---- + +. Create the `ConfigMap` object in the management cluster by running the following command: ++ +[source,terminal] +---- +$ oc --kubeconfig="$MGMT_KUBECONFIG" apply -f my-hosted-cp-performance-profile.yaml +---- + +. Edit the `NodePool` object in the `clusters` namespace adding the `spec.tuningConfig` field and the name of the created performance profile in that field by running the following command: ++ +[source,terminal] +---- +$ oc edit np -n clusters +---- ++ +[source,yaml] +---- +apiVersion: hypershift.openshift.io/v1beta1 +kind: NodePool +metadata: + annotations: + hypershift.openshift.io/nodePoolCurrentConfig: 2f752a2c + hypershift.openshift.io/nodePoolCurrentConfigVersion: 998aa3ce + hypershift.openshift.io/nodePoolPlatformMachineTemplate: democluster-us-east-1a-3dff55ec + creationTimestamp: "2025-04-09T09:41:55Z" + finalizers: + - hypershift.openshift.io/finalizer + generation: 1 + labels: + hypershift.openshift.io/auto-created-for-infra: democluster + name: democluster-us-east-1a + namespace: clusters + ownerReferences: + - apiVersion: hypershift.openshift.io/v1beta1 + kind: HostedCluster + name: democluster + uid: af77e390-c289-433c-9d29-3aee8e5dc76f + resourceVersion: "53056" + uid: 11efa47c-5a7b-476c-85cf-a274f748a868 +spec: + tuningConfig: + - name: performance + arch: amd64 + clusterName: democluster + management: +---- ++ +[NOTE] +==== +You can reference the same profile in multiple node pools. In {hcp}, the Node Tuning Operator appends a hash of the node pool name and namespace to the name of the `Tuned` custom resources to distinguish them. After you make the changes, the system detects that a configuration change is required and starts a rolling update of the nodes in that pool to apply the new configuration. +==== + +.Verification + +. List all node pools across all namespaces by running the following command: ++ +[source,terminal] +---- +$ oc --kubeconfig="$MGMT_KUBECONFIG" get np -A +---- ++ +.Example output +[source,terminal] +---- +NAMESPACE NAME CLUSTER DESIRED NODES CURRENT NODES AUTOSCALING AUTOREPAIR VERSION UPDATINGVERSION UPDATINGCONFIG MESSAGE +clusters democluster-us-east-1a democluster 1 1 False False 4.17.0 False True +---- ++ +[NOTE] +==== +The `UPDATINGCONFIG` field indicates whether the node pool is in the process of updating its configuration. During this update, the `UPDATINGCONFIG` field in the node pool's status becomes `True`. The new configuration is considered fully applied only when the `UPDATINGCONFIG` field returns to `False`. +==== + +. List all config maps in the `clusters-democluster` namespace by running the following command: ++ +[source,terminal] +---- +$ oc --kubeconfig="$MGMT_KUBECONFIG" get cm -n clusters-democluster +---- ++ +.Example output +[source,terminal] +---- +NAME DATA AGE +aggregator-client-ca 1 69m +auth-config 1 68m +aws-cloud-config 1 68m +aws-ebs-csi-driver-trusted-ca-bundle 1 66m +... 1 67m +kubelet-client-ca 1 69m +kubeletconfig-performance-democluster-us-east-1a 1 22m +... +ovnkube-identity-cm 2 66m +performance-democluster-us-east-1a 1 22m +... +tuned-performance-democluster-us-east-1a 1 22m +---- ++ +The output shows a kubeletconfig `kubeletconfig-performance-democluster-us-east-1a` and a performance profile `performance-democluster-us-east-1a` has been created. The Node Tuning Operator syncs the `Tuned` objects into the hosted cluster. You can verify which `Tuned` objects are defined and which profiles are applied to each node. + +. List available secrets on the management cluster by running the following command: ++ +[source,terminal] +---- +$ oc get secrets -n clusters +---- ++ +.Example output +[source,terminal] +---- +NAME TYPE DATA AGE +builder-dockercfg-25qpp kubernetes.io/dockercfg 1 128m +default-dockercfg-mkvlz kubernetes.io/dockercfg 1 128m +democluster-admin-kubeconfig Opaque 1 127m +democluster-etcd-encryption-key Opaque 1 128m +democluster-kubeadmin-password Opaque 1 126m +democluster-pull-secret Opaque 1 128m +deployer-dockercfg-8lfpd kubernetes.io/dockercfg 1 128m +---- + +. Extract the `kubeconfig` file for the hosted cluster by running the following command: ++ +[source,terminal] +---- +$ oc get secret -n clusters -o jsonpath='{.data.kubeconfig}' | base64 -d > hosted-cluster-kubeconfig +---- ++ +.Example +[source,terminal] +---- +$ oc get secret democluster-admin-kubeconfig -n clusters -o jsonpath='{.data.kubeconfig}' | base64 -d > hosted-cluster-kubeconfig +---- + +. Export the hosted cluster kubeconfig by running the following command: ++ +[source,terminal] +---- +$ export HC_KUBECONFIG= +---- + +. Verify that the kubeletconfig is mirrored in the hosted cluster by running the following command: ++ +[source,terminal] +---- +$ oc --kubeconfig="$HC_KUBECONFIG" get cm -n openshift-config-managed | grep kubelet +---- ++ +.Example output +[source,terminal] +---- +kubelet-serving-ca 1 79m +kubeletconfig-performance-democluster-us-east-1a 1 15m +---- + +. Verify that the `single-numa-node` policy is set on the hosted cluster by running the following command: ++ +[source,terminal] +---- +$ oc --kubeconfig="$HC_KUBECONFIG" get cm kubeletconfig-performance-democluster-us-east-1a -o yaml -n openshift-config-managed | grep single +---- ++ +.Example output +[source,terminal] +---- + topologyManagerPolicy: single-numa-node +---- diff --git a/modules/cnf-gathering-data-about-hosted-cluster-using-must-gather.adoc b/modules/cnf-gathering-data-about-hosted-cluster-using-must-gather.adoc new file mode 100644 index 000000000000..9a801b3b6613 --- /dev/null +++ b/modules/cnf-gathering-data-about-hosted-cluster-using-must-gather.adoc @@ -0,0 +1,105 @@ +// Module included in the following assemblies: +// +// * scalability_and_performance/low_latency_tuning/cnf-tuning-low-latency-nodes-with-perf-profile.adoc + +:_mod-docs-content-type: PROCEDURE +[id="gathering-data-about-your-hosted-cluster-using-must-gather_{context}"] += Gathering data about your hosted control planes cluster for the PPC + +The Performance Profile Creator (PPC) tool requires `must-gather` data. As a cluster administrator, run the `must-gather` command to capture information about your cluster. + +.Prerequisites + +* You have `cluster-admin` role access to the management cluster. +* You installed the {oc-first}. + +.Procedure + +. Export the management cluster `kubeconfig` file by running the following command: ++ +[source,terminal] +---- +$ export MGMT_KUBECONFIG= +---- + +. List all node pools across all namespaces by running the following command: ++ +[source,terminal] +---- +$ oc --kubeconfig="$MGMT_KUBECONFIG" get np -A +---- ++ +.Example output +[source,terminal] +---- +NAMESPACE NAME CLUSTER DESIRED NODES CURRENT NODES AUTOSCALING AUTOREPAIR VERSION UPDATINGVERSION UPDATINGCONFIG MESSAGE +clusters democluster-us-east-1a democluster 1 1 False False 4.17.0 False True +---- ++ +* The output shows the namespace `clusters` in the management cluster where the `NodePool` resource is defined. +* The name of the `NodePool` resource, for example `democluster-us-east-1a`. +* The `HostedCluster` this `NodePool` belongs to. For example, `democluster`. + +. On the management cluster, run the following command to list available secrets: ++ +[source,terminal] +---- +$ oc get secrets -n clusters +---- ++ +.Example output +[source,terminal] +---- +NAME TYPE DATA AGE +builder-dockercfg-25qpp kubernetes.io/dockercfg 1 128m +default-dockercfg-mkvlz kubernetes.io/dockercfg 1 128m +democluster-admin-kubeconfig Opaque 1 127m +democluster-etcd-encryption-key Opaque 1 128m +democluster-kubeadmin-password Opaque 1 126m +democluster-pull-secret Opaque 1 128m +deployer-dockercfg-8lfpd kubernetes.io/dockercfg 1 128m +---- + +. Extract the `kubeconfig` file for the hosted cluster by running the following command: ++ +[source,terminal] +---- +$ oc get secret -n -o jsonpath='{.data.kubeconfig}' | base64 -d > hosted-cluster-kubeconfig +---- ++ +.Example +[source,terminal] +---- +$ oc get secret democluster-admin-kubeconfig -n clusters -o jsonpath='{.data.kubeconfig}' | base64 -d > hosted-cluster-kubeconfig +---- + +. To create a `must-gather` bundle for the hosted cluster, open a separate terminal window and run the following commands: + +.. Export the hosted cluster `kubeconfig` file: ++ +[source,terminal] +---- +$ export HC_KUBECONFIG= +---- ++ +.Example +[source,terminal] +---- +$ export HC_KUBECONFIG=~/hostedcpkube/hosted-cluster-kubeconfig +---- + +.. Navigate to the directory where you want to store the `must-gather` data. + +.. Gather the troubleshooting data for your hosted cluster: ++ +[source,terminal] +---- +$ oc --kubeconfig="$HC_KUBECONFIG" adm must-gather +---- + +.. Create a compressed file from the `must-gather` directory that was just created in your working directory. For example, on a computer that uses a Linux operating system, run the following command: ++ +[source,terminal] +---- +$ tar -czvf must-gather.tar.gz must-gather.local.1203869488012141147 +---- \ No newline at end of file diff --git a/modules/cnf-running-the-performance-creator-profile-hosted.adoc b/modules/cnf-running-the-performance-creator-profile-hosted.adoc new file mode 100644 index 000000000000..d56643570210 --- /dev/null +++ b/modules/cnf-running-the-performance-creator-profile-hosted.adoc @@ -0,0 +1,127 @@ +// Module included in the following assemblies: +// +// * scalability_and_performance/low_latency_tuning/cnf-tuning-low-latency-nodes-with-perf-profile.adoc + +:_mod-docs-content-type: PROCEDURE +[id="running-the-performance-profile-profile-hosted-cluster-using-podman_{context}"] += Running the Performance Profile Creator on a hosted cluster using Podman + +As a cluster administrator, you can use Podman with the Performance Profile Creator (PPC) tool to create a performance profile. + +For more information about PPC arguments, see "Performance Profile Creator arguments". + +The PPC tool is designed to be hosted-cluster aware. When it detects a hosted cluster from the `must-gather` data it automatically takes the following actions: + +* Recognizes that there is no machine config pool (MCP). +* Uses node pools as the source of truth for compute node configurations instead of MCPs. +* Does not require you to specify the `node-pool-name` value explicitly unless you want to target a specific pool. + +[IMPORTANT] +==== +The PPC uses the `must-gather` data from your hosted cluster to create the performance profile. If you make any changes to your cluster, such as relabeling a node targeted for performance configuration, you must re-create the `must-gather` data before running PPC again. +==== + +.Prerequisites + +* Access to the cluster as a user with the `cluster-admin` role. +* A hosted cluster is installed. +* Installation of Podman and the {oc-first}. +* Access to the Node Tuning Operator image. +* Access to the `must-gather` data for your cluster. + +.Procedure + +. On the hosted cluster, use Podman to authenticate to `registry.redhat.io` by running the following command: ++ +[source,terminal] +---- +$ podman login registry.redhat.io +---- ++ +[source,bash] +---- +Username: +Password: +---- + +. Create a performance profile on the hosted cluster, by running the following command. The example uses sample PPC arguments and values: ++ +[source,terminal,subs="attributes+"] +---- +$ podman run --entrypoint performance-profile-creator \ + -v /path/to/must-gather:/must-gather:z \// <1> + registry.redhat.io/openshift4/ose-cluster-node-tuning-rhel9-operator:v{product-version} \ + --must-gather-dir-path /must-gather \ + --reserved-cpu-count=2 \// <2> + --rt-kernel=false \// <3> + --split-reserved-cpus-across-numa=false \ <4> + --topology-manager-policy=single-numa-node \// <5> + --node-pool-name=democluster-us-east-1a \ + --power-consumption-mode=ultra-low-latency \// <6> + --offlined-cpu-count=1 \// <7> + > my-hosted-cp-performance-profile.yaml +---- ++ +<1> Mounts the local directory where the output of an `oc adm must-gather` was created into the container. +<2> Specifies two reserved CPUs. +<3> Disables the real-time kernel. +<4> Disables reserved CPUs splitting across NUMA nodes. +<5> Specifies the NUMA topology policy. If installing the NUMA Resources Operator, this must be set to `single-numa-node`. +<6> Specifies minimal latency at the cost of increased power consumption. +<7> Specifies one offlined CPU. ++ +.Example output +[source,terminal] +---- +level=info msg="Nodes names targeted by democluster-us-east-1a pool are: ip-10-0-129-110.ec2.internal " +level=info msg="NUMA cell(s): 1" +level=info msg="NUMA cell 0 : [0 2 1 3]" +level=info msg="CPU(s): 4" +level=info msg="2 reserved CPUs allocated: 0,2 " +level=info msg="1 isolated CPUs allocated: 1" +level=info msg="Additional Kernel Args based on configuration: [] +---- + +. Review the created YAML file by running the following command: ++ +[source,terminal] +---- +$ cat my-hosted-cp-performance-profile +---- +.Example output ++ +[source,yaml] +---- +--- +apiVersion: v1 +data: + tuning: | + apiVersion: performance.openshift.io/v2 + kind: PerformanceProfile + metadata: + creationTimestamp: null + name: performance + spec: + cpu: + isolated: "1" + offlined: "3" + reserved: 0,2 + net: + userLevelNetworking: false + nodeSelector: + node-role.kubernetes.io/worker: "" + numa: + topologyPolicy: single-numa-node + realTimeKernel: + enabled: false + workloadHints: + highPowerConsumption: true + perPodPowerManagement: false + realTime: true + status: {} +kind: ConfigMap +metadata: + name: performance + namespace: clusters +---- + diff --git a/scalability_and_performance/cnf-tuning-low-latency-hosted-cp-nodes-with-perf-profile.adoc b/scalability_and_performance/cnf-tuning-low-latency-hosted-cp-nodes-with-perf-profile.adoc new file mode 100644 index 000000000000..737e61cf23c0 --- /dev/null +++ b/scalability_and_performance/cnf-tuning-low-latency-hosted-cp-nodes-with-perf-profile.adoc @@ -0,0 +1,39 @@ +:_mod-docs-content-type: ASSEMBLY +[id="cnf-tuning-low-latency-hosted-cp-nodes-with-perf-profile"] += Tuning hosted control planes for low latency with the performance profile +include::_attributes/common-attributes.adoc[] +:context: cnf-low-latency-perf-profile-hosted-cp + +toc::[] + +Tune hosted control planes for low latency by applying a performance profile. With the performance profile, you can restrict CPUs for infrastructure and application containers and configure huge pages, Hyper-Threading, and CPU partitions for latency-sensitive processes. + +[id="cnf-create-performance-profiles-hosted-cp"] +== Creating a performance profile for hosted control planes + +You can create a cluster performance profile by using the Performance Profile Creator (PPC) tool. The PPC is a function of the Node Tuning Operator. + +The PPC combines information about your cluster with user-supplied configurations to generate a performance profile that is appropriate to your hardware, topology, and use-case. + +The following is a high-level workflow for creating and applying a performance profile in your cluster: + +. Gather information about your cluster using the `must-gather` command. + +. Use the PPC tool to create a performance profile. + +. Apply the performance profile to your cluster. + +include::modules/cnf-gathering-data-about-hosted-cluster-using-must-gather.adoc[leveloffset=+2] + +[role="_additional-resources"] +.Additional resources + +* xref:../support/gathering-cluster-data.adoc#nodes-nodes-managing[Gathering data about your cluster] + +* xref:../hosted_control_planes/hcp-troubleshooting.adoc#hcp-must-gather-cli[Gathering data for a hosted cluster by using the CLI]. + +include::modules/cnf-running-the-performance-creator-profile-hosted.adoc[leveloffset=+2] + +* xref:../scalability_and_performance/cnf-tuning-low-latency-nodes-with-perf-profile.adoc#performance-profile-creator-arguments_cnf-low-latency-perf-profile[Performance Profile Creator arguments] + +include::modules/apply-performance-profile-hosted-cluster.adoc[leveloffset=+2] diff --git a/scalability_and_performance/cnf-tuning-low-latency-nodes-with-perf-profile.adoc b/scalability_and_performance/cnf-tuning-low-latency-nodes-with-perf-profile.adoc index 8080b04d4d2e..35d6d0710552 100644 --- a/scalability_and_performance/cnf-tuning-low-latency-nodes-with-perf-profile.adoc +++ b/scalability_and_performance/cnf-tuning-low-latency-nodes-with-perf-profile.adoc @@ -44,8 +44,9 @@ include::modules/cnf-gathering-data-about-cluster-using-must-gather.adoc[levelof [role="_additional-resources"] .Additional resources -* For more information about the `must-gather` tool, -see xref:../support/gathering-cluster-data.adoc#nodes-nodes-managing[Gathering data about your cluster]. +* xref:../support/gathering-cluster-data.adoc#nodes-nodes-managing[Gathering data about your cluster] + +* xref:../hosted_control_planes/hcp-troubleshooting.adoc#hcp-must-gather-cli[Gathering data for a hosted cluster by using the CLI] include::modules/cnf-running-the-performance-creator-profile.adoc[leveloffset=+2]