Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions _topic_maps/_topic_map_rosa_hcp.yml
Original file line number Diff line number Diff line change
Expand Up @@ -252,6 +252,8 @@ Topics:
File: nodes-cluster-resource-configure
- Name: Configuring PID limits
File: rosa-configuring-pid-limits
- Name: Managing multi-architecture clusters
File: rosa-multi-arch-cluster-managing
---
Name: Security and compliance
Dir: security
Expand Down
135 changes: 11 additions & 124 deletions modules/multi-architecture-scheduling-examples.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -4,136 +4,23 @@

:_mod-docs-content-type: CONCEPT
[id="multi-architecture-scheduling-examples_{context}"]

= Sample multi-architecture node workload deployments

Before you schedule workloads on a cluster with compute nodes of different architectures, consider the following use cases:
Scheduling a workload to an appropriate node based on architecture works in the same way as scheduling based on any other node characteristic.
ifndef::openshift-dedicated,openshift-rosa,openshift-rosa-hcp[]
Consider the following options when determining how to schedule your workloads.
endif::openshift-dedicated,openshift-rosa,openshift-rosa-hcp[]

Using node affinity to schedule workloads on a node:: You can allow a workload to be scheduled on only a set of nodes with architectures supported by its images, you can set the `spec.affinity.nodeAffinity` field in your pod's template specification.
+
.Example deployment with the `nodeAffinity` set to certain architectures
[source,yaml]
----
apiVersion: apps/v1
kind: Deployment
metadata: # ...
spec:
# ...
template:
# ...
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/arch
operator: In
values: <1>
- amd64
- arm64
----
<1> Specify the supported architectures. Valid values include `amd64`,`arm64`, or both values.
include::snippets/multi-arch-schedule-nodeaffinity.adoc[leveloffset=+0]

Tainting every node for a specific architecture:: You can taint a node to avoid workloads that are not compatible with its architecture to be scheduled on that node. In the case where your cluster is using a `MachineSet` object, you can add parameters to the `.spec.template.spec.taints` field to avoid workloads being scheduled on nodes with non-supported architectures.
ifndef::openshift-dedicated,openshift-rosa,openshift-rosa-hcp[]

* Before you can taint a node, you must scale down the `MachineSet` object or remove available machines. You can scale down the machine set by using one of following commands:
+
[source,terminal]
----
$ oc scale --replicas=0 machineset <machineset> -n openshift-machine-api
----
+
Or:
+
[source,terminal]
----
$ oc edit machineset <machineset> -n openshift-machine-api
----
For more information on scaling machine sets, see "Modifying a compute machine set".
include::snippets/multi-arch-schedule-taint.adoc[leveloffset=+0]

+
--
.Example `MachineSet` with a taint set
[source,yaml]
----
apiVersion: machine.openshift.io/v1beta1
kind: MachineSet
metadata: # ...
spec:
# ...
template:
# ...
spec:
# ...
taints:
- effect: NoSchedule
key: multi-arch.openshift.io/arch
value: arm64
----
You can also set a taint on a specific node by running the following command:
[source,terminal]
----
$ oc adm taint nodes <node-name> multi-arch.openshift.io/arch=arm64:NoSchedule
----
--
include::snippets/multi-arch-schedule-default-toleration-namespace.adoc[leveloffset=+0]

Creating a default toleration:: You can annotate a namespace so all of the workloads get the same default toleration by running the following command:
+
[source,terminal]
----
$ oc annotate namespace my-namespace \
'scheduler.alpha.kubernetes.io/defaultTolerations'='[{"operator": "Exists", "effect": "NoSchedule", "key": "multi-arch.openshift.io/arch"}]'
----
include::snippets/multi-arch-schedule-toleration.adoc[leveloffset=+0]

Tolerating architecture taints in workloads:: On a node with a defined taint, workloads will not be scheduled on that node. However, you can allow them to be scheduled by setting a toleration in the pod's specification.
+
.Example deployment with a toleration
[source,yaml]
----
apiVersion: apps/v1
kind: Deployment
metadata: # ...
spec:
# ...
template:
# ...
spec:
tolerations:
- key: "multi-arch.openshift.io/arch"
value: "arm64"
operator: "Equal"
effect: "NoSchedule"
----
+
This example deployment can also be allowed on nodes with the `multi-arch.openshift.io/arch=arm64` taint specified.
include::snippets/multi-arch-schedule-affinity-taint-toleration.adoc[leveloffset=+0]

Using node affinity with taints and tolerations:: When a scheduler computes the set of nodes to schedule a pod, tolerations can broaden the set while node affinity restricts the set. If you set a taint to the nodes of a specific architecture, the following example toleration is required for scheduling pods.
+
.Example deployment with a node affinity and toleration set.
[source,yaml]
----
apiVersion: apps/v1
kind: Deployment
metadata: # ...
spec:
# ...
template:
# ...
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/arch
operator: In
values:
- amd64
- arm64
tolerations:
- key: "multi-arch.openshift.io/arch"
value: "arm64"
operator: "Equal"
effect: "NoSchedule"
----
endif::openshift-dedicated,openshift-rosa,openshift-rosa-hcp[]
9 changes: 9 additions & 0 deletions modules/multi-architecture-scheduling.adoc
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
:_mod-docs-content-type: CONCEPT
[id="multi-architecture-scheduling_{context}"]
= Scheduling workloads on clusters with multi-architecture compute machines

When you deploy workloads on a cluster with compute nodes that use different architectures, you must align pod architecture with the architecture of the underlying node. Your workload may also require additional configuration to particular resources depending on the underlying node architecture.

ifndef::openshift-dedicated,openshift-rosa,openshift-rosa-hcp[]
You can use the Multiarch Tuning Operator to enable architecture-aware scheduling of workloads on clusters with multi-architecture compute machines. The Multiarch Tuning Operator implements additional scheduler predicates in the pods specifications based on the architectures that the pods can support at creation time.
endif::openshift-dedicated,openshift-rosa,openshift-rosa-hcp[]
Original file line number Diff line number Diff line change
Expand Up @@ -6,28 +6,23 @@ include::_attributes/common-attributes.adoc[]

toc::[]

== Scheduling workloads on clusters with multi-architecture compute machines
Managing a cluster that has nodes with multiple architectures requires you to consider node architecture as you monitor the cluster and manage your workloads. This requires you to take additional considerations into account when you
ifndef::openshift-dedicated,openshift-rosa,openshift-rosa-hcp[]
configure cluster resource requirements and behavior, or
endif::openshift-dedicated,openshift-rosa,openshift-rosa-hcp[]
schedule workloads in a multi-architecture cluster.

Deploying a workload on a cluster with compute nodes of different architectures requires attention and monitoring of your cluster. There might be further actions you need to take in order to successfully place pods in the nodes of your cluster.

You can use the Multiarch Tuning Operator to enable architecture-aware scheduling of workloads on clusters with multi-architecture compute machines. The Multiarch Tuning Operator implements additional scheduler predicates in the pods specifications based on the architectures that the pods can support at creation time. For more information, see xref:../../post_installation_configuration/configuring-multi-arch-compute-machines/multiarch-tuning-operator.adoc#multiarch-tuning-operator[Managing workloads on multi-architecture clusters by using the Multiarch Tuning Operator].

For more information on node affinity, scheduling, taints and tolerations, see the following documentation:

ifndef::openshift-dedicated,openshift-rosa[]
* xref:../../nodes/scheduling/nodes-scheduler-taints-tolerations.adoc#nodes-scheduler-taints-tolerations[Controlling pod placement using node taints].
endif::openshift-dedicated,openshift-rosa[]

* xref:../../nodes/scheduling/nodes-scheduler-node-affinity.adoc#nodes-scheduler-node-affinity[Controlling pod placement on nodes using node affinity]

* xref:../../nodes/scheduling/nodes-scheduler-about.adoc#nodes-scheduler-about[Controlling pod placement using the scheduler]
include::modules/multi-architecture-scheduling.adoc[leveloffset=+1]

include::modules/multi-architecture-scheduling-examples.adoc[leveloffset=+2]

.Additional resources

* xref:../../post_installation_configuration/configuring-multi-arch-compute-machines/multiarch-tuning-operator.adoc#multiarch-tuning-operator[Managing workloads on multi-architecture clusters by using the Multiarch Tuning Operator]
* xref:../../nodes/scheduling/nodes-scheduler-taints-tolerations.adoc#nodes-scheduler-taints-tolerations[Controlling pod placement using node taints]
* xref:../../nodes/scheduling/nodes-scheduler-node-affinity.adoc#nodes-scheduler-node-affinity[Controlling pod placement on nodes using node affinity]
* xref:../../nodes/scheduling/nodes-scheduler-about.adoc#nodes-scheduler-about[Controlling pod placement using the scheduler]
* xref:../../machine_management/modifying-machineset.adoc#machineset-modifying_modifying-machineset[Modifying a compute machine set]

include::modules/multi-architecture-enabling-64k-pages.adoc[leveloffset=+1]

include::modules/multi-architecture-import-imagestreams.adoc[leveloffset=+1]
include::modules/multi-architecture-import-imagestreams.adoc[leveloffset=+1]
20 changes: 20 additions & 0 deletions rosa_cluster_admin/rosa-multi-arch-cluster-managing.adoc
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
:_mod-docs-content-type: ASSEMBLY
//This assembly duplicates the following file to avoid symbolic links:
// * post_installation_configuration/configuring-multi-arch-compute-machines/multi-architecture-compute-managing.adoc

:context: rosa-multi-arch-compute-managing
[id="rosa-multi-arch-managing"]
= Managing a cluster with multi-architecture compute machines
include::_attributes/common-attributes.adoc[]

toc::[]

Managing a cluster that has nodes with multiple architectures requires you to consider node architecture as you monitor the cluster and manage your workloads. This requires you to take additional considerations into account when you schedule workloads in a multi-architecture cluster.

include::modules/multi-architecture-scheduling.adoc[leveloffset=+1]

include::modules/multi-architecture-scheduling-examples.adoc[leveloffset=+2]

// TODO OSDOCS-11411: Add back when HCP split in Nodes doc is complete
// .Additional resources
// * xref ../nodes/scheduling/nodes-scheduler-node-affinity.adoc#nodes-scheduler-node-affinity[Controlling pod placement on nodes using node affinity]
33 changes: 33 additions & 0 deletions snippets/multi-arch-schedule-affinity-taint-toleration.adoc
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
:_mod-docs-content-type: SNIPPET

Using node affinity with taints and tolerations:: When a scheduler computes the set of nodes to schedule a pod, tolerations can broaden the set while node affinity restricts the set. If you set a taint on nodes that have a specific architecture, you must also add a toleration to workloads that you want to be scheduled there.
+
.Example deployment with node affinity and toleration set
--
[source,yaml]
----
apiVersion: apps/v1
kind: Deployment
metadata: # ...
spec:
# ...
template:
# ...
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/arch
operator: In
values:
- amd64
- arm64
tolerations:
- key: "multiarch.openshift.io/arch"
value: "arm64"
operator: "Equal"
effect: "NoSchedule"
----
--
12 changes: 12 additions & 0 deletions snippets/multi-arch-schedule-default-toleration-namespace.adoc
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
:_mod-docs-content-type: SNIPPET

Creating a default toleration in a namespace:: When a node or machine set has a taint, only workloads that tolerate that taint can be scheduled. You can annotate a namespace so all of the workloads get the same default toleration by running the following command:
+
.Example default toleration set on a namespace
--
[source,terminal]
----
$ oc annotate namespace my-namespace \
'scheduler.alpha.kubernetes.io/defaultTolerations'='[{"operator": "Exists", "effect": "NoSchedule", "key": "multiarch.openshift.io/arch"}]'
----
--
30 changes: 30 additions & 0 deletions snippets/multi-arch-schedule-nodeaffinity.adoc
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
:_mod-docs-content-type: SNIPPET

Using `nodeAffinity` to schedule nodes with specific architectures:: You can allow a workload to be scheduled on only a set of nodes with architectures supported by its images, you can set the `spec.affinity.nodeAffinity` field in your pod's template specification.
+
.Example deployment with node affinity set
--
[source,yaml]
----
apiVersion: apps/v1
kind: Deployment
metadata: # ...
spec:
# ...
template:
# ...
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/arch
operator: In
values: <1>
- amd64
- arm64
----
<1> Specify the supported architectures. Valid values include `amd64`,`arm64`, or both values.
[source,yaml]
--
32 changes: 32 additions & 0 deletions snippets/multi-arch-schedule-taint.adoc
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
:_mod-docs-content-type: SNIPPET

Tainting each node for a specific architecture:: You can taint a node to avoid the node scheduling workloads that are incompatible with its architecture. When your cluster uses a `MachineSet` object, you can add parameters to the `.spec.template.spec.taints` field to avoid workloads being scheduled on nodes with non-supported architectures.
+
Before you add a taint to a node, you must scale down the `MachineSet` object or remove existing available machines. For more information, see _Modifying a compute machine set_.
+
.Example machine set with taint set
--
[source,yaml]
----
apiVersion: machine.openshift.io/v1beta1
kind: MachineSet
metadata: # ...
spec:
# ...
template:
# ...
spec:
# ...
taints:
- effect: NoSchedule
key: multiarch.openshift.io/arch
value: arm64
----
--
+
You can also set a taint on a specific node by running the following command:
+
[source,terminal]
----
$ oc adm taint nodes <node-name> multiarch.openshift.io/arch=arm64:NoSchedule
----
24 changes: 24 additions & 0 deletions snippets/multi-arch-schedule-toleration.adoc
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
:_mod-docs-content-type: SNIPPET

Tolerating architecture taints in workloads:: When a node or machine set has a taint, only workloads that tolerate that taint can be scheduled. You can configure your workload with a `toleration` so that it is scheduled on nodes with specific architecture taints.
+
.Example deployment with toleration set
--
[source,yaml]
----
apiVersion: apps/v1
kind: Deployment
metadata: # ...
spec:
# ...
template:
# ...
spec:
tolerations:
- key: "multiarch.openshift.io/arch"
value: "arm64"
operator: "Equal"
effect: "NoSchedule"
----
This example deployment can be scheduled on nodes and machine sets that have the `multiarch.openshift.io/arch=arm64` taint specified.
--