Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 7 additions & 2 deletions machine_management/deploying-machine-health-checks.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@ include::_attributes/common-attributes.adoc[]

toc::[]

[role="_abstract"]
You can configure and deploy a machine health check to automatically repair damaged machines in a machine pool.

include::snippets/machine-user-provisioned-limitations.adoc[leveloffset=+1]
Expand All @@ -20,8 +21,12 @@ include::modules/machine-health-checks-about.adoc[leveloffset=+1]
include::modules/machine-health-checks-resource.adoc[leveloffset=+1]

include::modules/machine-health-checks-creating.adoc[leveloffset=+1]
include::modules/machine-health-checks-short-circuiting.adoc[leveloffset=+1]

You can configure and deploy a machine health check to detect and repair unhealthy bare metal nodes.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Moved to a module.

include::modules/machine-health-checks-creating.adoc[leveloffset=+1]

include::modules/mgmt-power-remediation-baremetal-about.adoc[leveloffset=+1]

include::modules/mgmt-power-remediation-baremetal-about-creating-mhc-baremetal.adoc[leveloffset=+1]

include::modules/mgmt-power-remediation-baremetal-about-troubleshooting.adoc[leveloffset=+1]
12 changes: 8 additions & 4 deletions modules/cpms-changing-openstack-flavor-type.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@
[id="cpms-changing-openstack-flavor-type_{context}"]
= Changing the {rh-openstack} compute flavor by using a control plane machine set

[role="_abstract"]
You can change the {rh-openstack-first} compute service (Nova) flavor that your control plane machines use by updating the specification in the control plane machine set custom resource.

In {rh-openstack}, flavors define the compute, memory, and storage capacity of computing instances. By increasing or decreasing the flavor size, you can scale your control plane vertically.
Expand All @@ -22,10 +23,13 @@ In {rh-openstack}, flavors define the compute, memory, and storage capacity of c
providerSpec:
value:
# ...
flavor: m1.xlarge <1>
flavor: m1.xlarge
----
<1> Specify a {rh-openstack} flavor type that has the same base as the existing selection. For example, you can change `m6i.xlarge` to `m6i.2xlarge` or `m6i.4xlarge`. You can choose larger or smaller flavors depending on your vertical scaling needs.
+
where:

. Save your changes.
providerSpec.value:: Specify a {rh-openstack} flavor type that has the same base as the existing selection. For example, you can change `m6i.xlarge` to `m6i.2xlarge` or `m6i.4xlarge`. You can choose larger or smaller flavors depending on your vertical scaling needs.

After you save your changes, machines are replaced with ones that use the flavor you chose.
. Save your changes.
+
After you save your changes, machines are replaced with ones that use the flavor you chose.
93 changes: 23 additions & 70 deletions modules/machine-health-checks-resource.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -3,12 +3,12 @@
// * machine_management/deploying-machine-health-checks.adoc
// * post_installation_configuration/node-tasks.adoc

:_mod-docs-content-type: CONCEPT
:_mod-docs-content-type: REFERENCE
[id="machine-health-checks-resource_{context}"]
= Sample MachineHealthCheck resource
= About the MachineHealthCheck custom resource

[role="_abstract"]
You can use the sample `MachineHealthCheck` resource to configure health criteria, remediation limits, and startup timeouts for machines in a targeted pool.
You control how a machine health check remediates unhealthy machines by using a `MachineHealthCheck` custom resource (CR) to configure health criteria, remediation limits, and startup timeouts for machines in a targeted pool.

The `MachineHealthCheck` resource for all cloud-based installation types, and other than bare metal, resembles the following YAML file:

Expand All @@ -17,85 +17,38 @@ The `MachineHealthCheck` resource for all cloud-based installation types, and ot
apiVersion: machine.openshift.io/v1beta1
kind: MachineHealthCheck
metadata:
name: example <1>
name: example
namespace: openshift-machine-api
spec:
selector:
matchLabels:
machine.openshift.io/cluster-api-machine-role: <role> <2>
machine.openshift.io/cluster-api-machine-type: <role> <2>
machine.openshift.io/cluster-api-machineset: <cluster_name>-<label>-<zone> <3>
machine.openshift.io/cluster-api-machine-role: <role>
machine.openshift.io/cluster-api-machine-type: <role>
machine.openshift.io/cluster-api-machineset: <cluster_name>-<label>-<zone>
unhealthyConditions:
- type: "Ready"
timeout: "300s" <4>
timeout: "300s"
status: "False"
- type: "Ready"
timeout: "300s" <4>
timeout: "300s"
status: "Unknown"
maxUnhealthy: "40%" <5>
nodeStartupTimeout: "10m" <6>
maxUnhealthy: "40%"
nodeStartupTimeout: "10m"
----
<1> Specify the name of the machine health check to deploy.
<2> Specify a label for the machine pool that you want to check.
<3> Specify the machine set to track in `<cluster_name>-<label>-<zone>` format. For example, `prod-node-us-east-1a`.
<4> Specify the timeout duration for a node condition. If a condition is met for the duration of the timeout, the machine will be remediated. Long timeouts can result in long periods of downtime for a workload on an unhealthy machine.
<5> Specify the amount of machines allowed to be concurrently remediated in the targeted pool. This can be set as a percentage or an integer. If the number of unhealthy machines exceeds the limit set by `maxUnhealthy`, remediation is not performed.
<6> Specify the timeout duration that a machine health check must wait for a node to join the cluster before a machine is determined to be unhealthy.
where:

--
`metadata.name`:: Specifies the name of the machine health check to deploy.
`spec.selector.matchLabels`:: Replace the parameters for the following labels:
* `machine.openshift.io/cluster-api-machine-role`. Specify a label for the machine pool that you want to check.
* `machine.openshift.io/cluster-api-machine-type`. Specify a label for the machine pool that you want to check.
* `machine.openshift.io/cluster-api-machineset`. Specify the machine set to track in the `<cluster_name>-<label>-<zone>` format. For example, `prod-node-us-east-1a`.
`spec.unhealthyConditions.timeout`:: Specifies the timeout duration for a node condition. If a condition is met for the duration of the timeout, the machine will be remediated. Long timeouts can result in long periods of downtime for a workload on an unhealthy machine.
`spec.maxUnhealthy`:: Specifies the amount of machines allowed to be concurrently remediated in the targeted pool. This can be set as a percentage or an integer. If the number of unhealthy machines exceeds the limit set by `maxUnhealthy`, remediation is not performed.
`spec.nodeStartupTimeout`:: Specifies the timeout duration that a machine health check must wait for a node to join the cluster before a machine is determined to be unhealthy.
--

[NOTE]
====
The `matchLabels` are examples only; you must map your machine groups based on your specific needs.
====

[id="machine-health-checks-short-circuiting_{context}"]
== Short-circuiting machine health check remediation

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Short-circuiting ensures that machine health checks remediate machines only when the cluster is healthy.
Short-circuiting is configured through the `maxUnhealthy` field in the `MachineHealthCheck` resource.

If the user defines a value for the `maxUnhealthy` field, before remediating any machines, the `MachineHealthCheck` compares the value of `maxUnhealthy` with the number of machines within its target pool that it has determined to be unhealthy. Remediation is not performed if the number of unhealthy machines exceeds the `maxUnhealthy` limit.

[IMPORTANT]
====
If `maxUnhealthy` is not set, the value defaults to `100%` and the machines are remediated regardless of the state of the cluster.
====

The appropriate `maxUnhealthy` value depends on the scale of the cluster you deploy and how many machines the `MachineHealthCheck` covers. For example, you can use the `maxUnhealthy` value to cover multiple compute machine sets across multiple availability zones so that if you lose an entire zone, your `maxUnhealthy` setting prevents further remediation within the cluster. In global Azure regions that do not have multiple availability zones, you can use availability sets to ensure high availability.

[IMPORTANT]
====
If you configure a `MachineHealthCheck` resource for the control plane, set the value of `maxUnhealthy` to `1`.
This configuration ensures that the machine health check takes no action when multiple control plane machines appear to be unhealthy. Multiple unhealthy control plane machines can indicate that the etcd cluster is degraded or that a scaling operation to replace a failed machine is in progress.
If the etcd cluster is degraded, manual intervention might be required. If a scaling operation is in progress, the machine health check should allow it to finish.
====

The `maxUnhealthy` field can be set as either an integer or percentage.
There are different remediation implementations depending on the `maxUnhealthy` value.

=== Setting maxUnhealthy by using an absolute value

If `maxUnhealthy` is set to `2`:

* Remediation will be performed if 2 or fewer nodes are unhealthy
* Remediation will not be performed if 3 or more nodes are unhealthy

These values are independent of how many machines are being checked by the machine health check.

=== Setting maxUnhealthy by using percentages

If `maxUnhealthy` is set to `40%` and there are 25 machines being checked:

* Remediation will be performed if 10 or fewer nodes are unhealthy
* Remediation will not be performed if 11 or more nodes are unhealthy

If `maxUnhealthy` is set to `40%` and there are 6 machines being checked:

* Remediation will be performed if 2 or fewer nodes are unhealthy
* Remediation will not be performed if 3 or more nodes are unhealthy

[NOTE]
====
The allowed number of machines is rounded down when the percentage of `maxUnhealthy` machines that are checked is not a whole number.
====
62 changes: 62 additions & 0 deletions modules/machine-health-checks-short-circuiting.adoc
Original file line number Diff line number Diff line change
@@ -0,0 +1,62 @@
// Module included in the following assemblies:
//
// * machine_management/deploying-machine-health-checks.adoc
// * post_installation_configuration/node-tasks.adoc

:_mod-docs-content-type: CONCEPT
[id="machine-health-checks-short-circuiting_{context}"]
= About short-circuiting machine health check remediation

[role="_abstract"]
You can use machine health check short-circuiting to ensure that machine health checks remediate machines only when the cluster is healthy, by configuring the `maxUnhealthy` field in the `MachineHealthCheck` resource.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Moved from https://github.com/openshift/openshift-docs/pull/117206/changes#diff-4dcba0b1e71613dc3607d7de39037ba614c0c2cd4f8de056ac50748191df095fL49. No need to review beyond the short desc. And changed the H3s to description list entries.

If you define a value for the `maxUnhealthy` field, before remediating any machines, the `MachineHealthCheck` compares the value of `maxUnhealthy` with the number of machines within its target pool that it has determined to be unhealthy. Remediation is not performed if the number of unhealthy machines exceeds the `maxUnhealthy` limit.

[IMPORTANT]
====
If `maxUnhealthy` is not set, the value defaults to `100%` and the machines are remediated regardless of the state of the cluster.
====

The appropriate `maxUnhealthy` value depends on the scale of the cluster you deploy and how many machines the `MachineHealthCheck` covers. For example, you can use the `maxUnhealthy` value to cover multiple compute machine sets across multiple availability zones so that if you lose an entire zone, your `maxUnhealthy` setting prevents further remediation within the cluster. In global Azure regions that do not have multiple availability zones, you can use availability sets to ensure high availability.

[IMPORTANT]
====
If you configure a `MachineHealthCheck` resource for the control plane, set the value of `maxUnhealthy` to `1`.

This configuration ensures that the machine health check takes no action when multiple control plane machines appear to be unhealthy. Multiple unhealthy control plane machines can indicate that the etcd cluster is degraded or that a scaling operation to replace a failed machine is in progress.

If the etcd cluster is degraded, manual intervention might be required. If a scaling operation is in progress, the machine health check should allow it to finish.
====

The `maxUnhealthy` field can be set as either an integer or percentage.
There are different remediation implementations depending on the `maxUnhealthy` value.

Setting maxUnhealthy by using an absolute value::
If `maxUnhealthy` is set to `2`:
+
--
* Remediation will be performed if 2 or fewer nodes are unhealthy
* Remediation will not be performed if 3 or more nodes are unhealthy
--
+
These values are independent of how many machines are being checked by the machine health check.

Setting maxUnhealthy by using percentages::
If `maxUnhealthy` is set to `40%` and there are 25 machines being checked:
+
--
* Remediation will be performed if 10 or fewer nodes are unhealthy
* Remediation will not be performed if 11 or more nodes are unhealthy
--
+
If `maxUnhealthy` is set to `40%` and there are 6 machines being checked:
+
--
* Remediation will be performed if 2 or fewer nodes are unhealthy
* Remediation will not be performed if 3 or more nodes are unhealthy
--
+
[NOTE]
====
The allowed number of machines is rounded down when the percentage of `maxUnhealthy` machines that are checked is not a whole number.
====
3 changes: 1 addition & 2 deletions modules/machineset-vsphere-multiple-nics.adoc
Original file line number Diff line number Diff line change
@@ -1,4 +1,3 @@

// Module included in the following assemblies:
//
// * machine_management/creating_machinesets/creating-machineset-vsphere.adoc
Expand Down Expand Up @@ -163,4 +162,4 @@ tag::controlplane[]
`spec.template.machines_v1beta1_machine_openshift_io.spec.providerSpec.value.network.workspace.server`:: Specifies the vCenter server IP or fully qualified domain name (FQDN).

end::controlplane[]
--

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, I have no idea....

--
Original file line number Diff line number Diff line change
@@ -0,0 +1,131 @@
// Module included in the following assemblies:

// * machine_management/deploying-machine-health-checks.adoc

:_mod-docs-content-type: PROCEDURE
[id="mgmt-power-remediation-baremetal-about-creating-mhc-baremetal_{context}"]
= Creating a MachineHealthCheck resource for bare metal

[role="_abstract"]
You control how a machine health check remediates unhealthy machines by using a `MachineHealthCheck` resource to configure health criteria, remediation limits, and startup timeouts for machines in a targeted pool.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Moved from https://github.com/openshift/openshift-docs/pull/117206/changes#diff-9035c4e32d3908a412d19cfcf79271e1f3cfd5bb7a002b8b59c58749d6c79a42L70. But, I did move some things around, re-wrote a few, and added a few. Please do review.

.Prerequisites

* The {product-title} is installed using installer-provisioned infrastructure (IPI).

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 [error] RedHat.TermsErrors: Use 'installer-provisioned infrastructure' rather than 'IPI'. For more information, see RedHat.TermsErrors.

* Access to Baseboard Management Controller (BMC) credentials (or BMC access to each node).
* Network access to the BMC interface of the unhealthy node.
* For a metal3-based remediation, a `Metal3RemediationTemplate` resource must exist.
+
.Sample `Metal3RemediationTemplate` resource for bare metal, metal3-based remediation
[source,yaml]
----
apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: Metal3RemediationTemplate
metadata:
name: metal3-remediation-template
namespace: openshift-machine-api
spec:
template:
spec:
strategy:
type: Reboot
retryLimit: 1
timeout: 5m0s
----
.Procedure

. Create a `healthcheck.yaml` file that contains the definition of your machine health check.
+
.Sample `MachineHealthCheck` resource for bare metal, annotation-based remediation
[source,yaml]
----
apiVersion: machine.openshift.io/v1beta1
kind: MachineHealthCheck
metadata:
name: example
namespace: openshift-machine-api
annotations:
machine.openshift.io/remediation-strategy: external-baremetal
spec:
selector:
matchLabels:
machine.openshift.io/cluster-api-machine-role: <role>
machine.openshift.io/cluster-api-machine-type: <role>
machine.openshift.io/cluster-api-machineset: <cluster_name>-<label>-<zone>
unhealthyConditions:
- type: "Ready"
timeout: "300s"
status: "False"
- type: "Ready"
timeout: "300s"
status: "Unknown"
maxUnhealthy: "40%"
nodeStartupTimeout: "10m"
----
+
where
+
--
`metadata.name`:: Specifies the name of the machine health check to deploy.
`metadata.annotations`:: For bare metal clusters, you must include the `machine.openshift.io/remediation-strategy: external-baremetal` annotation in the `annotations` section to enable power-cycle remediation. With this remediation strategy, unhealthy hosts are rebooted instead of removed from the cluster.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 [error] RedHat.TermsErrors: Use 'bare-metal clusters' rather than 'bare metal clusters'. For more information, see RedHat.TermsErrors.

`spec.selector.matchLabels`:: Replace the parameters for the following labels:
* `machine.openshift.io/cluster-api-machine-role`. Specify a label for the machine pool that you want to check.
* `machine.openshift.io/cluster-api-machine-type`. Specify a label for the machine pool that you want to check.
* `machine.openshift.io/cluster-api-machineset`. Specify the machine set to track in the `<cluster_name>-<label>-<zone>` format. For example, `prod-node-us-east-1a`.
`spec.unhealthyConditions.timeout`:: Specifies the timeout duration for a node condition. If a condition is met for the duration of the timeout, the machine will be remediated. Long timeouts can result in long periods of downtime for a workload on an unhealthy machine.
`spec.maxUnhealthy`:: Specifies the amount of machines allowed to be concurrently remediated in the targeted pool. This can be set as a percentage or an integer. If the number of unhealthy machines exceeds the limit set by `maxUnhealthy`, remediation is not performed.
`spec.nodeStartupTimeout`:: Specifies the timeout duration that a machine health check must wait for a node to join the cluster before a machine is determined to be unhealthy.
--
+
.Sample `MachineHealthCheck` resource for bare metal, metal3-based remediation
[source,yaml]
----
apiVersion: machine.openshift.io/v1beta1
kind: MachineHealthCheck
metadata:
name: example
namespace: openshift-machine-api
spec:
selector:
matchLabels:
machine.openshift.io/cluster-api-machine-role: <role>
machine.openshift.io/cluster-api-machine-type: <role>
machine.openshift.io/cluster-api-machineset: <cluster_name>-<label>-<zone>
remediationTemplate:
apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: Metal3RemediationTemplate
name: metal3-remediation-template
namespace: openshift-machine-api
unhealthyConditions:
- type: "Ready"
timeout: "300s"
----
+
where:
+
--
`metadata.name`:: Specifies the name of the machine health check to deploy.
`spec.selector.matchLabels`:: Replace the parameters for the following labels:
* `machine.openshift.io/cluster-api-machine-role`. Specify a label for the machine pool that you want to check.
* `machine.openshift.io/cluster-api-machine-type`. Specify a label for the machine pool that you want to check.
* `machine.openshift.io/cluster-api-machineset`. Specify the machine set to track in the `<cluster_name>-<label>-<zone>` format. For example, `prod-node-us-east-1a`.
`spec.remediationTemplate`:: The metal3 remediation template to use. Specify the following information:
* `apiVersion`. Specify the API version as `infrastructure.cluster.x-k8s.io/v1beta1`.
* `kind`. Specify `Metal3RemediationTemplate`.
* `name`. Specify the name of the template.
* `namespace`. Specify the namespace of the template.
`spec.unhealthyConditions.timeout`:: Specifies the timeout duration for a node condition. If a condition is met for the duration of the timeout, the machine will be remediated. Long timeouts can result in long periods of downtime for a workload on an unhealthy machine.
--
+
[NOTE]
====
The `matchLabels` are examples only; you must map your machine groups based on your specific needs.
====

. Apply the `healthcheck.yaml` file to your cluster using the following command:
+
[source,terminal]
----
$ oc apply -f healthcheck.yaml
----
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
// Module included in the following assemblies:

// * machine_management/deploying-machine-health-checks.adoc

:_mod-docs-content-type: CONCEPT
[id="mgmt-power-remediation-baremetal-about-troubleshooting_{context}"]
= Troubleshooting issues with power-based remediation

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.


[role="_abstract"]
To troubleshoot issues you are having with power-based remediation, check the connection to the Baseboard Management Controller (BMC).

Verify the following conditions:

* You have access to the BMC.
* The BMC is connected to the control plane node that is responsible for running the remediation task.
Loading