WIP [MGMT-2101] Added remediation history to MachineHealthChecks

[slintes]

In order to better understand remediations, we want to track a limited amount of remediation in the MHC status, with the target machine / node, reason, and the timestamps when the unhealthy machine / node was detected, when remediation started, when the node is fenced (=deleted), and when remediation is done (node is ready again).

TODO: add tests

initial feedback welcome :)

Example:

apiVersion: machine.openshift.io/v1beta1
kind: MachineHealthCheck
metadata:
  annotations:
    machine.openshift.io/remediation-strategy: external-baremetal
  creationTimestamp: "2020-11-24T16:31:22Z"
spec:
  maxUnhealthy: 100%
  nodeStartupTimeout: 60m
  selector:
    matchLabels:
      machine.openshift.io/cluster-api-machine-role: worker
  unhealthyConditions:
  - status: "False"
    timeout: 20s
    type: Ready
  - status: Unknown
    timeout: 20s
    type: Ready
status:
  conditions:
  - lastTransitionTime: "2020-11-24T17:17:32Z"
    status: "True"
    type: RemediationAllowed
  currentHealthy: 2
  expectedMachines: 2
  remediationHistory:
  - conditionStatus: Unknown
    conditionType: Ready
    detected: "2020-11-24T17:33:43Z"
    fenced: "2020-11-24T17:34:35Z"
    finished: "2020-11-24T17:35:09Z"
    remediationType: external
    started: "2020-11-24T17:34:04Z"
    targetKind: Node
    targetName: worker-1
  remediationsAllowed: 2

[openshift-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
To complete the pull request process, please assign danil-grigorev after the PR has been reviewed.
You can assign the PR to them by writing /assign @danil-grigorev in a comment when ready.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[beekhof]

targetKind: Node

Maybe Master vs. Worker would be a more useful destinction?

[beekhof]

also, how many entries do we keep?

[slintes]

targetKind: Node

Maybe Master vs. Worker would be a more useful destinction?

The idea is to track whether a unhealthy machine (e.g. [0]) or a node with unhealthy conditions ([1]) triggered remediation.
If it's a master or worker should be visible by the name.

also, how many entries do we keep?

atm hardcoded 5 ([2] and [3])

[0] https://github.com/openshift/machine-api-operator/pull/760/files#diff-8b3f455c5e13c63eb4eb480e223a91c2051decc3269398fd2f53196196a6033fR623
[1] https://github.com/openshift/machine-api-operator/pull/760/files#diff-8b3f455c5e13c63eb4eb480e223a91c2051decc3269398fd2f53196196a6033fR666
[2] https://github.com/openshift/machine-api-operator/pull/760/files#diff-3c70c3e9ec89f59adc85598b5ea78b19a08270743277d631bb0dcfad4d622e49R12
[3] https://github.com/openshift/machine-api-operator/pull/760/files#diff-3c70c3e9ec89f59adc85598b5ea78b19a08270743277d631bb0dcfad4d622e49R74

[slintes]

/retest

[openshift-merge-robot]

@slintes: The following tests failed, say /retest to rerun all failed tests:

Test name	Commit	Details	Rerun command
ci/prow/e2e-libvirt	883fb39	link	/test e2e-libvirt
ci/prow/e2e-aws-workers-rhel7	883fb39	link	/test e2e-aws-workers-rhel7

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

[elmiko]

i'm just starting to take a look at this, and as i was reading the description and comments i was wondering if our recent efforts to add more metrics on the mhc would help answer some of the underlying questions.

for example, we recently added a metric for successful remediations (see #754), this metric will update as machines are remediated. currently the metric only contains labels for the name and namespace of the mhc (not the nodes remediated). i don't know that it would solve the issues this pr is addressing, but it is another piece of data.

[slintes]

Hi Michael, thanks for pointing out the new metrics.
The context for our work is that we'd like to show the remediation history in the UI. It will we be hard for them to parse metrics (or events) and calculate the actual state of recent and ongoing remediations from it. That's why we want to add them to the MHC status.

[elmiko]

The context for our work is that we'd like to show the remediation history in the UI. It will we be hard for them to parse metrics (or events) and calculate the actual state of recent and ongoing remediations from it. That's why we want to add them to the MHC status.

makes sense to me

[openshift-bot]

Issues go stale after 90d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle stale

[slintes]

/remove-lifecycle stale
/lifecycle frozen

[openshift-ci]

@slintes: PR needs rebase.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[openshift-ci]

@slintes: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/e2e-aws	883fb39	link	/test e2e-aws
ci/prow/e2e-aws-operator	883fb39	link	/test e2e-aws-operator
ci/prow/e2e-aws-upgrade	883fb39	link	/test e2e-aws-upgrade
ci/prow/e2e-metal-ipi	883fb39	link	/test e2e-metal-ipi
ci/prow/e2e-metal-ipi-ovn-dualstack	883fb39	link	/test e2e-metal-ipi-ovn-dualstack
ci/prow/e2e-metal-ipi-upgrade	883fb39	link	/test e2e-metal-ipi-upgrade
ci/prow/e2e-metal-ipi-virtualmedia	883fb39	link	/test e2e-metal-ipi-virtualmedia
ci/prow/e2e-vsphere-serial	883fb39	link	/test e2e-vsphere-serial
ci/prow/verify-crds-sync	883fb39	link	true	/test verify-crds-sync
ci/prow/e2e-aws-ovn	883fb39	link	true	/test e2e-aws-ovn
ci/prow/e2e-vsphere-ovn-serial	883fb39	link	true	/test e2e-vsphere-ovn-serial
ci/prow/e2e-aws-ovn-upgrade	883fb39	link	true	/test e2e-aws-ovn-upgrade

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

[slintes]

/close

Our team is more focused on NHC than MHC these days. I will create a new PR once we decide to revisit this topic.

[openshift-ci]

@slintes: Closed this PR.

In response to this:

/close

Our team is more focused on NHC than MHC these days. I will create a new PR once we decide to revisit this topic.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[elmiko]

thanks @slintes