OCPBUGS-2086: Detect failure in wait-for on transition back to ready by bfournie · Pull Request #6470 · openshift/installer

bfournie · 2022-10-06T17:24:25Z

If a cluster moves from "preparing-for-installation" back to "ready" it indicates that a failure has occurred and installation will not continue. Note that in order to check the status the return from the validation check is moved to after the handling of the status errors.

zaneb · 2022-10-06T19:15:20Z

pkg/agent/cluster.go

The error message should be something like "Failed to prepare cluster installation".

Unfortunately, returning an err doesn't actually do anything because the return value is not being checked. We need another mechanism that will result in (a) the error message being printed (instead of the warning above), and (b) the process exiting.

That could mean replacing all of the other errors in this function with nil, since they don't do anything, and then actually checking the return value. Or we could add another return value. Or something else.

I changed to return an additional value to indicate whether the process should exit. This allows logs to be output which will not result in an exit, I've also removed some logs (like the validation) that happen every cycle.

When the error occurs the wait-for now exits with:
level=error msg=Bootstrap failed to complete: : bootstrap process returned error: Failed to prepare cluster installation

zaneb · 2022-10-06T21:15:05Z

We also need to handle the case where wait-for bootstrap-complete is run only after assisted has already failed to prepare the installation. See the Jira ticket for more details. That could be in another PR, since it's quite a bit more complicated.

bfournie · 2022-10-06T21:32:37Z

We also need to handle the case where wait-for bootstrap-complete is run only after assisted has already failed to prepare the installation. See the Jira ticket for more details. That could be in another PR, since it's quite a bit more complicated.

Yes I was planning on handling that after we got agreement on the this first one.

celebdor · 2022-10-06T21:54:22Z

/title OCPBUGS-2086: Detect failure in wait-for on transition back to ready

openshift-ci-robot · 2022-10-06T23:11:40Z

@bfournie: This pull request references Jira Issue OCPBUGS-2086, which is valid.

3 validation(s) were run on this bug

bug is open, matching expected state (open)
bug target version (4.12.0) matches configured target version for branch (4.12.0)
bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, POST)

The bug has been updated to refer to the pull request using the external bug tracker.

Details

In response to this:

If a cluster moves from "preparing-for-installation" back to "ready" it indicates that a failure has occurred and installation will not continue. Note that in order to check the status the return from the validation check is moved to after the handling of the status errors.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

zaneb · 2022-10-07T18:29:52Z

pkg/agent/waitfor.go

I think we should only have this Info log if exit_on_err is false.

Changed to log here if not exiting.

zaneb · 2022-10-07T18:31:33Z

linter is not happy: https://prow.ci.openshift.org/view/gs/origin-ci-test/pr-logs/pull/openshift_installer/6470/pull-ci-openshift-installer-master-golint/1578161321176731648#1:build-log.txt%3A32-37

If a cluster moves from "preparing-for-installation" back to "ready" it indicates that a failure has occurred and installation will not continue. Note that in order to check the status the return from the validation check is moved to after the handling of the status errors.

bfournie · 2022-10-10T10:54:59Z

linter is not happy: https://prow.ci.openshift.org/view/gs/origin-ci-test/pr-logs/pull/openshift_installer/6470/pull-ci-openshift-installer-master-golint/1578161321176731648#1:build-log.txt%3A32-37

Fixed

openshift-ci · 2022-10-10T13:41:11Z

@bfournie: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/okd-scos-e2e-gcp-ovn-upgrade	7e15daee895c84f842344db0676daa45eb1b4a6f	link	false	`/test okd-scos-e2e-gcp-ovn-upgrade`
ci/prow/okd-scos-e2e-vsphere	7e15daee895c84f842344db0676daa45eb1b4a6f	link	false	`/test okd-scos-e2e-vsphere`
ci/prow/okd-scos-e2e-gcp	7e15daee895c84f842344db0676daa45eb1b4a6f	link	false	`/test okd-scos-e2e-gcp`

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

pkg/agent/cluster.go

lranjbar · 2022-10-24T14:33:49Z

/approve

openshift-ci · 2022-10-24T14:36:33Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: lranjbar

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~pkg/agent/OWNERS~~ [lranjbar]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

zaneb · 2022-10-25T15:13:34Z

/lgtm

openshift-ci-robot · 2022-10-25T22:34:15Z

@bfournie: Jira Issue OCPBUGS-2086 is in an unrecognized state (ON_QA) and will not be moved to the MODIFIED state.

Details

In response to this:

If a cluster moves from "preparing-for-installation" back to "ready" it indicates that a failure has occurred and installation will not continue. Note that in order to check the status the return from the validation check is moved to after the handling of the status errors.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

Fix unexpected behavior from merging #6470 and #6500

In openshift#6470 we detected a failure when the cluster state moves back to Ready after an installation has been initiated. This adds an automatic retry when that condition occurs. It will help resolve issues like https://issues.redhat.com/browse/OCPBUGS-3280 and, in general, any problems that cause a cluster prepare failure.

openshift-ci bot requested review from dhellmann and zaneb October 6, 2022 17:26

zaneb reviewed Oct 6, 2022

View reviewed changes

bfournie force-pushed the detect-ready-transition-fail branch from 7e15dae to a57d4b9 Compare October 6, 2022 23:06

bfournie changed the title ~~AGENT-353: Detect failure in wait-for on transition back to ready~~ OCPBUGS-2086: Detect failure in wait-for on transition back to ready Oct 6, 2022

zaneb reviewed Oct 7, 2022

View reviewed changes

bfournie force-pushed the detect-ready-transition-fail branch from a57d4b9 to 9787b10 Compare October 10, 2022 10:49

lranjbar suggested changes Oct 18, 2022

View reviewed changes

pkg/agent/cluster.go Outdated Show resolved Hide resolved

add check for command run after failure happens

9d4fbc0

bfournie force-pushed the detect-ready-transition-fail branch from 666ce0e to 9d4fbc0 Compare October 18, 2022 15:33

openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Oct 24, 2022

openshift-ci bot assigned zaneb Oct 25, 2022

openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Oct 25, 2022

openshift-merge-robot merged commit a2c4d48 into openshift:master Oct 25, 2022

zaneb mentioned this pull request Oct 26, 2022

Agent-321: Cluster and Host validations rework #6500

Merged

lranjbar added a commit to lranjbar/installer that referenced this pull request Oct 26, 2022

Fix unexpected behavior from merging openshift#6470 and openshift#6500

00c3068

openshift-merge-robot added a commit that referenced this pull request Oct 27, 2022

Merge pull request #6526 from lranjbar/fix-merge-errors-2022-10-26

15b1759

Fix unexpected behavior from merging #6470 and #6500

bfournie deleted the detect-ready-transition-fail branch October 28, 2022 17:49

bfournie mentioned this pull request Nov 7, 2022

OCPBUGS-3280: Automatically retry install #6567

Merged

Conversation

bfournie commented Oct 6, 2022

Uh oh!

zaneb Oct 6, 2022

Choose a reason for hiding this comment

Uh oh!

bfournie Oct 6, 2022

Choose a reason for hiding this comment

Uh oh!

zaneb commented Oct 6, 2022

Uh oh!

bfournie commented Oct 6, 2022

Uh oh!

celebdor commented Oct 6, 2022

Uh oh!

openshift-ci-robot commented Oct 6, 2022

Uh oh!

zaneb Oct 7, 2022

Choose a reason for hiding this comment

Uh oh!

bfournie Oct 10, 2022

Choose a reason for hiding this comment

Uh oh!

zaneb commented Oct 7, 2022

Uh oh!

bfournie commented Oct 10, 2022

Uh oh!

openshift-ci bot commented Oct 10, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

lranjbar commented Oct 24, 2022

Uh oh!

openshift-ci bot commented Oct 24, 2022

Uh oh!

zaneb commented Oct 25, 2022

Uh oh!

openshift-ci-robot commented Oct 25, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

openshift-ci bot commented Oct 10, 2022 •

edited

Loading