Support oc adm upgrade rollback in CI #51287
Conversation
shellyyang1989
force-pushed
the
rollback-oc
branch
from
55ec891
to
fad80b6
Compare
|
/pj-rehearse periodic-ci-openshift-openshift-tests-private-release-4.16-amd64-rollback-ec-aws-ipi-byo-route53-f28 |
|
@shellyyang1989: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
ci-operator/step-registry/cucushift/upgrade/rollback/cucushift-upgrade-rollback-commands.sh
Outdated
Show resolved
Hide resolved
ci-operator/step-registry/cucushift/upgrade/rollback/cucushift-upgrade-rollback-commands.sh
Show resolved
Hide resolved
ci-operator/step-registry/cucushift/upgrade/rollback/cucushift-upgrade-rollback-commands.sh
Outdated
Show resolved
Hide resolved
ci-operator/step-registry/cucushift/upgrade/rollback/cucushift-upgrade-rollback-commands.sh
Outdated
Show resolved
Hide resolved
ci-operator/step-registry/cucushift/upgrade/rollback/cucushift-upgrade-rollback-commands.sh
Show resolved
Hide resolved
...enshift-tests-private/openshift-openshift-tests-private-release-4.16__amd64-rollback-ec.yaml
Outdated
Show resolved
Hide resolved
...ift/upgrade/qe/sanity/rollback-stable/openshift-upgrade-qe-sanity-rollback-stable-chain.yaml
Outdated
Show resolved
Hide resolved
shellyyang1989
force-pushed
the
rollback-oc
branch
from
fad80b6
to
30c68eb
Compare
| res=$(run_command "oc adm upgrade rollback") | ||
| out="Requested rollback from ${TARGET_VERSION} to ${SOURCE_VERSION}" | ||
| if [[ ${res} == *"${out}"* ]]; then | ||
| echo "Rolling back cluster gets started..." | ||
| else | ||
| echo "Rolling back cluster doesn't start..." |
There was a problem hiding this comment.
perhaps we should fail (return 1) if we're getting unexpected output, as well as capture stderr (2>&1) into res and print the error vs the expected? wdyt?
| res=$(run_command "oc adm upgrade rollback") | |
| out="Requested rollback from ${TARGET_VERSION} to ${SOURCE_VERSION}" | |
| if [[ ${res} == *"${out}"* ]]; then | |
| echo "Rolling back cluster gets started..." | |
| else | |
| echo "Rolling back cluster doesn't start..." | |
| res=$(oc adm upgrade rollback 2>&1 || true) | |
| out="Requested rollback from ${TARGET_VERSION} to ${SOURCE_VERSION}" | |
| if [[ ${res} == *"${out}"* ]]; then | |
| echo "Rolling back cluster from ${TARGET_VERSION} to ${SOURCE_VERSION} started..." | |
| else | |
| echo "Rolling back cluster returned unexpected:\n${res}\nexpecting: ${out}" | |
| return 1 |
ci-operator/step-registry/cucushift/upgrade/rollback/cucushift-upgrade-rollback-commands.sh
Outdated
Show resolved
Hide resolved
shellyyang1989
force-pushed
the
rollback-oc
branch
from
30c68eb
to
aff6db2
Compare
ci-operator/step-registry/cucushift/upgrade/rollback/cucushift-upgrade-rollback-commands.sh
Show resolved
Hide resolved
| local version state | ||
| version=$(oc get clusterversion/version -o jsonpath='{.status.history[0].version}') | ||
| state=$(oc get clusterversion/version -o jsonpath='{.status.history[0].state}') | ||
| if [[ ${version} == "${TARGET_VERSION}" && ${state} == "Completed" ]]; then |
There was a problem hiding this comment.
the same here with https://github.com/openshift/release/pull/51287/files#r1593479250
| version=$(oc get clusterversion/version -o jsonpath='{.status.history[0].version}') | ||
| state=$(oc get clusterversion/version -o jsonpath='{.status.history[0].state}') | ||
| if [[ ${version} == "${TARGET_VERSION}" && ${state} == "Completed" ]]; then | ||
| echo "History check PASSED, cluster is now rollbacked to ${TARGET_VERSION}" && return 0 |
| TARGET_VERSION="$(env "NO_PROXY=*" "no_proxy=*" oc adm release info "${TARGET}" --output=json | jq -r '.metadata.version')" | ||
|
|
||
| SOURCE_VERSION="$(oc get clusterversion --no-headers | awk '{print $2}')" | ||
| SOURCE_MINOR_VERSION="$(echo "${SOURCE_VERSION}" | cut -f2 -d.)" |
There was a problem hiding this comment.
It looks like SOURCE_MINOR_VERSION and TARGET_MINOR_VERSION are not used in any other places except in log, so could be dropped, right?
| # Rollback the cluster to target release | ||
| function rollback() { | ||
| res=$(run_command "oc adm upgrade rollback") | ||
| out="Requested rollback from ${TARGET_VERSION} to ${SOURCE_VERSION}" |
There was a problem hiding this comment.
refer to https://github.com/openshift/release/pull/51287/files#r1593479250, should adjust ${TARGET_VERSION} with ${SOURCE_VERSION}, right?
|
/pj-rehearse periodic-ci-openshift-openshift-tests-private-release-4.16-amd64-rollback-ec-aws-ipi-byo-route53-f28 periodic-ci-openshift-openshift-tests-private-release-4.16-amd64-rollback-nightly-azure-upi-f14 |
|
@shellyyang1989: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
shellyyang1989
force-pushed
the
rollback-oc
branch
3 times, most recently
from
7d2c4da
to
3778e0c
Compare
|
/pj-rehearse periodic-ci-openshift-openshift-tests-private-release-4.16-amd64-rollback-ec-aws-ipi-byo-route53-f28 |
|
@shellyyang1989: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
|
maybe worth adding status dump such as sounds like rollback failures may be a valuable insight however this may go into another PR as well, just an idea |
|
all in all lgtm, leaving to @jiajliu for a final review. |
| cpu: 100m | ||
| memory: 200Mi | ||
| - ref: cucushift-chainupgrade-toimage | ||
| - chain: openshift-upgrade-qe-sanity-rollback | ||
| workflow: cucushift-installer-rehearse-ibmcloud-ipi | ||
| - as: nutanix-ipi-f28 |
There was a problem hiding this comment.
Shall we update the nutanix job name to reflect the install configurations together in this PR?
| @@ -38397,7 +38397,7 @@ periodics: | |||
| ci.openshift.io/generator: prowgen | |||
| job-release: "4.16" | |||
| pj-rehearse.openshift.io/can-be-rehearsed: "true" | |||
| name: periodic-ci-openshift-openshift-tests-private-release-4.16-amd64-rollback-nightly-azure-upi-f14 | |||
| name: periodic-ci-openshift-openshift-tests-private-release-4.16-amd64-rollback-nightly-azure-upi-f28 | |||
There was a problem hiding this comment.
we had a discussion around nightly job in #51287 (comment), wondering the signature problem is not an issue now?
There was a problem hiding this comment.
ah, good catch! I should have updated the latest image to rc.0 but I forgot. Rolling back from nightly to rc should be okay.
| env: | ||
| - name: TIMEOUT | ||
| default: "130" | ||
| documentation: Time to wait for upgrade finish |
|
Rollback step passed in the reheasal. |
shellyyang1989
force-pushed
the
rollback-oc
branch
from
d920cb7
to
b8946b5
Compare
|
[REHEARSALNOTIFIER]
Interacting with pj-rehearseComment: Once you are satisfied with the results of the rehearsals, comment: |
|
[REHEARSALNOTIFIER]
Interacting with pj-rehearseComment: Once you are satisfied with the results of the rehearsals, comment: |
|
@shellyyang1989: The following test failed, say
Full PR test history. Your PR dashboard. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
|
lgtm |
|
/lgtm |
|
@liangxia PTAL. Thank you! |
|
/lgtm |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: jianlinliu, liangxia, shellyyang1989 The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
openshift-ci
bot
added
the
approved
|
/pj-rehearse ack |
|
@jianlinliu: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel. |
openshift-ci-robot
added
the
rehearsals-ack
The change adds support for oc adm upgrade rollback cli.
Refers OTA-1071