[release-4.8] Bug 1975078: Gracefully shutdown taking around 6-7 mins (libvirt provider) #2636

openshift-cherrypick-robot · 2021-06-23T05:25:31Z

This is an automated cherry-pick of #2631

/assign praveenkumar

The machine-config-daemon gets stuck blocking SIGTERM on rebootless updatesbecause it only removes its SIGTERM protection when it reboots or when it encounters an error in the triggerUpdateWithmachineConfig-> update->performPostConfigChangeAction cycle. This changes the behavior such that it will remove the protection on a successful rebootless update and adds some logging messages so it's more clear when it starts and stops protecting itself.

openshift-ci · 2021-06-23T05:25:42Z

@openshift-cherrypick-robot: Bugzilla bug 1965992 has been cloned as Bugzilla bug 1975078. Retitling PR to link against new bug.
/retitle [release-4.8] Bug 1975078: Gracefully shutdown taking around 6-7 mins (libvirt provider)

In response to this:

[release-4.8] Bug 1965992: Gracefully shutdown taking around 6-7 mins (libvirt provider)

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-ci · 2021-06-23T05:25:50Z

@openshift-cherrypick-robot: This pull request references Bugzilla bug 1975078, which is valid. The bug has been moved to the POST state. The bug has been updated to refer to the pull request using the external bug tracker.

6 validation(s) were run on this bug

bug is open, matching expected state (open)
bug target release (4.8.0) matches configured target release for branch (4.8.0)
bug is in the state NEW, which is one of the valid states (NEW, ASSIGNED, ON_DEV, POST, POST)
dependent bug Bugzilla bug 1965992 is in the state MODIFIED, which is one of the valid states (MODIFIED, ON_QA, VERIFIED)
dependent Bugzilla bug 1965992 targets the "4.9.0" release, which is one of the valid target releases: 4.9.0
bug has dependents

Requesting review from QA contact:
/cc @mike-nguyen

In response to this:

[release-4.8] Bug 1975078: Gracefully shutdown taking around 6-7 mins (libvirt provider)

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

praveenkumar · 2021-06-23T08:24:49Z

/retest

sinnykumari

/lgtm

openshift-ci · 2021-06-23T13:09:51Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: openshift-cherrypick-robot, sinnykumari

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [sinnykumari]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

openshift-bot · 2021-06-23T13:24:07Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2021-06-23T13:37:07Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2021-06-23T14:16:08Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2021-06-23T14:29:09Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2021-06-23T15:08:06Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2021-06-23T15:47:09Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2021-06-23T16:39:07Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2021-06-23T18:23:09Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2021-06-23T19:38:06Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2021-06-23T19:50:06Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2021-06-23T21:20:19Z

/retest

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2021-06-23T21:59:20Z

/retest

Please review the full test history for this PR and help us cut down flakes.

praveenkumar · 2021-06-29T09:20:35Z

Since this backport PR is not critical for 4.8 release, it will go into 4.8 z stream release. Putting hold until it gets cherry-pick approved for z stream release and avoid retesting failing jobs.

@sinnykumari It is quite critical for us (CRC side) to move forward, right now we are using a hack as workaround which is suggested on the original BZ ( Deleting the MCD pods just before shutdown). We really want to avoid those hack on CRC side which are issues on openshift and fixed. Also I tested 4.9 nightly which have this fix and it worked. I would like to have it as part of 4.8 release side instead of any z stream so we can also ship CRC with 4.8 without any hacky workaround.

sinnykumari · 2021-06-29T14:41:57Z

I understand your concern, but since we are close to 4.8 going GA, to avoid any regression only critical bugs get approved from group lead and this bug severity is marked as medium. If you still think it should be merged in 4.8 GA instead of waiting for z stream, it would require priority escalation from CRC team.

yuqi-zhang · 2021-06-30T19:25:19Z

I agree with what Sinny said. 4.8 is already past final freeze, so we cannot merge any code into 4.8.0 anymore barring extraordinary circumstances

kikisdeliveryservice · 2021-07-13T23:27:36Z

@sinnykumari since we need approval for this to merge, I think we can remove the hold now? WDYT

sinnykumari · 2021-07-14T09:34:28Z

@sinnykumari since we need approval for this to merge, I think we can remove the hold now? WDYT

PR is already approved, it doesn't have group-lead approval but that doesn't prevent bot to keep retesting. Hold is mainly to keep bot away 😆

sinnykumari · 2021-07-14T09:36:47Z

Since, all failed tests have completed, hopefully skip will work as expected now
/skip

sinnykumari · 2021-07-14T09:37:02Z

/hold cancel

sdodson · 2021-07-22T18:36:19Z

Re-validating bugzilla status after prow config updates.
/bugzilla refresh

openshift-ci · 2021-07-22T18:38:25Z

@sdodson: This pull request references Bugzilla bug 1975078, which is valid.

6 validation(s) were run on this bug

bug is open, matching expected state (open)
bug target release (4.8.z) matches configured target release for branch (4.8.z)
bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, ON_DEV, POST, POST)
dependent bug Bugzilla bug 1965992 is in the state VERIFIED, which is one of the valid states (VERIFIED, RELEASE_PENDING, CLOSED (ERRATA), CLOSED (CURRENTRELEASE))
dependent Bugzilla bug 1965992 targets the "4.9.0" release, which is one of the valid target releases: 4.9.0
bug has dependents

Requesting review from QA contact:
/cc @mike-nguyen

In response to this:

Re-validating bugzilla status after prow config updates.
/bugzilla refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

ecordell · 2021-08-04T19:03:21Z

[patch-manager] ⌛ This pull request was not picked by the patch manager for the current z-stream window and have to wait for the next window.

skipped

Score: 0.20
Reason: missed cutoff today, may make it tomorrow

NOTE: This message was automatically generated, if you have questions please ask on #forum-release

ecordell · 2021-08-05T18:54:48Z

[patch-manager] ⌛ This pull request was not picked by the patch manager for the current z-stream window and have to wait for the next window.

skipped

Score: 0.20
Reason: missed cutoff today, may make it tomorrow

NOTE: This message was automatically generated, if you have questions please ask on #forum-release

ecordell · 2021-08-05T18:55:26Z

[patch-manager] ⌛ This pull request was not picked by the patch manager for the current z-stream window and have to wait for the next window.

skipped

Score: 0.20
Reason: missed cutoff for today, may make tomorrow

NOTE: This message was automatically generated, if you have questions please ask on #forum-release

ecordell · 2021-08-06T16:28:10Z

[patch-manager] ⌛ This pull request was not picked by the patch manager for the current z-stream window and have to wait for the next window.

skipped

Score: 0.20
Reason: skipping for now, may still be tagged today

NOTE: This message was automatically generated, if you have questions please ask on #forum-release

ecordell · 2021-08-06T17:55:40Z

[patch-manager] ⌛ This pull request was not picked by the patch manager for the current z-stream window and have to wait for the next window.

skipped

Score: 0.20
Reason: missed cutoff this week

NOTE: This message was automatically generated, if you have questions please ask on #forum-release

openshift-bot · 2021-08-11T20:00:06Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2021-08-12T02:08:08Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2021-08-12T02:20:13Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2021-08-12T05:31:09Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2021-08-12T05:44:07Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2021-08-12T07:51:09Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2021-08-12T10:52:34Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2021-08-12T17:18:35Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2021-08-12T17:57:34Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

openshift-ci · 2021-08-12T18:24:22Z

@openshift-cherrypick-robot: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Rerun command
ci/prow/e2e-gcp-single-node	`a7ef54f`	link	`/test e2e-gcp-single-node`
ci/prow/e2e-metal-ipi-ovn-dualstack	`a7ef54f`	link	`/test e2e-metal-ipi-ovn-dualstack`
ci/prow/okd-e2e-upgrade	`a7ef54f`	link	`/test okd-e2e-upgrade`

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

openshift-bot · 2021-08-12T18:36:35Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

openshift-bot · 2021-08-12T18:49:35Z

/retest-required

Please review the full test history for this PR and help us cut down flakes.

openshift-ci · 2021-08-12T20:45:34Z

@openshift-cherrypick-robot: All pull requests linked via external trackers have merged:

openshift/machine-config-operator#2636

Bugzilla bug 1975078 has been moved to the MODIFIED state.

In response to this:

[release-4.8] Bug 1975078: Gracefully shutdown taking around 6-7 mins (libvirt provider)

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-ci bot assigned praveenkumar Jun 23, 2021

openshift-cherrypick-robot mentioned this pull request Jun 23, 2021

Bug 1965992: Gracefully shutdown taking around 6-7 mins (libvirt provider) #2631

Merged

openshift-ci bot changed the title ~~[release-4.8] Bug 1965992: Gracefully shutdown taking around 6-7 mins (libvirt provider)~~ [release-4.8] Bug 1975078: Gracefully shutdown taking around 6-7 mins (libvirt provider) Jun 23, 2021

openshift-ci bot requested review from kikisdeliveryservice and yuqi-zhang June 23, 2021 05:25

openshift-ci bot added bugzilla/severity-medium Referenced Bugzilla bug's severity is medium for the branch this PR is targeting. bugzilla/valid-bug Indicates that a referenced Bugzilla bug is valid for the branch this PR is targeting. labels Jun 23, 2021

openshift-ci bot requested a review from mike-nguyen June 23, 2021 05:25

praveenkumar removed their assignment Jun 23, 2021

sinnykumari approved these changes Jun 23, 2021

View reviewed changes

openshift-ci bot assigned sinnykumari Jun 23, 2021

openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Jun 23, 2021

openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jun 23, 2021

openshift-ci bot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jul 14, 2021

bparees added the cherry-pick-approved Indicates a cherry-pick PR into a release branch has been approved by the release branch manager. label Aug 11, 2021

openshift-ci bot merged commit 8423356 into openshift:release-4.8 Aug 12, 2021

[release-4.8] Bug 1975078: Gracefully shutdown taking around 6-7 mins (libvirt provider) #2636

[release-4.8] Bug 1975078: Gracefully shutdown taking around 6-7 mins (libvirt provider) #2636

Conversation

openshift-cherrypick-robot commented Jun 23, 2021

openshift-ci bot commented Jun 23, 2021

openshift-ci bot commented Jun 23, 2021

praveenkumar commented Jun 23, 2021

sinnykumari left a comment

Choose a reason for hiding this comment

openshift-ci bot commented Jun 23, 2021

openshift-bot commented Jun 23, 2021

openshift-bot commented Jun 23, 2021

openshift-bot commented Jun 23, 2021

openshift-bot commented Jun 23, 2021

openshift-bot commented Jun 23, 2021

openshift-bot commented Jun 23, 2021

openshift-bot commented Jun 23, 2021

openshift-bot commented Jun 23, 2021

openshift-bot commented Jun 23, 2021

openshift-bot commented Jun 23, 2021

openshift-bot commented Jun 23, 2021

openshift-bot commented Jun 23, 2021

praveenkumar commented Jun 29, 2021

sinnykumari commented Jun 29, 2021

yuqi-zhang commented Jun 30, 2021

kikisdeliveryservice commented Jul 13, 2021

sinnykumari commented Jul 14, 2021

sinnykumari commented Jul 14, 2021

sinnykumari commented Jul 14, 2021

sdodson commented Jul 22, 2021

openshift-ci bot commented Jul 22, 2021

ecordell commented Aug 4, 2021

ecordell commented Aug 5, 2021

ecordell commented Aug 5, 2021

ecordell commented Aug 6, 2021

ecordell commented Aug 6, 2021

openshift-bot commented Aug 11, 2021

openshift-bot commented Aug 12, 2021

openshift-bot commented Aug 12, 2021

openshift-bot commented Aug 12, 2021

openshift-bot commented Aug 12, 2021

openshift-bot commented Aug 12, 2021

openshift-bot commented Aug 12, 2021

openshift-bot commented Aug 12, 2021

openshift-bot commented Aug 12, 2021

openshift-ci bot commented Aug 12, 2021 • edited

openshift-bot commented Aug 12, 2021

openshift-bot commented Aug 12, 2021

openshift-ci bot commented Aug 12, 2021

openshift-ci bot commented Aug 12, 2021 •

edited