Bug 2018053: tuned: add timeout and restarts #286

jmencak · 2021-10-28T05:52:17Z

TuneD profile application typically takes ~0.5s and should never take
more than ~5s. However, there were cases where TuneD daemon got stuck
during application of a profile. Experience shows that subsequent
restarts of TuneD can resolve this in certain situations. TuneD itself,
however, has no mechanism for restarting a profile application that is
taking too long.

This change implements full restarts for TuneD daemon.

Other changes:

tunedStop() is now fully synchronous. Previously, when TuneD
refused to terminate within 10s, SIGKILL was sent without waiting
for TuneD process to terminate.

Also see: rhbz#2013940

TuneD profile application typically takes ~0.5s and should never take more than ~5s. However, there were cases where TuneD daemon got stuck during application of a profile. Experience shows that subsequent restarts of TuneD can resolve this in certain situations. TuneD itself, however, has no mechanism for restarting a profile application that is taking too long. This change implements full restarts for TuneD daemon. Other changes: - tunedStop() is now fully synchronous. Previously, when TuneD refused to terminate within 10s, SIGKILL was sent without waiting for TuneD process to terminate. Also see: rhbz#2013940

openshift-ci · 2021-10-28T05:52:18Z

@jmencak: No Bugzilla bug is referenced in the title of this pull request.
To reference a bug, add 'Bug XXX:' to the title of this pull request and request another bug refresh with /bugzilla refresh.

In response to this:

tuned: add timeout and restarts

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-ci · 2021-10-28T05:52:52Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: jmencak

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [jmencak]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

jmencak · 2021-10-28T06:00:33Z

Notes to reviewers/QE.

Automated cherry-pick was not possible due to conflicts.
Exponential backoff (implemented in 4.9/4.10) was dropped due to too complete event model rewrite from 4.8->4.9. Backporting to 4.8 would require too intrusive/risky changes.
For testing, you could use a similar CR as in the original 4.10 PR. However, note, the profile will that CR (sleep >60) will never get applied as the timeout period will not increase and the sleep will always start from 0. This is also important to know for @skordas . Please check the functionality by watching the tuned pod logs.

jmencak · 2021-10-28T06:01:04Z

/uncc @sjug
/assign @dagrayvid

openshift-ci · 2021-10-28T06:01:54Z

@jmencak: This pull request references Bugzilla bug 2018053, which is invalid:

expected dependent Bugzilla bug 2017488 to be in one of the following states: VERIFIED, RELEASE_PENDING, CLOSED (ERRATA), CLOSED (CURRENTRELEASE), but it is NEW instead

Comment /bugzilla refresh to re-evaluate validity if changes to the Bugzilla bug are made, or edit the title of this pull request to link to a different bug.

In response to this:

Bug 2018053: tuned: add timeout and restarts

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

dagrayvid · 2021-10-29T20:03:46Z

Thanks @jmencak. This looks good to me.

/lgtm

openshift-bot · 2021-10-30T01:00:43Z

/bugzilla refresh

Recalculating validity in case the underlying Bugzilla bug has changed.

openshift-ci · 2021-10-30T01:00:47Z

@openshift-bot: This pull request references Bugzilla bug 2018053, which is invalid:

expected dependent Bugzilla bug 2017488 to be in one of the following states: VERIFIED, RELEASE_PENDING, CLOSED (ERRATA), CLOSED (CURRENTRELEASE), but it is ON_QA instead

Comment /bugzilla refresh to re-evaluate validity if changes to the Bugzilla bug are made, or edit the title of this pull request to link to a different bug.

In response to this:

/bugzilla refresh

Recalculating validity in case the underlying Bugzilla bug has changed.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-bot · 2021-10-31T01:00:23Z

/bugzilla refresh

Recalculating validity in case the underlying Bugzilla bug has changed.

openshift-ci · 2021-10-31T01:00:27Z

@openshift-bot: This pull request references Bugzilla bug 2018053, which is invalid:

expected dependent Bugzilla bug 2017488 to be in one of the following states: VERIFIED, RELEASE_PENDING, CLOSED (ERRATA), CLOSED (CURRENTRELEASE), but it is ON_QA instead

Comment /bugzilla refresh to re-evaluate validity if changes to the Bugzilla bug are made, or edit the title of this pull request to link to a different bug.

In response to this:

/bugzilla refresh

Recalculating validity in case the underlying Bugzilla bug has changed.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-bot · 2021-11-01T01:00:23Z

/bugzilla refresh

Recalculating validity in case the underlying Bugzilla bug has changed.

openshift-ci · 2021-11-01T01:00:27Z

@openshift-bot: This pull request references Bugzilla bug 2018053, which is invalid:

expected dependent Bugzilla bug 2017488 to be in one of the following states: VERIFIED, RELEASE_PENDING, CLOSED (ERRATA), CLOSED (CURRENTRELEASE), but it is ON_QA instead

Comment /bugzilla refresh to re-evaluate validity if changes to the Bugzilla bug are made, or edit the title of this pull request to link to a different bug.

In response to this:

/bugzilla refresh

Recalculating validity in case the underlying Bugzilla bug has changed.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-bot · 2021-11-02T01:00:28Z

/bugzilla refresh

Recalculating validity in case the underlying Bugzilla bug has changed.

openshift-ci · 2021-11-02T01:00:35Z

@openshift-bot: This pull request references Bugzilla bug 2018053, which is invalid:

expected dependent Bugzilla bug 2017488 to be in one of the following states: VERIFIED, RELEASE_PENDING, CLOSED (ERRATA), CLOSED (CURRENTRELEASE), but it is ON_QA instead

Comment /bugzilla refresh to re-evaluate validity if changes to the Bugzilla bug are made, or edit the title of this pull request to link to a different bug.

In response to this:

/bugzilla refresh

Recalculating validity in case the underlying Bugzilla bug has changed.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

jmencak · 2021-11-02T13:08:20Z

/bugzilla refresh

openshift-ci · 2021-11-02T13:08:28Z

@jmencak: This pull request references Bugzilla bug 2018053, which is valid. The bug has been moved to the POST state. The bug has been updated to refer to the pull request using the external bug tracker.

6 validation(s) were run on this bug

bug is open, matching expected state (open)
bug target release (4.8.z) matches configured target release for branch (4.8.z)
bug is in the state NEW, which is one of the valid states (NEW, ASSIGNED, ON_DEV, POST, POST)
dependent bug Bugzilla bug 2017488 is in the state VERIFIED, which is one of the valid states (VERIFIED, RELEASE_PENDING, CLOSED (ERRATA), CLOSED (CURRENTRELEASE))
dependent Bugzilla bug 2017488 targets the "4.9.z" release, which is one of the valid target releases: 4.9.0, 4.9.z
bug has dependents

No GitHub users were found matching the public email listed for the QA contact in Bugzilla (skordas@redhat.com), skipping review request.

In response to this:

/bugzilla refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

jmencak · 2021-11-02T13:18:14Z

Backport risk assessment

If the Bugzilla associated with the PR has the "FastFix" keyword, the subjective assessment on the issue has already been done and a customer is impacted. These PRs should be prioritized for merge.
- not applicable
For branches that are in the Maintenance lifecycle phase:
- not applicable
The bug has significant impact either through severity, reduction in supportability, or number of users affected.
- yes
The severity field of the bug must be set to accurately reflect criticality.
- bugzilla/severity-high
The PR is merged in the next newer release branch and the bug for that newer release is VERIFIED by QE.
- yes
The PR’s description is well formed with user-focused release notes that state the bug number, impact, cause, and resolution. Where appropriate, it should also contain information about how a user can identify whether a particular cluster is affected.
- yes

jmencak · 2021-11-02T13:18:25Z

/label backport-risk-assessed

jmencak · 2021-11-02T13:19:04Z

@wabouhamad , can we get cherry-pick-approved? Many thanks!

wabouhamad · 2021-11-03T12:35:40Z

/label cherry-pick-approved

openshift-ci · 2021-11-03T12:40:50Z

@jmencak: All pull requests linked via external trackers have merged:

openshift/cluster-node-tuning-operator#286

Bugzilla bug 2018053 has been moved to the MODIFIED state.

In response to this:

Bug 2018053: tuned: add timeout and restarts

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-ci bot requested review from kpouget and sjug October 28, 2021 05:52

openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Oct 28, 2021

openshift-ci bot assigned dagrayvid Oct 28, 2021

openshift-ci bot removed the request for review from sjug October 28, 2021 06:01

jmencak changed the title ~~tuned: add timeout and restarts~~ Bug 2018053: tuned: add timeout and restarts Oct 28, 2021

openshift-ci bot added the bugzilla/severity-high Referenced Bugzilla bug's severity is high for the branch this PR is targeting. label Oct 28, 2021

openshift-ci bot added the bugzilla/invalid-bug Indicates that a referenced Bugzilla bug is invalid for the branch this PR is targeting. label Oct 28, 2021

openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Oct 29, 2021

openshift-ci bot added bugzilla/valid-bug Indicates that a referenced Bugzilla bug is valid for the branch this PR is targeting. and removed bugzilla/invalid-bug Indicates that a referenced Bugzilla bug is invalid for the branch this PR is targeting. labels Nov 2, 2021

openshift-ci bot added the backport-risk-assessed Indicates a PR to a release branch has been evaluated and considered safe to accept. label Nov 2, 2021

openshift-ci bot added the cherry-pick-approved Indicates a cherry-pick PR into a release branch has been approved by the release branch manager. label Nov 3, 2021

openshift-merge-robot merged commit 8e9c873 into openshift:release-4.8 Nov 3, 2021

jmencak deleted the 4.8-tuned-application-timeout branch November 3, 2021 13:56

jmencak mentioned this pull request Nov 5, 2021

Bug 2020518: tuned: add timeout and restarts #287

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Bug 2018053: tuned: add timeout and restarts #286

Bug 2018053: tuned: add timeout and restarts #286

jmencak commented Oct 28, 2021

openshift-ci bot commented Oct 28, 2021

openshift-ci bot commented Oct 28, 2021

jmencak commented Oct 28, 2021

jmencak commented Oct 28, 2021

openshift-ci bot commented Oct 28, 2021

dagrayvid commented Oct 29, 2021

openshift-bot commented Oct 30, 2021

openshift-ci bot commented Oct 30, 2021

openshift-bot commented Oct 31, 2021

openshift-ci bot commented Oct 31, 2021

openshift-bot commented Nov 1, 2021

openshift-ci bot commented Nov 1, 2021

openshift-bot commented Nov 2, 2021

openshift-ci bot commented Nov 2, 2021

jmencak commented Nov 2, 2021

openshift-ci bot commented Nov 2, 2021

jmencak commented Nov 2, 2021

jmencak commented Nov 2, 2021

jmencak commented Nov 2, 2021

wabouhamad commented Nov 3, 2021

openshift-ci bot commented Nov 3, 2021

Bug 2018053: tuned: add timeout and restarts #286

Bug 2018053: tuned: add timeout and restarts #286

Conversation

jmencak commented Oct 28, 2021

openshift-ci bot commented Oct 28, 2021

openshift-ci bot commented Oct 28, 2021

jmencak commented Oct 28, 2021

jmencak commented Oct 28, 2021

openshift-ci bot commented Oct 28, 2021

dagrayvid commented Oct 29, 2021

openshift-bot commented Oct 30, 2021

openshift-ci bot commented Oct 30, 2021

openshift-bot commented Oct 31, 2021

openshift-ci bot commented Oct 31, 2021

openshift-bot commented Nov 1, 2021

openshift-ci bot commented Nov 1, 2021

openshift-bot commented Nov 2, 2021

openshift-ci bot commented Nov 2, 2021

jmencak commented Nov 2, 2021

openshift-ci bot commented Nov 2, 2021

jmencak commented Nov 2, 2021

jmencak commented Nov 2, 2021

jmencak commented Nov 2, 2021

wabouhamad commented Nov 3, 2021

openshift-ci bot commented Nov 3, 2021