Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bug 2078969: Ignore Profile updates triggered by old operands #355

Merged
merged 1 commit into from Apr 28, 2022

Conversation

jmencak
Copy link
Contributor

@jmencak jmencak commented Apr 27, 2022

Depending on TuneD configuration and version, TuneD may calculate
different kernel command-line parameters for the same TuneD profile.
This is a problem during NTO operator upgrades. The old (still running)
and newly started NTO operands (already running with newer version of
TuneD) will compete to push their specific version of calculated kernel
command-line parameters. The new version will always win, but this
"competition" may trigger extra unwanted node reboots.

This change prevents the operator from accepting the
MachineConfig-specific changes from old (not-yet-upgraded) operands.
This prevents the NTO-generated MachineConfigs from being flipped
between the old and new versions of kernel command-line parameters.

Other changes:

  • Fix a potential SEGV when a Profile fails to be retrieved by the
    operand.

Resolves rhbz#2078969.

@openshift-ci openshift-ci bot added bugzilla/severity-high Referenced Bugzilla bug's severity is high for the branch this PR is targeting. bugzilla/invalid-bug Indicates that a referenced Bugzilla bug is invalid for the branch this PR is targeting. labels Apr 27, 2022
@openshift-ci
Copy link
Contributor

openshift-ci bot commented Apr 27, 2022

@jmencak: This pull request references Bugzilla bug 2078969, which is invalid:

  • expected the bug to target the "4.11.0" release, but it targets "---" instead

Comment /bugzilla refresh to re-evaluate validity if changes to the Bugzilla bug are made, or edit the title of this pull request to link to a different bug.

In response to this:

Bug 2078969: Ignore Profile updates triggered by old operands

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@openshift-ci openshift-ci bot requested review from ffromani and kpouget April 27, 2022 12:32
@openshift-ci
Copy link
Contributor

openshift-ci bot commented Apr 27, 2022

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: jmencak

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-ci openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Apr 27, 2022
@jmencak
Copy link
Contributor Author

jmencak commented Apr 27, 2022

/bugzilla refresh

@openshift-ci openshift-ci bot added bugzilla/valid-bug Indicates that a referenced Bugzilla bug is valid for the branch this PR is targeting. and removed bugzilla/invalid-bug Indicates that a referenced Bugzilla bug is invalid for the branch this PR is targeting. labels Apr 27, 2022
@openshift-ci
Copy link
Contributor

openshift-ci bot commented Apr 27, 2022

@jmencak: This pull request references Bugzilla bug 2078969, which is valid. The bug has been moved to the POST state. The bug has been updated to refer to the pull request using the external bug tracker.

3 validation(s) were run on this bug
  • bug is open, matching expected state (open)
  • bug target release (4.11.0) matches configured target release for branch (4.11.0)
  • bug is in the state ASSIGNED, which is one of the valid states (NEW, ASSIGNED, ON_DEV, POST, POST)

No GitHub users were found matching the public email listed for the QA contact in Bugzilla (liqcui@redhat.com), skipping review request.

In response to this:

/bugzilla refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@jmencak
Copy link
Contributor Author

jmencak commented Apr 27, 2022

/cc @dagrayvid
Additional info. I've noticed this issue yesterday when upgrading from 4.8->4.9 and using realtime profile (other upgrades up to 4.11 were fine). We need to get this backported to at least 4.9 (that will fix the 4.8->4.9 upgrade issue). Didn't see any issues from 4.7->4.8 upgrades, but ideally, I'd like to backport this to 4.8 too.

@openshift-ci openshift-ci bot requested a review from dagrayvid April 27, 2022 12:40
Depending on TuneD configuration and version, TuneD may calculate
different kernel command-line parameters for the same TuneD profile.
This is a problem during NTO operator upgrades.  The old (still running)
and newly started NTO operands (already running with newer version of
TuneD) will compete to push their specific version of calculated kernel
command-line parameters.  The new version will always win, but this
"competition" may trigger extra unwanted node reboots.

This change prevents the operator from accepting the
MachineConfig-specific changes from old (not-yet-upgraded) operands.
This prevents the NTO-generated MachineConfigs from being flipped
between the old and new versions of kernel command-line parameters.

Other changes:
  - Fix a potential SEGV when a Profile fails to be retrieved by the
    operand.

Resolves rhbz#2078969.
@openshift-ci
Copy link
Contributor

openshift-ci bot commented Apr 27, 2022

@jmencak: This pull request references Bugzilla bug 2078969, which is valid.

3 validation(s) were run on this bug
  • bug is open, matching expected state (open)
  • bug target release (4.11.0) matches configured target release for branch (4.11.0)
  • bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, ON_DEV, POST, POST)

No GitHub users were found matching the public email listed for the QA contact in Bugzilla (liqcui@redhat.com), skipping review request.

In response to this:

Bug 2078969: Ignore Profile updates triggered by old operands

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@jmencak
Copy link
Contributor Author

jmencak commented Apr 27, 2022

Piggy-backing a fix (I know it is not ideal...) for a SEGV issue found by @fromanirh. Thank you for the report.

@jmencak
Copy link
Contributor Author

jmencak commented Apr 27, 2022

/retest

1 similar comment
@jmencak
Copy link
Contributor Author

jmencak commented Apr 28, 2022

/retest

@jmencak
Copy link
Contributor Author

jmencak commented Apr 28, 2022

Infra issues.

error: timed out waiting for the condition on clusteroperators/network

/joke
/retest

@openshift-ci
Copy link
Contributor

openshift-ci bot commented Apr 28, 2022

@jmencak: I've been trying to come up with a dad joke about momentum . . . but I just can't seem to get it going.

In response to this:

Infra issues.

error: timed out waiting for the condition on clusteroperators/network

/joke
/retest

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@jmencak
Copy link
Contributor Author

jmencak commented Apr 28, 2022

/retest

@jmencak
Copy link
Contributor Author

jmencak commented Apr 28, 2022

Infra issues Cluster operator monitoring is not available. I'm not going to and waste any more money on retesting until this is reviewed. @dagrayvid could you please take a look? Thank you!

@dagrayvid
Copy link
Contributor

Thanks Jiri. These fixes looks good to me.

/lgtm

@jmencak
Copy link
Contributor Author

jmencak commented Apr 28, 2022

Thank you David.
/retest

@openshift-ci openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Apr 28, 2022
@jmencak
Copy link
Contributor Author

jmencak commented Apr 28, 2022

Failed to wait for bootstrapping to complete. 

/retest

@jmencak
Copy link
Contributor Author

jmencak commented Apr 28, 2022

/retest

@jmencak
Copy link
Contributor Author

jmencak commented Apr 28, 2022

Bot, please give me a favour and once this merges
/cherry-pick release-4.10

@openshift-cherrypick-robot

@jmencak: once the present PR merges, I will cherry-pick it on top of release-4.10 in a new PR and assign it to you.

In response to this:

Bot, please give me a favour and once this merges
/cherry-pick release-4.10

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@openshift-ci
Copy link
Contributor

openshift-ci bot commented Apr 28, 2022

@jmencak: all tests passed!

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@openshift-merge-robot openshift-merge-robot merged commit 74331b5 into openshift:master Apr 28, 2022
@openshift-ci
Copy link
Contributor

openshift-ci bot commented Apr 28, 2022

@jmencak: All pull requests linked via external trackers have merged:

Bugzilla bug 2078969 has been moved to the MODIFIED state.

In response to this:

Bug 2078969: Ignore Profile updates triggered by old operands

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@openshift-cherrypick-robot

@jmencak: #355 failed to apply on top of branch "release-4.10":

Applying: Ignore Profile updates triggered by old operands
Using index info to reconstruct a base tree...
M	pkg/apis/tuned/v1/tuned_types.go
M	pkg/operator/controller.go
M	pkg/tuned/controller.go
Falling back to patching base and 3-way merge...
Auto-merging pkg/tuned/controller.go
Auto-merging pkg/operator/controller.go
CONFLICT (content): Merge conflict in pkg/operator/controller.go
Auto-merging pkg/apis/tuned/v1/tuned_types.go
error: Failed to merge in the changes.
hint: Use 'git am --show-current-patch=diff' to see the failed patch
Patch failed at 0001 Ignore Profile updates triggered by old operands
When you have resolved this problem, run "git am --continue".
If you prefer to skip this patch, run "git am --skip" instead.
To restore the original branch and stop patching, run "git am --abort".

In response to this:

Bot, please give me a favour and once this merges
/cherry-pick release-4.10

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@jmencak jmencak deleted the 4.11-bz2078969 branch April 29, 2022 05:06
IlyaTyomkin pushed a commit to IlyaTyomkin/cluster-node-tuning-operator that referenced this pull request May 23, 2023
Depending on TuneD configuration and version, TuneD may calculate
different kernel command-line parameters for the same TuneD profile.
This is a problem during NTO operator upgrades.  The old (still running)
and newly started NTO operands (already running with newer version of
TuneD) will compete to push their specific version of calculated kernel
command-line parameters.  The new version will always win, but this
"competition" may trigger extra unwanted node reboots.

This change prevents the operator from accepting the
MachineConfig-specific changes from old (not-yet-upgraded) operands.
This prevents the NTO-generated MachineConfigs from being flipped
between the old and new versions of kernel command-line parameters.

Other changes:
  - Fix a potential SEGV when a Profile fails to be retrieved by the
    operand.

Resolves rhbz#2078969.

Co-authored-by: Jiri Mencak <jmencak@users.noreply.github.com>
IlyaTyomkin pushed a commit to IlyaTyomkin/cluster-node-tuning-operator that referenced this pull request Jun 13, 2023
Depending on TuneD configuration and version, TuneD may calculate
different kernel command-line parameters for the same TuneD profile.
This is a problem during NTO operator upgrades.  The old (still running)
and newly started NTO operands (already running with newer version of
TuneD) will compete to push their specific version of calculated kernel
command-line parameters.  The new version will always win, but this
"competition" may trigger extra unwanted node reboots.

This change prevents the operator from accepting the
MachineConfig-specific changes from old (not-yet-upgraded) operands.
This prevents the NTO-generated MachineConfigs from being flipped
between the old and new versions of kernel command-line parameters.

Other changes:
  - Fix a potential SEGV when a Profile fails to be retrieved by the
    operand.

Resolves rhbz#2078969.

Co-authored-by: Jiri Mencak <jmencak@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. bugzilla/severity-high Referenced Bugzilla bug's severity is high for the branch this PR is targeting. bugzilla/valid-bug Indicates that a referenced Bugzilla bug is valid for the branch this PR is targeting. lgtm Indicates that a PR is ready to be merged.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants