Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

MCO-681, OCPBUGS-20427: Add Key State Metrics, No datapoint found when querying up certain registered metrics #3965

Merged
merged 1 commit into from Nov 29, 2023

Conversation

inesqyx
Copy link
Contributor

@inesqyx inesqyx commented Oct 10, 2023

In this PR, I have enabled several metrics to report key state of the MCC & MCD

  • Metrics for MCC sub-controllers
    • Introduced state metric: MCCSubControllerState, which has label "subcontroller, state object, to record the action of the subcontroller on certain object e.g. machine-config-controller-render Sync Machine Config Pool master timestampA --> the render controller has synced the master pool at timestamp A.
  • Timestamps for MCD config drifts
  • Timestamps for pivot error

I have also fixed the bug reported in https://issues.redhat.com/browse/OCPBUGS-20427:

Before: when query for mcc_drain_err, it returns no datapoint found
Now: when query for mcc_drain_err, it returns the initial value set until change happened

@openshift-ci-robot openshift-ci-robot added the jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. label Oct 10, 2023
@openshift-ci-robot
Copy link
Contributor

openshift-ci-robot commented Oct 10, 2023

@inesqyx: This pull request references MCO-681 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.15.0" version, but no target version was set.

In response to this:

WIP

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@openshift-ci openshift-ci bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Oct 10, 2023
@openshift-ci
Copy link
Contributor

openshift-ci bot commented Oct 10, 2023

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

@openshift-merge-robot openshift-merge-robot added needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. and removed needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. labels Nov 23, 2023
@inesqyx inesqyx changed the title MCO-681: Add Key State Metrics MCO-681, OCPBUGS-20427: Add Key State Metrics, No datapoint found when querying up certain registered metrics Nov 23, 2023
@openshift-ci-robot openshift-ci-robot added the jira/valid-bug Indicates that a referenced Jira bug is valid for the branch this PR is targeting. label Nov 23, 2023
@openshift-ci-robot
Copy link
Contributor

@inesqyx: This pull request references Jira Issue OCPBUGS-20427, which is valid. The bug has been moved to the POST state.

3 validation(s) were run on this bug
  • bug is open, matching expected state (open)
  • bug target version (4.15.0) matches configured target version for branch (4.15.0)
  • bug is in the state New, which is one of the valid states (NEW, ASSIGNED, POST)

Requesting review from QA contact:
/cc @sergiordlr

The bug has been updated to refer to the pull request using the external bug tracker.

In response to this:

WIP

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@openshift-ci-robot
Copy link
Contributor

@inesqyx: This pull request references Jira Issue OCPBUGS-20427, which is valid.

3 validation(s) were run on this bug
  • bug is open, matching expected state (open)
  • bug target version (4.15.0) matches configured target version for branch (4.15.0)
  • bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, POST)

Requesting review from QA contact:
/cc @sergiordlr

In response to this:

In this PR, I have enabled several metrics to report key state of the MCC & MCD

  • Metrics for MCC sub-controllers
    • Introduced state metric: MCCSubControllerState

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@inesqyx inesqyx marked this pull request as ready for review November 23, 2023 16:58
@openshift-ci openshift-ci bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Nov 23, 2023
@inesqyx inesqyx force-pushed the MCO-681 branch 2 times, most recently from bb308fa to 3b1a64c Compare November 23, 2023 17:38
@inesqyx
Copy link
Contributor Author

inesqyx commented Nov 23, 2023

/retest-required

@inesqyx
Copy link
Contributor Author

inesqyx commented Nov 23, 2023

/test verify

@inesqyx
Copy link
Contributor Author

inesqyx commented Nov 23, 2023

/retest-required

@inesqyx
Copy link
Contributor Author

inesqyx commented Nov 23, 2023

/test ci/prow/okd-scos-images

Copy link
Contributor

openshift-ci bot commented Nov 23, 2023

@inesqyx: The specified target(s) for /test were not found.
The following commands are available to trigger required jobs:

  • /test 4.12-upgrade-from-stable-4.11-images
  • /test cluster-bootimages
  • /test e2e-aws-ovn
  • /test e2e-aws-ovn-upgrade
  • /test e2e-gcp-op
  • /test e2e-gcp-op-single-node
  • /test e2e-hypershift
  • /test images
  • /test okd-scos-images
  • /test unit
  • /test verify

The following commands are available to trigger optional jobs:

  • /test 4.12-upgrade-from-stable-4.11-e2e-aws-ovn-upgrade
  • /test bootstrap-unit
  • /test e2e-alibabacloud-ovn
  • /test e2e-aws-disruptive
  • /test e2e-aws-ovn-fips
  • /test e2e-aws-ovn-fips-op
  • /test e2e-aws-ovn-workers-rhel8
  • /test e2e-aws-proxy
  • /test e2e-aws-serial
  • /test e2e-aws-single-node
  • /test e2e-aws-upgrade-single-node
  • /test e2e-aws-workers-rhel8
  • /test e2e-azure
  • /test e2e-azure-ovn-upgrade
  • /test e2e-azure-upgrade
  • /test e2e-gcp-op-layering
  • /test e2e-gcp-ovn-rt-upgrade
  • /test e2e-gcp-rt
  • /test e2e-gcp-rt-op
  • /test e2e-gcp-single-node
  • /test e2e-gcp-upgrade
  • /test e2e-metal-assisted
  • /test e2e-metal-ipi
  • /test e2e-metal-ipi-ovn-dualstack
  • /test e2e-metal-ipi-ovn-ipv6
  • /test e2e-openstack
  • /test e2e-openstack-dualstack
  • /test e2e-openstack-externallb
  • /test e2e-openstack-parallel
  • /test e2e-ovirt
  • /test e2e-ovirt-upgrade
  • /test e2e-ovn-step-registry
  • /test e2e-vsphere
  • /test e2e-vsphere-upgrade
  • /test e2e-vsphere-upi
  • /test e2e-vsphere-upi-zones
  • /test e2e-vsphere-zones
  • /test okd-e2e-aws
  • /test okd-e2e-gcp-op
  • /test okd-e2e-upgrade
  • /test okd-e2e-vsphere
  • /test okd-images
  • /test okd-scos-e2e-aws-ovn
  • /test okd-scos-e2e-gcp-op
  • /test okd-scos-e2e-gcp-ovn-upgrade
  • /test okd-scos-e2e-vsphere

Use /test all to run the following jobs that were automatically triggered:

  • pull-ci-openshift-machine-config-operator-master-bootstrap-unit
  • pull-ci-openshift-machine-config-operator-master-e2e-aws-ovn
  • pull-ci-openshift-machine-config-operator-master-e2e-aws-ovn-upgrade
  • pull-ci-openshift-machine-config-operator-master-e2e-gcp-op
  • pull-ci-openshift-machine-config-operator-master-e2e-gcp-op-layering
  • pull-ci-openshift-machine-config-operator-master-e2e-gcp-op-single-node
  • pull-ci-openshift-machine-config-operator-master-e2e-hypershift
  • pull-ci-openshift-machine-config-operator-master-images
  • pull-ci-openshift-machine-config-operator-master-okd-images
  • pull-ci-openshift-machine-config-operator-master-okd-scos-e2e-aws-ovn
  • pull-ci-openshift-machine-config-operator-master-okd-scos-images
  • pull-ci-openshift-machine-config-operator-master-unit
  • pull-ci-openshift-machine-config-operator-master-verify

In response to this:

/test ci/prow/okd-scos-images

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@inesqyx
Copy link
Contributor Author

inesqyx commented Nov 23, 2023

/test okd-scos-images

1 similar comment
@inesqyx
Copy link
Contributor Author

inesqyx commented Nov 23, 2023

/test okd-scos-images

@inesqyx
Copy link
Contributor Author

inesqyx commented Nov 23, 2023

/retest-required

1 similar comment
@inesqyx
Copy link
Contributor Author

inesqyx commented Nov 24, 2023

/retest-required

@inesqyx
Copy link
Contributor Author

inesqyx commented Nov 27, 2023

/test unit

1 similar comment
@inesqyx
Copy link
Contributor Author

inesqyx commented Nov 27, 2023

/test unit

@sergiordlr
Copy link

sergiordlr commented Nov 28, 2023

There is a problem with the way we clean the mcd_config_drift metric when the drift is fixed.

  1. Force a config drfit in a node
  2. The mcd_config_drift is reported properly
  3. Remove the MCD pod running in the node where the drift happened
  4. The metric is removed (it should not be removed, since the drift is not fixed)

We will track this behaviour here:
https://issues.redhat.com/browse/OCPBUGS-24003

@inesqyx inesqyx force-pushed the MCO-681 branch 2 times, most recently from 1ae8079 to 5230cbd Compare November 28, 2023 21:25
Copy link
Contributor

@yuqi-zhang yuqi-zhang left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm

@yuqi-zhang
Copy link
Contributor

/hold

For qe-approval

@openshift-ci openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Nov 29, 2023
Copy link
Contributor

openshift-ci bot commented Nov 29, 2023

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: inesqyx, yuqi-zhang

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-ci openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Nov 29, 2023
@inesqyx
Copy link
Contributor Author

inesqyx commented Nov 29, 2023

/test e2e-aws-ovn-upgrade

@sergiordlr
Copy link

Given the date of the feature freeze, we have decided to approve this PR now and fix the problem regarding the mcd_config_drift metric later using another PR later.

The behaviour regarding the mcd_config_drift metric will be tracked here:

https://issues.redhat.com/browse/OCPBUGS-24003

/cc @rioliu-rh @sinnykumari

/label qe-approved

@openshift-ci openshift-ci bot added the qe-approved Signifies that QE has signed off on this PR label Nov 29, 2023
@sinnykumari
Copy link
Contributor

Sounds good, thanks Sergio!
Removing hold as we have qe approval.
/hold cancel

@openshift-ci openshift-ci bot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Nov 29, 2023
@openshift-ci-robot
Copy link
Contributor

/retest-required

Remaining retests: 0 against base HEAD 637f757 and 2 for PR HEAD 85f6487 in total

@inesqyx
Copy link
Contributor Author

inesqyx commented Nov 29, 2023

/test e2e-hypershift

Copy link
Contributor

openshift-ci bot commented Nov 29, 2023

@inesqyx: all tests passed!

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@openshift-merge-bot openshift-merge-bot bot merged commit 16b16ef into openshift:master Nov 29, 2023
14 checks passed
@openshift-ci-robot
Copy link
Contributor

@inesqyx: Jira Issue OCPBUGS-20427: All pull requests linked via external trackers have merged:

Jira Issue OCPBUGS-20427 has been moved to the MODIFIED state.

In response to this:

In this PR, I have enabled several metrics to report key state of the MCC & MCD

  • Metrics for MCC sub-controllers
  • Introduced state metric: MCCSubControllerState, which has label "subcontroller, state object, to record the action of the subcontroller on certain object e.g. machine-config-controller-render Sync Machine Config Pool master timestampA --> the render controller has synced the master pool at timestamp A.
  • Timestamps for MCD config drifts
  • Timestamps for pivot error

I have also fixed the bug reported in https://issues.redhat.com/browse/OCPBUGS-20427:

Before: when query for mcc_drain_err, it returns no datapoint found
Now: when query for mcc_drain_err, it returns the initial value set until change happened

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@openshift-bot
Copy link
Contributor

[ART PR BUILD NOTIFIER]

This PR has been included in build openshift-proxy-pull-test-container-v4.15.0-202311292108.p0.g16b16ef.assembly.stream for distgit openshift-proxy-pull-test.
All builds following this will include this PR.

stbenjam added a commit to stbenjam/machine-config-operator that referenced this pull request Nov 30, 2023
This reverts commit 16b16ef, reversing
changes made to 637f757.
openshift-merge-bot bot added a commit that referenced this pull request Nov 30, 2023
TRT-1375: Revert #3965 "MCO-681, OCPBUGS-20427: Add Key State Metrics, No datapoint found when querying up certain registered metrics"
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. jira/valid-bug Indicates that a referenced Jira bug is valid for the branch this PR is targeting. jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. lgtm Indicates that a PR is ready to be merged. qe-approved Signifies that QE has signed off on this PR
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

7 participants