Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bug 1907475: Add recording rules for ingress traffic and error rate #1019

Merged

Conversation

smarterclayton
Copy link
Contributor

The overall rates of ingress traffic to the cluster are relevant in
assessing health and usage - these rules add calculations for the
rate of requests that error (5xx and other types of errors) across
the frontends, the total requests passing across the frontends, the
bandwidth used in both directions, and the number of open connections.
Error rate is split for openshift and workloads (like CPU is) in order
to better categorize trends before, during, and after upgrades.

This data is relevant for both admins and engineering teams to classify
and categorize the overall health and traffic of the cluster. This adds
seven new series to telemetry via the cluster:usage: prefix.

  • I added CHANGELOG entry for this change.
  • No user facing changes, so no entry in CHANGELOG was needed.

The overall rates of ingress traffic to the cluster are relevant in
assessing health and usage - these rules add calculations for the
rate of requests that error (5xx and other types of errors) across
the frontends, the total requests passing across the frontends, the
bandwidth used in both directions, and the number of open connections.
Error rate is split for openshift and workloads (like CPU is) in order
to better categorize trends before, during, and after upgrades.

This data is relevant for both admins and engineering teams to classify
and categorize the overall health and traffic of the cluster. This adds
seven new series to telemetry via the cluster:usage: prefix.
@openshift-ci-robot openshift-ci-robot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Dec 11, 2020
@smarterclayton
Copy link
Contributor Author

/retest

1 similar comment
@smarterclayton
Copy link
Contributor Author

/retest

@smarterclayton smarterclayton changed the title WIP: jsonnet/rules: Add recording rules for ingress traffic and error rate WIP: Bug 1907475: Add recording rules for ingress traffic and error rate Dec 14, 2020
@openshift-ci-robot openshift-ci-robot added bugzilla/severity-high Referenced Bugzilla bug's severity is high for the branch this PR is targeting. bugzilla/invalid-bug Indicates that a referenced Bugzilla bug is invalid for the branch this PR is targeting. labels Dec 14, 2020
@openshift-ci-robot
Copy link
Contributor

@smarterclayton: This pull request references Bugzilla bug 1907475, which is invalid:

  • expected the bug to target the "4.7.0" release, but it targets "---" instead

Comment /bugzilla refresh to re-evaluate validity if changes to the Bugzilla bug are made, or edit the title of this pull request to link to a different bug.

In response to this:

WIP: Bug 1907475: Add recording rules for ingress traffic and error rate

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@smarterclayton
Copy link
Contributor Author

/bugzilla refresh

@openshift-ci-robot openshift-ci-robot added the bugzilla/valid-bug Indicates that a referenced Bugzilla bug is valid for the branch this PR is targeting. label Dec 14, 2020
@openshift-ci-robot
Copy link
Contributor

@smarterclayton: This pull request references Bugzilla bug 1907475, which is valid. The bug has been moved to the POST state. The bug has been updated to refer to the pull request using the external bug tracker.

3 validation(s) were run on this bug
  • bug is open, matching expected state (open)
  • bug target release (4.7.0) matches configured target release for branch (4.7.0)
  • bug is in the state NEW, which is one of the valid states (NEW, ASSIGNED, ON_DEV, POST, POST)

In response to this:

/bugzilla refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@openshift-ci-robot openshift-ci-robot removed the bugzilla/invalid-bug Indicates that a referenced Bugzilla bug is invalid for the branch this PR is targeting. label Dec 14, 2020
@smarterclayton smarterclayton changed the title WIP: Bug 1907475: Add recording rules for ingress traffic and error rate Bug 1907475: Add recording rules for ingress traffic and error rate Dec 14, 2020
@openshift-ci-robot openshift-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Dec 14, 2020
@smarterclayton
Copy link
Contributor Author

@Miciah can you or someone else on the team take a look? I want to quantify "happy" and "non-disrupted" ingress (this won't catch the nodeport / local traffic problem, but it will catch lower level problems like ovs)

@Miciah
Copy link

Miciah commented Dec 16, 2020

@sgreene570, could you take a look? @RiRa12621, I figure you might be interested in well.

@RiRa12621
Copy link
Contributor

Thanks for letting me know @Miciah .
Had a chat with Clayton about it before already and it's only recording rules for now so that's fine from my perspective.

@smarterclayton
Copy link
Contributor Author

Can one of the network edge reviewers confirm these make sense to capture given the reasoning in the bug and PR? Just want to make sure before I get final review on the rule structure.

@Miciah
Copy link

Miciah commented Dec 21, 2020

Can one of the network edge reviewers confirm these make sense to capture given the reasoning in the bug and PR? Just want to make sure before I get final review on the rule structure.

Yeah, it makes sense to me.

/lgtm

@openshift-ci-robot openshift-ci-robot added the lgtm Indicates that a PR is ready to be merged. label Dec 21, 2020
@quarterpin
Copy link

/bugzilla cc-qa

@openshift-ci-robot
Copy link
Contributor

@quarterpin: This pull request references Bugzilla bug 1907475, which is valid.

3 validation(s) were run on this bug
  • bug is open, matching expected state (open)
  • bug target release (4.7.0) matches configured target release for branch (4.7.0)
  • bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, ON_DEV, POST, POST)

Requesting review from QA contact:
/cc @quarterpin

In response to this:

/bugzilla cc-qa

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@quarterpin
Copy link

/lgtm

Verified via pre-merge verification workflow, more reference related to the test can be found in:
https://bugzilla.redhat.com/show_bug.cgi?id=1907475#c1

@s-urbaniak
Copy link
Contributor

/approve

@openshift-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: Miciah, quarterpin, s-urbaniak, smarterclayton

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-ci-robot openshift-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jan 4, 2021
@openshift-bot
Copy link
Contributor

/retest

Please review the full test history for this PR and help us cut down flakes.

@openshift-merge-robot openshift-merge-robot merged commit 49e98d5 into openshift:master Jan 4, 2021
@openshift-ci-robot
Copy link
Contributor

@smarterclayton: All pull requests linked via external trackers have merged:

Bugzilla bug 1907475 has been moved to the MODIFIED state.

In response to this:

Bug 1907475: Add recording rules for ingress traffic and error rate

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. bugzilla/severity-high Referenced Bugzilla bug's severity is high for the branch this PR is targeting. bugzilla/valid-bug Indicates that a referenced Bugzilla bug is valid for the branch this PR is targeting. lgtm Indicates that a PR is ready to be merged.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

8 participants