Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Namespace tag the disruptions.gauge metric #847

Merged
merged 5 commits into from
Mar 15, 2024

Conversation

ptnapoleon
Copy link
Contributor

@ptnapoleon ptnapoleon commented Mar 14, 2024

What does this PR do?

  • Adds new functionality
  • Alters existing functionality
  • Fixes a bug
  • Improves documentation or testing

Please briefly describe your changes as well as the motivation behind them:

  • Users want a metric to see how many disruptions are running in their namespace. They can sort of do so with ongoing_duration, but that's a bit indirect. They should be able to check the disruptions.gauge and filter by a tag on their namespace.
  • We don't currently tag disruptions.gauge at all. I've added a namespace tag

Code Quality Checklist

  • The documentation is up to date.
  • My code is sufficiently commented and passes continuous integration checks.
  • I have signed my commit (see Contributing Docs).

Testing

  • I leveraged continuous integration testing
    • by depending on existing unit tests or end-to-end tests.
    • by adding new unit tests or end-to-end tests.
  • I manually tested the following steps:
    • x
    • locally.
    • as a canary deployment to a cluster.

@datadog-datadog-prod-us1
Copy link

datadog-datadog-prod-us1 bot commented Mar 14, 2024

Datadog Report

Branch report: philip/namespace-tag-on-gauge
Commit report: cd7a088
Test service: chaos-controller

✅ 0 Failed, 720 Passed, 2 Skipped, 7m 13.45s Wall Time

@ptnapoleon
Copy link
Contributor Author

I've confirmed this works! the namespace tag for this metric now lists the number of disruptions in that namespace, no longer attributing them all to chaos-engineering

@ptnapoleon ptnapoleon marked this pull request as ready for review March 14, 2024 14:34
@ptnapoleon ptnapoleon requested a review from a team as a code owner March 14, 2024 14:34
}

// send metrics
if err := r.MetricsSink.MetricStuckOnRemovalGauge(float64(stuckOnRemoval)); err != nil {
r.BaseLog.Errorw("error sending stuck_on_removal_total metric", "error", err)
}

if err := r.MetricsSink.MetricDisruptionsGauge(float64(len(l.Items))); err != nil {
r.BaseLog.Errorw("error sending disruptions.gauge metric", "error", err)
if len(namespaces) > 0 {
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've tested and this works in the case of 0 disruptions, disruptions in 1 namespace, and disruptions in many namespaces

Copy link
Contributor

@clairecng clairecng left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

@ptnapoleon ptnapoleon merged commit 4d98cfd into main Mar 15, 2024
18 checks passed
@ptnapoleon ptnapoleon deleted the philip/namespace-tag-on-gauge branch March 15, 2024 12:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants