Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Don't scrape from headless services #895

Merged
merged 2 commits into from Sep 11, 2020

Conversation

pmalek-sumo
Copy link
Contributor

Description

This prevents prometheus double scraping fluentd pods by removing the annotations from headless services.

Testing performed
  • ci/build.sh
  • Redeploy fluentd and fluentd-events pods
  • Confirm events, logs, and metrics are coming in

@pmalek-sumo
Copy link
Contributor Author

Fixes #776

@pmalek-sumo pmalek-sumo linked an issue Sep 9, 2020 that may be closed by this pull request
Copy link
Contributor

@sumo-drosiek sumo-drosiek left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think thats the correct solution. We could eventually add additional label for scraping metrics
@perk thoughts?

@perk-sumo
Copy link
Contributor

Yeah, I think that works for now but let's fix it properly in 2.0: #896

@perk-sumo perk-sumo added this to the v1.3 milestone Sep 10, 2020
@vsinghal13
Copy link
Contributor

If we change the labels now, will it be considered a breaking change?

Copy link
Contributor

@frankreno frankreno left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Copy link
Contributor

@sumo-drosiek sumo-drosiek left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Valid point from @vsinghal13
@pmalek-sumo could you verify that?

Copy link
Contributor

@sumo-drosiek sumo-drosiek left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verified, no need for --force and change is not considered as breaking change

@pmalek-sumo pmalek-sumo merged commit 43ae186 into master Sep 11, 2020
@pmalek-sumo pmalek-sumo deleted the dont-scrape-from-headless-services branch September 11, 2020 15:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Custom Metrics From Fluentd are scraped 2x in Prometheus
6 participants