Skip to content

/prometheus

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Kubernetes endpoints SD fails to detect empty replicas #3659

Closed
simonpasquier opened this Issue Jan 5, 2018 · 2 comments

Comments

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
1 participant
@simonpasquier
@simonpasquier
Copy link
Member

simonpasquier commented Jan 5, 2018

What did you do?

Using the K8S endpoints SD to monitor a service + statefulset (replicas=1). After deployment, Prometheus scrapes the associated pods as expected. After a while, I scale the statefulset to zero replicas and check that the pods have been deleted.

What did you expect to see?

Prometheus should stop scraping the pods since none exists anymore.

What did you see instead? Under which circumstances?

Prometheus keeps scraping the pods (and fails) as if it wasn't aware of the replica update.

Environment

  • Prometheus version:

master
@simonpasquier

This comment has been minimized.

Sign in to view
Copy link
Member Author

simonpasquier commented Jan 5, 2018

I have a PR for this.

simonpasquier added a commit to simonpasquier/prometheus that referenced this issue Jan 5, 2018

@simonpasquier
Fix Kubernetes endpoints SD for empty subsets 
When an endpoints object has no associated pods (replica scaled to zero
for instance), the endpoints SD should return a target group with no
targets so that the SD manager propagates this information to the scrape
manager.

Fixes prometheus#3659
c3f133a

@brian-brazil brian-brazil closed this in bde64cf Jan 30, 2018

@lock

This comment has been minimized.

Sign in to view
Copy link

lock bot commented Mar 22, 2019

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

@lock lock bot locked and limited conversation to collaborators Mar 22, 2019

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
You can’t perform that action at this time.