Improve Prometheus's internal metrics

[fabxc]

Many parts of Prometheus itself are not instrumented as well they should be.

[juliusv]

/cc @brian-brazil because he has mentioned this at times and probably has opinions about what should be changed (besides, ehem, changing everything to seconds!).

[brian-brazil]

I maintain that the correct way to measure memory usage is in terms of how much of it you can read in a second!

[juliusv]

@brian-brazil Haha, 👍 💯 Let's make it so!

[brian-brazil]

On a more serious note, I'll be doing up consoles templates for prometheus itself in the next few days so I should have some concrete suggestions from that.

[fabxc]

There are also parts which are not instrumented at all – namely the
different SD mechanisms and the query engine.
The more generic the SD instrumentation the better. But there may be some
reasonable specific metrics.

On Tue, Aug 4, 2015 at 2:37 PM Brian Brazil notifications@github.com
wrote:

On a more serious note, I'll be doing up consoles templates for prometheus
itself in the next few days so I should have some concrete suggestions from
that.

—
Reply to this email directly or view it on GitHub
#961 (comment)
.

[mwitkow]

Can I drop a feature request here?

• number of targets per job
• number of metrics collected and retained last scrape per job

:)

[brian-brazil]

Per-instance/job metrics are something that should come from the scrape, and don't belong on the /metrics for prometheus as they're not about prometheus at a systems level.

number of targets per job

count by(job)(up{job="job"}

number of metrics collected and retained last scrape per job

count by (job)({job="job"})

Be careful with that one, it's going to be slow.

[mwitkow]

Yea, the last one is precisely the reason why we'd like a counter :)

[brian-brazil]

The way to do that would be as scrape_samples_collected similar to scrape_duration_seconds.

[brian-brazil]

scrape_samples_scraped has been added. The rest of this ticket is rather generic, so if there's specific metrics you're looking for please open issues for then.

[beorn7]

I guess @fabxc original intention was to keep this as a reminder to eventually rewrite the instrumentation from scratch. I don't know if anybody ever will have the time and nerves to do so. We could just keep improving instrumentation iteratively as we see fit.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.