I am missing about 10% in the _count of a prometheus.SummaryVec

[markfink]

thank you for prometheus, I love it!

I am very new to prometheus so probably I am doing something wrong but I am consistently loosing 10% counts in the SummaryVec I report with "github.com/prometheus/client_golang/prometheus".

a summary from my test (32.826 events over a duration of 180 sec):

01_01_teststep, 50.195683, 50.017329, 60.545467, 17774
02_01_teststep, 100.196233, 100.025423, 105.679511, 9029
03_01_teststep, 150.200282, 150.034356, 160.413986, 6023

Querying the prometheus graph tool I get the following _counts: 16411, 8335, 5560 (30306). This is about 10% loss in event counts. I have a logfile of the events which shows the 32.826 events. I am not exactly sure about how prometheus handles the metrics. So is a loss in count something to be expected?

As I said most likely I am doing something wrong. Any tips on how to investigate this further? Has the prometheus reporter a logger or something?

The relevant part of my code to register the SummaryVec:

    elapsed := prometheus.NewSummaryVec(prometheus.SummaryOpts{
        Name: "gogrinder_elapsed_ms",
        Help: "Current time elapsed of gogrinder teststep in ms.",
        Objectives: map[float64]float64{0.5: 0.05, 0.9: 0.01, 0.95: 0.005, 0.99: 0.001},
    }, []string{"teststep"})  // , "timestamp"
    regElapsed := prometheus.MustRegisterOrGet(elapsed).(*prometheus.SummaryVec)

I am using your docker container to run Prometheus - some build info:

branch  stable
buildDate   20160122-21:30:47
buildUser   @9f8f0f8d724a
goVersion   1.5.3
revision    287d9b2
version 0.16.2

[brian-brazil]

We expect some loss due to application crashes and the like, but that shouldn't happen when everything is healthy. Can you confirm the /metrics page is showing the wrong values for the count?

[beorn7]

As @brian-brazil said, please confirm that the /metrics endpoint on your target has the same numbers. If yes, this is a pure client-side issue (so we should then move the issue over to prometheus/client_golang).

The client-side metric objects are supposed to count losslessly. So this would either be a bug in the client library or in your code (like concurrent registration of multiple summaries ore something – the way you register would allow multiple summaries, it should in principle work with a simple MustRegister, perhaps try it that way and see if you get a registration error).

[markfink]

@beorn7 @brian-bazil thank you for your help
I did not really think through what it means to have a pull model. Problem was that I did not run the exporter long enough for prometheus to scrape all metrics. Looking at /metrics like you suggested helped me to see that.

sorry for the noise
Mark

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.