Counter not start from zero

[chocopowwwa]

What did you do?
Tailing Nginx log and parse and publish it using Mtail (mine quiet similar like https://github.com/ipfs/infrastructure/blob/master/mtail/progs/nginx.mtail), the probem is that when the mtail starts it publish metrics like this

TYPE http_requests_total counter

TYPE http_request_duration_milliseconds_sum counter

TYPE http_response_size_bytes_sum counter

and after i make several http requests ( for example 15 http requests ), mtail will read the logs and publish metrics like

TYPE http_requests_total counter

http_requests_total{airlines="ZumaAir", code="404",method="get",vhost="localhost",prog="nginxlog.mtail",instance="funbox"} 15

TYPE http_request_duration_milliseconds_sum counter

TYPE http_response_size_bytes_sum counter

but prometheus will assume that the counters starts from 15 not 0
MY Query:
irate(http_requests_total{job="apache2",airlines=~"$airlines",code!="500"}[$interval])
NB: $airlines is ZumaAir

as you see on the image, becausei didn't introduce the metrics first to prometheus and set its initial value ( like for example 0 ) the total values of the query it will be less than the actual sum of the metrics

What did you expect to see?
i expect the total of the query is the same as the total of the vents, is the any possible to assume that if the metrics hasn't introduced to promethus it will assume that the metrics starts from zero?

What did you see instead? Under which circumstances?
the total of the query isn't the same as the total of the metrics
also it makes no change the the graph

Environment

• System information:
  Linux 4.4.12-boot2docker x86_64
• Prometheus version:
  1.0.1
• Alertmanager version:
• Prometheus configuration file:
  • job_name: "apache2"
    scrape_interval: 5s
    static_configs:
    • targets: ["192.168.50.4:3903"]
• Alertmanager configuration file:
• Logs:

[brian-brazil]

There's not much we can do here, irate only looks at two samples at a time. Without a sample to go before the 15 value, we don't have anything to compare it to.

If we did what you suggested it'd be far worse, as every network blip would lead to a massive spurious spike in your graphs.

[chocopowwwa]

hmm sorry ask this because i really confuse,
then how do i suppose to show request per second/minute foreach http code from nginx log? i tried to use increase but the numbers doesnt make sense to me the only make sense is irate

[brian-brazil]

You appear to be doing this correctly, this is what this sort of query produces on a sparse metric where you don't know the label values in advance.

If you can pre-populate the metric with 0 for all potential timeseries you would get better results, but I'm guessing that's not possible here. See https://prometheus.io/docs/practices/instrumentation/#avoid-missing-metrics

[chocopowwwa]

Oh now i got it, thanks
any plan for the future for this ( like adding special prefix to the metrics name maybe? )

[brian-brazil]

There's #1673 discussing this, but I don't see any way to make this workable in a sane way.

There's other features in future that may let you implement such a thing yourself.

[chocopowwwa]

thank you 😃

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.