Un-incremented counters don't have value 0

[paulborza]

When I navigate to my metrics endpoint, some counters don't have values because they were never incremented. Such a counter is the unknown_failure_total which gets incremented on unhandled exceptions.

I have Prometheus hooked up to Grafana and it keeps alerting me that there's missing data for some counters. I don't want to suppress this alert because missing data could also mean that Grafana is not able to reach my Prometheus server.

The HELP and TYPE statements are printed even for un-incremented counters, but their zero value is not. Should a 0 value be printed for counters which have yet to be incremented?

# HELP postgres_upsert_failure_total postgres_upsert_failure_total
# TYPE postgres_upsert_failure_total counter
postgres_upsert_failure_total{environment="production"} 2

# HELP unknown_failure_total cli_unknown_failure_total
# TYPE unknown_failure_total counter

Thanks!

[siimon]

Not sure what the "correct" behaviour is in this case, you could also argue that it shouldn't have any value at all until you have set any.

A workaround in your case is to inc with 0 just after you created it

[paulborza]

Thanks for the workaround suggestion. Here are my thoughts:

1. According to the official Prometheus documentation about writing client libraries: "Counters MUST start at 0." So when a new counter is initialized, its value has to be zero. And it is zero, because otherwise the inc method wouldn't make sense unless a starting value is offered.
2. Its odd to see HELP and TYPE metadata displayed but the counter value not.

I think we should always print the value of counters, even if they're zero. A missing counter is != than a counter which was never initialized. What do you think?

[SimenB]

I agree with @paulborza. It also matches what the java client does.

import io.prometheus.client.CollectorRegistry;
import io.prometheus.client.Counter;
import io.prometheus.client.exporter.common.TextFormat;

import java.io.StringWriter;
import java.util.Collections;

public class App {
    static Counter c = Counter.build("name", "help").register();

    public static void main(String[] args) throws Exception {
        StringWriter writer = new StringWriter();
        TextFormat.write004(writer, CollectorRegistry.defaultRegistry.filteredMetricFamilySamples(Collections.<String>emptySet()));
        System.out.println(writer.toString());
    }
}

Prints

# HELP name help
# TYPE name counter
name 0.0

[SimenB]

Should be the same for all types of metrics.

import io.prometheus.client.CollectorRegistry;
import io.prometheus.client.Counter;
import io.prometheus.client.Gauge;
import io.prometheus.client.Histogram;
import io.prometheus.client.Summary;
import io.prometheus.client.exporter.common.TextFormat;

import java.io.StringWriter;
import java.util.Collections;

public class App {
    static {
        Counter.build("counter", "help").register();
        Gauge.build("gauge", "help").register();
        Histogram.build("histogram", "help").register();
        Summary.build("summary", "help").register();
    }

    public static void main(String[] args) throws Exception {
        StringWriter writer = new StringWriter();
        TextFormat.write004(writer, CollectorRegistry.defaultRegistry.filteredMetricFamilySamples(Collections.<String>emptySet()));
        System.out.println(writer.toString());
    }
}

# HELP summary help
# TYPE summary summary
summary_count 0.0
summary_sum 0.0
# HELP gauge help
# TYPE gauge gauge
gauge 0.0
# HELP counter help
# TYPE counter counter
counter 0.0
# HELP histogram help
# TYPE histogram histogram
histogram_bucket{le="0.005",} 0.0
histogram_bucket{le="0.01",} 0.0
histogram_bucket{le="0.025",} 0.0
histogram_bucket{le="0.05",} 0.0
histogram_bucket{le="0.075",} 0.0
histogram_bucket{le="0.1",} 0.0
histogram_bucket{le="0.25",} 0.0
histogram_bucket{le="0.5",} 0.0
histogram_bucket{le="0.75",} 0.0
histogram_bucket{le="1.0",} 0.0
histogram_bucket{le="2.5",} 0.0
histogram_bucket{le="5.0",} 0.0
histogram_bucket{le="7.5",} 0.0
histogram_bucket{le="10.0",} 0.0
histogram_bucket{le="+Inf",} 0.0
histogram_count 0.0
histogram_sum 0.0

[siimon]

Alright 🙂 then we should update prom-client to follow Prometheus docs 🙂

[SimenB]

One can call inc/set(0) for counters and gauges, but observe(0) doesn't work for summaries and histograms (they both set count, and histogram adds to a bucket), so we should probably solve this properly instead of relying on setting stuff to zero.

[SimenB]

I did discover something interesting though; if adding a label, they are not shown. This might be a bug in the java implementation though, not sure. Seems inconsistent.

import io.prometheus.client.CollectorRegistry;
import io.prometheus.client.Counter;
import io.prometheus.client.Gauge;
import io.prometheus.client.Histogram;
import io.prometheus.client.Summary;
import io.prometheus.client.exporter.common.TextFormat;

import java.io.StringWriter;
import java.util.Collections;

public class App {
    static {
        Counter.build("counter", "help").labelNames("label").register();
        Gauge.build("gauge", "help").labelNames("label").register();
        Histogram.build("histogram", "help").labelNames("label").register();
        Summary.build("summary", "help").labelNames("label").register();
    }

    public static void main(String[] args) throws Exception {
        StringWriter writer = new StringWriter();
        TextFormat.write004(writer, CollectorRegistry.defaultRegistry.filteredMetricFamilySamples(Collections.<String>emptySet()));
        System.out.println(writer.toString());
    }
}

# HELP summary help
# TYPE summary summary
# HELP counter help
# TYPE counter counter
# HELP histogram help
# TYPE histogram histogram
# HELP gauge help
# TYPE gauge gauge

[dm3]

With labels present there's no way to know the label values to display a default zero. However, once you've initialized the "child" with the label values by calling the SimpleCollector#labels method, you should see zero as in the no-labels case.

[siimon]

Published in 10.0.1!