Alpha.2 Missing metrics on scrape when a sample limit is set (even when sample limit is not being actually hit)

[checketts]

What did you do?
Running the v2.0.0-alpha.2 and comparing the results against my existing v1.6.2 deployment

What did you expect to see?
Using v1.6.2 I can see the scrape samples coming in regularly with the query scrape_samples_scraped{app="drydock"}

What did you see instead? Under which circumstances?
With alpha.2, scraping the same targets I see metrics being missed

Environment
Running in Docker container in Kubernetes 1.4.1

[brian-brazil]

Do you know which metrics are missing? Is it always the same ones?

[checketts]

I believe they are the same metrics that are missing. The red line is my non-leader server which reports fewer metrics and notice how it is not showing the fluctuation.

On the alpha.2 server there is a log line that corresponds to the dropped metrics of

time="2017-05-25T15:37:20Z" level=error msg="append failed" err="sample limit of 7500 exceeded" source="scrape.go:518" target="{__address__="192.168.198.182:8180", __metrics_path__="/prometheus", __scheme__="http", app="drydock", app_version="1.0.809_master", instance="192.168.198.182:8180", job="kubernetes-pods", kubernetes_namespace="drydock", kubernetes_pod_name="drydock-2762539047-fc53l", pod_template_hash="2762539047"}"

So that would imply it is passing the 7500 limit I had set, but as we can see from the v1.6.2 server, there are <1000 metrics.

Also The query prometheus_target_scrapes_exceeded_sample_limit_total{app="drydock"} is returning no datapoints on either server, so that log message seems to be in error.

[brian-brazil]

Does the problem go away if you remove the limit? I see some issues with this code in 2.0, but nothing that'd cause this.

[brian-brazil]

Ah, we're passing along the wrong error message.

[checketts]

Removing the sample limit completely made the problem go away. Increasing the limit didn't work (though I had only tried tripling the value)

[fabxc]

Is this fixed with #2772 @brian-brazil

[brian-brazil]

No, I've got the code to fix just need to write the unittests.

[fabxc]

Fixed in #2787

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.