`prometheus_target_scrapes_exceeded_sample_limit_total` not incremented

[ant1441]

What did you do?
When a target goes over the configured sample_limit, I looked at the prometheus_target_scrapes_exceeded_sample_limit_total metric.

What did you expect to see?
I expected this to increase for each scrape which hit the sample limit, as per the help on the metric:

Total number of scrapes that hit the sample limit and were rejected.

(from scrape.go#L83)

What did you see instead? Under which circumstances?
This metric was not incremented.

Environment

• System information:

  Linux 4.4.0-103-generic x86_64

• Prometheus version:

Seen on the prom/prometheus:v2.0.0 Docker image, reproduced with local build from master branch:

prometheus, version 2.0.0 (branch: master, revision: 97464236c76301dab933ab8407f7ebe0ee1ec292)

• Alertmanager version:
  NA

• Prometheus configuration file:

global:
  scrape_interval:     15s
  evaluation_interval: 15s

scrape_configs:
  - job_name: 'some_job'

    static_configs:
    - targets: ['localhost:9091']

    sample_limit: 100

• Alertmanager configuration file:
  NA

• Logs:

level=warn ts=2018-01-09T11:36:18.344766728Z caller=scrape.go:683 component="scrape manager" scrape_pool=some_job target=http://localhost:9091/metrics msg="append failed" err="sample limit exceeded"

Notes
It appears to be the case that at scrape.go#L907 the metric will be incremented if err == nil && sampleLimitErr != nil.
However, if I am understanding correctly, err will always be the same as sampleLimitErr as once the limit has been hit all further samples will also result in that error. Though there is the possibility that a different error is raised at the final stage of the loop.

My guess would be that scrape.go#L907 should be changed to just sampleLimitErr != nil , but I'm not certain how the line err = sampleLimitErr should be changed. If I am understanding things correctly, err would always already be sampleLimitErr, so that line is unneeded.

[brian-brazil]

Thanks for reporting this. Would you like to send a PR fixing it? A unittest would be great too.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.