Prometheus get different time metrics

[andreyradygin]

What did you do?
I'm scraping metrics with Prometheus from netdata hosts with average function method.

What did you expect to see?
Common metrics and labels for them for each netdata host. There are multiple the same metrics with only instance label different.

What did you see instead? Under which circumstances?
I observe a strange thing - every request to build a graph, give me different set of time for metric points, therefore I've got different variants of graphic (every Execute in Prom web ui or Update in grafana web ui), because metrics values are little vary.

For example, we have:
netdata_system_cpu_percentage_average{chart="system.cpu",dimension=~".*",family="cpu",instance="$host",job="netdata-scrape"}

and first execute give me some set of metrics, one or them - 7:15:39GMT and have value of system.cpu guest = 0.40122... second time I've Execute, and getting total different set of times and point for 7:15:39GMT already disappear and I have 7:15:44GMT, or 7:15:27GMT with another values.

Here are three screenshots of three executes in a row, why are they so different?

Environment
I have netdata 1.9+prometheus 2.0 + grafana set up.
Collection method - average.
Scrape interval - 5 seconds.
Centos 7

• System information:
  Linux 3.10.0-514.26.2.el7.x86_64 x86_64

• Prometheus version:
  docker exec -it 3c2dab634358 prometheus --version
  prometheus, version 2.0.0 (branch: HEAD, revision: 0a74f98)
  build user: root@615b82cb36b6
  build date: 20171108-07:11:59
  go version: go1.9.2

• Prometheus configuration file:

global:
  scrape_interval:     5s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
  evaluation_interval: 5s # Evaluate rules every 15 seconds. The default is every 1 minute.
....

- job_name: 'netdata-scrape'

    metrics_path: '/api/v1/allmetrics'
    params:
      # format: prometheus | prometheus_all_hosts
      format: [prometheus]
      #
      # sources: as-collected | raw | average | sum | volume
      # default is: average
      source: [average]
      #
      # server name for this prometheus - the default is the client IP
      # for netdata to uniquely identify it
      #server: ['prometheus1']
    honor_labels: true

    file_sd_configs:
      - files:
        - hosts.yml
        refresh_interval: 10m

    metric_relabel_configs:
      - source_labels: [ instance ]
        regex: '(.*):(.*)'
        replacement: ${1}
        target_label: instance

• Logs:

level=warn ts=2018-01-14T19:06:02.805141096Z caller=scrape.go:905 component="target manager" scrape_pool=netdata-scrape target="http://HOSTNAME:19999/api/v1/allmetrics?format=prometheus&source=average" msg="Error on ingesting samples with different value but same timestamp" num_dropped=3

I've asked developer of netdata about this strange behavior:
netdata/netdata#3287

hm... I think this is a question about prometheus. netdata sends only one value to prometheus every >time and prometheus keeps tracks of them as a series.
This seems like prometheus is receiving metrics from multiple netdata, but it thinks it is the same.`

[brian-brazil]

Your metric_relabel_configs is likely removing uniqueness from your metrics.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.