Docker instance won't scrape

[MihaMarkic]

What did you do?
Run prometheus as docker instance.

What did you expect to see?
Scrapes running.

What did you see instead? Under which circumstances?
No scrapes are run, both prometheus and mine target are listed as unknown.
Note that service of mine (airpoll) is very simple.

Environment
Ubuntu/docker 17.05.0-ce

• System information:

  Linux 4.4.0-83-generic x86_64

• Prometheus version:

  1.7.1

• Prometheus configuration file:

insert configuration here

• Alertmanager configuration file:

scrape_configs:
  # Scrape Prometheus itself every 5 seconds.
  - job_name: 'prometheus'
    scrape_interval: 5s
    static_configs:
      - targets: ['localhost:9090']

  # Scrape the Node Exporter every 5 seconds.
  - job_name: 'airpoll'
    scrape_interval: 5s
    static_configs:
      - targets: ['airpoll:1234']

• Logs:

airprometheus_1  | time="2017-07-12T08:46:08Z" level=warning msg="Flag -storage.local.memory-chunks is deprecated. Its value 2500 is used to override -storage.local.target-heap-size to 7680000." source="config.go:317"
airprometheus_1  | time="2017-07-12T08:46:08Z" level=info msg="Starting prometheus (version=1.7.1, branch=master, revision=3afb3fffa3a29c3de865e1172fb740442e9d0133)" source="main.go:88"
airprometheus_1  | time="2017-07-12T08:46:08Z" level=info msg="Build context (go=go1.8.3, user=root@0aa1b7fc430d, date=20170612-11:44:05)" source="main.go:89"
airprometheus_1  | time="2017-07-12T08:46:08Z" level=info msg="Host details (Linux 4.4.0-83-generic #106-Ubuntu SMP Mon Jun 26 17:54:43 UTC 2017 x86_64 953a61dde346 (none))" source="main.go:90"
airprometheus_1  | time="2017-07-12T08:46:08Z" level=info msg="Loading configuration file /etc/prometheus/prometheus.yml" source="main.go:252"
airprometheus_1  | time="2017-07-12T08:46:08Z" level=info msg="Loading series map and head chunks..." source="storage.go:428"
airprometheus_1  | time="2017-07-12T08:46:08Z" level=info msg="0 series loaded." source="storage.go:439"
airprometheus_1  | time="2017-07-12T08:46:08Z" level=info msg="Starting target manager..." source="targetmanager.go:63"
airprometheus_1  | time="2017-07-12T08:46:08Z" level=info msg="Listening on :9090" source="web.go:259"
airprometheus_1  | time="2017-07-12T08:46:09Z" level=warning msg="Storage has entered rushed mode." chunksToPersist=0 memoryChunks=0 source="storage.go:1867" urgencyScore=1
airprometheus_1  | time="2017-07-12T08:46:09Z" level=error msg="Storage needs throttling. Scrapes and rule evaluations will be skipped." chunksToPersist=0 memoryChunks=0 source="storage.go:1007" urgencyScore=1

[brian-brazil]

Usage questions should be asked on https://groups.google.com/forum/#!aboutgroup/prometheus-users

[gouthamve]

From your logs it is clear that you don't have enough memory. But what is surprising is that you ran out of memory for just 2 targets.

Give your prometheus enough memory and it should work. This has been closed as it is not a bug in Prometheus itself.

[MihaMarkic]

@gouthamve Yes, it is odd even more so because before the update (to 1.7.1) it just worked. Ctop is showing that prometheus is using 17M/983M available. By give it more memory what did you think: at docker level?

[MihaMarkic]

Looks like I've found the issue.
First I was using (deprecated)
-storage.local.memory-chunks=2500
It was working with the older prometheus but not with the current one. So I removed and tried this one:
-storage.local.target-heap-size=20485760
It worked sometimes, almost randomly.

At the end I just removed the last one and now it works.

[timchenxiaoyu]

disk io ,bule is read,green is write

time="2017-07-21T05:14:09Z" level=info msg="Storage does not need throttling anymore." chunksToPersist=11127 memoryChunks=247856 source="storage.go:990" urgencyScore=0.583
time="2017-07-21T05:14:17Z" level=warning msg="Storage has entered rushed mode." chunksToPersist=11122 memoryChunks=242511 source="storage.go:1842" urgencyScore=1
time="2017-07-21T05:14:17Z" level=error msg="Storage needs throttling. Scrapes and rule evaluations will be skipped." chunksToPersist=11090 memoryChunks=242511 source="storage.go:982" urgencyScore=1
time="2017-07-21T05:14:44Z" level=info msg="Storage has left rushed mode." chunksToPersist=9858 memoryChunks=242297 source="storage.go:1832" urgencyScore=0.644

prometheus version is 1.6.1 run in docker container
why io is continuously reduce ?

[suanziliu]

Hi @MihaMarkic ,
I almost met the same issue and fixing method with you, I was using v1.8.2.
I used to use storage.local.memory-chunks but now still using -storage.local.target-heap-size.
It's always crasing these days and tells me OOM.

Can I ask:

At the end I just removed the last one and now it works.

Which one did you remove and what is the configuration you are using?

Thanks,

[MihaMarkic]

@suanziliu Probably I just removed -storage.local.target-heap-size.