Remote Storage Queue won't flush

[fmoessbauer]

What did you do?

Update prometheus to commit 404b306

What did you expect to see?

Remote storage working properly.

What did you see instead? Under which circumstances?

Writes to the remote storage are not flushed. The remote_storage_queue fills up until no more elements fit in and then samples are dropped.

This is probably related to PR #3731

Environment

• System information:

Linux 4.13.0-25-generic x86_64

• Prometheus version:

prometheus, version 2.1.0 (branch: master, revision: 404b306fb9d5c0976cc83f0dff39254c0918493e)                                            
  build user:       root@3dc6e252e6c2                                
  build date:       20180207-10:31:30                                
  go version:       go1.9.2

• Prometheus configuration file:

global:
  scrape_interval:     15s # By default, scrape targets every 15 seconds.
  external_labels:
    monitor: 'codelab-monitor'

scrape_configs:
  - job_name: 'prometheus'

    scrape_interval: 15s

    static_configs:
      - targets: ['localhost:9090', 'collectd-exporter:9103']

  - job_name: 'linux-server'
    scrape_interval: 60s
    static_configs:
      - targets:
        - '192.168.2.xx:xxxx'

# [...]

rule_files:
  - alert-rules/*.rule

alerting:
  alertmanagers:
    - static_configs:
      - targets:
        - alertmanager:9093

# Remote write configuration (for Graphite, OpenTSDB, or InfluxDB).
remote_write:
  - url: "http://prom-io:9201/write"

# Remote read configuration (for InfluxDB only at the moment).
remote_read:
  - url: "http://prom-io:9201/read"

• Logs:

level=warn ts=2018-02-07T13:55:17.944112632Z caller=queue_manager.go:225 component=remote msg="Remote storage queue full, discarding sample. Multiple subsequent messages of this kind may be suppressed."

[krasi-georgiev]

@tomwilkie might give some ideas here

[juliusv]

Both I and @tomwilkie can reproduce this from master, so it's also in 2.2.

[AndreaGiardini]

Same thing here. V2.2 is not able to flush remote_storage queue

[Jean-Daniel]

Same here. And this is especially bothering as it is not possible to revert to 2.1…

[tomwilkie]

@Jean-Daniel @AndreaGiardini try the docker image tomwilkie/prometheus:3809-correctly-stop-timer-3 in the mean time. Working on a fix.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.