Retention cutoff failed leading to compaction failed/spiralling disk usage

[alxmk]

What did you do?
I am running Prometheus 2.0.0 in Kubernetes with a 7 day retention interval.

What did you expect to see?
After 7 days, Prometheus expires time series data older than that.

What did you see instead? Under which circumstances?
At 7 days and ~3 hours:

level=error ts=2017-11-20T15:00:57.971198569Z caller=db.go:255 component=tsdb msg="retention cutoff failed" err="remove /prometheus/01BYTQE4494NH3HD88TFMEW2SD/chunks: directory not empty"

Subsequently seeing repeated:

level=error ts=2017-11-20T15:00:57.973249858Z caller=db.go:260 component=tsdb msg="compaction failed" err="plan compaction: open /prometheus/01BYTQE4494NH3HD88TFMEW2SD/meta.json: no such file or directory"
level=error ts=2017-11-20T15:01:59.977086398Z caller=db.go:255 component=tsdb msg="retention cutoff failed" err="read block meta /prometheus/01BYTQE4494NH3HD88TFMEW2SD: open /prometheus/01BYTQE4494NH3HD88TFMEW2SD/meta.json: no such file or directory"

And disk utilisation growing incredibly quickly (~250GB in 24 hours).

Directory looks like:

/prometheus # ls -la 01BYTQE4494NH3HD88TFMEW2SD
total 61436
drwxr-xr-x    3 nobody   42949672      4096 Nov 20 15:00 .
drwxr-xr-x 3193 nobody   42949672    249856 Nov 23 11:34 ..
-rw-r--r--    1 nobody   42949672  62397330 Nov 13 13:00 .nfs0000000000000068000012da
drwxr-xr-x    2 nobody   42949672      4096 Nov 20 15:00 chunks
/prometheus # ls -la 01BYTQE4494NH3HD88TFMEW2SD/chunks/
total 19996
drwxr-xr-x    2 nobody   42949672      4096 Nov 20 15:00 .
drwxr-xr-x    3 nobody   42949672      4096 Nov 20 15:00 ..
-rw-r--r--    1 nobody   42949672  20380371 Nov 13 13:00 .nfs0000000000000df2000012d9

The file preventing deletion of chunks directory is opened by Prometheus:

/prometheus # fuser 01BYTQE4494NH3HD88TFMEW2SD/chunks/.nfs0000000000000df2000012d9 
17 
/prometheus # ps -ef | grep 17
   17 root     7751:2 /bin/prometheus --config.file=/config/prometheus-2/prometheus --storage.tsdb.path=/prometheus --storage.tsdb.retention=168h --web.console.libraries=/usr/share/prometheus/console_libraries --web.console.templates=/usr/share/prometheus/consoles --web.listen-address=:9091 --web.enable-lifecycle

Looks like a similar result to #3487 but potentially a different root cause.

Environment
Kubernetes 1.7.4

• System information:

/prometheus # uname -srm
Linux 4.4.0-98-generic x86_64

• Prometheus version:

/prometheus # prometheus --version
prometheus, version 2.0.0 (branch: HEAD, revision: 0a74f98628a0463dddc90528220c94de5032d1a0)
  build user:       root@615b82cb36b6
  build date:       20171108-07:11:59
  go version:       go1.9.2

• Alertmanager version:

Not relevant

• Prometheus configuration file:

global:
  scrape_interval: 30s
  scrape_timeout: 10s
  evaluation_interval: 30s
  external_labels:
    core_env: kube
alerting:
  alertmanagers:
  - kubernetes_sd_configs:
    - api_server: null
      role: pod
      namespaces:
        names: []
    scheme: http
    timeout: 10s
    relabel_configs:
    - source_labels: [__meta_kubernetes_pod_label_app]
      separator: ;
      regex: alertmanager
      replacement: $1
      action: keep
    - source_labels: [__meta_kubernetes_namespace]
      separator: ;
      regex: default
      replacement: $1
      action: keep
    - source_labels: [__meta_kubernetes_pod_container_port_number]
      separator: ;
      regex: null
      replacement: $1
      action: drop
rule_files:
- /config/prometheus-2/rules
metric_relabel_configs:
  - source_labels: [__name__]
    separator: ;
    regex: go_.+
    replacement: $1
    action: drop
  - source_labels: [__name__]
    separator: ;
    regex: http_.+
    replacement: $1
    action: drop
scrape_configs:
- job_name: prometheus
  scrape_interval: 30s
  scrape_timeout: 10s
  metrics_path: /metrics
  scheme: http
  static_configs:
  - targets:
    - localhost:9091
- job_name: job-N
  scrape_interval: 30s
  scrape_timeout: 10s
  metrics_path: /metrics
  scheme: http
  static_configs:
  - targets:
    -  <snip>
    labels:
      <snip>
  - targets:
    - <snip>
    labels:
      <snip>
.
.
.

We have 100 jobs with around 10000 targets in total.

• Alertmanager configuration file:

Not relevant

• Logs:
  prom.log

[brian-brazil]

It is not recommended to use Prometheus with NFS, as we need a POSIX filesystem. We also expect to be the only thing creating files in our data directory.

[alxmk]

@brian-brazil Prometheus is the only thing creating anything in the data directory.

We have no option but to use NFS to get the IOPS required.

[brian-brazil]

NFS is creating files in the directory.

I'd suggest using a local SSD.

[alxmk]

@brian-brazil Local SSD is not fast enough (or it wasn't for Prometheus v1, has the IOPS requirement dropped for the new back end?).

Sorry you'll have to excuse my ignorance on NFS, but if that file is created by the NFS why does Prometheus have it pinned open?

[brian-brazil]

Yes, 2.0 has a new storage backend with a much reduced need for iops for writes.

I'm not familiar enough with this peculiar detail of NFS to know what's going on.

[alxmk]

https://serverfault.com/questions/201294/nfsxxxx-files-appearing-what-are-those

Found the above reading around the subject - seems like the NFS file's there because of an open file, suggests perhaps there's an issue with files not being closed out? Not sure.

Since we're running this on Kubernetes I don't know that SSD is an option for us even if it's performant enough. The only way would be to pin it to a specific Node and add the SSD there, but then we lose resiliency. Tricky.

[Dominik-K]

@alxmk You can add a SSD as a persistent volume to your pods. Then Kubernetes handles to mount the SSD to the node where the Prometheus pod is running. Did you consider that?

[alxmk]

My understanding is that still has the issue that when the node that has the SSD physically attached to it goes down, we lose the volume, even if the prometheus pod is running elsewhere.

[gouthamve]

This should be fixed by prometheus/tsdb#213 and #3508

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.