Prometheus data never stop growing even with a tiny retention time

[astraios]

What did you do?

I'm using prometheus to scrape metrics from my kubernetes cluster. It's a small setup, one master, 5 nodes with around 40 containers spread across the cluster.

I configured a small retention time, 1h.

What did you expect to see?

It should work without filing my disk.

What did you see instead? Under which circumstances?

Even with a small retention period, the datastore never stop growing until my /var is full and my linux crash.

If I dive into the docker volume, I can see that the folder labelpair_to_fingerprints never stop growing and prometheus deployment has been running for days.

Environment

Kubernetes v1.5.2

• Prometheus version:

  v1.3.1

• Alertmanager version:

  insert output of alertmanager -version here (if relevant to the issue)

• Prometheus deployment file:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  labels:
    name: prometheus-deployment
  name: prometheus
  namespace: kube-system
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: prometheus
    spec:
      containers:
      - image: quay.io/prometheus/prometheus:v1.3.1
        name: prometheus
        command:
        - "/bin/prometheus"
        args:
        - "-config.file=/etc/prometheus/prometheus.yml"
        - "-storage.local.path=/prometheus"
        - "-storage.local.retention=1h"
        ports:
        - containerPort: 9090
          protocol: TCP
        volumeMounts:
        - mountPath: "/prometheus"
          name: data
        - mountPath: "/etc/prometheus"
          name: config-volume
        resources:
          requests:
            cpu: 1000m
            memory: 3000Mi
          limits:
            cpu: 2000m
            memory: 3000Mi
      volumes:
      - emptyDir: {}
        name: data
      - configMap:
          name: prometheus-config
        name: config-volume```

[beorn7]

What disk size are we talking about here?

Note that there is a certain base footprint for the LevelDB indices, the checkpointing, the directory structure and such.
Then you have to take into account that the series files are chunked, so each series has to take at least one chunk, which is a bit more than 1kiB on disk. A chunk will easily span a handful of hours, so any retention time shorter than the duration a chunk covers will not reduce disk usage. Also, series are truncated in a cycle, which takes a while to get through if you have many series (in particular on HDD as a seek to the series file is involved). Look for lines starting with "Completed maintenance sweep through..." to find completion times for a cycle through archived and in-memory series (the latter will be most relevant to you). If that's, let's say, 12h, each series will only be looked at every 12h.

[astraios]

For now, prometheus disk usage reached 690MB total and it's been running for 3days.

Here is the last "Completion message" I got from the prometheus logs:

Completed maintenance sweep through 102366 in-memory fingerprints in 8m4.819083756s.

Even if we reach a 12h cycle, wouldn't the data size decrease after 3 days?
I also should mention that I didn't proceed in any upsizing of any kind that would result in a increase of collected metrics.

Regards,
Thomas

[beorn7]

For now, prometheus disk usage reached 690MB total and it's been running for 3days.

Is it still growing, or is 690MB the steady state?

With 100k series, I would expect disk usage to not go beyond 500MiB or so...

Could you check how the disk usage is distributed over sub-directories in your data dir?
Also an interesting metric is prometheus_local_storage_chunk_ops_total{type="drop"}, to see if and how many chunks get deleted at all.

And please double-check that the flag value for retention is really set (via http://my.prometheus:9090/flags ).

[astraios]

Disk usage in the data dir:
https://gist.github.com/notuscloud/76537ad381725cf54aecc5e5f910118a

I'll tell you tomorrow if the disk usage increased or not, for now, it is still around 690MB.

prometheus_local_storage_chunk_ops_total{type="drop"} return this:
prometheus_local_storage_chunk_ops_total{type="drop"} 703563

I double-checked to flags, and storage.local.retention is set to 1h0m0s

Is there a way to know how many series my prometheus collects?
Is this metric a good indicator prometheus_local_storage_memory_series 98224 ?

Many thanks,
Thomas

[astraios]

Hi,

I checked the prometheus disk consumption this morning and we are now at 713MB.
It's still growing.

Regards,
Thomas

[beorn7]

prometheus_local_storage_memory_series is the number of series currently in memory.

You could just count the *.db files in the various two-hexadecimal-digits directories beneath your data directory to find out how many series are represented as files in your storage. You can also see how large they are individually. Note that, depending on the filesystem and its parameters, there will be an overhead per file, i.e. the file size is smaller than what's taken in the filesystem. The Prometheus storage is designed with sizes in mind that are three orders of magnitude larger than what you have now.

Let's say you have 100k files there. Each is at least 1kiB in size. That's already 100MiB. About the same size for the checkpoint. And then LevelDB will have a certain baseline. Plus the overhead as explained above. Not entirely unreasonable to end up with something around 1GiB.

The value of prometheus_local_storage_chunk_ops_total{type="drop"} tells you that chunks are deleted for real. That makes it highly unlikely that we have a clean-up bug that triggers with short retention times. You could take rate(prometheus_local_storage_chunk_ops_total{type="drop"}[5m]) and then rate(prometheus_local_storage_chunk_ops_total{type="persist"}[5m]) to compere "incoming" and "outgoing" rate of chunks.

[astraios]

Hi,

Sorry for this late answer. I have 14970 series in my storage. As we speak, the storage size increased to reach 870MB.

I gave 2CPU and 3GB of RAM to the pod and both resources are used up to 100% most of the time.

Regards,
Thomas

[beorn7]

I don't know how I can assist you any further with troubleshooting. Ever growing disk space usage doesn't seem to be a problem with Prometheus in general. We need more evidence to find anything that might be wrong with Prometheus here.

[astraios]

What can I provide you?

[beorn7]

Ideally a PR to fix the bug. :->

Otherwise a smoking gun, that let's us know on a more low level detail what's going wrong and that something is going wrong in the first place. Disk, memory, CPU usage might all be reasonable in your setup. We needed something like "this goroutine spins on the CPU where it shouldn't" or "This file is still on disk but it shouldn't" or "this data structure is allocated in memory but it shouldn't".

Go has a lot of troubleshooting tooling for that but I have no capacity to teach you using them.

[astraios]

Ok, thx.

I'm gonna go learn a few things about Go troubleshouting and come back with more data.

Thanks for you time, it's really appreciated.

Regards,
Thomas

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.