no space left on device for prometheus

[ShurikAg]

Hi all,

We are running on this stack for a while now. After some time I noticed that we ran out of space on prometheus storage.

ts=2022-06-01T07:36:18.482Z caller=main.go:1064 level=error err="opening storage failed: open /prometheus/wal/00000554: no space left on device"

The storage configuration is exactly as it was proposed from the starter kit. So, it is set to 5G.

Now, I tried adjusting the storage size to 20G through the same repo that drives the cluster config, however, the storage still stays at 5G. How can I resize the storage allocated for it?

[v-ctiutiu]

Hi @ShurikAg

Let me check and get back to you.

Thanks.

[mtiutiu-heits]

I see. Did you look at the volume in the DO panel ? Was it resized ?

I'm testing the resize right now via Helm, and indeed the PVC size is not changed for some reason. I noticed that volume expansion is set for DO block storage, so if the PVC value is increased the DO block storage volume should be increased as well:

$ kubectl get sc

NAME                         PROVISIONER                 RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGE
do-block-storage (default)   dobs.csi.digitalocean.com   Delete          Immediate           true                   10d

So, after I changed the volume size in the helm values file, I see the new value reflected in the output (10Gi):

$ helm get values kube-prom-stack -n monitoring

...
prometheus:
  enabled: true
  prometheusSpec:
    storageSpec:
      volumeClaimTemplate:
        spec:
          accessModes:
          - ReadWriteOnce
          resources:
            requests:
              storage: 10Gi
          storageClassName: do-block-storage
...

Then, if I list the Prometheus PVC parameters, it still shows the old size of 5Gi:

$ kubectl get pvc -n monitoring

NAME                                                                                                     STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS       AGE
prometheus-kube-prom-stack-kube-prome-prometheus-db-prometheus-kube-prom-stack-kube-prome-prometheus-0   Bound    pvc-150c7961-18ec-497f-8a3e-bb5e0d29299c   5Gi        RWO            do-block-storage   88m

For me it seems that for some reason the Prometheus operator doesn't see the new storage spec value.

Still investigating.

[mtiutiu-heits]

@ShurikAg

Seems there's a discussion about this issue on the prometheus operator page. You can check that as well.

[mtiutiu-heits]

@ShurikAg

In a nutshell, it seems that the Prometheus operator is using StatefulSets under the hood. StatefulSets are immutable in general, and changing PVC size attached to them is subject to same restrictions.

After a quick search I found this blog post which explains the same thing.

Let me know if it's the same issue, and if it helps.

Thanks.

[ShurikAg]

@mtiutiu-heits OK, that starts making a bit more sense. However, as I mentioned before, I accidentally deleted the storage class and I don't have the config that it used to have. Is there any way to resurrect it?

[ShurikAg]

@mtiutiu-heits Thank you for pointing out that article. It was super helpful for information. However, it didn't work as described. I had to delete everything related to Prometheus, except the operator, to fix it. Not a big deal in our case, we didn't have downtime for the application. Also, since we are managing everything through Flux CD, recreating everything was as simple as committing new config with changes.