Error opening memory series storage

[Spritekin]

Hi,
I got this message while restarting my prometheus.

time="2016-04-28T03:55:26Z" level=error msg="Error opening memory series storage: resource temporarily unavailable" source="main.go:139"

The data is on a NFS storage at /prometheus/data.

I changed the target directory to /prometheus/data2 and it starts fine. But of course no data available.

I'm running prometheus 0.17 from the official docker image (only with my yaml files added).

I was hoping anyone would shed a light.

[brian-brazil]

We do not recommend using NFS for Prometheus.

This is likely a file locking issue, which is notoriously difficult to get working with NFS.

[Spritekin]

Alriiight... thank you Brian!

[beorn7]

Prometheus keeps a file lock to protect against concurrent use of the same storage directory by two different Prometheus instances. I assume this is what broke here.

[waqark3389]

Any workaround to this? I moved my prometheus directory including everything inside it from /opt/prometheus/prometheus-1.0.0.linux-amd64/ to /opt/prometheus/ and am getting the same above error while attempting to start.

[brian-brazil]

Are you sure you killed the old prometheus?

[waqark3389]

Yeah I did. I think its due to the fact that I am using systemd file. I will double check my systemd service file as I've just read that two instances can cause this.

[waqark3389]

Apologies, was a problem with my systemd script. Changed to the following and all works fine.
[Unit]
Description=Prometheus Systemd File
After=network.target

[Service]
User=prometheus
Group=prometheus
Type=simple
WorkingDirectory=/opt/prometheus
LimitNOFILE=65536
Environment='ALLOUT=/opt/prometheus/prometheus.log'
EnvironmentFile=-/etc/default/prometheus
ExecStart=/bin/sh -c "exec /opt/prometheus/prometheus -config.file=/opt/prometheus/prometheus.yml ${PROMOTHEUS_OPTS} > ${ALLOUT} 2>&1"
KillMode=control-group
#Restart=always

[Install]
WantedBy=multi-user.target
Alias=prometheus.service

[dalgibbard]

FWIW; we've hit this issue or similar twice now whilst using MapR (NFS) as the backend storage for running Prometheus in Mesos/Marathon. Seems that prometheus locks up if the backend storage has a 'moment'. And then it's a bit hit and miss as to whether the lock remains in the backend or not. Moving the 'DIRTY' file away, and back again seems to be sufficient to get Prometheus running again if the lock is persisted, but the Prometheus process refuses to stop; even when stopping docker the process becomes defunct/zombie...
This is prometheus 1.2.1 at the mo (soon to update) - do you happen to know if newer versions handle backend storage troubles more gracefully? (and specifically which version(s) it was fixed in if so?)

Thanks! :)

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.