Could not lock /prometheus/DIRTY, Prometheus already running?

[djenriquez]

Hello,

I am having some trouble configuring Prometheus. I currently seem to keep running into this problem:

prometheus, version 0.15.1 (branch: master, revision: 64349aa)
  build user:       julius@julius-thinkpad
  build date:       20150727-17:56:51
  go version:       1.4.2
time="2015-09-08T05:40:04Z" level=info msg="Loading configuration file /etc/prometheus/prometheus.yml" file=main.go line=173 
time="2015-09-08T05:40:04Z" level=info msg="Loading series map and head chunks..." file=storage.go line=263 
time="2015-09-08T05:40:04Z" level=info msg="0 series loaded." file=storage.go line=268 
time="2015-09-08T05:40:04Z" level=info msg="Starting target manager..." file=targetmanager.go line=75 
time="2015-09-08T05:40:04Z" level=info msg="Listening on :9090" file=web.go line=186 
prometheus, version 0.15.1 (branch: master, revision: 64349aa)
  build user:       julius@julius-thinkpad
  build date:       20150727-17:56:51
  go version:       1.4.2
time="2015-09-08T05:40:05Z" level=info msg="Loading configuration file /etc/prometheus/prometheus.yml" file=main.go line=173 
time="2015-09-08T05:40:05Z" level=error msg="Could not lock /prometheus/DIRTY, Prometheus already running?" file=persistence.go line=178 
time="2015-09-08T05:40:05Z" level=error msg="Error opening memory series storage: resource temporarily unavailable" file=main.go line=116 
prometheus, version 0.15.1 (branch: master, revision: 64349aa)
  build user:       julius@julius-thinkpad
  build date:       20150727-17:56:51
  go version:       1.4.2
time="2015-09-08T05:40:06Z" level=info msg="Loading configuration file /etc/prometheus/prometheus.yml" file=main.go line=173 
time="2015-09-08T05:40:06Z" level=error msg="Could not lock /prometheus/DIRTY, Prometheus already running?" file=persistence.go line=178 
time="2015-09-08T05:40:06Z" level=error msg="Error opening memory series storage: resource temporarily unavailable" file=main.go line=116 
prometheus, version 0.15.1 (branch: master, revision: 64349aa)
  build user:       julius@julius-thinkpad
  build date:       20150727-17:56:51
  go version:       1.4.2
time="2015-09-08T05:40:07Z" level=info msg="Loading configuration file /etc/prometheus/prometheus.yml" file=main.go line=173 
time="2015-09-08T05:40:07Z" level=error msg="Could not lock /prometheus/DIRTY, Prometheus already running?" file=persistence.go line=178 
time="2015-09-08T05:40:07Z" level=error msg="Error opening memory series storage: resource temporarily unavailable" file=main.go line=116 
prometheus, version 0.15.1 (branch: master, revision: 64349aa)
  build user:       julius@julius-thinkpad
  build date:       20150727-17:56:51
  go version:       1.4.2

When I use the example configuration from the Getting Started guide, everything works fine:

global:
  scrape_interval:     15s # By default, scrape targets every 15 seconds.
  evaluation_interval: 15s # By default, scrape targets every 15 seconds.
  # scrape_timeout is set to the global default (10s).

  # Attach these extra labels to all timeseries collected by this Prometheus instance.
  labels:
    monitor: 'codelab-monitor'

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: 'prometheus'

    # Override the global default and scrape targets from this job every 5 seconds.
    scrape_interval: 5s
    scrape_timeout: 10s

    target_groups:
      - targets: ['localhost:9090']

However, it seems that if i change this file in ANYWAY, for example, remove the # Override the global defaults and scrape targets from this job every 5 seconds. line, the logs start reporting this DIRTY issue.

In fact, even removing the blank lines like this configuration:

global:
  scrape_interval:     15s # By default, scrape targets every 15 seconds.
  evaluation_interval: 15s # By default, scrape targets every 15 seconds.
  # scrape_timeout is set to the global default (10s).

  # Attach these extra labels to all timeseries collected by this Prometheus instance.
  labels:
    monitor: 'codelab-monitor'

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: 'prometheus'
    # Override the global default and scrape targets from this job every 5 seconds.
    scrape_interval: 5s
    scrape_timeout: 10s
    target_groups:
      - targets: ['localhost:9090']

ends up throwing the same error.

I am running Prometheus in a docker container, incase that matters.

Any idea what is going on?

[beorn7]

Looks like Prometheus cannot write into the directory you have configured as storage location (configured by flag -storage.local.path). Why changing the config changes that, is a mystery to me. Perhaps Docker somehow changes write access if you change something in your container?

[djenriquez]

The problem was related to how I was building my docker image from my Dockerfile. Using the RUN command, I created the /prometheus folder which i then extracted the tar.gz file into. I then moved the items into their appropriate locations in /bin/ and /etc/prometheus. After the RUN command, I set VOLUME ["/prometheus"].

I did some rearranging, by setting VOLUME ["/prometheus"] before the RUN command, and used a different folder to extract the tar.gz contents into: /tmp/prometheus/

After this, everything was fine.

[beorn7]

Glad to hear. :)

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.