[Error] missing a file at start-up

[JeromeVi]

What did you do?
I runed Prometheus + nodeexporter + cAdvisor + AlertManager on Docker

What did you expect to see?
My containers running

What did you see instead? Under which circumstances?
I see the Prometheus container restarting in loop

Environment

• System information:

    Darwin 16.4.0 x86_64
  

• Prometheus version:
  latest

• Logs:

time="2017-02-17T08:08:03Z" level=info msg="Starting prometheus (version=1.5.2, branch=master, revision=bd1182d29f462c39544f94cc822830e1c64cf55b)" source="main.go:75"
time="2017-02-17T08:08:03Z" level=info msg="Build context (go=go1.7.5, user=root@1a01c5f68840, date=20170210-16:23:28)" source="main.go:76"
time="2017-02-17T08:08:03Z" level=info msg="Loading configuration file /etc/prometheus/prometheus.yml" source="main.go:248"
time="2017-02-17T08:08:03Z" level=error msg="Error opening memory series storage: leveldb: manifest corrupted (field 'comparer'): missing [file=MANIFEST-000009]" source="main.go:182"

[JeromeVi]

I have tried to remove all the containers + images and made the docker-compose again and nothing has changed.

docker-compose.yml

version: '2'

networks:
  monitor-net:
    driver: bridge

volumes:
    prometheus_data: {}

services:

  prometheus:
    image: prom/prometheus
    container_name: prometheus
    volumes:
      - ./prometheus/:/etc/prometheus/
      - prometheus_data:/prometheus
    command:
      - '-config.file=/etc/prometheus/prometheus.yml'
      - '-storage.local.path=/prometheus'
      - '-alertmanager.url=http://alertmanager:9093'
      - '-storage.local.memory-chunks=100000'
    restart: unless-stopped
    expose:
      - 9090
    ports:
      - 9090:9090
    networks:
      - monitor-net
    labels:
      org.label-schema.group: "monitoring"

  alertmanager:
    image: prom/alertmanager
    container_name: alertmanager
    volumes:
      - ./alertmanager/:/etc/alertmanager/
    command:
      - '-config.file=/etc/alertmanager/config.yml'
      - '-storage.path=/alertmanager'
    restart: unless-stopped
    expose:
      - 9093
    ports:
      - 9093:9093
    networks:
      - monitor-net
    labels:
      org.label-schema.group: "monitoring"

  nodeexporter:
    image: prom/node-exporter
    container_name: nodeexporter
    restart: unless-stopped
    expose:
      - 9100
    networks:
      - monitor-net
    labels:
      org.label-schema.group: "monitoring"

  cadvisor:
    image: google/cadvisor:v0.24.1
    container_name: cadvisor
    volumes:
      - /:/rootfs:ro
      - /var/run:/var/run:rw
      - /sys:/sys:ro
      - /var/lib/docker/:/var/lib/docker:ro
    restart: unless-stopped
    expose:
      - 8080
    networks:
      - monitor-net
    labels:
      org.label-schema.group: "monitoring"

prometheus.yml

global:
  scrape_interval:     15s
  evaluation_interval: 15s

  external_labels:
      monitor: 'docker-host-alpha'

rule_files:
  - "targets.rules"
  - "host.rules"
  - "containers.rules"

scrape_configs:
  - job_name: 'nodeexporter'
    scrape_interval: 5s
    static_configs:
      - targets: ['nodeexporter:9100']

  - job_name: 'cadvisor'
    scrape_interval: 5s
    static_configs:
      - targets: ['cadvisor:8080']

  - job_name: 'prometheus'
    scrape_interval: 10s
    static_configs:
      - targets: ['localhost:9090']

[brian-brazil]

You database got corrupted. Blow it away.

[JeromeVi]

It changed nothing I said I have removed all the containers + images @brian-brazil

[beorn7]

Your setup must corrupt the database then. Perhaps some accidental read-only or "clean up on restart" thing? Or you mapped two things to the same location?

I'm not an expert in parsing docker-compose config, but this is almost certainly an external setup problem rather than a Prometheus problem.

[JeromeVi]

So I think the problem comes from my vbox machine who run Prometheus. Thank you for the help

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.