Prometheus starts but port 9090 not listening!

[stewartbarrett]

After resizing the volume on one of our Prometheus nodes (due to a full disk) Prometheus seems to start but doesn't listen on port 9090. How can I recover this?

Prometheus is started with:

/usr/bin/prometheus -config.file=/etc/prometheus/config.yml -storage.local.path=/var/lib/prometheus -storage.local.memory-chunks=1572864 -storage.local.max-chunks-to-persist=1572864 -web.console.libraries=/usr/share/prometheus/console_libraries -web.console.templates=/usr/share/prometheus/consoles -alertmanager.url=http://alertmanager.meer-spacestation.local

The log file shows the following: (the crashrecovery doesn't recover)

time="2016-08-31T14:15:27Z" level=warning msg="Truncating file /var/lib/prometheus/05/5b69110db860f7.db to exactly 169 chunks, trimming 199 extraneous bytes." source="crashrecovery.go:220"
time="2016-08-31T14:15:28Z" level=warning msg="Recovered metric prometheus_local_storage_indexing_batch_sizes_count{datacenter="eu-west-1b", instance="platform.prometheus-host:eu-west-1b:", job="prometheus", product="platform", service="prometheus-host"}, fingerprint 055b69110db860f7: all 169 chunks recovered from series file." source="crashrecovery.go:319"
time="2016-08-31T14:15:28Z" level=warning msg="Recovered metric instance:server_http_response_time_milliseconds_count:increase15m{exported_service="customer-account.personal-details", http_status_code="404", path="/clientaccesspolicy.xml", product="customer-account", service="web"}, fingerprint 0511bc78303fc0fb: all 9 chunks recovered from series file." source="crashrecovery.go:319"
time="2016-08-31T14:15:28Z" level=warning msg="Truncating file /var/lib/prometheus/05/1c7ed9ab7c5040.db to exactly 495 chunks, trimming 801 extraneous bytes." source="crashrecovery.go:220"
time="2016-08-31T14:15:29Z" level=warning msg="Recovered metric server_http_response_time_milliseconds{datacenter="bre", environment="prod", exported_service="customer-account.myctm-journey", http_status_code="500", http_verb="GET", instance="172.30.1.136:9190", job="log_exporter_bgl", log_exporter_source="f:/logs/iis/myctm/W3SVC3", path="/cgi-bin", product="customer-account", quantile="0.99", service="web", webserver="iis"}, fingerprint 051c7ed9ab7c5040: all 495 chunks recovered from series file." source="crashrecovery.go:319"

prometheus, version 0.16.2 (branch: release-0.16, revision: 287d9b2)
build user: xxxxxxxxx@macpro
build date: 20160118-13:10:31
go version: 1.5.3

Im running this in AWS on Ubuntu on an m4.2xlarge with 500GB EBS volume.

[zouyee]

need waiting for recover

[beorn7]

Yeah, as long as the recovery runs, the server doesn't listen on port 9090.

Your storage might be corrupt beyond repair (which could happen if you run out of disk space), in which case, you need to start from scratch.

[juliusv]

Seems answered - closing. Please reopen if necessary.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.