Docker: unable to mount /prometheus

[rtoma]

Using boot2docker and docker 1.7.0.

Gets me this:

$ docker run --rm -v $PWD/data:/prometheus -p 9090:9090 prom/prometheus
prometheus, version 0.14.0 (branch: stable, revision: 67e7741)
  build user:       root
  build date:       20150603-06:20:33
  go version:       1.4.2
time="2015-07-03T13:54:05Z" level=warning msg="No remote storage URLs provided; not sending any samples to long-term storage" file=main.go line=126
time="2015-07-03T13:54:05Z" level=info msg="Loading configuration file /etc/prometheus/prometheus.yml" file=main.go line=220
time="2015-07-03T13:54:05Z" level=error msg="Error opening memory series storage: fsync: invalid argument" file=main.go line=242

AFAIK this is result of LevelDB on a Virtualbox mount? Couldn't find a fix/workaround.

Any help welcome!

[juliusv]

Hmm, not sure. Works for me with native Docker (not boot2docker) on Ubuntu Linux:

$ docker run --rm -v $PWD/data:/prometheus -p 9090:9090 prom/prometheus
prometheus, version 0.14.0 (branch: stable, revision: 67e7741)
  build user:       root
  build date:       20150603-06:20:33
  go version:       1.4.2
time="2015-07-03T13:58:54Z" level=warning msg="No remote storage URLs provided; not sending any samples to long-term storage" file=main.go line=126 
time="2015-07-03T13:58:54Z" level=info msg="Loading configuration file /etc/prometheus/prometheus.yml" file=main.go line=220 
time="2015-07-03T13:58:54Z" level=info msg="Loading series map and head chunks..." file=storage.go line=227 
time="2015-07-03T13:58:54Z" level=info msg="224 series loaded." file=storage.go line=232 
time="2015-07-03T13:58:54Z" level=info msg="Starting target manager..." file=targetmanager.go line=75 
time="2015-07-03T13:58:54Z" level=info msg="Listening on :9090" file=web.go line=118

[sdurrheimer]

Volume are kind of special with boot2docker. Lot of problems about permissions.

I will test it tonight.

[KoichiSenada]

I have the same bug with the latest Kitematic and Docker Hub Prom Prometheus
It happens whenever I mount the expected /prometheus folder.
My OS is Windows 8.1 x64

prometheus, version 0.14.0 (branch: stable, revision: 67e7741)
  build user:       root
  build date:       20150603-06:20:33
  go version:       1.4.2
time="2015-07-08T10:19:15Z" level=warning msg="No remote storage URLs provided; not sending any samples to long-term storage" file=main.go line=126 
time="2015-07-08T10:19:15Z" level=info msg="Loading configuration file /etc/prometheus/prometheus.yml" file=main.go line=220 
time="2015-07-08T10:19:15Z" level=error msg="Error opening memory series storage: fsync: invalid argument" file=main.go line=242 

[gesellix]

another test case on Mac/boot2docker 1.7.1:

$ docker run --rm -v $PWD/data:/prometheus -p 9090:9090 prom/prometheus
prometheus, version 0.14.0 (branch: stable, revision: 67e7741)
  build user:       root
  build date:       20150603-06:20:33
  go version:       1.4.2
time="2015-07-16T22:28:26Z" level=warning msg="No remote storage URLs provided; not sending any samples to long-term storage" file=main.go line=126 
time="2015-07-16T22:28:26Z" level=info msg="Loading configuration file /etc/prometheus/prometheus.yml" file=main.go line=220 
time="2015-07-16T22:28:26Z" level=info msg="Loading series map and head chunks..." file=storage.go line=227 
time="2015-07-16T22:28:26Z" level=info msg="223 series loaded." file=storage.go line=232 
time="2015-07-16T22:28:26Z" level=info msg="Starting target manager..." file=targetmanager.go line=75 
time="2015-07-16T22:28:26Z" level=info msg="Listening on :9090" file=web.go line=118 

[beorn7]

Docker experts (like @discordianfish ), any insights here?

[discordianfish]

I think I had the same/similar issue when trying to use sshfs to store prometheus metrics on. boot2docker and kitematic probably use some network filesystem as well, so probably the same issue.
Maybe prometheus should ignore the error from fsync if on "uncommon" filesystems?

[brian-brazil]

Was there any resolution here?

If we need a filesystem that supports fsync then we should document that somewhere.

[discordianfish]

I think we discussed this back then and the sentiment was that it's probably safe to ignore and other are doing the same, but we thought it only affected nfs which was deemed to be less of a problem.

[brian-brazil]

Okay, I'll close for now. If it comes up again we can discuss it.

[rtoma]

The issue is closed, but I cannot find a solution in prior comments.

Prometheus still does not work for me with a volume mount. Without an -v it all works just fine.

$ b2d version
Boot2Docker-cli version: v1.8.0

$ docker version
Git commit: 9a26066
Client:
 Version:      1.8.1
 API version:  1.20
 Go version:   go1.4.2
 Git commit:   d12ea79
 Built:        Thu Aug 13 02:49:29 UTC 2015
 OS/Arch:      darwin/amd64

Server:
 Version:      1.8.1
 API version:  1.20
 Go version:   go1.4.2
 Git commit:   d12ea79
 Built:        Thu Aug 13 02:49:29 UTC 2015
 OS/Arch:      linux/amd64

$ docker run --rm -v $PWD/data:/prometheus -p 9090:9090 prom/prometheus
prometheus, version 0.15.1 (branch: stable, revision: 64349aa)
  build user:       @bfbdc5abbb9f
  build date:       20150727-15:58:57
  go version:       1.4.2
time="2015-08-21T07:07:49Z" level=info msg="Loading configuration file /etc/prometheus/prometheus.yml" file=main.go line=173
time="2015-08-21T07:07:49Z" level=error msg="Error opening memory series storage: fsync: invalid argument" file=main.go line=116

[brian-brazil]

@rtoma It appears you need a filesystem that supports fsync, which is going to be a requirement for any database-type system such as prometheus. An strace will help you know exactly which syscall is failing.

[matthiasr]

Since Prometheus is not a very consistent database anyway, I would say it should log a warning (once) on a failed fsync, but carry on and hope for the best.

[matthiasr]

Alternatively, it could provide a -no-sync flag for YOLO operations. This may also be helpful when throughput is more important than consistency.

[brian-brazil]

Since Prometheus is not a very consistent database anyway, I would say it should log a warning (once) on a failed fsync, but carry on and hope for the best.

Consistency at the application level isn't the same thing as consistency at the filesystem level. Mostly silently disabling a key safety feature could cause us problems and support hassle, we already seem to have quite a few users who are KILLing rather than TERMing prometheus, and a few reports of corruption so let's not add more noise to the mix.

[matthiasr]

Then let's at least allow to override this. It prevents anyone on a docker/mac or network storage from using Prometheus.

[brian-brazil]

Then let's at least allow to override this. It prevents anyone on a docker/mac or network storage from using Prometheus.

We first need to ask if that's a use case we want to support. There are more than a few users out there who'll blindly disable fsync without realising they no longer have a database they can rely on. I think the better question is why docker on mac doesn't support fsync.

[gesellix]

as mentioned in #875 (comment) it seemed to work on my mac.

[brian-brazil]

It doesn't look like there's much we can safely do here, it's not practical to run any sort of stateful system on top of a filesystem that doesn't support fsync. I'd recommend using a more standard local filesystem rather than the more complex network ones.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.