Allow for volumes without host-mapping them to VirtualBox

[rmb938]

Databases like mongo, mysql and rabbitmq get permission denied when trying to start. This has to do with the volume permissions. Being that having access to that data isn't really necessary I would remove the volume but there doesn't seem to be a way to do that either.

[jmorganca]

👍 Thanks for reporting this!

Seems like many of the official hub repositories such as mysql or redis run the db process as a separate user, which won't have access to a mounted volume.

It seems like a quick fix here is just to add the ability to disable a container's volume. Keep an eye out for a fix!

[rmb938]

Alright and thanks for the quick reply

[thaJeztah]

Currently, it's not possible to "disable" a volume if an image defines one. There's work in progress to allow UNSET-ing properties. See moby/moby#8177 for more information on that.

However, volumes are used deliberately in those images for a number of reasons;

• For persistent data; volumes "outlive" the lifespan of a container; if the container "dies", the data is not lost
• Being able to "share" data (ie, --volumes-from), for example to use a one-off container to initialize the data/database. And another container to actually use or run the data(base)
• Being able to "upgrade" a container (stop the old database container, start a new one, using --volumes-from=old-container, and remove the old container)
• For performance; and this is an important reason, the layered/copy-on-write filesystem in the container has a huge performance penalty for data that's changing often, which is the case with databases.

Using a "regular" volume in stead of a bind-mounted volume could solve the permissions issue. Another solution is to chown the directory/files using the uid:gid of the database user, but if VBox shared directories are used, performance and permissions can still be tricky.

[rmb938]

I know you can't disable a volume, but if you run a container without giving it volume information it will create a random volume inside the docker storage directory. This can be an easy and quick fix for now especially if the user does not care about having an accessible volume

[thaJeztah]

@rmb938 👍 alright! This put me on the "wrong foot";

It seems like a quick fix here is just to add the ability to disable a container's volume.

[jmorganca]

:+1 Wrong wording on my part! Sorry! I meant adding away to disable the host binding altogether.

Sent from my iPhone

On Tue, Mar 3, 2015 at 12:33 PM, Sebastiaan van Stijn
notifications@github.com wrote:

@rmb938 👍 alright! This put me on the "wrong foot";

It seems like a quick fix here is just to add the ability to disable a container's volume.

Reply to this email directly or view it on GitHub:
#209 (comment)

[thaJeztah]

No problem, and no harm done! my comment may be useful for people finding this issue and are new to Docker and not familiar with all aspects of volumes 😄

[jmorganca]

@thaJeztah Hugely useful! That comment should be a blog post post somewhere 😃

[g12n]

Another example for this Problem is the elsdoerfer / thumbor container.
It automatically exposes 3 volumes, but cant write into it. This way you cant run the tool at all.

[tslater]

+1 this is a pretty important thing here...I have the same issue with many containers
ex. mkdir: cannot create directory `/var/lib/rabbitmq/mnesia': Permission denied

[ghost]

+1 for the comment above , Getting no love from the rabbit container setup ,

Cannot create directory `/var/lib/rabbitmq/mnesia': Permission denied.

permissions error occurring on many other containers , making most unuseable

[jmorganca]

Sorry for the delay in getting a patch out and thanks for all the feedback
on this. This is priority for 0.5.11

On Saturday, March 14, 2015, Paul Mortimer notifications@github.com wrote:

• for the comment above , Getting no love from the rabbit container
  setup ,

Cannot create directory `/var/lib/rabbitmq/mnesia': Permission denied.

permissions error occurring on many other containers , making most
unuseable

—
Reply to this email directly or view it on GitHub
#209 (comment).

[sebgoa]

+1 for a fix to this. This is a petty big blocker.

[stereosupersonic]

+1 please fix this, i want to use mongodb

[IBMRob]

+1 please fix so I can try mqadvanced

[g12n]

@boot2docker ist struggeling with the same Problem 😒
compare: boot2docker/boot2docker#581

[ezegolub]

+1. This is a huge blocker :(

[adamcypher]

Does Kitematic use Boot2Docker underneath or replace it?

This thread has several suggestions for fixing the issue with Boot2Docker. A promising suggestion is to use NFS.

boot2docker/boot2docker#581

[adamcypher]

What are the solid work-arounds to date?

Was going to try the NFS solution, but not sure how to create the Kitematic-VM equivalent for:
/var/lib/boot2docker/bootlocal.sh

[adamcypher]

What's the release ETA for 0.5.11?

[danielchatfield]

This is a massive blocker, I think virtualbox share should be ditched and NFS (or other) used instead (NFS even performs better).

[kipelovets]

+1 official/mysql doesn't work because of permission problems

[DominicBoettger]

+1 Mongodb official does not work

[amrox]

+1 Mongodb official or postgres official does not seem to work

[DominicBoettger]

Solved it for me with another image which does not export the data volume.
https://registry.hub.docker.com/u/inspirationlabs/mongodb/

[jmorganca]

ETA for 0.5.11 is Friday. Sorry again in a delay with the update

[lypht]

+1 for mongodb. No issues with Rabbitmq in the same Kitematic instance, though.

[lypht]

Well this is interesting...

By defaulting the advanced options to TTY and STDIN open I was able to get all of my containers to stay running. I deselected both and restarted without issue. Can anyone else replicate this behavior? I've changed no permissions or otherwise altered these containers (beyond renaming them for dev purposes). The screenshot is useless other than to demonstrate that the containers are renamed and running.

[otype]

+1 for elasticsearch 1.4.5, mysql 5.6.26, rabbitmq 3.5.4-management ... all of these won't start up if the volume has been enabled.

Redis 3.0.3, though, seems to be able to run with an enabled volume ...

[fotisp]

+1

[marksomething]

Is there a work around?

[solisoft]

My work around : install Linux on your laptop and use terminal ;)

[ghostrocket]

+1 kind of makes this exercise futile

[pierre-b]

+1 also ;)

[alexbeletsky]

+1 blocks rabbitmq, will be happy to see workaround.

[alexbeletsky]

That issue exists for quite long time, basically preventing to use docker-compose for devepment environments. It seems to me, that it's not related to Kitematic itself, but rather docker (docker-machine) problem.

Are there any connected issues?

[sahin]

+1

I am using latest version in Kitematic. After we are setting "Enable Volumes", we are getting the following the errors.

Postgres

The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.
The database cluster will be initialized with locale "en_US.utf8".
The default database encoding has accordingly been set to "UTF8".
The default text search configuration will be set to "english".
Data page checksums are disabled.
fixing permissions on existing directory /var/lib/postgresql/data ... ok
creating subdirectories ... initdb: could not create directory "/var/lib/postgresql/data/global": Permission denied
initdb: removing contents of data directory "/var/lib/postgresql/data"
The files belonging to this database system will be owned by user "postgres".

Mysql

Initializing database
2015-12-28T01:24:37.592834Z 0 [Warning] Setting lower_case_table_names=2 because file system for /var/lib/mysql/ is case insensitive
2015-12-28T01:24:37.644847Z 0 [ERROR] InnoDB: Operating system error number 13 in a file operation.
2015-12-28T01:24:37.645642Z 0 [ERROR] InnoDB: The error means mysqld does not have the access rights to the directory.
2015-12-28T01:24:37.646569Z 0 [ERROR] InnoDB: Operating system error number 13 in a file operation.
2015-12-28T01:24:37.647002Z 0 [ERROR] InnoDB: The error means mysqld does not have the access rights to the directory.
2015-12-28T01:24:37.647438Z 0 [ERROR] InnoDB: Cannot open datafile './ibdata1'
2015-12-28T01:24:37.647820Z 0 [ERROR] InnoDB: Could not open or create the system tablespace. If you tried to add new data files to the system tablespace, and it failed here, you should now edit innodb_data_file_path in my.cnf back to what it was, and remove the new ibdata files InnoDB created in this failed attempt. InnoDB only wrote those files full of zeros, but did not yet use them in any way. But be careful: do not remove old data files which contain your precious data!
2015-12-28T01:24:37.648064Z 0 [ERROR] InnoDB: InnoDB Database creation was aborted with error Cannot open a file. You may need to delete the ibdata1 file before trying to start up again.
2015-12-28T01:24:38.252569Z 0 [ERROR] Plugin 'InnoDB' init function returned error.
2015-12-28T01:24:38.252688Z 0 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
2015-12-28T01:24:38.253262Z 0 [ERROR] Failed to initialize plugins.
2015-12-28T01:24:38.253551Z 0 [ERROR] Aborting

Mongo

2015-12-28T01:26:54.678+0000 I CONTROL  [initandlisten] MongoDB starting : pid=1 port=27017 dbpath=/data/db 64-bit host=a484d3b2a3e0
2015-12-28T01:26:54.680+0000 I CONTROL  [initandlisten] db version v3.2.0
2015-12-28T01:26:54.682+0000 I CONTROL  [initandlisten] git version: 45d947729a0315accb6d4f15a6b06be6d9c19fe7
2015-12-28T01:26:54.683+0000 I CONTROL  [initandlisten] OpenSSL version: OpenSSL 1.0.1e 11 Feb 2013
2015-12-28T01:26:54.683+0000 I CONTROL  [initandlisten] allocator: tcmalloc
2015-12-28T01:26:54.683+0000 I CONTROL  [initandlisten] modules: none
2015-12-28T01:26:54.683+0000 I CONTROL  [initandlisten] build environment:
2015-12-28T01:26:54.683+0000 I CONTROL  [initandlisten]     distmod: debian71
2015-12-28T01:26:54.683+0000 I CONTROL  [initandlisten]     distarch: x86_64
2015-12-28T01:26:54.683+0000 I CONTROL  [initandlisten]     target_arch: x86_64
2015-12-28T01:26:54.683+0000 I CONTROL  [initandlisten] options: {}
2015-12-28T01:26:54.702+0000 I STORAGE  [initandlisten] exception in initAndListen: 98 Unable to create/open lock file: /data/db/mongod.lock errno:13 Permission denied Is a mongod instance already running?, terminating
2015-12-28T01:26:54.702+0000 I CONTROL  [initandlisten] dbexit:  rc: 100

[sahin]

is there any ETA on this issue?, this is very simple and serious hassle for average dev to use kinematic.

@JeffDM @elesant

[gildor]

Really big issue, you just can't use Mac as host with dir shared with container (really useful for development).

[eturino]

+1, I can't really use this with mariadb without persisting the data...

[FrenchBen]

@eturino I understand the issue at hand, but you should be able to create a data container and simply map mariadb to it - You can copy your data to the container and back it up pretty easily as well.
If you need a new mariadb container, just scale it and allow it to grab the volume from the above data container.

[soosap]

+1

[koushikckm]

+1 Struggling a lot with mysql on docker :( due to this.
Any work around pls?

[agborkowski]

its funny one solution to solve this issue is get back to one vm image and dont waste time for trying change this behavior ;)

[Lenzcrafter]

ya... this is a joke. done wasting hours with docker... 5 minutes with vmware and im good to go...

[FrenchBen]

@Lenzcrafter I'd be interested in knowing how 5 minutes with vmware and im good to go solved your issue - Could you clarify?
Also there's an on-going discussion regarding mounting and its complexity here: #1202

[emad-elsaid]

I'm stuck with Postgres for 2 days, is there any workaround that can work with both development (OSX) and production (Linux ubuntu) machines?

[FrenchBen]

@blazeeboy You can setup a data volume and have Postgres write to it. The same volume can be used on a production machine as well.

[emad-elsaid]

@FrenchBen How can we do that? because as for now the official Postgres image doesn't boot and produces this output

The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.
The database cluster will be initialized with locale "en_US.utf8".
The default database encoding has accordingly been set to "UTF8".
The default text search configuration will be set to "english".
Data page checksums are disabled.
fixing permissions on existing directory /var/lib/postgresql/data ... ok
initdb: could not create directory "/var/lib/postgresql/data/pg_xlog": Permission denied
initdb: removing contents of data directory "/var/lib/postgresql/data"

On OSX 10.11.4, Docker tools installed 24 hours ago, so it's the latest I think.

[FrenchBen]

@blazeeboy Something like this would work:

# Create a volume for all of the Postgres Data
docker run -d --name pgsql-volume -v /var/lib/postgresql/data tianon/true
# Create your postgres container using the volume created above
docker run --name some-postgres -e POSTGRES_PASSWORD=mysecretpassword --volumes-from pgsql-volume -d postgres

If you need to backup/restore/migrate your data, follow the following:
https://docs.docker.com/engine/userguide/containers/dockervolumes/#backup-restore-or-migrate-data-volumes

container logs:

...
The database cluster will be initialized with locale "en_US.utf8".
The default database encoding has accordingly been set to "UTF8".
The default text search configuration will be set to "english".

Data page checksums are disabled.

fixing permissions on existing directory /var/lib/postgresql/data ... ok
creating subdirectories ... ok
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting dynamic shared memory implementation ... posix
creating configuration files ... ok
creating template1 database in /var/lib/postgresql/data/base/1 ... ok
initializing pg_authid ... ok
initializing dependencies ... ok
...

If you remove the postgres container and create a new one, the container logs will be (notice no DB creation) 🎉 :

LOG:  database system was shut down at 2016-04-13 00:50:46 UTC
LOG:  MultiXact member wraparound protections are now enabled
LOG:  database system is ready to accept connections
LOG:  autovacuum launcher started

[thaJeztah]

Instead of a data-only container, you can also use a named volume;

# create a volume for the postgres data
docker volume create --name postgres-data

# and use that volume when starting the container
docker run --name some-postgres -v postgres-data:/var/lib/postgresql/data/ -e POSTGRES_PASSWORD=mysecretpassword -d postgres

[jchilders]

For mongodb, upgrading to the Docker Beta (1.13.0-beta39 (15140)) fixed the issues I was having and others have reported above. Docker Beta for Mac