Permission denied error when mounting /export/ directory to local disk

[lparsons]

NOTE: I am using boot2docker v1.6.1 on OSX 10.9.5.

Command: docker run -p 8080:80 -v /Users/lparsons/galaxy_storage/:/export/ bgruening/galaxy-stable

Result: /Users/lparsons/galaxy_storage/ is created on my OSX machine and populated with galaxy_central, postgresql, shed_tools and var directories. However, I then run into the following error:

==> /home/galaxy/uwsgi.log <==
Traceback (most recent call last):
  File "lib/galaxy/webapps/galaxy/buildapp.py", line 51, in app_factory
    app = galaxy.app.UniverseApplication( global_conf=global_conf, **kwargs )
  File "lib/galaxy/app.py", line 38, in __init__
    self.config.check()
  File "lib/galaxy/config.py", line 618, in check
    self._ensure_directory( path )
  File "lib/galaxy/config.py", line 599, in _ensure_directory
    raise ConfigurationError( "Unable to create missing directory: %s\n%s" % ( path, e ) )
ConfigurationError: Unable to create missing directory: /export/galaxy-central/database/files
[Errno 13] Permission denied: '/export/galaxy-central/database'

Various other logs report errors due to the directory /export/galaxy-central/database/files not existing.

I've attempted to manually create this directory, but then I get similar errors about other directories and I wasn't ever able to get things to startup.

[dpryan79]

Make sure that things under /export are owned by the galaxy user (uid 1450 I think). uwsgi and most other things are running as that user, so that should allow it to do whatever it needs to.

[lparsons]

The directory is created entirely by the the docker container, it doesn't exist at all prior to executing docker run and many directories do get created.

[bgruening]

@lparsons can you provide me with more informations? Docker version? kinematic? Galaxy Docker version? I haven't seen this before.
Also can you look at the permissions for /export/galaxy-central/database/?

Thanks and sorry for the trouble!
Bjoern

[lparsons]

I ran into this error yesterday with boot2docker/docker version 1.6.1. Today, there was an update to boot2docker/docker version 1.7 and I'm running into the same problem:

$ boot2docker version
Boot2Docker-cli version: v1.7.0
Git commit: 7d89508

$ docker version
Client version: 1.7.0
Client API version: 1.19
Go version (client): go1.4.2
Git commit (client): 0baf609
OS/Arch (client): darwin/amd64
Server version: 1.7.0
Server API version: 1.19
Go version (server): go1.4.2
Git commit (server): 0baf609
OS/Arch (server): linux/amd64

I am not using Kinematic. I don't know how to determine the Galaxy Docker version, but it was freshly downloaded yesterday and again today.

[lparsons]

Permissions

Docker Container

$ docker exec grave_fermat ls -la /export
total 4
drwxr-xr-x   1 1000 staff  238 Jun 30 15:02 .
drwxr-xr-x 102 root root  4096 Jun 30 15:02 ..
drwxr-xr-x   1 1000 staff 1020 Jun 17 22:55 .distribution_config
drwxr-xr-x   1 1000 staff  204 Jun 29 17:23 galaxy-central
drwxr-xr-x   1 1000 staff  102 Jun 29 17:23 postgresql
drwxr-xr-x   1 1000 staff   68 Jun 17 22:56 shed_tools
drwxr-xr-x   1 1000 staff  102 Jun 29 17:23 var

$ docker exec grave_fermat ls -la /export/galaxy-central
total 0
drwxr-xr-x 1 1000 staff  204 Jun 29 17:23 .
drwxr-xr-x 1 1000 staff  238 Jun 30 15:02 ..
drwxr-xr-x 1 1000 staff 1088 Jun 29 17:23 config
drwxr-xr-x 1 1000 staff  272 Jun 17 05:54 display_applications
drwxr-xr-x 1 1000 staff 1904 Jun 17 22:55 tool-data
drwxr-xr-x 1 1000 staff   68 Jun 17 22:56 tool_deps

Boot2Docker VM

docker@boot2docker:/Users/lparsons/Documents/projects/galaxy_tools/galaxy_storage$ ls -la
total 0
drwxr-xr-x    1 docker   staff          238 Jun 30 15:02 ./
drwxr-xr-x    1 docker   staff          272 Jun 29 17:23 ../
drwxr-xr-x    1 docker   staff         1020 Jun 17 22:55 .distribution_config/
drwxr-xr-x    1 docker   staff          204 Jun 29 17:23 galaxy-central/
drwxr-xr-x    1 docker   staff          102 Jun 29 17:23 postgresql/
drwxr-xr-x    1 docker   staff           68 Jun 17 22:56 shed_tools/
drwxr-xr-x    1 docker   staff          102 Jun 29 17:23 var/

OSX Host

$ ls -la
total 0
drwxr-xr-x  7 lparsons staff  238 Jun 30 11:02 ./
drwxr-xr-x  8 lparsons staff  272 Jun 29 13:23 ../
drwxr-xr-x 30 lparsons staff 1020 Jun 17 18:55 .distribution_config/
drwxr-xr-x  6 lparsons staff  204 Jun 29 13:23 galaxy-central/
drwxr-xr-x  3 lparsons staff  102 Jun 29 13:23 postgresql/
drwxr-xr-x  2 lparsons staff   68 Jun 17 18:56 shed_tools/
drwxr-xr-x  3 lparsons staff  102 Jun 29 13:23 var/

[chambm]

I'm thinking this is the same issue I had with sharing /export between my VirtualBox VM (either Kitematic Boot2Docker or Vagrant Ubuntu) and my Windows host. Virtual machine shared folders commonly have problems keeping uid/gid straight between the host and guest. As you see in your B2D VM's ls, the shared folder is mapped with docker:staff (1000:50). Your Docker container has the staff group but not the docker user. But that doesn't really matter, because those things should be owned by the galaxy user (1450 by default) and postgres (default varies). I just submitted a patch to Bjorn that allows you to configure what the postgres user's UID/GID will be in the docker container (default 1550). Then, in the VM, you make sure to map /export/postgresql with 1550:1550, and /export/galaxy-central and /export/shed_tools with 1450:1450. /export/var should be mounted with root:root. In Kitematic on Windows, I couldn't tell it what UID/GID to use for the mounts, so I ended up having to use Vagrant Ubuntu and now it's working.

[bgruening]

xref: #71

[bgruening]

@lparsons can you check if #71 is of any help for you?

[ksuderman]

@bgruening I've been having the same problem and unfortunately #71 doesn't solve it for me.

I believe the root cause is Docker and boot2docker/boot2docker#581 . It seems that it is not possible to change the owner of the /export directories from within the VM (or a container running in the VM) and so postgres refuses to start. If I ssh into the container (see Problem #3) I can see the directories properly mounted under /export, but they are all owned by docker:staff and any attempt to sudo chown them results in a "permission denied" error.

Another (almost) related problem is that the /export/galaxy-central/database/* directory hierarchy is not being created which causes other odd "permission denied" errors when Galaxy tries to start. The relevant stack trace is:

Traceback (most recent call last):
  File "lib/galaxy/webapps/galaxy/buildapp.py", line 64, in paste_app_factory
    app = galaxy.app.UniverseApplication( global_conf=global_conf, **kwargs )
  File "lib/galaxy/app.py", line 38, in __init__
    self.config.check()
  File "lib/galaxy/config.py", line 618, in check
    self._ensure_directory( path )
  File "lib/galaxy/config.py", line 599, in _ensure_directory
    raise ConfigurationError( "Unable to create missing directory: %s\n%s" % ( path, e ) )
ConfigurationError: Unable to create missing directory: /export/galaxy-central/database/files
[Errno 13] Permission denied: '/export/galaxy-central/database'

Everything else under /export is being populated correctly so I don't know what is special about /export/galaxy-central/database. If I create the directories before running Docker at least I get to the "Galaxy won't start because Postgres isn't running" problem. Otherwise Galaxy won't start due to the missing /export/galaxy-central/database/* directories.

[bgruening]

@ksuderman which UID/GID does your postgres user have? Are you running :dev or :15.07? If not can you try one of these new versions?

[ksuderman]

Sorry @bgruening, I should have mentioned that I have been basing my images on galaxy-stable:dev. I just tried 15.07 image and the problems are still there. The postgres user is 1550:1550 and the galaxy user is 1450:1450 (see below).

However, I have made progress.

To "fix" (hack around) the "permission denied" problems I cloned the docker-galaxy-stable repo and changed the GALAXY_UID and GALAXY_GID to 1000 (the docker UID in VirtualBox). This solves the problem of Galaxy not being able to create files in /export/galaxy-central/database/* directories.

I had assumed Postgres wasn't starting due to permissions problems in the /export/postgresql directory, and that is sort of correct. I can start postgres with service postgresql start but it seems supervisord doesn't like the permissions on the /export directory. If I try to use the same command supvisord uses (from /etc/supervisor/conf.d/galaxy.conf) I get the following: (note postmaster refuses to run as the root user):

> sudo -u postgres /usr/lib/postgresql/9.3/bin/postmaster -D "/export/postgresql/9.3/main"
LOG:  could not open configuration file "/export/postgresql/9.3/main/postgresql.conf": Permission denied
FATAL:  configuration file "/export/postgresql/9.3/main/postgresql.conf" contains errors

If I try as the galaxy user (which has now has r/w access to /export) I get:

> sudo -u galaxy /usr/lib/postgresql/9.3/bin/postmaster -D "/export/postgresql/9.3/main"
FATAL:  data directory "/export/postgresql/9.3/main" has group or world access
DETAIL:  Permissions should be u=rwx (0700).

Due to the VirtualBox/Docker/OS X bugs it is not possible to change the permissions/owner of any files/directories in the shared volume. So my workaround is to simply add service postgresql start to the top of the /usr/bin/startup script.

With the above two changes I can now use shared volumes, however I don't think these "fixes" should be merged into docker-galaxy-stable. Another suggested work around is to remount /Users with NFS in the VBox VM, but I haven't explored that yet.

[bgruening]

@ksuderman this is a really great analysis. Thanks a bunch for sharing. I suppose these bugs are known to the Docker guys? I hope they fix it soon. Let me know if I can support you.

What we can try is to figure out if the container is running under VBox/OS-X and squeeze in your workaround - only in this case? What do you think?

@lparsons any opinion here?

Sorry that I can not help much as I'm missing the a proper test environment.

[chambm]

Keith, in my case, on Windows, I originally tried to share everything with
the host. First I gave up on having the Postgres share persist in Windows
(due to the same issue you ran into with inability to change permissions)
and opted to use a semi-permanent Linux VM to handle that persistence. Then
I eventually gave up on having the galaxy-central directories shared with
the host as well because of symbolic link shenanigans. You might consider
using a Vagrant VM like me to set up a semi-permanent Linux VM for your
Docker environment (it takes the place of boot2docker).

NFS would fix the postgres problems for you most likely on OSX, but Winnfsd
would not fix the issue because everything maps to a single global UID/GID.

On Sun, Sep 6, 2015 at 6:41 PM, Björn Grüning notifications@github.com
wrote:

@ksuderman https://github.com/ksuderman this is a really great
analysis. Thanks a bunch for sharing. I suppose these bugs are known to the
Docker guys? I hope they fix it soon. Let me know if I can support you.

What we can try is to figure out if the container is running under
VBox/OS-X and squeeze in your workaround - only in this case? What do you
think?

@lparsons https://github.com/lparsons any opinion here?

Sorry that I can not help much as I'm missing the a proper test
environment.

—
Reply to this email directly or view it on GitHub
#68 (comment)
.

[ksuderman]

@bgruening Yes, this turns out to be a known problem in Docker (boot2docker actually) but it seems the root cause is the way VirtualBox does shared folders so I don't know how quickly a fix will be coming.

If anyone wants to test my hacks I've pushed an image to the Docker hub:

FROM ksuderman/galaxy-stable:osx
...

@chambm I will take look at Vagrant. Our use case right now is to develop something that can be used for course work and demos so I want to keep things as simple as possible for our users. Vagrant looks promising if it solves the permission problems.

[lparsons]

@bgruening I don't see an elegant solution (at least not near term). The idea of using the workaround in the VBox/OS-X case seems like a reasonable and practical solution if it works.

@chambm The Vagrant idea does indeed seem promising. My main goal is to be able to quickly setup clean Galaxy environments where I can test things like tool integrations and installations. Semi-permanent would be fine in those cases. Am I understanding correctly that you are creating a Vagrantfile that creates a Linux VM, then vagrant ssh in and then running the docker image with shared folder there?

[chambm]

@lparsons Check out https://github.com/chambm/bingomics-galaxy/tree/master/vagrant - it's the Vagrantfile I use for developing and testing my lab's docker-galaxy flavor. It has a lot of stuff needed for a cluster-based configuration commented out since that's a bit resource intensive. But that is a possibility. There are some shell provisioning commands to prep a fresh VM and then it runs the docker image on vagrant up. I log in with Putty rather than vagrant ssh, but if your vagrant ssh will work, that's even easier. My Bingomics docker image installs planemo, so it really makes tool development smooth (that being a relative term).

[ksuderman]

Another work around, particularly if you just want some persistence during testing and development is to mount a directory from the VM. I.E.

> docker run -v /var/lib/galaxy:/export -d -p 8000:80 bgruening/galaxy-stable:dev

Then use docker-machine scp to moves file in/out of the VM directory. As long as you don't delete the VM the /var/lib/galaxy directory will be persisted and available for multiple containers.

[bgruening]

@ksuderman @lparsons is there anything I can do here?
Can this be closed?

[ksuderman]

I think you can close this. There is not much you can do other than document the problem and possible work arounds.

I've simply been mounting volumes I need to write to in the VirtualBox VM and then using docker-machine scp or docker-machine ssh to configure things in the VM if needed. However, I discovered that Docker mounts '/' as a 3Gb partition so volumes should really by mounted under /mnt/sda1/ in the VM, which is where the remaining virtual disk space is mounted; e.g. /var/lib/docker is really a link to /mnt/sda1/var/lib/docker

[bgruening]

@lparsons @ksuderman we have added a small section about data containers to the Docker readme file: https://github.com/bgruening/docker-galaxy-stable#usage

This might be a good solution for this problem.