Cannot Start Containers: file permission denied

[metanull-operator]

I cannot start either the beacon-chain or validator using docker. After trying to get latest earlier today, I got the following errors:

[2019-12-19 19:09:06] ERROR main: mkdir /data/beaconchaindata: permission denied

and

[2019-12-19 20:22:19] ERROR main: could not get private key: open /data/validatorprivatekey82fc5770988d: permission denied

I started my beacon-chain with the following command:

docker run --user $(id -u):$(id -g) -d --name beacon-chain -t -v $HOME/prysm-data:/data -p 4000:4000 -p 13000:13000 -p 12000:12000 gcr.io/prysmaticlabs/prysm/beacon-chain:latest --datadir=/data --p2p-host-ip XXX.XXX.XXX.XXX --snappy

and my validator with:

docker run --user $(id -u):$(id -g) --name validator -dt -v $HOME/prysm-docker:/data --network="host" gcr.io/prysmaticlabs/prysm/validator:latest --graffiti="metanull (Prysm)" --beacon-rpc-provider=127.0.0.1:4000 --keystore-path=/data --password=PASSWORD

I deleted my ~/prysm-data directory prior to starting the beacon-chain so that I would not have an existing folder with existing permissions. The mkdir command fails with permission denied.

I believe I even received this error when run as root, which I thought was odd.

validator version Prysm/Git commit: da63766
beacon-chain version Prysm/Git commit: 20168ad

@prestonvanloon confirmed this independently, I believe.

Let me know what additional information you need.

[prestonvanloon]

This is certainly caused by #4320 where we changed the default docker container user to be uid 1001. This user likely does not have the permissions to write/read in those directories. I recommended setting the docker --user flag to be your user ID and group ID which is what $(id -u):($id -g) is doing, but this seems not to work for some users.

Read more about this feature here: https://docs.docker.com/engine/reference/run/#user

I do not have a solid fix at this time, but once we find a resolution then we need to update the documentation for running Prysm. We should not have been using the default docker user (root) in the first place so this temporary pain is acceptable trade off for the potential security increase gain from running as a non-privileged user.

[metanull-operator]

This may not be a problem with the code. I tried an older build (8701c835e2e6990f6362fc705b8c147ff78e56e629732e134bfefef951aa8e90), from prior to the root user change, and I still get the same error.

[torfbolt]

Had the same problem today, temporary workaround for me was to do a chmod 0777 on the data dir.

[metanull-operator]

Thank you. That workaround got me back up and running. My own troubleshooting has not resolved this issue yet, but chmod 777 at least stopped the bleeding to penalties. I am tempted to try a clean install of Ubuntu to try again, because I have probably broken my docker installation in other ways in an attempt to fix this/get up and running again.

[ethgasstation]

I was unable to create the validator account due to this permission denied problem. The workaround for me was to use 'docker run --user 0:0 ...' - which runs the container as root

[ethgasstation]

FYI- I think you will need to revert #4320 - docker daemon should always run as root. The only way to get the current version to work on Ubuntu is to use the flag '--user 0:0' which essentially reverts #4320

[prestonvanloon]

yeah, let's just revert #4320. It has been causing a lot of issues.