docker plugin install errors out on Docker Desktop on Mac

[metadaddy]

The associated forum post URL from https://forum.rclone.org

https://forum.rclone.org/t/utilizing-the-rclone-docker-volume-driver-with-backblaze-b2/37870/2?u=metadaddy

What is the problem you are having with rclone?

docker plugin install errors out on Docker Desktop on Mac

What is your rclone version (output from rclone version)

rclone v1.59.2
- os/version: darwin 13.3.1 (64 bit)
- os/kernel: 22.4.0 (arm64)
- os/type: darwin
- os/arch: arm64
- go/version: go1.18.6
- go/linking: dynamic
- go/tags: cmount

Which OS you are using and how many bits (e.g. Windows 7, 64 bit)

macOS Ventura 13.3.1 (22E261) 64 bit

Which cloud storage system are you using? (e.g. Google Drive)

Backblaze B2

The command you were trying to run (e.g. rclone copy /tmp remote:tmp)

% docker plugin install rclone/docker-volume-rclone:arm64 args="-v" --alias rclone --grant-all-permissions

A log from the command with the -vv flag (e.g. output from rclone -vv copy /tmp remote:tmp)

arm64: Pulling from rclone/docker-volume-rclone
Digest: sha256:0e64406b030ad1653cdfe7ab3c449b464c498f9a2c0c284614fba67b9e74276c
0515e2589b33: Complete 
Error response from daemon: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error mounting "/var/lib/docker-plugins/rclone/config" to rootfs at "/data/config": stat /var/lib/docker-plugins/rclone/config: no such file or directory: unknown

Further detail

Docker Desktop version 4.18.0 (104112)
Docker Engine: 20.10.24

The rclone Docker Volume Plugin doc instructs you to create a pair of directories on the host:

% sudo mkdir -p /var/lib/docker-plugins/rclone/config
% sudo mkdir -p /var/lib/docker-plugins/rclone/cache

I did so, and tried to install the plugin, but:

% docker plugin install rclone/docker-volume-rclone:arm64 args="-v" --alias rclone --grant-all-permissions
latest: Pulling from rclone/docker-volume-rclone
Digest: sha256:fd1396468c3b3613081a1b10a2b4173219f9b3d43b2aedceab6830c84d821094
c3ef44c55cda: Complete 
Error response from daemon: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error mounting "/var/lib/docker-plugins/rclone/config" to rootfs at "/data/config": stat /var/lib/docker-plugins/rclone/config: no such file or directory: unknown

The problem is that the Mac is not the Docker host. The Docker host is a Linux instance running in a VM. This Linux instance has its own filesystem that overlaps with the Mac filesystem in some places, but not in others. In particular, /var on the Mac does not map to the same directory as /var on the Docker host.

When Docker installs the plugin, it tries to mount the host's /var/lib/docker-plugins/rclone/config directory as /data/config in the plugin, and fails, since that directory does not exist on the host. There is a workaround, though. On the Mac, /var is a soft link to /private/var, and the Mac's /private directory is accessible from the host, and therefore the plugin.

Following advice here, I uninstalled and reinstalled the plugin, configuring it to use the /private directories for its config and cache:

% docker plugin install rclone/docker-volume-rclone:arm64 \
    --alias rclone --grant-all-permissions args="-v" \
    config="/private/var/lib/docker-plugins/rclone/config" \
    cache="/private/var/lib/docker-plugins/rclone/cache"
arm64: Pulling from rclone/docker-volume-rclone
Digest: sha256:0e64406b030ad1653cdfe7ab3c449b464c498f9a2c0c284614fba67b9e74276c
0515e2589b33: Complete 
Installed plugin rclone/docker-volume-rclone:arm64

This allowed the plugin to install cleanly; the next step was to create a volume:

% docker volume create test -d rclone -o type=b2 -o b2-account=${B2_APPLICATION_KEY_ID} -o b2-key=${B2_APPLICATION_KEY}
Error response from daemon: create test: VolumeDriver.Create: failed to save state: open /data/cache/docker-plugin.state: operation not permitted

Now the issue is that, even though the host has made the config and cache directories available to the plugin, the plugin can't write to them, since the Linux host seems to run as the current Mac user rather than as root.

Note that the volume appears to be there, but you can't mount it:

% docker volume ls
DRIVER          VOLUME NAME
local           0da0dac570c2b2d55981b2924851555a84d929857566eba12424f67c67e10a3e
local           9b025e15c1530e1b977814d30d7d3d1b5b380818f856325deb1edb3a18afb967
local           120a61a4886fdd954d06a2fd4259b335beb120863ce9cda5b9e6627cd5aba5f2
rclone:latest   test
% docker run --rm -it -v test:/mnt --workdir /mnt ubuntu:latest bash
docker: Error response from daemon: VolumeDriver.Mount: failed to save state: open /data/cache/docker-plugin.state: operation not permitted.
See 'docker run --help'.

It turned out that the fix was easy - change the owner of the plugin's directories to the current user:

% sudo chown -R $(whoami):staff /var/lib/docker-plugins/rclone

Alternatively, you could give all users access to those directories.

After removing the non-functional test volume, I was able to create a volume and then access it without any problems:

% docker volume create test -d rclone -o type=b2 -o b2-account=${B2_APPLICATION_KEY_ID} -o b2-key=${B2_APPLICATION_KEY}
test
% docker run --rm -it -v test:/mnt --workdir /mnt ubuntu:latest bash
root@eee9adc6b4ad:/mnt# ls
...all the buckets in my B2 account...
root@eee9adc6b4ad:/mnt# cat metadaddy-public/hello.txt 
Hello from US West
root@eee9adc6b4ad:/mnt# echo 'Hello from docker' > metadaddy-public/docker.txt

Check it out: https://f004.backblazeb2.com/file/metadaddy-public/docker.txt

Suggested Resolution

Add specific documentation for Docker Desktop for Mac:

Explanation that Docker Desktop for Mac runs a Linux VM as the Docker host. The Docker host has access to portions of the Mac filesystem as the current user.

After creating the two directories, you must either change their owner to the current user:

% sudo chown -R $(whoami):staff /var/lib/docker-plugins/rclone

or, if you prefer, make then accessible to all users:

% sudo chmod -R a+rwx /var/lib/docker-plugins/rclone

When you start the plugin, you must set the config and cache settings to point to the correct locations in the Docker host:

docker plugin install rclone/docker-volume-rclone:arm64 \
    args="-v" --alias rclone --grant-all-permissions \
    config="/private/var/lib/docker-plugins/rclone/config" \
    cache="/private/var/lib/docker-plugins/rclone/cache"

Replace arm64 with amd64 for Intel-based Macs.

How to use GitHub

• Please use the 👍 reaction to show that you are affected by the same issue.
• Please don't comment if you have no relevant information to add. It's just extra noise for everyone subscribed to this issue.
• Subscribe to receive notifications on status change and new comments.

[ncw]

Excellent investigation - thank you :-)

Do you want to add this to the docs?

The source for the docs which need editing are here https://github.com/rclone/rclone/blob/master/docs/content/docker.md - a new section on installing on mac sounds like a great idea. You can edit the docs online (click the pencil icon on the page) or send a pull request in the normal way.

Any help gratefully received (I don't have a mac!).

[metadaddy]

Sure - I'll put it on my list 👍🏻