New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

S3FS + RexRay + Third party S3 integration possible? #3083

Closed
piscis opened this Issue Nov 23, 2017 · 7 comments

Comments

Projects
None yet
3 participants
@piscis

piscis commented Nov 23, 2017

Hi I have a quick question, I played around with

https://wasabi.com/

A 100% compatible, as they claim, S3 storage service. I noticed that rexray recently exposed the S3FS_ENDPOINT setting for s3fs via environment variable.

When I create a docker volume with kontena (config below) the buckets get created but the deployment fails because of a error the rexray/s3fs throws. Deploying a container manually and mounting it works.

Do you support wasabi as storage backend or is there some example how to greate a s3fs volumn driver with kontena?

If I do the same with a AWS integration everything works.

Docker plugin install

docker plugin install rexray/s3fs S3FS_ENDPOINT='https://s3.wasabisys.com' S3FS_MAXRETRIES=20 S3FS_ACCESSKEY=XXXX S3FS_SECRETKEY=YYYY  REXRAY_LOGLEVEL=debug

Create driver

kontena volume  create --driver rexray/s3fs --scope instance testvolume

Create driver

kontena volume  create --driver rexray/s3fs --scope instance testvolume
@jnummelin

This comment has been minimized.

Show comment
Hide comment
@jnummelin

jnummelin Nov 24, 2017

Member

If wasabi.com is really 100% S3 compatible, I don't see any reason why it shouldn't work. Kontena "just" instructs Docker to create and mount the volumes, really the same operation you do manually with docker volume create + docker run -v my-s3-vol:/data ...

When I create a docker volume with kontena (config below) the buckets get created but the deployment fails because of a error the rexray/s3fs throws.

What kind or error you see?

Member

jnummelin commented Nov 24, 2017

If wasabi.com is really 100% S3 compatible, I don't see any reason why it shouldn't work. Kontena "just" instructs Docker to create and mount the volumes, really the same operation you do manually with docker volume create + docker run -v my-s3-vol:/data ...

When I create a docker volume with kontena (config below) the buckets get created but the deployment fails because of a error the rexray/s3fs throws.

What kind or error you see?

@jnummelin

This comment has been minimized.

Show comment
Hide comment
@jnummelin

jnummelin Nov 24, 2017

Member

I was able to get volumes running to the point where the bucket is created on wasabi side.

BUT

There seems to be something wrong with rexray, it is able to create the bucket, but when mounting it on a container it actually gets mounted as local storage only. See: rexray/rexray#908
I just included some debug info on the issue, let's see if rexray folks are able to pin down the issue.

One other thing I encountered was this:

core@wild-shadow-74 ~ $ docker run -ti -v wasabi-rexray-test:/data alpine sh
/ # ls -lah /data/
total 16
drwx------    2 root     root        4.0K Nov 24 08:38 .
drwxr-xr-x    1 root     root        4.0K Nov 24 08:38 ..
/ # df -h
Filesystem                Size      Used Available Use% Mounted on
overlay                 133.1G      1.1G    126.4G   1% /
overlay                 133.1G      1.1G    126.4G   1% /
tmpfs                    64.0M         0     64.0M   0% /dev
tmpfs                     3.9G         0      3.9G   0% /sys/fs/cgroup
/dev/sda9               133.1G      1.1G    126.4G   1% /data
/dev/sda9               133.1G      1.1G    126.4G   1% /etc/resolv.conf
/dev/sda9               133.1G      1.1G    126.4G   1% /etc/hostname
/dev/sda9               133.1G      1.1G    126.4G   1% /etc/hosts
shm                      64.0M         0     64.0M   0% /dev/shm
tmpfs                    64.0M         0     64.0M   0% /proc/kcore
tmpfs                    64.0M         0     64.0M   0% /proc/latency_stats
tmpfs                    64.0M         0     64.0M   0% /proc/timer_list
tmpfs                    64.0M         0     64.0M   0% /proc/sched_debug
tmpfs                     3.9G         0      3.9G   0% /sys/firmware
/ # exit
$ docker run -ti -v wasabi-rexray-test:/data alpine sh
/run/torcx/bin/docker: Error response from daemon: error while mounting volume '/var/lib/docker/plugins/3ddde09ebb0b23c0011ceb685cd3699b18ba019c7954493cd1a58b88d10081b3/rootfs': VolumeDriver.Mount: docker-legacy: Mount: wasabi-rexray-test: failed: error mounting s3fs bucket.

So I'm only able to mount the volume once. Mounting it second time always fails. :/

Member

jnummelin commented Nov 24, 2017

I was able to get volumes running to the point where the bucket is created on wasabi side.

BUT

There seems to be something wrong with rexray, it is able to create the bucket, but when mounting it on a container it actually gets mounted as local storage only. See: rexray/rexray#908
I just included some debug info on the issue, let's see if rexray folks are able to pin down the issue.

One other thing I encountered was this:

core@wild-shadow-74 ~ $ docker run -ti -v wasabi-rexray-test:/data alpine sh
/ # ls -lah /data/
total 16
drwx------    2 root     root        4.0K Nov 24 08:38 .
drwxr-xr-x    1 root     root        4.0K Nov 24 08:38 ..
/ # df -h
Filesystem                Size      Used Available Use% Mounted on
overlay                 133.1G      1.1G    126.4G   1% /
overlay                 133.1G      1.1G    126.4G   1% /
tmpfs                    64.0M         0     64.0M   0% /dev
tmpfs                     3.9G         0      3.9G   0% /sys/fs/cgroup
/dev/sda9               133.1G      1.1G    126.4G   1% /data
/dev/sda9               133.1G      1.1G    126.4G   1% /etc/resolv.conf
/dev/sda9               133.1G      1.1G    126.4G   1% /etc/hostname
/dev/sda9               133.1G      1.1G    126.4G   1% /etc/hosts
shm                      64.0M         0     64.0M   0% /dev/shm
tmpfs                    64.0M         0     64.0M   0% /proc/kcore
tmpfs                    64.0M         0     64.0M   0% /proc/latency_stats
tmpfs                    64.0M         0     64.0M   0% /proc/timer_list
tmpfs                    64.0M         0     64.0M   0% /proc/sched_debug
tmpfs                     3.9G         0      3.9G   0% /sys/firmware
/ # exit
$ docker run -ti -v wasabi-rexray-test:/data alpine sh
/run/torcx/bin/docker: Error response from daemon: error while mounting volume '/var/lib/docker/plugins/3ddde09ebb0b23c0011ceb685cd3699b18ba019c7954493cd1a58b88d10081b3/rootfs': VolumeDriver.Mount: docker-legacy: Mount: wasabi-rexray-test: failed: error mounting s3fs bucket.

So I'm only able to mount the volume once. Mounting it second time always fails. :/

@piscis

This comment has been minimized.

Show comment
Hide comment
@piscis

piscis Nov 30, 2017

I'm getting this error:

Deployment of service vicoli/wordpress/redis failed:
  - halting deploy of vicoli/wordpress/redis, one or more instances failed
  - GridServiceInstanceDeployer::ServiceError: Docker::Error::ServerError: VolumeDriver.Mount: docker-legacy: Mount: wordpress.redis.testvolume13-1: failed: error mounting s3fs bucket (on node company)

I tried to deploy 3 instances like so:

  redis:
     image: redis
     command: touch /data/appendonly.aof && redis-server --appendonly yes
     instances: 3
     volumes:
       - redis-data:/data

The first bucket gets created then the volume driver fails and you see the above error.

I also tried to install the docker plugin like mentioned here:

rexray/rexray#849 (comment)

docker plugin install rexray/s3fs:0.11.0 S3FS_OPTIONS="-o url=https://s3.wasabisys.com" S3FS_ENDPOINT='https://s3.wasabisys.com' S3FS_ACCESSKEY=XXX S3FS_SECRETKEY=YYY

and creating the volumn via kontena cli:

kontena volume  create --driver rexray/s3fs --scope instance testvolume13

But no luck

piscis commented Nov 30, 2017

I'm getting this error:

Deployment of service vicoli/wordpress/redis failed:
  - halting deploy of vicoli/wordpress/redis, one or more instances failed
  - GridServiceInstanceDeployer::ServiceError: Docker::Error::ServerError: VolumeDriver.Mount: docker-legacy: Mount: wordpress.redis.testvolume13-1: failed: error mounting s3fs bucket (on node company)

I tried to deploy 3 instances like so:

  redis:
     image: redis
     command: touch /data/appendonly.aof && redis-server --appendonly yes
     instances: 3
     volumes:
       - redis-data:/data

The first bucket gets created then the volume driver fails and you see the above error.

I also tried to install the docker plugin like mentioned here:

rexray/rexray#849 (comment)

docker plugin install rexray/s3fs:0.11.0 S3FS_OPTIONS="-o url=https://s3.wasabisys.com" S3FS_ENDPOINT='https://s3.wasabisys.com' S3FS_ACCESSKEY=XXX S3FS_SECRETKEY=YYY

and creating the volumn via kontena cli:

kontena volume  create --driver rexray/s3fs --scope instance testvolume13

But no luck

@doncicuto

This comment has been minimized.

Show comment
Hide comment
@doncicuto

doncicuto Nov 30, 2017

Hi @piscis and @jnummelin,
I copy here my answer for rexray#849 issue maybe it can solve your issue in Kontena.

If you want to install the plugin using a 3rd party like minio or wasabi, you should use S3FS_OPTIONS and specidy the URL that you use in the S3FS_ENDPOINT as @clintkitson mentioned, but there's a typo it should be S3FS_OPTIONS="url=http://..." removing the -o. For example:

docker plugin install rexray/s3fs S3FS_OPTIONS="allow_other,use_path_request_style,nonempty,url=http://127.0.0.1:9000" S3FS_ENDPOINT="http://127.0.0.1:9000" S3FS_ACCESSKEY="xxxxxxxxxxxxxxxx" S3FS_SECRETKEY="xxxxxxxxxxxxxxxxxx" --alias s3fs --grant-all-permissions

Hope that helps!

doncicuto commented Nov 30, 2017

Hi @piscis and @jnummelin,
I copy here my answer for rexray#849 issue maybe it can solve your issue in Kontena.

If you want to install the plugin using a 3rd party like minio or wasabi, you should use S3FS_OPTIONS and specidy the URL that you use in the S3FS_ENDPOINT as @clintkitson mentioned, but there's a typo it should be S3FS_OPTIONS="url=http://..." removing the -o. For example:

docker plugin install rexray/s3fs S3FS_OPTIONS="allow_other,use_path_request_style,nonempty,url=http://127.0.0.1:9000" S3FS_ENDPOINT="http://127.0.0.1:9000" S3FS_ACCESSKEY="xxxxxxxxxxxxxxxx" S3FS_SECRETKEY="xxxxxxxxxxxxxxxxxx" --alias s3fs --grant-all-permissions

Hope that helps!

@jnummelin

This comment has been minimized.

Show comment
Hide comment
@jnummelin

jnummelin Dec 15, 2017

Member

@piscis did you test with the S3FS_OPTIONS pointing to Wasabi?

I was able to use volumes with Wasabi, installed the plugin with:

docker plugin install rexray/s3fs S3FS_ACCESSKEY=xxxxxx S3FS_SECRETKEY=xyzxyzxyzxyz S3FS_ENDPOINT=https://s3.wasabisys.com S3FS_OPTIONS=url=https://s3.wasabisys.com
Member

jnummelin commented Dec 15, 2017

@piscis did you test with the S3FS_OPTIONS pointing to Wasabi?

I was able to use volumes with Wasabi, installed the plugin with:

docker plugin install rexray/s3fs S3FS_ACCESSKEY=xxxxxx S3FS_SECRETKEY=xyzxyzxyzxyz S3FS_ENDPOINT=https://s3.wasabisys.com S3FS_OPTIONS=url=https://s3.wasabisys.com
@piscis

This comment has been minimized.

Show comment
Hide comment
@piscis

piscis Dec 15, 2017

@jnummelin yes it works for me thank you

piscis commented Dec 15, 2017

@jnummelin yes it works for me thank you

@jnummelin

This comment has been minimized.

Show comment
Hide comment
@jnummelin

jnummelin Dec 15, 2017

Member

Closing, revealed to be docs/usability thing in rexray side

Member

jnummelin commented Dec 15, 2017

Closing, revealed to be docs/usability thing in rexray side

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment