New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

FUSE volumes #7890

Open
rsokolowski opened this Issue May 7, 2015 · 53 comments

Comments

Projects
None yet
@rsokolowski
Contributor

rsokolowski commented May 7, 2015

I have an use-case where I would like to mount Google Cloud Storage (GCS) bucket (and a directory in that bucket) in my container and use it as a regular FS. Currently, it seems doable using s3fs-fuse (https://github.com/s3fs-fuse/s3fs-fuse) - thanks @brendanburns. It would be great if GCS was supported as a first class Volume in Kubernetes.

@zmerlynn

This comment has been minimized.

Member

zmerlynn commented Oct 19, 2015

This is also possible now through the https://github.com/GoogleCloudPlatform/gcsfuse project, it looks like.

@rboyd

This comment has been minimized.

rboyd commented Oct 19, 2015

@zmerlynn were you able to get gcsfuse to work? I'm expecting this won't work until 1.1 release http://stackoverflow.com/questions/31124368/allow-privileged-containers-in-kubernetes-on-google-container-gke

@zmerlynn

This comment has been minimized.

Member

zmerlynn commented Oct 19, 2015

I haven't tried yet. I was merely noting that looked like it may be an option over s3fs-fuse for GCS, which would require the same privileges, presumably.

@thockin

This comment has been minimized.

Member

thockin commented Oct 20, 2015

We'll want to decide how we want to handle FUSE mounts in general - there
are potentially a LOT of neat things we can do, but FUSE is (historically)
known to be less than 100% reliable. The simplest is that we push it all
into the user's space and require privileges. Perhaps there are more
interesting ways to manage it?

On Mon, Oct 19, 2015 at 4:27 PM, Zach Loafman notifications@github.com
wrote:

I haven't tried yet. I was merely noting that looked like it may be an
option over s3fs-fuse for GCS, which would require the same privileges,
presumably.


Reply to this email directly or view it on GitHub
#7890 (comment)
.

@nickschuch

This comment has been minimized.

nickschuch commented Dec 14, 2015

Im about halfway through coding a volume for https://github.com/s3fs-fuse/s3fs-fuse, based it off the nfs implementation while also drawing some inspiration from #17221. Is this something people can see as a viable solution?

@thockin

This comment has been minimized.

Member

thockin commented Dec 14, 2015

I'd like to see us shake out a solid design for FUSE-based volumes BEFORE
we argue about the merit of any one volume.

On Sun, Dec 13, 2015 at 5:01 PM, Nick Schuch notifications@github.com
wrote:

Im about halfway through coding a volume for
https://github.com/s3fs-fuse/s3fs-fuse, based it off the nfs
implementation while also drawing some inspiration from #17221
#17221. Is this something
people can see as a viable solution?


Reply to this email directly or view it on GitHub
#7890 (comment)
.

@nickschuch

This comment has been minimized.

nickschuch commented Dec 14, 2015

No argurments from me :) How can I help?

@thockin

This comment has been minimized.

Member

thockin commented Dec 14, 2015

ideate :)

On Mon, Dec 14, 2015 at 1:45 PM, Nick Schuch notifications@github.com
wrote:

No argurments from me :) How can I help?


Reply to this email directly or view it on GitHub
#7890 (comment)
.

@nickschuch

This comment has been minimized.

nickschuch commented Dec 14, 2015

My main goal was to defer the fuse implementations to packages on the host and then mount them just like the other volumes eg. NFS.

Maybe we could make a higher level volume which took properties similar to an fstab/mount? That way users are free to use there own mount implementations and we are just using those. That would cut down on duplication of writing multiple volumes with the same scaffolding, as well as support gcsfuse, s3fs-fuse, azure files etc.... Essentially, if you can mount it, we can run it.

@nickschuch

This comment has been minimized.

nickschuch commented Dec 17, 2015

Hmm, scratch that, that was a very raw thought, I see now we pretty much have that via the "mount" package and volumes provide a higher level.

Currently updated my goal to creating a "fuse" volume, going to write some code and see what other thoughts come from there. That will allow us to also mount the other fuse filesystems.

@pnovotnak

This comment has been minimized.

Contributor

pnovotnak commented Jan 22, 2016

I just wanted to chime in and say that this would be a huge boon when running on GCE. Right now I'm looking into storage options for the company I work for... There are a number of inferior options but this would be by far the best for our case.

@thockin

This comment has been minimized.

Member

thockin commented Jan 22, 2016

@kubernetes/sig-storage We discussed FUSE a bit recently and worked out a basic model for it, but it is somewhat complicated to do correctly.

Some notes:

We need a FUSE daemon per volume (maybe we can find a way to flatten to per-pod, but not sure that is ALWAYS ok)

FUSE daemons need privileges

This FUSE daemon must run in the pod’s cgroups and net namespace (chargeback), but must NOT run in the pod’s IPC or PID namespace (for security)

It must be killed when the pod terminates.

We need a way to report this container:

  • add it to pod.spec.containers?
  • pod.status.adminContainers?
    • bad - what image was run?  can we recreate this if it were lost (the status litmus test)

A less-perfect alternative might be to run GCS FUSE on every Google VM and treat it as a special form of hostPath. I don't really want to special case it, though, so it's somewhat less attractive.

@bgrant0607 bgrant0607 changed the title from Mounting Google Cloud Storage into a container to FUSE volumes May 18, 2016

@bgrant0607

This comment has been minimized.

Member

bgrant0607 commented May 18, 2016

Related: #831

@bgrant0607

This comment has been minimized.

Member

bgrant0607 commented May 18, 2016

FUSE is mainly needed for writes? Otherwise, it seems simpler to just fetch a tar.gz and unpack it into an emptyDir.

@jefflaplante

This comment has been minimized.

jefflaplante commented Oct 26, 2016

Has anyone worked on this further? I would like to be able mount FUSE volumes like other PersistentVolumes in kubernetes as an alternative to NFS or glusterfs for multi-container Read-write.

@Stono

This comment has been minimized.

Stono commented Mar 1, 2017

For those still playing with this I ended up using the preStop and postStart lifecycle hooks and running the fuse command, which results in a very similar behaviour.

https://karlstoney.com/2017/03/01/fuse-mount-in-kubernetes/

It'd be awesome however to have the ability to do fuse mounts as PersistentVolumes in Kubernetes.

@baracoder

This comment has been minimized.

baracoder commented Mar 24, 2017

I have a similar issue. I would like to use ceph-fuse because of requent problems with the kernel driver.

Using life cycle hooks is a valid workaround but the issue is still valid because the docker images need to be modified this way. Is it possible to use privileged init containers for this somehow?

@sunshinekitty

This comment has been minimized.

sunshinekitty commented Apr 4, 2017

+1

@nickschuch

This comment has been minimized.

nickschuch commented Apr 4, 2017

We managed to get this going at a flexvolume, Ill look at what it will take for us to publish to code (bit rough, but demonstrates the point).

@matesitox

This comment has been minimized.

matesitox commented Apr 18, 2017

+1

1 similar comment
@davidberardozzi

This comment has been minimized.

davidberardozzi commented Apr 21, 2017

+1

@ivanjaros

This comment has been minimized.

ivanjaros commented Apr 22, 2017

damn, two years and no interest from google in his.

@maxekman

This comment has been minimized.

maxekman commented Apr 22, 2017

+1, this would indeed be a very useful feature. We have a current use case for storing DB backups directly in GCS for example.

@danielqsj

This comment has been minimized.

Contributor

danielqsj commented May 9, 2017

+1 for ceph-fuse

@tomi-vanek

This comment has been minimized.

tomi-vanek commented Jul 31, 2017

+1

@paultiplady

This comment has been minimized.

paultiplady commented Jul 31, 2017

Folks, a PSA: you can add a 👍 reaction to the original issue to signal your agreement, without triggering an email for everyone who's following this issue. Please do that instead of spamming us.

@sunshinekitty

This comment has been minimized.

sunshinekitty commented Jul 31, 2017

Bueller

@MattMS

This comment has been minimized.

MattMS commented Aug 1, 2017

I'm guessing people are adding the +1 comments to spam Owners/Members into paying attention to this, since there are already 4.4k+ Issues to contend with and this has not received any official responses in over 1 year.
It should be noted that the Issue search allows sorting by "Most reactions" (with 👍 as an option), but this is somewhat hidden.

I would humbly suggest people add their desired use-cases with their +1 comments.

I personally want to store Postgres backups in Google Storage.

@amlinux

This comment has been minimized.

amlinux commented Aug 1, 2017

Also the issue tags are confusing. Priority - needs more evidence. What kind of evidence is needed? Team - cluster (deprecated). Maybe some other, not deprecated team should take over this issue? Could you please re-triage.

@zhangxiaoyu-zidif

This comment has been minimized.

Member

zhangxiaoyu-zidif commented Nov 11, 2017

I implement ceph fuse mount, but faied to pass CI. But the function is tested well in 1.5.1, 1.6.9, 1.8.1

@tobsch

This comment has been minimized.

tobsch commented Nov 20, 2017

+1

1 similar comment
@nlassaux

This comment has been minimized.

nlassaux commented Nov 28, 2017

+1

@fejta-bot

This comment has been minimized.

fejta-bot commented Feb 26, 2018

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

@MattMS

This comment has been minimized.

MattMS commented Feb 27, 2018

Please keep this issue open or redirect us to a more appropriate issue.
/remove-lifecycle stale

@jamiejackson

This comment has been minimized.

jamiejackson commented Mar 28, 2018

I'm containerizing an applications's components, one of which accesses a remote file system over SSHFS with autofs, so I could have used such a feature, too.

@alexpirine

This comment has been minimized.

alexpirine commented Mar 30, 2018

I would need this feature to simply store user-uploaded files on google cloud storage. It would be very easy by mounting the bucket into a container folder, and simply writing files to it (for instance by configuring MEDIA_ROOT in a Django app).

@spacebel

This comment has been minimized.

spacebel commented Apr 12, 2018

What evidence are you looking for, guys ? Cloud storage is used by so many people for the persitence.
Looking for a use case ? I need to dynamically create Jobs in a Kubernetes cluster and mount bucket for these Jobs.
Please, hurry up :)

@buckhx

This comment has been minimized.

buckhx commented Apr 27, 2018

Another use case: We have a data pipeline that relies on third parties providing exports to GCS bucket. We want to mount the exports bucket as a volume that our k8s jobs can read from directly instead of copying the files to the container on run.

@bergey

This comment has been minimized.

bergey commented May 7, 2018

What's the status of this? Are maintainers waiting for more evidence that users want this feature? Are there open design questions, per @thockin's 2+ year old comment above? Are maintainers hoping someone will submit a PR? Is there a workable solution (postStart hook?) such that maintainers don't want to add a second way?

Like others in this thread, I'm trying to get my DB backups into cloud storage, one way or another.

@dims

This comment has been minimized.

Member

dims commented May 7, 2018

Looking through material above, looks like there are a few ways of doing things:

  1. postStart / preStop : https://karlstoney.com/2017/03/01/fuse-mount-in-kubernetes/ (and) https://github.com/maciekrb/gcs-fuse-sample
  2. CephFS : https://github.com/kubernetes/kubernetes/blob/master/pkg/volume/cephfs/cephfs.go (One could adapt this code to gcsfuse)
  3. Daemonset with bidirectional bind-mount idea : pangeo-data/pangeo#190 (comment)
  4. Init containers : https://github.com/ageapps/k8s-storage-buckets#nginx---gcsfuse-in-k8s-with-init-containers-just-read-access
@spacebel

This comment has been minimized.

spacebel commented May 7, 2018

The cons of these approaches areit requires adding the libraries on the container image (therefore not working on all the use cases of launching containers provided by third parties).

Also using postStart means the container may read the bucket immediatly but it may not already be mounted.

@dims

This comment has been minimized.

Member

dims commented May 7, 2018

@spacebel the #3 does not need libraries on containers provided by third parties i think

@spacebel

This comment has been minimized.

spacebel commented May 8, 2018

I'm not sure, but I read in the procedure as step 1 "Make a container that has all the software for doing the GCS Mount".

But this solution indeed remove one other disadvantage of (1) which is privileges required.

@dims

This comment has been minimized.

Member

dims commented May 8, 2018

@spacebel that container is for the daemonset. not the app itself.

@spacebel

This comment has been minimized.

spacebel commented Aug 2, 2018

For feedback, I have tried approach 3 (daemonset). If it helps:

Unfortunately, it does not work with Google Cloud (at least with the COS machines), even when manually modifying the docker.service to set the MountFlags to shared (and restart docker).
Note that it also requires Kubernetes v1.10.

@CTrox

This comment has been minimized.

CTrox commented Aug 6, 2018

I wrote a CSI driver for S3 (and compatible object stores) which dynamically provisions buckets and mounts them to any (and unprivileged) containers. https://github.com/CTrox/csi-s3

@neyz

This comment has been minimized.

neyz commented Oct 9, 2018

Following this thread too. Use case is simple data sharing between front web servers without having to rely on complicated HA NFS / gluster setups.

It would really be nice to have a simple supported solution.

@mcortinas

This comment has been minimized.

mcortinas commented Oct 23, 2018

no plans if we can avoid gcsfuse in order to mount volumes GCS?

I think should be very useful for the customers if we can use GCS such as these other kind of volumes https://kubernetes.io/docs/concepts/storage/volumes/#types-of-volumes

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment