FUSE volumes

[rsokolowski]

I have an use-case where I would like to mount Google Cloud Storage (GCS) bucket (and a directory in that bucket) in my container and use it as a regular FS. Currently, it seems doable using s3fs-fuse (https://github.com/s3fs-fuse/s3fs-fuse) - thanks @brendanburns. It would be great if GCS was supported as a first class Volume in Kubernetes.

[zmerlynn]

This is also possible now through the https://github.com/GoogleCloudPlatform/gcsfuse project, it looks like.

[rboyd]

@zmerlynn were you able to get gcsfuse to work? I'm expecting this won't work until 1.1 release http://stackoverflow.com/questions/31124368/allow-privileged-containers-in-kubernetes-on-google-container-gke

[zmerlynn]

I haven't tried yet. I was merely noting that looked like it may be an option over s3fs-fuse for GCS, which would require the same privileges, presumably.

[thockin]

We'll want to decide how we want to handle FUSE mounts in general - there
are potentially a LOT of neat things we can do, but FUSE is (historically)
known to be less than 100% reliable. The simplest is that we push it all
into the user's space and require privileges. Perhaps there are more
interesting ways to manage it?

On Mon, Oct 19, 2015 at 4:27 PM, Zach Loafman notifications@github.com
wrote:

I haven't tried yet. I was merely noting that looked like it may be an
option over s3fs-fuse for GCS, which would require the same privileges,
presumably.

—
Reply to this email directly or view it on GitHub
#7890 (comment)
.

[nickschuch]

Im about halfway through coding a volume for https://github.com/s3fs-fuse/s3fs-fuse, based it off the nfs implementation while also drawing some inspiration from #17221. Is this something people can see as a viable solution?

[thockin]

I'd like to see us shake out a solid design for FUSE-based volumes BEFORE
we argue about the merit of any one volume.

On Sun, Dec 13, 2015 at 5:01 PM, Nick Schuch notifications@github.com
wrote:

Im about halfway through coding a volume for
https://github.com/s3fs-fuse/s3fs-fuse, based it off the nfs
implementation while also drawing some inspiration from #17221
#17221. Is this something
people can see as a viable solution?

—
Reply to this email directly or view it on GitHub
#7890 (comment)
.

[nickschuch]

No argurments from me :) How can I help?

[thockin]

ideate :)

On Mon, Dec 14, 2015 at 1:45 PM, Nick Schuch notifications@github.com
wrote:

No argurments from me :) How can I help?

—
Reply to this email directly or view it on GitHub
#7890 (comment)
.

[nickschuch]

My main goal was to defer the fuse implementations to packages on the host and then mount them just like the other volumes eg. NFS.

Maybe we could make a higher level volume which took properties similar to an fstab/mount? That way users are free to use there own mount implementations and we are just using those. That would cut down on duplication of writing multiple volumes with the same scaffolding, as well as support gcsfuse, s3fs-fuse, azure files etc.... Essentially, if you can mount it, we can run it.

[nickschuch]

Hmm, scratch that, that was a very raw thought, I see now we pretty much have that via the "mount" package and volumes provide a higher level.

Currently updated my goal to creating a "fuse" volume, going to write some code and see what other thoughts come from there. That will allow us to also mount the other fuse filesystems.

[pnovotnak]

I just wanted to chime in and say that this would be a huge boon when running on GCE. Right now I'm looking into storage options for the company I work for... There are a number of inferior options but this would be by far the best for our case.

[thockin]

@kubernetes/sig-storage We discussed FUSE a bit recently and worked out a basic model for it, but it is somewhat complicated to do correctly.

Some notes:

We need a FUSE daemon per volume (maybe we can find a way to flatten to per-pod, but not sure that is ALWAYS ok)

FUSE daemons need privileges

This FUSE daemon must run in the pod’s cgroups and net namespace (chargeback), but must NOT run in the pod’s IPC or PID namespace (for security)

It must be killed when the pod terminates.

We need a way to report this container:

• add it to pod.spec.containers?
• pod.status.adminContainers?
  • bad - what image was run?  can we recreate this if it were lost (the status litmus test)

A less-perfect alternative might be to run GCS FUSE on every Google VM and treat it as a special form of hostPath. I don't really want to special case it, though, so it's somewhat less attractive.

[bgrant0607]

Related: #831

[bgrant0607]

FUSE is mainly needed for writes? Otherwise, it seems simpler to just fetch a tar.gz and unpack it into an emptyDir.

[jefflaplante]

Has anyone worked on this further? I would like to be able mount FUSE volumes like other PersistentVolumes in kubernetes as an alternative to NFS or glusterfs for multi-container Read-write.

[Stono]

For those still playing with this I ended up using the preStop and postStart lifecycle hooks and running the fuse command, which results in a very similar behaviour.

https://karlstoney.com/2017/03/01/fuse-mount-in-kubernetes/

It'd be awesome however to have the ability to do fuse mounts as PersistentVolumes in Kubernetes.

[baracoder]

I have a similar issue. I would like to use ceph-fuse because of requent problems with the kernel driver.

Using life cycle hooks is a valid workaround but the issue is still valid because the docker images need to be modified this way. Is it possible to use privileged init containers for this somehow?

[sunshinekitty]

+1

[nickschuch]

We managed to get this going at a flexvolume, Ill look at what it will take for us to publish to code (bit rough, but demonstrates the point).

[matesitox]

+1

[davidberardozzi]

+1

[ivanjaros]

damn, two years and no interest from google in his.

[maxekman]

+1, this would indeed be a very useful feature. We have a current use case for storing DB backups directly in GCS for example.

[danielqsj]

+1 for ceph-fuse

[tomi-vanek]

+1

[paultiplady]

Folks, a PSA: you can add a 👍 reaction to the original issue to signal your agreement, without triggering an email for everyone who's following this issue. Please do that instead of spamming us.

[sunshinekitty]

Bueller

[MattMS]

I'm guessing people are adding the +1 comments to spam Owners/Members into paying attention to this, since there are already 4.4k+ Issues to contend with and this has not received any official responses in over 1 year.
It should be noted that the Issue search allows sorting by "Most reactions" (with 👍 as an option), but this is somewhat hidden.

I would humbly suggest people add their desired use-cases with their +1 comments.

I personally want to store Postgres backups in Google Storage.

[amlinux]

Also the issue tags are confusing. Priority - needs more evidence. What kind of evidence is needed? Team - cluster (deprecated). Maybe some other, not deprecated team should take over this issue? Could you please re-triage.

[bgrant0607]

https://stackoverflow.com/questions/35966832/mount-google-storage-bucket-in-google-container

[zhangxiaoyu-zidif]

I implement ceph fuse mount, but faied to pass CI. But the function is tested well in 1.5.1, 1.6.9, 1.8.1

[tobsch]

+1

[nlassaux]

+1

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[MattMS]

Please keep this issue open or redirect us to a more appropriate issue.
/remove-lifecycle stale

[jamiejackson]

I'm containerizing an applications's components, one of which accesses a remote file system over SSHFS with autofs, so I could have used such a feature, too.

[alexpirine]

I would need this feature to simply store user-uploaded files on google cloud storage. It would be very easy by mounting the bucket into a container folder, and simply writing files to it (for instance by configuring MEDIA_ROOT in a Django app).

[spacebel]

What evidence are you looking for, guys ? Cloud storage is used by so many people for the persitence.
Looking for a use case ? I need to dynamically create Jobs in a Kubernetes cluster and mount bucket for these Jobs.
Please, hurry up :)

[buckhx]

Another use case: We have a data pipeline that relies on third parties providing exports to GCS bucket. We want to mount the exports bucket as a volume that our k8s jobs can read from directly instead of copying the files to the container on run.

[bergey]

What's the status of this? Are maintainers waiting for more evidence that users want this feature? Are there open design questions, per @thockin's 2+ year old comment above? Are maintainers hoping someone will submit a PR? Is there a workable solution (postStart hook?) such that maintainers don't want to add a second way?

Like others in this thread, I'm trying to get my DB backups into cloud storage, one way or another.

[dims]

Looking through material above, looks like there are a few ways of doing things:

1. postStart / preStop : https://karlstoney.com/2017/03/01/fuse-mount-in-kubernetes/ (and) https://github.com/maciekrb/gcs-fuse-sample
2. CephFS : https://github.com/kubernetes/kubernetes/blob/master/pkg/volume/cephfs/cephfs.go (One could adapt this code to gcsfuse)
3. Daemonset with bidirectional bind-mount idea : pangeo-data/pangeo#190 (comment)
4. Init containers : https://github.com/ageapps/k8s-storage-buckets#nginx---gcsfuse-in-k8s-with-init-containers-just-read-access

[spacebel]

The cons of these approaches areit requires adding the libraries on the container image (therefore not working on all the use cases of launching containers provided by third parties).

Also using postStart means the container may read the bucket immediatly but it may not already be mounted.

[dims]

@spacebel the #3 does not need libraries on containers provided by third parties i think

[spacebel]

I'm not sure, but I read in the procedure as step 1 "Make a container that has all the software for doing the GCS Mount".

But this solution indeed remove one other disadvantage of (1) which is privileges required.

[dims]

@spacebel that container is for the daemonset. not the app itself.

[spacebel]

For feedback, I have tried approach 3 (daemonset). If it helps:

• Project example (with kubernetes daemonset): https://github.com/spacebel/gcsfuse-slim
• Image of a slim gcsfuse: https://hub.docker.com/r/cnlspacebel/gcsfuse-slim/

Unfortunately, it does not work with Google Cloud (at least with the COS machines), even when manually modifying the docker.service to set the MountFlags to shared (and restart docker).
Note that it also requires Kubernetes v1.10.

[CTrox]

I wrote a CSI driver for S3 (and compatible object stores) which dynamically provisions buckets and mounts them to any (and unprivileged) containers. https://github.com/CTrox/csi-s3

[neyz]

Following this thread too. Use case is simple data sharing between front web servers without having to rely on complicated HA NFS / gluster setups.

It would really be nice to have a simple supported solution.

[mcortinas]

no plans if we can avoid gcsfuse in order to mount volumes GCS?

I think should be very useful for the customers if we can use GCS such as these other kind of volumes https://kubernetes.io/docs/concepts/storage/volumes/#types-of-volumes