Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

It should be possible to use the ReadWriteMany mode #543

Closed
luxas opened this issue Apr 4, 2017 · 28 comments
Closed

It should be possible to use the ReadWriteMany mode #543

luxas opened this issue Apr 4, 2017 · 28 comments

Comments

@luxas
Copy link
Member

@luxas luxas commented Apr 4, 2017

Right now the preferred mode seems to be ReadWriteOnce for a PV, but preferably Rook should be able to create RWX Volumes as well

Is this hard technically with Ceph or haven't you just highlighted it?

@bassam

This comment has been minimized.

Copy link
Member

@bassam bassam commented Apr 4, 2017

Its technically possible as I understand it, however, it requires client side settings (at rbd level) to turn off caching. You wouldn't want two clients to be writing and run into consistency issues. Also this will likely impact performance.

@luxas

This comment has been minimized.

Copy link
Member Author

@luxas luxas commented Apr 4, 2017

Seems like the k8s CephFS supports this but not RBD out-of-the-box: https://kubernetes.io/docs/concepts/storage/persistent-volumes/#access-modes

@bassam

This comment has been minimized.

Copy link
Member

@bassam bassam commented Apr 4, 2017

Yes for CephFS its straightforward since the MDS is policing every write to the file system.

@luxas

This comment has been minimized.

Copy link
Member Author

@luxas luxas commented Apr 4, 2017

Ok, and CephFS for Rook is in the works, right?

@bassam

This comment has been minimized.

Copy link
Member

@bassam bassam commented Apr 4, 2017

For K8S yes. CephFS is functional today in standalone mode. Im curious which scenario you care about for RWX. Can you elaborate?

@luxas

This comment has been minimized.

Copy link
Member Author

@luxas luxas commented May 7, 2017

@bassam For example, I might want to run three WordPress replicas and let them share the same storage volume. When I upload new files, it shouldn't matter which replica I hit; they should all be able to write to the volume.

Is there any way to do this with RBD or do I have to use CephFS?
Also when will CephFS be usable with Kubernetes?

@luxas

This comment has been minimized.

Copy link
Member Author

@luxas luxas commented May 7, 2017

I've also seen others who've asked for this feature...

@bassam

This comment has been minimized.

Copy link
Member

@bassam bassam commented May 21, 2017

@kokhang can we add support for this in the rbd plugin before 1.7 freeze? I think it might be a matter of disabling fencing if RWM is set.

@jamiehannaford

This comment has been minimized.

Copy link
Member

@jamiehannaford jamiehannaford commented Jun 14, 2017

@bassam @kokhang Thanks for working on this. Will this be ready for 1.7?

@hunter

This comment has been minimized.

Copy link
Contributor

@hunter hunter commented Jun 14, 2017

I'd imagine 1.7 has feature frozen now that beta1 is out.

Could the external-storage CephFS provisioner be utilised (and/or ported for Rook)? https://github.com/kubernetes-incubator/external-storage/tree/master/ceph/cephfs

@bassam

This comment has been minimized.

Copy link
Member

@bassam bassam commented Jun 14, 2017

@hunter @jamiehannaford the path we are taking here is to add a rook volume plugin to Kubernetes in the 1.8 timeframe. The plugin will support block (including ReadWriteMany) and filesystem. The design is outlined here #432 (comment) and the initial PR (which is still under design review by the Kubernetes team) is here kubernetes/kubernetes#46843.

@jamiehannaford

This comment has been minimized.

Copy link
Member

@jamiehannaford jamiehannaford commented Jun 14, 2017

@bassam Awesome, thanks for the update!

@hunter

This comment has been minimized.

Copy link
Contributor

@hunter hunter commented Jun 14, 2017

Yep, understood @bassam.

I thought it might be worth sharing the external-storage plugin if anyone wants to test out RWM before 1.8. I'm sure there will be a few questions about it since we're still a ways off :)

@FaKod

This comment has been minimized.

Copy link

@FaKod FaKod commented Oct 2, 2017

kubernetes/kubernetes#46843 has been closed by stating that the rook team will use a flexvolume. So ReadWriteMany will still be available with K8s 1.8? Or is the timeline independent from K8s now?

@bassam

This comment has been minimized.

Copy link
Member

@bassam bassam commented Oct 2, 2017

@FaKod yes we are going to use a flex volume (the work is well underway see #882). ReadWriteMany will be supported by the flex volume and is independent of K8S. One feature that would help RWM is the ability to mount devices directly into containers which looks like will come in v1.9.

What is you scenario for RWM? How do you plan to use it?

@luxas

This comment has been minimized.

Copy link
Member Author

@luxas luxas commented Oct 2, 2017

@bassam When is the next Rook release scheduled?

@FaKod

This comment has been minimized.

Copy link

@FaKod FaKod commented Oct 3, 2017

@bassam I think there is no use case in microservice architectures. However, some customers are using file transfer from one container to another, are implementing distributed locks via filesystem or use a specific common (and slow) file store for pictures/videos. What I tested is to grab html files from a repository, store them in a "shared" volume and use it in a nginx container (just for testing and as an example in our Kubernetes training). The results are filesystem corruptions from time to time (in the volume). So it definitely doesn't work.

@kokhang

This comment has been minimized.

Copy link
Member

@kokhang kokhang commented Oct 3, 2017

@FaKod The thing about RWX is that Kubernetes does not support returning raw block volumes. It is always expects a block volume formatted and mounted with a filesystem. So it cannot support RWX for block. Here is the feature that tracks this. kubernetes/enhancements#351

However, we rook you can create ceph filesystems. These are RWX. Can you leverage this for your use case instead?

@FaKod

This comment has been minimized.

Copy link

@FaKod FaKod commented Oct 4, 2017

@kokhang "with rook you can create ceph filesystems". Do you have an example how to do that?

@kokhang

This comment has been minimized.

Copy link
Member

@kokhang kokhang commented Oct 4, 2017

@FaKod This is how you can setup and consume a filesystem. https://github.com/kokhang/rook/blob/master/Documentation/k8s-filesystem.md.

This is changing very soon and it is being made easier via CRD. #1029

@jbw976

This comment has been minimized.

Copy link
Member

@jbw976 jbw976 commented Oct 5, 2017

@kokhang do we also have documentation for the current way (before Rook flexvolume plugin gets CephFS support) to use CephFS with a persistent volume claim for a pod?

@kokhang

This comment has been minimized.

Copy link
Member

@kokhang kokhang commented Oct 9, 2017

@jbw976 ceph filesystem cannot be dynamically created with pvc. This is created using a CRD. https://github.com/rook/rook/blob/master/Documentation/filesystem-crd.md

Now that we support multiple filesystem, perhaps we should leverage the PVC instead of CRD. But that is a different discussion.

@FaKod

This comment has been minimized.

Copy link

@FaKod FaKod commented Oct 10, 2017

@kokhang Which version do you mean? In the last release, there is no CRD other than cluster, right?

@jbw976

This comment has been minimized.

Copy link
Member

@jbw976 jbw976 commented Oct 12, 2017

@FaKod in the last release (0.5.1) we only had cluster and pool CRDs. But now in the latest master builds, we have added a couple new CRDs very recently to declaratively create other objects in a Rook cluster, such as object storage and shared filesystems.

Object store: https://rook.io/docs/rook/master/object-store-crd.html
Shared filesystem: https://rook.io/docs/rook/master/filesystem-crd.html

@galexrt

This comment has been minimized.

Copy link
Member

@galexrt galexrt commented Jan 15, 2018

@luxas @kokhang @jbw976 @bassam As far as I understand the topic from the comments, this is about being able to create/use CephFS/Filesystem by a StorageClass and PVCs.
Anything against closing this issue and using #1125 to track this feature?
Also there has been a design doc (#1152) by @kokhang which is mentioned in #1125.

@t3hmrman

This comment has been minimized.

Copy link

@t3hmrman t3hmrman commented Apr 30, 2018

Hey all,

I'm not sure if this belongs here, but was looking through the issues -- I wanted to note that if I set up a PVC and use it from two different pods (in my case, actually 2 replicas in the same Deployment), the second replica fails with an error stating that the volume is attached to another pod (the first replica to boot), on rook v0.7.0.

I've confirmed that the PV is RWX, so I was a bit surprised that the second replica wasn't binding correctly. I can make a new issue if that's preferred but thought this issue might be more relevant.

I'm very happy with rook as the StorageClass (default type FileSystem) + PVC + Dynamically created PV + binding is working just fine, except in this RWX case -- not being able to set replicas to a value greater than one is a bit of an issue

@galexrt

This comment has been minimized.

Copy link
Member

@galexrt galexrt commented May 10, 2018

@t3hmrman Block storage is not available for RWX, that is where a network filesystem such as CephFS comes in.
Currently you can only use CephFS directly in the volumes section of a Deployment, #1125 covers usage of CephFS through PVCs.


Closing this issue as there have been no comments against closing this and using #1125 to track being able to consume CephFS through PVCs.

@galexrt galexrt closed this May 10, 2018
@t3hmrman

This comment has been minimized.

Copy link

@t3hmrman t3hmrman commented May 11, 2018

Hey @galexrt thank you very much for clarifying!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
9 participants
You can’t perform that action at this time.