Add support for libStorage as a Kubernetes volume plugin. #28599

Open
wants to merge 3 commits into
from
@vladimirvivien
vladimirvivien commented Jul 7, 2016 edited

Analytics
This change is to implement the libStorage storage provisioner and orchestrator framework as a Kubernetes volume plugin. LibStorage is an open source project from EMC {code}. One of its primary goals is provide persistent volume access to containerized applications. The libStorage framework comes with a stable API and a robust set of storage primitives including:

  • Volume creation
  • Volume inspection
  • Mount and unmount
  • Attach and detach

LibStorage offers a simple model for creating drivers for storage provisioning and orchestration. Storage drivers use a small but versioned API to implement some or all storage primitives. The drivers are automatically managed by the libStorage infrastructure. LibStorage exists to provide uniform access to the sizable and diverse storage platforms found in enterprise deployments. A libStorage volume plugin would allow Kubernetes to support many of these platforms while decoupling their implementations from the Kubernetes codebase.

The goals of the libStorage Kubernetes volume plugin are:

  • Mitigate the efforts of implementing k8s volume services
  • Grow the list of enterprise storage platforms available to k8s
  • Provide a portable storage API to third-party storage implementers

The libStorage volume plugin is designed to be cloud and provider agnostic. The volume is able to handle both PVs and PVCs. To enable libStorage as a dynamic persistent volume provisioner, use the --enable-libstorage-provisioner flag when launching the kube-controller-manager. This will use libStorage to automatically provision volumes defined in a Persistent Volume Claim.


This change is Reviewable

@googlebot

Thanks for your pull request. It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

📝 Please visit https://cla.developers.google.com/ to sign.

Once you've signed, please reply here (e.g. I signed it!) and we'll verify. Thanks.


  • If you've already signed a CLA, it's possible we don't have your GitHub username or you're using a different email address. Check your existing CLA data and verify that your email is set on your git commits.
  • If you signed the CLA as a corporation, please let us know the company's name.
@googlebot googlebot added the cla: no label Jul 7, 2016
@eparis
Member
eparis commented Jul 7, 2016

ok to test

@eparis
Member
eparis commented Jul 7, 2016

@kubernetes/sig-storage

@pmorie pmorie added the area/storage label Jul 7, 2016
@pmorie
Member
pmorie commented Jul 7, 2016

Suggestion: this would probably be easier to review if it was split into multiple commits like so:

  1. API changes
  2. Generated code
  3. Volume plugin
@eparis
Member
eparis commented Jul 7, 2016

godeps updates/changes in a single commit.

@vladimirvivien

Working on breaking out auto-gen code from real code changes.

@k8s-merge-robot k8s-merge-robot added size/XS and removed size/XXL labels Jul 8, 2016
@vladimirvivien

@pmorie I followed your suggestion and reset the feature branch and broke out the code into 3 separate commits to make it easy to review .

  • See 7f435b2 - libStorage source changes
  • See 56790fa - Godep dependencies
  • See 9005111 - Auto-generated sources
@k8s-bot
k8s-bot commented Jul 11, 2016

Can one of the admins verify that this patch is reasonable to test? If so, please reply "ok to test".
(Note: "add to whitelist" is no longer supported. Please update configurations in kubernetes/test-infra/jenkins/job-configs/kubernetes-jenkins-pull instead.)

This message may repeat a few times in short succession due to jenkinsci/ghprb-plugin#292. Sorry.

Otherwise, if this message is too spammy, please complain to ixdy.

@vladimirvivien

I signed it!

@googlebot

CLAs look good, thanks!

@googlebot googlebot added cla: yes and removed cla: no labels Jul 11, 2016
@vladimirvivien

@kubernetes/sig-storage

@rootfs
Contributor
rootfs commented Jul 17, 2016

About enable-libstorage-provisioner option, please coordinate with #29006

@thockin thockin assigned saad-ali and unassigned zmerlynn Jul 18, 2016
@thockin thockin and 1 other commented on an outdated diff Jul 18, 2016
Godeps/Godeps.json
@@ -18,6 +18,11 @@
"Rev": "'75cd24fc2f2c2a2088577d12123ddee5f54e0675'"
},
{
+ "ImportPath": "github.com/BurntSushi/toml",
@thockin
thockin Jul 18, 2016 Member

For a start we CAN NOT use this dep - it is licensed under the WTFPL which is not allowed.

https://news.ycombinator.com/item?id=5733050#up_5733477

@cantbewong
cantbewong Jul 18, 2016 edited

Thanks for reviewing, I have asked the developer associated with the code using this WTFPL licensed dep to investigate alternatives

@k8s-merge-robot k8s-merge-robot added size/XXL and removed size/XS labels Jul 18, 2016
@vladimirvivien

Please 'keep-open'. This PR is still active.

@k8s-bot
k8s-bot commented Aug 23, 2016

Can one of the admins verify that this patch is reasonable to test? If so, please reply "ok to test".
(Note: "add to whitelist" is no longer supported. Please update configurations in kubernetes/test-infra/jenkins/job-configs/kubernetes-jenkins-pull instead.)

This message will repeat several times in short succession due to jenkinsci/ghprb-plugin#292. Sorry.

@vladimirvivien
vladimirvivien commented Sep 9, 2016 edited

@kubernetes/sig-storage @saad-ali
Sorry for the XXL tag. But I try to categorized all commits into API Type, Dependencies, and Volume Plugin Impl. Hope this will make things easier to review.

This PR is ready to start the review process. It is in a WIP phase, there are more code coming (testing, dynamic provisioning, etc), but wanted to get some feedback early in this release cycle. Changes are grouped into 3 categories including new API type, dependencies, and volume plugin code.

Looking forward to hear back.

@vladimirvivien
vladimirvivien commented Sep 9, 2016 edited

@kubernetes/sig-storage @saad-ali The build system seems to run into some GCE Firewall issues:

ERROR: (gcloud.compute.firewall-rules.delete) Some requests did not succeed:
 - The resource 'projects/k8s-jkns-pr-gce/global/firewalls/e2e-gce-agent-pr-47-0-minion-e2e-gce-agent-pr-47-0-http-alt' was not found

ERROR: (gcloud.compute.firewall-rules.delete) Some requests did not succeed:
 - The resource 'projects/k8s-jkns-pr-gce/global/firewalls/e2e-gce-agent-pr-47-0-minion-e2e-gce-agent-pr-47-0-nodeports' was not found
@k8s-bot
k8s-bot commented Sep 16, 2016

GCE e2e build/test failed for commit 3f11315.

Please reference the list of currently known flakes when examining this failure. If you request a re-test, you must reference the issue describing the flake.

@vladimirvivien

@saad-ali, @kubernetes/sig-storge,
Hi any help on getting passed FAILED hack/make-rules/../../hack/verify-api-reference-docs.sh ? When I run the hack/update-api-reference-docs.sh script, it reports that everything is fixed, but the verify script still fails. Any pointers here would be helpful.

@saad-ali
Member

Hi any help on getting passed FAILED hack/make-rules/../../hack/verify-api-reference-docs.sh ? When I run the hack/update-api-reference-docs.sh script, it reports that everything is fixed, but the verify script still fails. Any pointers here would be helpful.

@vladimirvivien Try make clean, rebase with HEAD, and then run hack/update-api-reference-docs.sh

@saad-ali saad-ali added the keep-open label Sep 26, 2016
@k8s-merge-robot k8s-merge-robot added size/XS and removed size/XXL labels Oct 4, 2016
@saad-ali saad-ali referenced this pull request in kubernetes/contrib Oct 19, 2016
Closed

Incorrect PR Size label #1892

@k8s-merge-robot k8s-merge-robot added size/XXL and removed size/XS labels Oct 20, 2016
@wattsteve
Contributor
wattsteve commented Oct 24, 2016 edited

Thanks Vladimir, we appreciate the contribution. A couple of points on the approach taken:

  • I think the Dynamic Provisioning enablement should use the Kubernetes beta Dynamic Provisioning implementation and not the Alpha. The README should be adapted to include an example with a Storage Class that uses the LibStorage Provisioner. I think this is important because we want to create a consistent user experience and process for dynamic provisioning via the Kubernetes control plane. The idea is that once the admin and the users learn how to configure and use dynamic provisioning for one storage platform in Kubernetes, the process is as close to identical as possible when enabling and using dynamic provisioning for other storage platforms.
  • I'm not sure why you have the environment variable boolean to enable lib storage provisioning. With the (newer and greatly improved) beta dynamic provisioning framework, if the admin doesn’t want cluster users to have access to a particular provisioner, then they do not create and offer a storage class for it. Using this approach, I think you can get rid of the need for the flag altogether.
  • Anecdotal and not related to the volume plugin but... have you thought about running the LibStorage runtime on all nodes as a DaemonSet? One of the benefits to having the Kubernetes storage drivers (volume plugins and provisioners) in-tree is that the approach offers a good deployment and management user experience as all the drivers are included in the Kubelet and therefore are all already configured and running on every Kube node. Even with the LibStorage Volume Plugin in-tree, users are still left with the need to configure the LibStorage runtime on every Kube Node. If you were able to containerize the LibStorage runtime and deploy it as a DaemonSet, you'd be able to use the Kubernetes Control Plane to deploy the runtime and ensure that it stays up and running and greatly decrease the overall deployment complexity of getting LibStorage configured in Kubernetes.
@vladimirvivien

@wattsteve Thank you for the thoughtful comments. I will address them with their respective bullet points below.

  • I am working on some code update that, I hope to push this week, will do several things including updating the code to use the Beta Dynamic Provisioner API instead of the Alpha. The README, in examples/volumes/libstorage does have an example of using Storage Class (with the Alpha API) which will be updated.

  • The environment variable you mentioned is a relic from when this first started, before dynamic provisioning was introduced. This approach has been removed in subsequent code, and instead we rely on the standard Dynamic provisioning API.

  • The libStorage infrastructure supports several deployment models including a client/server option, which we will use with the Kubernetes integration. As such, a libStorage server runtime (REX-Ray in this instance) acts as an intermediator between k8s and a storage backend for volume-related management operations. The server does not have have to be deployed on every node in a cluster (it may have been something I poorly described previously). Instead, the server can run on one or more (HA configuration) nodes.

    Since a libStorage provider's binary (REX-Ray) is not tightly coupled to the in-tree volume libStorage plugin, it can be deployed in any way that is conducive to proper k8s deployment practices. Whether it's a DeamonSet, Pod, or a deployment external to k8s, we are exploring diverse methods that best fit libStorage/REX-Ray running inside Kubernetes while providing a management experience that remains faithful to the Kubernetes way.

Thanks again for your input!

@vladimirvivien

@k8s-bot verify test this

@vladimirvivien

@k8s-bot cvm gke e2e test this

@vladimirvivien

@k8s-bot gci gke e2e test this

@vladimirvivien

@saad-ali
The latest push for this CR is ready to be reviewed. The most important update in this push is considerable reduction in PR size. The CI seems to be flaking out, is there a way to force a clean rebuild? All my local verifications passed without any failures.

@saad-ali
Member

@saad-ali The latest push for this CR is ready to be reviewed. The most important update in this push is considerable reduction in PR size. The CI seems to be flaking out, is there a way to force a clean rebuild? All my local verifications passed without any failures.

Tests won't run until you rebase.

This branch has conflicts that must be resolved

@apelisse apelisse removed the do-not-merge label Nov 11, 2016
@vladimirvivien

@saad-ali Please take a look at this PR for review. Also, the number of files in this PR has gone down considerably (64 instead including all the generated source code). But it is still marked size/XXL.

@vladimirvivien

@k8s-bot cvm gke e2e test this

@k8s-ci-robot
Collaborator

@vladimirvivien: you can't request testing unless you are a kubernetes member.

In response to [this comment](https://github.com/kubernetes/kubernetes/pull/28599#issuecomment-261290574):

@k8s-bot cvm gke e2e test this

If you have questions or suggestions related to this bot's behavior, please file an issue against the kubernetes/test-infra repository.

@dims
Member
dims commented Nov 17, 2016

@k8s-bot cvm gke e2e test this

@dims
Member
dims commented Nov 17, 2016

there you go @vladimirvivien

@vladimirvivien

@k8s-bot cvm gke e2e test this

@k8s-ci-robot
Collaborator

@vladimirvivien: you can't request testing unless you are a kubernetes member.

In response to [this comment](https://github.com/kubernetes/kubernetes/pull/28599#issuecomment-261384978):

@k8s-bot cvm gke e2e test this

If you have questions or suggestions related to this bot's behavior, please file an issue against the kubernetes/test-infra repository.

@vladimirvivien

@dims thanks!

@vladimirvivien

@k8s-bot unit test this

@k8s-ci-robot
Collaborator

@vladimirvivien: you can't request testing unless you are a kubernetes member.

In response to [this comment](https://github.com/kubernetes/kubernetes/pull/28599#issuecomment-261651365):

@k8s-bot unit test this

If you have questions or suggestions related to this bot's behavior, please file an issue against the kubernetes/test-infra repository.

@eparis
Member
eparis commented Nov 19, 2016

@k8s-bot test this

@mikedanese
Member

@k8s-bot bazel test this, issue kubernetes/test-infra#1152

@saad-ali
Member

@saad-ali Please take a look at this PR for review. Also, the number of files in this PR has gone down considerably (64 instead including all the generated source code). But it is still marked size/XXL.

Someone else from @kubernetes/sig-storage will have to take a look. I'm slammed at the moment with 1.5 release czar duties.

@k8s-ci-robot
Collaborator

Jenkins GCE etcd3 e2e failed for commit 0b397c6. Full PR test history.

The magic incantation to run this job again is @k8s-bot gce etcd3 e2e test this. Please help us cut down flakes by linking to an open flake issue when you hit one in your PR.

@k8s-ci-robot
Collaborator

Jenkins GKE smoke e2e failed for commit 0b397c6. Full PR test history.

The magic incantation to run this job again is @k8s-bot cvm gke e2e test this. Please help us cut down flakes by linking to an open flake issue when you hit one in your PR.

@k8s-ci-robot
Collaborator

Jenkins GCI GKE smoke e2e failed for commit 0b397c6. Full PR test history.

The magic incantation to run this job again is @k8s-bot gci gke e2e test this. Please help us cut down flakes by linking to an open flake issue when you hit one in your PR.

@k8s-ci-robot
Collaborator

Jenkins GCE Node e2e failed for commit 0b397c6. Full PR test history.

The magic incantation to run this job again is @k8s-bot node e2e test this. Please help us cut down flakes by linking to an open flake issue when you hit one in your PR.

@k8s-ci-robot
Collaborator

Jenkins GCE e2e failed for commit 0b397c6. Full PR test history.

The magic incantation to run this job again is @k8s-bot cvm gce e2e test this. Please help us cut down flakes by linking to an open flake issue when you hit one in your PR.

@k8s-ci-robot
Collaborator

Jenkins CRI GCE Node e2e failed for commit 0b397c6. Full PR test history.

The magic incantation to run this job again is @k8s-bot cri node e2e test this. Please help us cut down flakes by linking to an open flake issue when you hit one in your PR.

@k8s-ci-robot
Collaborator

Jenkins GCI GCE e2e failed for commit 0b397c6. Full PR test history.

The magic incantation to run this job again is @k8s-bot gci gce e2e test this. Please help us cut down flakes by linking to an open flake issue when you hit one in your PR.

@k8s-ci-robot
Collaborator

Jenkins kops AWS e2e failed for commit cfe847a. Full PR test history.

The magic incantation to run this job again is @k8s-bot kops aws e2e test this. Please help us cut down flakes by linking to an open flake issue when you hit one in your PR.

@vladimirvivien

@k8s-bot unit test this

@k8s-ci-robot
Collaborator

@vladimirvivien: you can't request testing unless you are a kubernetes member.

In response to this comment:

@k8s-bot unit test this

If you have questions or suggestions related to this bot's behavior, please file an issue against the kubernetes/test-infra repository.

@vladimirvivien

@k8s-bot kops aws e2e test this

@k8s-ci-robot
Collaborator

@vladimirvivien: you can't request testing unless you are a kubernetes member.

In response to this comment:

@k8s-bot kops aws e2e test this

If you have questions or suggestions related to this bot's behavior, please file an issue against the kubernetes/test-infra repository.

@k8s-ci-robot
Collaborator

Jenkins CRI GCE e2e failed for commit f918ce6. Full PR test history.

The magic incantation to run this job again is @k8s-bot cri e2e test this. Please help us cut down flakes by linking to an open flake issue when you hit one in your PR.

@k8s-ci-robot
Collaborator

Jenkins verification failed for commit 7d48a6e. Full PR test history.

The magic incantation to run this job again is @k8s-bot verify test this. Please help us cut down flakes by linking to an open flake issue when you hit one in your PR.

@k8s-ci-robot
Collaborator

Jenkins Kubemark GCE e2e failed for commit 7d48a6e. Full PR test history.

The magic incantation to run this job again is @k8s-bot kubemark e2e test this. Please help us cut down flakes by linking to an open flake issue when you hit one in your PR.

@k8s-merge-robot
Collaborator

Adding label:do-not-merge because PR changes docs prohibited to auto merge
See http://kubernetes.io/editdocs/ for information about editing docs

vladimirvivien added some commits Nov 15, 2016
@vladimirvivien vladimirvivien LibStorage Kubernetes volume plugin source code
This commit captures the code changes for implementating the Kubernetes libStorage volume plugin including:
- LibStoragetVolumeSource API type
- LibStorage Plugin Impl
- LibStorage impl of Mounter, Unmounter, Provisioner, etc
- LibStorage client code
7991080
@vladimirvivien vladimirvivien LibStorage generated artifacts
This commit encapsulates all source and doc artifacts generated by the build scripts including
- API documentations
- Source code for API types
- Other build artifacts
- Etc
fada4bc
@vladimirvivien vladimirvivien LibStorage volume plugin example documentation
This commit includes all example documentation showing how to use the libStorage Kuberneties volume plugin.
4d073df
@k8s-ci-robot
Collaborator

Jenkins Bazel Build failed for commit 4d073df. Full PR test history.

The magic incantation to run this job again is @k8s-bot bazel test this. Please help us cut down flakes by linking to an open flake issue when you hit one in your PR.

If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@vladimirvivien

@k8s-bot unit test this

@k8s-ci-robot
Collaborator

Jenkins unit/integration failed for commit 4d073df. Full PR test history.

The magic incantation to run this job again is @k8s-bot unit test this. Please help us cut down flakes by linking to an open flake issue when you hit one in your PR.

If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@k8s-merge-robot
Collaborator

@vladimirvivien PR needs rebase

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment