New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Out-of-tree CSI Volume Plugins #178

Open
saad-ali opened this Issue Jan 24, 2017 · 35 comments

Comments

@saad-ali
Member

saad-ali commented Jan 24, 2017

  • One-line feature description (can be used as a release note): Add support for Out-of-Tree CSI Volume Plugins in Kubernetes
  • Primary contact (assignee): @saad-ali
  • Responsible SIGs: storage
  • Design proposal link (community repo): kubernetes/community#1258
  • Reviewer(s) - (for LGTM) recommend having 2+ reviewers (at least one from code-area OWNERS file) agreed to review. Reviewers from multiple companies preferred:
    @childsb
    @jsafrane
  • Approver (likely from SIG/area to which feature belongs):
    @thockin
  • Feature target (which target equals to which milestone):
    • Alpha release target (x.y): 1.9
    • Beta release target (x.y): 1.10
    • Stable release target (x.y): 1.13

Background

Kubernetes volume plugins are currently all "in-tree" meaning that their source code is included in the main Kubernetes repo. All volume plugins are compiled and ship along with kubernetes binaries.

The drawback to this approach is that it requires third-party storage vendors wanting to support Kubernetes to commit code to the Kubernetes repo, and thus be locked into Kubernetes release schedules. It requires them to make their source code public/open-source.

While the Flex volume plugin already provides a mechanism for Plugin developers to experiment with out-of-tree plugins, it provides no guarantees of backwards compatibility (since it is alpha), and is completely exec based (driver installation requires ability to deploy files to specific locations on node and master machines).

This feature aims to create a new (or adopt an existing) API for Volume Plugins in Kubernetes that:

  • Will allow volume plugins to be developed out-of-tree.
  • Not require building volume plugins (or their dependencies) into Kubernetes binaries.
  • Not requiring direct machine access to deploy new volume plugins (drivers).

CC @kubernetes/sig-storage-feature-requests @kubernetes/sig-storage-proposals

@saad-ali saad-ali added this to the next-milestone milestone Jan 24, 2017

@saad-ali

This comment has been minimized.

Show comment
Hide comment
@saad-ali

saad-ali Jan 24, 2017

Member

We aim to have a design doc for this by 1.6.

Member

saad-ali commented Jan 24, 2017

We aim to have a design doc for this by 1.6.

@calebamiles calebamiles modified the milestones: v1.6, next-milestone Feb 7, 2017

@idvoretskyi

This comment has been minimized.

Show comment
Hide comment
@idvoretskyi

idvoretskyi Mar 8, 2017

Member

@saad-ali any update on this feature? Docs and release notes are required (please, provide them to the features spreadsheet.

Member

idvoretskyi commented Mar 8, 2017

@saad-ali any update on this feature? Docs and release notes are required (please, provide them to the features spreadsheet.

@saad-ali saad-ali removed this from the v1.6 milestone Mar 13, 2017

@saad-ali

This comment has been minimized.

Show comment
Hide comment
@saad-ali

saad-ali Mar 13, 2017

Member

This does not need to be tracked for 1.6. It's design only. No bits shipping as part of 1.6. Removing 1.6 label.

Member

saad-ali commented Mar 13, 2017

This does not need to be tracked for 1.6. It's design only. No bits shipping as part of 1.6. Removing 1.6 label.

@idvoretskyi idvoretskyi added this to the next-milestone milestone Mar 14, 2017

@idvoretskyi

This comment has been minimized.

Show comment
Hide comment
@idvoretskyi

idvoretskyi Mar 14, 2017

Member

@saad-ali thank you for clarifying.

Member

idvoretskyi commented Mar 14, 2017

@saad-ali thank you for clarifying.

@jeffvance

This comment has been minimized.

Show comment
Hide comment
@jeffvance

jeffvance Mar 21, 2017

@saad-ali is the idea behind the 3rd bullet Not requiring direct machine access to deploy new volume plugins (drivers) to be able to pull plugin dependencies (eg. client libraries) from a defined registry or repo and push them to each node?

jeffvance commented Mar 21, 2017

@saad-ali is the idea behind the 3rd bullet Not requiring direct machine access to deploy new volume plugins (drivers) to be able to pull plugin dependencies (eg. client libraries) from a defined registry or repo and push them to each node?

@krmayankk

This comment has been minimized.

Show comment
Hide comment
@krmayankk

krmayankk Mar 26, 2017

@saad-ali also its not clear, how is this different from out -of tree dynamic provisioners ?

krmayankk commented Mar 26, 2017

@saad-ali also its not clear, how is this different from out -of tree dynamic provisioners ?

@saad-ali saad-ali changed the title from Out-of-tree Volume Plugins (CVI) to Out-of-tree Volume Plugins (CSI) May 18, 2017

@lpabon

This comment has been minimized.

Show comment
Hide comment
@lpabon

lpabon Jun 14, 2017

@saad-ali How does this compare with your CSI work on the CNCF ?

lpabon commented Jun 14, 2017

@saad-ali How does this compare with your CSI work on the CNCF ?

@saad-ali

This comment has been minimized.

Show comment
Hide comment
@saad-ali

saad-ali Jun 14, 2017

Member

@saad-ali is the idea behind the 3rd bullet Not requiring direct machine access to deploy new volume plugins (drivers) to be able to pull plugin dependencies (eg. client libraries) from a defined registry or repo and push them to each node?

Yep!

@saad-ali also its not clear, how is this different from out -of tree dynamic provisioners ?

This feature (CSI) will enable completely out-of-tree volume plugins. Out-of-tree dynamic provisioners exist today as an alpha mechinism to enable volume provisioning (one functionality of volume plugins) to 1) be implmented out-of-tree, and 2) be decoupled from the rest of the volume plugin (potentially enabling multiple provisioners per volume type). Once Kubernetes supports CSI volume plugins, the only remaining benefit of Out-of-tree dynamic provisioners will be 2 (decoupled from the rest of the volume plugin, potentially enabling multiple provisioners per volume type). We can decide at that point if it makes sense to continue to support them or deprecate them.

@saad-ali How does this compare with your CSI work on the CNCF ?

CSI is the proposed protocol spec. The intention (or at least one of the intentions) of this feature is to implement that interface in Kubernetes.

Member

saad-ali commented Jun 14, 2017

@saad-ali is the idea behind the 3rd bullet Not requiring direct machine access to deploy new volume plugins (drivers) to be able to pull plugin dependencies (eg. client libraries) from a defined registry or repo and push them to each node?

Yep!

@saad-ali also its not clear, how is this different from out -of tree dynamic provisioners ?

This feature (CSI) will enable completely out-of-tree volume plugins. Out-of-tree dynamic provisioners exist today as an alpha mechinism to enable volume provisioning (one functionality of volume plugins) to 1) be implmented out-of-tree, and 2) be decoupled from the rest of the volume plugin (potentially enabling multiple provisioners per volume type). Once Kubernetes supports CSI volume plugins, the only remaining benefit of Out-of-tree dynamic provisioners will be 2 (decoupled from the rest of the volume plugin, potentially enabling multiple provisioners per volume type). We can decide at that point if it makes sense to continue to support them or deprecate them.

@saad-ali How does this compare with your CSI work on the CNCF ?

CSI is the proposed protocol spec. The intention (or at least one of the intentions) of this feature is to implement that interface in Kubernetes.

@luxas luxas referenced this issue Sep 1, 2017

Closed

Flex volume API and Improved lifecycle (flexvolume) #93

0 of 23 tasks complete
@luxas

This comment has been minimized.

Show comment
Hide comment
@luxas

luxas Sep 1, 2017

Member

@saad-ali Are you targeting alpha for v1.9? Is that information up-to-date?
Thanks!

Member

luxas commented Sep 1, 2017

@saad-ali Are you targeting alpha for v1.9? Is that information up-to-date?
Thanks!

@saad-ali saad-ali modified the milestones: next-milestone, 1.9 Oct 20, 2017

@saad-ali saad-ali changed the title from Out-of-tree Volume Plugins (CSI) to Out-of-tree CSI Volume Plugins Oct 23, 2017

@saad-ali

This comment has been minimized.

Show comment
Hide comment
@saad-ali

saad-ali Oct 23, 2017

Member

@saad-ali Are you targeting alpha for v1.9? Is that information up-to-date?

Yes! All info is up to date.

Member

saad-ali commented Oct 23, 2017

@saad-ali Are you targeting alpha for v1.9? Is that information up-to-date?

Yes! All info is up to date.

@saad-ali

This comment has been minimized.

Show comment
Hide comment
@saad-ali

saad-ali Oct 23, 2017

Member

This feature (CSI) will enable completely out-of-tree volume plugins. Out-of-tree dynamic provisioners exist today as an alpha mechinism to enable volume provisioning (one functionality of volume plugins) to 1) be implmented out-of-tree, and 2) be decoupled from the rest of the volume plugin (potentially enabling multiple provisioners per volume type). Once Kubernetes supports CSI volume plugins, the only remaining benefit of Out-of-tree dynamic provisioners will be 2 (decoupled from the rest of the volume plugin, potentially enabling multiple provisioners per volume type). We can decide at that point if it makes sense to continue to support them or deprecate them.

An update on this: we've incorporated external-provisioners in to the CSI design!

Member

saad-ali commented Oct 23, 2017

This feature (CSI) will enable completely out-of-tree volume plugins. Out-of-tree dynamic provisioners exist today as an alpha mechinism to enable volume provisioning (one functionality of volume plugins) to 1) be implmented out-of-tree, and 2) be decoupled from the rest of the volume plugin (potentially enabling multiple provisioners per volume type). Once Kubernetes supports CSI volume plugins, the only remaining benefit of Out-of-tree dynamic provisioners will be 2 (decoupled from the rest of the volume plugin, potentially enabling multiple provisioners per volume type). We can decide at that point if it makes sense to continue to support them or deprecate them.

An update on this: we've incorporated external-provisioners in to the CSI design!

@govint

This comment has been minimized.

Show comment
Hide comment
@govint

govint Oct 24, 2017

@saad-ali can you point to the CSI design that has the external provisioners incorporated. Is it here kubernetes/community#1258?

govint commented Oct 24, 2017

@saad-ali can you point to the CSI design that has the external provisioners incorporated. Is it here kubernetes/community#1258?

@luxas

This comment has been minimized.

Show comment
Hide comment
@luxas

luxas Oct 24, 2017

Member

@saad-ali Will this work be in v1.9? I assume as alpha as the initial step.

Member

luxas commented Oct 24, 2017

@saad-ali Will this work be in v1.9? I assume as alpha as the initial step.

@kokhang

This comment has been minimized.

Show comment
Hide comment
@kokhang

kokhang Nov 9, 2017

Member

@luxas @saad-ali does "alpha" mean that csi would need to be enabled via a feature gate like all the other alpha features?

Also the CSI plugin depends on other features, which are currently in alpha, like mount-propagation and perhaps containerized-mount? Will these be promoted to beta? or does this mean that in order to consume the CSI plugin, these features have to be enabled via the feature gates?

Member

kokhang commented Nov 9, 2017

@luxas @saad-ali does "alpha" mean that csi would need to be enabled via a feature gate like all the other alpha features?

Also the CSI plugin depends on other features, which are currently in alpha, like mount-propagation and perhaps containerized-mount? Will these be promoted to beta? or does this mean that in order to consume the CSI plugin, these features have to be enabled via the feature gates?

@Bradamant3

This comment has been minimized.

Show comment
Hide comment
@Bradamant3

Bradamant3 Mar 2, 2018

Member

@saad-ali any doc updates needed for 1.10? Please update the feature tracking spreadsheet. And please get any needed doc PR in as soon as possible (this reminder is late, too ...). Thanks!

Member

Bradamant3 commented Mar 2, 2018

@saad-ali any doc updates needed for 1.10? Please update the feature tracking spreadsheet. And please get any needed doc PR in as soon as possible (this reminder is late, too ...). Thanks!

@Bradamant3

This comment has been minimized.

Show comment
Hide comment
@Bradamant3

Bradamant3 Mar 6, 2018

Member

@saad-ali we need doc updates this week, please! See previous comment. Docs PRs are due to be merged by Friday, March 9 and we need to allow time to review. Thanks! cc @idvoretskyi

Member

Bradamant3 commented Mar 6, 2018

@saad-ali we need doc updates this week, please! See previous comment. Docs PRs are due to be merged by Friday, March 9 and we need to allow time to review. Thanks! cc @idvoretskyi

@CalvinHartwell

This comment has been minimized.

Show comment
Hide comment
@CalvinHartwell

CalvinHartwell Apr 11, 2018

@saad-ali do we have any example working CSI plugins yet?

CalvinHartwell commented Apr 11, 2018

@saad-ali do we have any example working CSI plugins yet?

@tamalsaha

This comment has been minimized.

Show comment
Hide comment
Member

tamalsaha commented Apr 11, 2018

@dims

This comment has been minimized.

Show comment
Hide comment
Member

dims commented Apr 11, 2018

@clintkitson

This comment has been minimized.

Show comment
Hide comment
@CalvinHartwell

This comment has been minimized.

Show comment
Hide comment
@CalvinHartwell

CalvinHartwell Apr 11, 2018

thanks @tamalsaha @dims @clintkitson don't forget: https://github.com/ceph/ceph-csi, it should go into that list.

Also that list is pretty class, it needs to go into the storage section of the upstream k8s manual. 👍 👍 💯

CalvinHartwell commented Apr 11, 2018

thanks @tamalsaha @dims @clintkitson don't forget: https://github.com/ceph/ceph-csi, it should go into that list.

Also that list is pretty class, it needs to go into the storage section of the upstream k8s manual. 👍 👍 💯

@justaugustus

This comment has been minimized.

Show comment
Hide comment
@justaugustus

justaugustus Apr 17, 2018

Member

@saad-ali
Any plans for this in 1.11?

If so, can you please ensure the feature is up-to-date with the appropriate:

  • Description
  • Milestone
  • Assignee(s)
  • Labels:
    • stage/{alpha,beta,stable}
    • sig/*
    • kind/feature

cc @idvoretskyi

Member

justaugustus commented Apr 17, 2018

@saad-ali
Any plans for this in 1.11?

If so, can you please ensure the feature is up-to-date with the appropriate:

  • Description
  • Milestone
  • Assignee(s)
  • Labels:
    • stage/{alpha,beta,stable}
    • sig/*
    • kind/feature

cc @idvoretskyi

@saad-ali

This comment has been minimized.

Show comment
Hide comment
@saad-ali

saad-ali Apr 25, 2018

Member

@justaugustus for 1.11 we are planning on keeping CSI in beta. Planning for GA in 1.12

Member

saad-ali commented Apr 25, 2018

@justaugustus for 1.11 we are planning on keeping CSI in beta. Planning for GA in 1.12

@saad-ali saad-ali modified the milestones: v1.10, v1.12 Apr 25, 2018

@justaugustus

This comment has been minimized.

Show comment
Hide comment
@justaugustus

justaugustus Jul 18, 2018

Member

@saad-ali --

It looks like this feature is currently in the Kubernetes 1.12 Milestone.

If that is still accurate, please ensure that this issue is up-to-date with ALL of the following information:

  • One-line feature description (can be used as a release note):
  • Primary contact (assignee):
  • Responsible SIGs:
  • Design proposal link (community repo):
  • Link to e2e and/or unit tests:
  • Reviewer(s) - (for LGTM) recommend having 2+ reviewers (at least one from code-area OWNERS file) agreed to review. Reviewers from multiple companies preferred:
  • Approver (likely from SIG/area to which feature belongs):
  • Feature target (which target equals to which milestone):
    • Alpha release target (x.y)
    • Beta release target (x.y)
    • Stable release target (x.y)

Set the following:

  • Description
  • Assignee(s)
  • Labels:
    • stage/{alpha,beta,stable}
    • sig/*
    • kind/feature

Once this feature is appropriately updated, please explicitly ping @justaugustus, @kacole2, @robertsandoval, @rajendar38 to note that it is ready to be included in the Features Tracking Spreadsheet for Kubernetes 1.12.


Please note that the Features Freeze is July 31st, after which any incomplete Feature issues will require an Exception request to be accepted into the milestone.

In addition, please be aware of the following relevant deadlines:

  • Docs deadline (open placeholder PRs): 8/21
  • Test case freeze: 8/28

Please make sure all PRs for features have relevant release notes included as well.

Happy shipping!

Member

justaugustus commented Jul 18, 2018

@saad-ali --

It looks like this feature is currently in the Kubernetes 1.12 Milestone.

If that is still accurate, please ensure that this issue is up-to-date with ALL of the following information:

  • One-line feature description (can be used as a release note):
  • Primary contact (assignee):
  • Responsible SIGs:
  • Design proposal link (community repo):
  • Link to e2e and/or unit tests:
  • Reviewer(s) - (for LGTM) recommend having 2+ reviewers (at least one from code-area OWNERS file) agreed to review. Reviewers from multiple companies preferred:
  • Approver (likely from SIG/area to which feature belongs):
  • Feature target (which target equals to which milestone):
    • Alpha release target (x.y)
    • Beta release target (x.y)
    • Stable release target (x.y)

Set the following:

  • Description
  • Assignee(s)
  • Labels:
    • stage/{alpha,beta,stable}
    • sig/*
    • kind/feature

Once this feature is appropriately updated, please explicitly ping @justaugustus, @kacole2, @robertsandoval, @rajendar38 to note that it is ready to be included in the Features Tracking Spreadsheet for Kubernetes 1.12.


Please note that the Features Freeze is July 31st, after which any incomplete Feature issues will require an Exception request to be accepted into the milestone.

In addition, please be aware of the following relevant deadlines:

  • Docs deadline (open placeholder PRs): 8/21
  • Test case freeze: 8/28

Please make sure all PRs for features have relevant release notes included as well.

Happy shipping!

@justaugustus

This comment has been minimized.

Show comment
Hide comment
@justaugustus

justaugustus Jul 31, 2018

Member

@saad-ali --
Feature Freeze is today. Are we planning on graduating this feature in Kubernetes 1.12?
If so, can you make sure everything is up-to-date, so I can include it on the 1.12 Feature tracking spreadsheet?

Member

justaugustus commented Jul 31, 2018

@saad-ali --
Feature Freeze is today. Are we planning on graduating this feature in Kubernetes 1.12?
If so, can you make sure everything is up-to-date, so I can include it on the 1.12 Feature tracking spreadsheet?

@saad-ali saad-ali modified the milestones: v1.12, v1.13 Aug 1, 2018

@saad-ali saad-ali added stage/stable and removed stage/beta labels Aug 1, 2018

@saad-ali

This comment has been minimized.

Show comment
Hide comment
@saad-ali

saad-ali Aug 1, 2018

Member

@justaugustus for 1.12 we are planning on keeping CSI in beta. Planning for GA in 1.13

Member

saad-ali commented Aug 1, 2018

@justaugustus for 1.12 we are planning on keeping CSI in beta. Planning for GA in 1.13

@AishSundar

This comment has been minimized.

Show comment
Hide comment
@AishSundar

AishSundar Oct 17, 2018

@saad-ali whats left of this work to go to GA in 1.13? Are kubernetes/kubernetes#69690 and kubernetes/kubernetes#69688 the only 2 pending PRs for this feature?

AishSundar commented Oct 17, 2018

@saad-ali whats left of this work to go to GA in 1.13? Are kubernetes/kubernetes#69690 and kubernetes/kubernetes#69688 the only 2 pending PRs for this feature?

@jimangel

This comment has been minimized.

Show comment
Hide comment
@jimangel

jimangel Oct 17, 2018

Member

@saad-ali when this goes GA will there need to be any doc changes (1.13)?

Member

jimangel commented Oct 17, 2018

@saad-ali when this goes GA will there need to be any doc changes (1.13)?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment