Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

enabling CSIMigrationvSphere feature by default #103523

Merged

Conversation

divyenpatel
Copy link
Member

@divyenpatel divyenpatel commented Jul 6, 2021

What type of PR is this?

/kind feature

What this PR does / why we need it:

This PR is enabling CSIMigrationvSphere feature by default for the upcoming Kubernetes Release 1.25.

Which issue(s) this PR fixes:

Fixes #

Special notes for your reviewer:

Test Logs

https://gist.github.com/divyenpatel/f8e7afe6c67068b008de2f6bbd8c82f9

Does this PR introduce a user-facing change?

enabling CSIMigrationvSphere feature by default.

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:


@k8s-ci-robot k8s-ci-robot added release-note Denotes a PR that will be considered when it comes time to generate release notes. kind/feature Categorizes issue or PR as related to a new feature. size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. do-not-merge/needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Jul 6, 2021
@k8s-ci-robot
Copy link
Contributor

k8s-ci-robot commented Jul 6, 2021

@divyenpatel: This issue is currently awaiting triage.

If a SIG or subproject determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@k8s-ci-robot k8s-ci-robot added the needs-priority Indicates a PR lacks a `priority/foo` label and requires one. label Jul 6, 2021
@divyenpatel
Copy link
Member Author

divyenpatel commented Jul 6, 2021

/milestone 1.22

@k8s-ci-robot
Copy link
Contributor

k8s-ci-robot commented Jul 6, 2021

@divyenpatel: You must be a member of the kubernetes/milestone-maintainers GitHub team to set the milestone. If you believe you should be able to issue the /milestone command, please contact your and have them propose you as an additional delegate for this responsibility.

In response to this:

/milestone 1.22

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@xing-yang
Copy link
Contributor

xing-yang commented Jul 6, 2021

/milestone 1.22

@k8s-ci-robot
Copy link
Contributor

k8s-ci-robot commented Jul 6, 2021

@xing-yang: The provided milestone is not valid for this repository. Milestones in this repository: [next-candidate, v1.16, v1.17, v1.18, v1.19, v1.20, v1.21, v1.22, v1.23, v1.24, v1.25, v1.26]

Use /milestone clear to clear the milestone.

In response to this:

/milestone 1.22

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@xing-yang
Copy link
Contributor

xing-yang commented Jul 6, 2021

/milestone v1.22

@k8s-ci-robot k8s-ci-robot added this to the v1.22 milestone Jul 6, 2021
@xing-yang
Copy link
Contributor

xing-yang commented Jul 6, 2021

/sig storage
/sig cloud-provider

@k8s-ci-robot k8s-ci-robot added sig/storage Categorizes an issue or PR as relevant to SIG Storage. sig/cloud-provider Categorizes an issue or PR as relevant to SIG Cloud Provider. and removed do-not-merge/needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. labels Jul 6, 2021
@xing-yang
Copy link
Contributor

xing-yang commented Jul 6, 2021

/assign @msau42
/assign @Jiawei0227

@xing-yang
Copy link
Contributor

xing-yang commented Jul 6, 2021

/retest

@xing-yang
Copy link
Contributor

xing-yang commented Jul 6, 2021

/assign @gnufied

@xing-yang
Copy link
Contributor

xing-yang commented Jul 7, 2021

/retest

@gnufied
Copy link
Member

gnufied commented Jul 7, 2021

Couple of questions:

  1. Did we solve the NFS datastore issues in CSI driver? I did not hear a resolution or a deprecation notice for nfs datastore.
  2. In last release we deprecated bunch of previously supported stuff - deprecating in-tree vsphere volume diskformat parameters, vsphere less than 67u3, vm hardware less than 15 and multi vCenter support #98546 and our deprecation notice contains following message - "Support for these deprecations will be available till Kubernetes v1.24.". But enabling migration by default in 1.22 basically will make those parameters unsupported in 1.22 itself unless users explicitly disable the feature gate. Was this discussed as agreed path? I personally think - this will catch many users off-guard.

@divyenpatel
Copy link
Member Author

divyenpatel commented Jul 7, 2021

Did we solve the NFS datastore issues in CSI driver? I did not hear a resolution or a deprecation notice for nfs datastore.

First Class Disk used by vSphere CSI Driver does not support NFSv4 datastore. Refer to known VSLM issues here.

In last release we deprecated bunch of previously supported stuff and our deprecation notice contains following message - "Support for these deprecations will be available till Kubernetes v1.24.". But enabling migration by default in 1.22 basically will make those parameters unsupported in 1.22 itself unless users explicitly disable the feature gate.

We will continue to support deprecated parameters until 1.24. Users still have an option to disable migration on their cluster while upgrading to 1.22.

Was this discussed as agreed path? I personally think - this will catch many users off-guard.

Yes this was discussed in the sig architecture meeting. Refer to #98546 (comment) and meeting notes. - Section: "Agenda notes for February 11, 2021".

@gnufied
Copy link
Member

gnufied commented Jul 7, 2021

First Class Disk used by vSphere CSI Driver does not support NFSv4 datastore. Refer to known VSLM issues here.

Thanks for the link. In which case - we may have to deprecate NFSv4 datastore support completely with the intree driver. Also - are NFSv3 datastores supported by CSI driver? Can customers use that as a fallback? It should be noted that - if you add a NFS datastore in vcenter, vcenter recommends NFSv3 only for ESXI hosts older than 6.0. So I suspect - many folks will try and use NFSv4 with vsphere and fail.

@divyenpatel
Copy link
Member Author

divyenpatel commented Feb 9, 2022

/milestone v1.24

@k8s-ci-robot
Copy link
Contributor

k8s-ci-robot commented Feb 9, 2022

@divyenpatel: You must be a member of the kubernetes/milestone-maintainers GitHub team to set the milestone. If you believe you should be able to issue the /milestone command, please contact your and have them propose you as an additional delegate for this responsibility.

In response to this:

/milestone v1.24

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@xing-yang
Copy link
Contributor

xing-yang commented Feb 16, 2022

/milestone v1.24

@k8s-ci-robot k8s-ci-robot added this to the v1.24 milestone Feb 16, 2022
@gnufied
Copy link
Member

gnufied commented Mar 2, 2022

We discussed this today in CSI implementation meeting and since we only deprecated vSphere versions < 6.7u3 previously ( #98546 ) and since migration requires 7.0.1 at minimum - we will have to deprecate all versions of vSphere prior to 7.0.1 if we want to enable migration by default. Please consider this requirement before we can enable the migration.

@DiptoChakrabarty
Copy link
Member

DiptoChakrabarty commented Mar 22, 2022

I am from the v1.24 bug triage team , code freeze is on 03/30 so just wanted to remind to get this PR merged before that to target v1.24 :)

@xing-yang
Copy link
Contributor

xing-yang commented Mar 25, 2022

@DiptoChakrabarty We are delaying this PR to a later release.

@xing-yang
Copy link
Contributor

xing-yang commented Mar 25, 2022

/milestone clear

@k8s-ci-robot k8s-ci-robot removed this from the v1.24 milestone Mar 25, 2022
@k8s-ci-robot k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label May 6, 2022
@xing-yang
Copy link
Contributor

xing-yang commented Jun 17, 2022

/milestone v1.25

@k8s-ci-robot k8s-ci-robot added this to the v1.25 milestone Jun 17, 2022
@divyenpatel divyenpatel force-pushed the enable-vsphere-csi-migration branch from 9a601df to dfba13a Compare Jun 17, 2022
@k8s-ci-robot k8s-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jun 17, 2022
@Jiawei0227
Copy link
Contributor

Jiawei0227 commented Jun 21, 2022

Have we addressed concern that @gnufied has?

@gnufied
Copy link
Member

gnufied commented Jul 7, 2022

I believe kubernetes-sigs/vsphere-csi-driver#1653 issue should be a blocker before we can enable CSI migration as default for vsphere.

@divyenpatel
Copy link
Member Author

divyenpatel commented Jul 8, 2022

I believe kubernetes-sigs/vsphere-csi-driver#1653 issue should be a blocker before we can enable CSI migration as default for vsphere.

@gnufied I have fixed this issue in the vSphere CSI Driver.
kubernetes-sigs/vsphere-csi-driver#1854

Note: After this change in the vSphere CSI Driver, In-tree static vSphere PV will get migrated and registered as FCD when vSphere CSI Driver is installed with csi-migration feature-gate enabled, even when migration is not enabled on the kube-controller or kubelet.

cc: @jsafrane

@xing-yang
Copy link
Contributor

xing-yang commented Jul 12, 2022

/milestone v1.25

@divyenpatel
Copy link
Member Author

divyenpatel commented Jul 12, 2022

@Jiawei0227

@gnufied 's concern is addressed in the vSphere CSI Driver.

@xing-yang
Copy link
Contributor

xing-yang commented Jul 12, 2022

@Jiawei0227
@gnufied 's concern is addressed in the vSphere CSI Driver.

Merging this PR.

/lgtm
/approve

@k8s-ci-robot k8s-ci-robot added the lgtm Indicates that a PR is ready to be merged. label Jul 12, 2022
@k8s-ci-robot
Copy link
Contributor

k8s-ci-robot commented Jul 12, 2022

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: divyenpatel, xing-yang

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jul 12, 2022
@k8s-ci-robot k8s-ci-robot merged commit ec849f4 into kubernetes:master Jul 13, 2022
15 checks passed
@gnufied
Copy link
Member

gnufied commented Jul 14, 2022

We should also document that - k8s 1.25 drops (we previously deprecated) support for older versions. We previously agreed that:

  1. 1.24 is last version will support < 6.7u3, since CSI migration does not work properly on that version. So we should mention that k8s 1.25 will completely drop support for vsphere 6.7u3. Anyone installing or upgrading to 1.25 should not be on 6.7 or older versions.
  2. Support for versions older than 7.0u2 is something we agreed was iffy. In last release we deprecated support for <7.0u2. Since migration only works with caveats in those versions, we should remove support for anything <7.0u2 in k8s 1.25.
  3. I am not sure about Windows support, since it is still alpha but since at basic level it works, I think we need to call out.

cc @msau42 @jsafrane

We probably need to communicate this to wider community

@xing-yang
Copy link
Contributor

xing-yang commented Jul 24, 2022

We should also document that - k8s 1.25 drops (we previously deprecated) support for older versions. We previously agreed that:

1.24 is last version will support < 6.7u3, since CSI migration does not work properly on that version. So we should mention that k8s 1.25 will completely drop support for vsphere 6.7u3. Anyone installing or upgrading to 1.25 should not be on 6.7 or older versions.
Support for versions older than 7.0u2 is something we agreed was iffy. In last release we deprecated support for <7.0u2. Since migration only works with caveats in those versions, we should remove support for anything <7.0u2 in k8s 1.25.
I am not sure about Windows support, since it is still alpha but since at basic level it works, I think we need to call out.
cc @msau42 @jsafrane

We probably need to communicate this to wider community

Addressed by this PR: #111255
cc @gnufied @msau42 @jsafrane @Jiawei0227 @jingxu97

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. kind/feature Categorizes issue or PR as related to a new feature. lgtm Indicates that a PR is ready to be merged. needs-priority Indicates a PR lacks a `priority/foo` label and requires one. needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. release-note Denotes a PR that will be considered when it comes time to generate release notes. sig/cloud-provider Categorizes an issue or PR as relevant to SIG Cloud Provider. sig/storage Categorizes an issue or PR as relevant to SIG Storage. size/XS Denotes a PR that changes 0-9 lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet