Feature: Implements Raw Block Volume Support for in-tree vSphere plugin #68761

fanzhangio · 2018-09-18T00:34:39Z

add volume mode checking and set fstype for any FSType
set VolumeMode in PV spec
implement BlockVolumePlugin, BlockVolumeMapper, BlockVolumeUnmapper interfaces.
add tests for vsphere block volume

What this PR does / why we need it:
vSphere should support raw block volume support
https://kubernetes.io/docs/concepts/storage/persistent-volumes/#raw-block-volume-support

Which issue(s) this PR fixes:
Fixes #68762

Special notes for your reviewer:

Release note:

vSphereVolume implements Raw Block Volume Support

fanzhangio · 2018-09-18T04:19:39Z

/test pull-kubernetes-e2e-gce-100-performance
/test pull-kubernetes-kubemark-e2e-gce-big

k8s-ci-robot · 2018-09-18T08:56:20Z

@fanzhangio: You must be a member of the kubernetes/kubernetes-milestone-maintainers github team to set the milestone.

In response to this:

/milestone v1.12

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

fanzhangio · 2018-09-21T06:39:53Z

@vladimirvivien Could you plz take a look at this PR ?

vladimirvivien · 2018-09-21T12:29:36Z

Hi @fanzhangio
For proper block volume implementation, you will need to implement a three interfaces:

BlockVolumePlugin - https://github.com/kubernetes/kubernetes/blob/master/pkg/volume/plugins.go#L255:6
BlockVolumeMapper - https://github.com/kubernetes/kubernetes/blob/master/pkg/volume/volume.go#L150:6
BlockVolumeUnmapper - https://github.com/kubernetes/kubernetes/blob/master/pkg/volume/volume.go#L171

Take a look at gce_pd_block to get an idea what is needed
https://github.com/kubernetes/kubernetes/blob/master/pkg/volume/gce_pd/gce_pd_block.go

SandeepPissay · 2018-10-01T03:10:20Z

pkg/volume/vsphere_volume/vsphere_volume_block.go

+	"k8s.io/kubernetes/pkg/volume/util/volumepathhandler"
+)
+
+var _ volume.BlockVolumePlugin = &vsphereVolumePlugin{}


The existing vsphere volume plugin already support creating/attaching/detaching/deleting block volumes. Could you explain what is the additional functionality specific to vSphere are we getting for block volumes with this PR?

@SandeepPissay I think this PR is for raw block device support.

https://kubernetes.io/docs/concepts/storage/persistent-volumes/#raw-block-volume-support

Thanks @divyenpatel ! Yes, it is raw block volume support. I am going to modify the PR title for more clarity. @SandeepPissay

vladimirvivien · 2018-11-05T21:51:15Z

/lgtm

vladimirvivien · 2018-11-05T21:52:59Z

@SandeepPissay @divyenpatel PTAL, this PR needs approval. Thanks.

divyenpatel · 2018-11-05T22:59:27Z

pkg/volume/vsphere_volume/vsphere_volume_block.go

+	blkUtil := volumepathhandler.NewBlockVolumePathHandler()
+	globalMapPathUUID, err := blkUtil.FindGlobalMapPathUUIDFromPod(pluginDir, mapPath, podUID)
+	if err != nil {
+		return nil, err


Would be good for debugging if we can print error before returning.

glog.Errorf("Failed to find GlobalMapPathUUID from Pod: %s with error: %+v", podUID, err) return nil, err

pkg/volume/vsphere_volume/vsphere_volume_block.go

divyenpatel · 2018-11-05T23:01:47Z

pkg/volume/vsphere_volume/vsphere_volume_block.go

+func (plugin *vsphereVolumePlugin) newBlockVolumeMapperInternal(spec *volume.Spec, podUID types.UID, manager vdManager, mounter mount.Interface) (volume.BlockVolumeMapper, error) {
+	volumeSource, _, err := getVolumeSource(spec)
+	if err != nil {
+		return nil, err


log error before return.

pkg/volume/vsphere_volume/vsphere_volume.go

divyenpatel · 2018-11-05T23:17:48Z

@fanzhangio can you please provide testing details?
Can you share YAMLs that you have tried out to access raw block volume in the pod with vSphere cloud provider?

fanzhangio · 2018-11-07T19:10:28Z

Hi @divyenpatel, here is E2E test by steps:

provision VC and Hosts.
provision VMs as k8s nodes
deploy k8s cluster enabling feature gates BlockVolume=true and cloud provider intree
Run intree e2e tests towards this VC
Test PV, PVC yaml on cluster

fanzhangio · 2018-11-07T19:12:57Z

YAMLs

Persistent Volume Claim requesting a Raw Block Volume

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: block-pvc
  annotations:
    volume.beta.kubernetes.io/storage-class: thin-disk
spec:
  accessModes:
    - ReadWriteOnce
  volumeMode: Block
  resources:
    requests:
      storage: 2Gi

Create Storage Class

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: thin-disk
provisioner: kubernetes.io/vsphere-volume
parameters:
    datastore: sharedVmfs-0

Persistent Volume

apiVersion: v1
kind: PersistentVolume
metadata:
  name: raw-pv
spec:
  volumeMode: Block
  capacity:
    storage: 2Gi
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  vsphereVolume:
    volumePath: “[datasotre1] volumes/myDisk”

divyenpatel · 2018-11-07T19:22:33Z

@fanzhangio I do not see in the diff, following check is removed pkg/volume/vsphere_volume/vsphere_volume.go

if util.CheckPersistentVolumeClaimModeBlock(v.options.PVC) {
 		return nil, fmt.Errorf("%s does not support block volume provisioning", v.plugin.GetPluginName())
 	}

- add raw block volume support for vsphere volume provisioner - set VolumeMode and any FSType in vsphere volume dynamic provision if feature BlockVolume enabled - implement BlockVolumePlugin, BlockVolumeMapper, BlockVolumeUnmapper interfaces. - add tests for vsphere block volume

fanzhangio · 2018-11-07T20:50:47Z

/test pull-kubernetes-integration

divyenpatel · 2018-11-07T20:57:54Z

/assign @divyenpatel

divyenpatel · 2018-11-07T20:58:04Z

/approve

k8s-ci-robot · 2018-11-07T20:58:12Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: divyenpatel, fanzhangio

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~pkg/volume/vsphere_volume/OWNERS~~ [divyenpatel]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

divyenpatel · 2018-11-07T21:02:49Z

/lgtm

fanzhangio · 2018-11-07T21:45:02Z

/kind feature

fanzhangio · 2018-11-08T00:05:38Z

/test pull-kubernetes-e2e-gce

k8s-ci-robot added release-note Denotes a PR that will be considered when it comes time to generate release notes. size/S Denotes a PR that changes 10-29 lines, ignoring generated files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. labels Sep 18, 2018

k8s-ci-robot requested review from rootfs and saad-ali September 18, 2018 00:34

fanzhangio force-pushed the vsphere-volume branch from 9a87a71 to 9bd0cf5 Compare September 18, 2018 00:35

fanzhangio force-pushed the vsphere-volume branch from 9bd0cf5 to 85e1b27 Compare September 25, 2018 06:50

k8s-ci-robot removed the size/S Denotes a PR that changes 10-29 lines, ignoring generated files. label Sep 25, 2018

fanzhangio changed the title ~~Add vsphere block volume support~~ (WIP) Add vsphere block volume support Sep 25, 2018

k8s-ci-robot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. labels Sep 25, 2018

fanzhangio force-pushed the vsphere-volume branch 7 times, most recently from 04b1d6e to 44a06bf Compare September 30, 2018 18:54

fanzhangio changed the title ~~(WIP) Add vsphere block volume support~~ Feature: Implement vSphere block volume Sep 30, 2018

k8s-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Sep 30, 2018

SandeepPissay reviewed Oct 1, 2018

View reviewed changes

fanzhangio changed the title ~~Feature: Implement vSphere block volume~~ Feature: Implement Raw vSphere Block Volume Support Oct 1, 2018

fanzhangio force-pushed the vsphere-volume branch from e2e0900 to 691f756 Compare October 29, 2018 15:57

k8s-ci-robot assigned vladimirvivien Nov 5, 2018

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Nov 5, 2018

divyenpatel reviewed Nov 5, 2018

View reviewed changes

pkg/volume/vsphere_volume/vsphere_volume.go Show resolved Hide resolved

fanzhangio force-pushed the vsphere-volume branch from 691f756 to 6c8dd1a Compare November 7, 2018 12:59

k8s-ci-robot removed the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Nov 7, 2018

fanzhangio force-pushed the vsphere-volume branch from 6c8dd1a to dcb3a5f Compare November 7, 2018 20:16

divyenpatel approved these changes Nov 7, 2018

View reviewed changes

k8s-ci-robot assigned divyenpatel Nov 7, 2018

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Nov 7, 2018

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Nov 7, 2018

k8s-ci-robot added kind/feature Categorizes issue or PR as related to a new feature. and removed needs-kind Indicates a PR lacks a `kind/foo` label and requires one. labels Nov 7, 2018

k8s-ci-robot merged commit e4c5d45 into kubernetes:master Nov 8, 2018

fanzhangio mentioned this pull request Nov 9, 2018

Implements Raw Block Volume Support for in-tree vSphere plugin kubernetes/enhancements#626

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Feature: Implements Raw Block Volume Support for in-tree vSphere plugin #68761

Feature: Implements Raw Block Volume Support for in-tree vSphere plugin #68761

fanzhangio commented Sep 18, 2018 •

edited

Loading

fanzhangio commented Sep 18, 2018

k8s-ci-robot commented Sep 18, 2018

fanzhangio commented Sep 21, 2018

vladimirvivien commented Sep 21, 2018

SandeepPissay Oct 1, 2018

divyenpatel Oct 1, 2018 •

edited

Loading

fanzhangio Oct 1, 2018

vladimirvivien commented Nov 5, 2018

vladimirvivien commented Nov 5, 2018

divyenpatel Nov 5, 2018

fanzhangio Nov 7, 2018

divyenpatel Nov 5, 2018

fanzhangio Nov 7, 2018

divyenpatel commented Nov 5, 2018

fanzhangio commented Nov 7, 2018 •

edited

Loading

fanzhangio commented Nov 7, 2018 •

edited

Loading

divyenpatel commented Nov 7, 2018 •

edited

Loading

fanzhangio commented Nov 7, 2018

divyenpatel commented Nov 7, 2018

divyenpatel commented Nov 7, 2018

k8s-ci-robot commented Nov 7, 2018

divyenpatel commented Nov 7, 2018

fanzhangio commented Nov 7, 2018

fanzhangio commented Nov 8, 2018

Feature: Implements Raw Block Volume Support for in-tree vSphere plugin #68761

Feature: Implements Raw Block Volume Support for in-tree vSphere plugin #68761

Conversation

fanzhangio commented Sep 18, 2018 • edited Loading

fanzhangio commented Sep 18, 2018

k8s-ci-robot commented Sep 18, 2018

fanzhangio commented Sep 21, 2018

vladimirvivien commented Sep 21, 2018

SandeepPissay Oct 1, 2018

Choose a reason for hiding this comment

divyenpatel Oct 1, 2018 • edited Loading

Choose a reason for hiding this comment

fanzhangio Oct 1, 2018

Choose a reason for hiding this comment

vladimirvivien commented Nov 5, 2018

vladimirvivien commented Nov 5, 2018

divyenpatel Nov 5, 2018

Choose a reason for hiding this comment

fanzhangio Nov 7, 2018

Choose a reason for hiding this comment

divyenpatel Nov 5, 2018

Choose a reason for hiding this comment

fanzhangio Nov 7, 2018

Choose a reason for hiding this comment

divyenpatel commented Nov 5, 2018

fanzhangio commented Nov 7, 2018 • edited Loading

fanzhangio commented Nov 7, 2018 • edited Loading

Persistent Volume Claim requesting a Raw Block Volume

Create Storage Class

Persistent Volume

divyenpatel commented Nov 7, 2018 • edited Loading

fanzhangio commented Nov 7, 2018

divyenpatel commented Nov 7, 2018

divyenpatel commented Nov 7, 2018

k8s-ci-robot commented Nov 7, 2018

divyenpatel commented Nov 7, 2018

fanzhangio commented Nov 7, 2018

fanzhangio commented Nov 8, 2018

fanzhangio commented Sep 18, 2018 •

edited

Loading

divyenpatel Oct 1, 2018 •

edited

Loading

fanzhangio commented Nov 7, 2018 •

edited

Loading

fanzhangio commented Nov 7, 2018 •

edited

Loading

divyenpatel commented Nov 7, 2018 •

edited

Loading