White paper on security and cluster isolation for kubernetes.

[kerneltime]

No description provided.

[kerneltime]

Ran info

• Configuration issues

• Disk in use

• UUID issue even though enableUUID seems to be set

{"log":"I0213 09:48:43.502621       1 operation_executor.go:620] AttachVolume.Attach succeeded for volume \"kubernetes.io/vsphere-volume/[netapp01ads02] k8s/myDisk\" (spec.Na
me: \"test-volume\") from node \"k8s.minionpp-01\".\n","stream":"stderr","time":"2017-02-13T09:48:43.502834346Z"}
{"log":"I0213 09:48:43.599644       1 node_status_updater.go:135] Updating status for node \"k8s.minionpp-01\" succeeded. patchBytes: \"{\\\"status\\\":{\\\"volumesAtt
ached\\\":[{\\\"devicePath\\\":\\\"/dev/disk/by-id/wwn-0x6000c2931824b17ebd31f2dc365f0d67\\\",\\\"name\\\":\\\"kubernetes.io/vsphere-volume/[netapp01ads02] k8s/myDisk\\\"}]}}
\" VolumesAttached: [{kubernetes.io/vsphere-volume/[netapp01ads02] k8s/myDisk /dev/disk/by-id/wwn-0x6000c2931824b17ebd31f2dc365f0d67}]\n","stream":"stderr","time":"2017-02-13
T09:48:43.599797118Z"}
{"log":"E0213 09:48:46.750024       1 vsphere.go:1062] disk uuid not found for [netapp01ads02] k8s/myDisk. err: No disk UUID found\n","stream":"stderr","time":"2017-02-13T09:
48:46.750250047Z"}
{"log":"E0213 09:48:46.750107       1 vsphere.go:1044] Failed to check whether disk is attached. err: No disk UUID found\n","stream":"stderr","time":"2017-02-13T09:48:46.7502
7874Z"}

[cvauvarin]

@kerneltime the problem was a permission issue on the vCenter. Do you think it is possible to add in the documentation a list of permissions that are needed?

[kerneltime]

@cvauvarin yes of course, can you elaborate on the permissions problem? The last you mentioned that the disk was attached to the VM, I am not sure why permissions made a difference.

[cvauvarin]

Yes the disk was attached to the VM but the kube-controller could not get the UUID of the disk. What I did is that I used another user with full permissions on the vCenter and it worked without any issue. Then we tried to add some permissions to the other user to make it worked.

Here is the list of the permissions we applied :

• Datastore :
  • Allocate space
  • Browse datastore
  • Low level file operations
  • Update virtual machines files
  • Update virtual machnes metadata
• Virtual machine
  • Configuration
    • Add existing disk
    • Add new disk
    • Remove disk

Don't think it was a problem of reading the uuid, do you think it can be a problem writing some metadata ?

[kerneltime]

Thank you for the additional info.

[tusharnt]

This issue tracks the list of privileges the user needs to specify in vSphere UI in order to configure vSphere cloud provider.

[kerneltime]

Relevant https://bugzilla.eng.vmware.com/show_bug.cgi?id=1791819

[kerneltime]

A hacky reference between API spec and UI spec
auth-privs.txt

[kerneltime]

Partial list

Privileges

FindByIp 	   => System.View
MakeDirectory => Datastore.FileManagement #  https://bugzilla.eng.vmware.com/show_bug.cgi?id=1791819

[kerneltime]

The goal here is to have a white paper that explains what is possible to isolate the credentials used in kubernetes and what is the level of isolation achieved and gaps that a customer should be aware about.

[divyenpatel]

Updated Getting Started Guide with minimal set of privileges required for vSphere Cloud Provider
kubernetes/website#2989

[divyenpatel]

Updated k8s-anywhere prerequisites section with privileges required for Kubernetes-Anywhere.
kubernetes-retired/kubernetes-anywhere#360

[divyenpatel]

Updated documentation with required set of roles and permissions required for Kubernetes vsphere cloud provider.