Change mechanism of zone selection for dynamic provisioning

[gnufied]

Using AWS as an example, the current zone selection mechanism looks like following (assuming no zone is specified in StorageClass)

1. Filter all running instances in the region. The region information is usually derived from zone information configured in cloudprovider's config.
2. Make a set of zones from running instances in the region.
3. select a zone from list of zones in step#2 using round robin method (more or less)

This creates bunch of problems if there are more than one clusters running in the region or there are nodes which don't participate in Kube cluster.

Our current fix to solve this problem is - tag all instances participating in clustering with some sort of cluster-id (we have 2 distinct ways of tagging instances right now but leaving that aside). The tagging mechanism is very much tool specific and relying on it completely leads to surprising behaviour.

In stead of relying on running instances, can we not instead keep a set of zones via node information present in node informer and use this set for determining zone of PV?

This will most likely result in change in Provision interface because for most part- volume plugins don't initialize or have access to informers, so zone has to be selected before making Provision call or some other related change has to be done.

cc @jsafrane @kubernetes/sig-storage-api-reviews

[jsafrane]

What Cinder provisioner does is that it lists nodes from API server. They have region+zone labels and we can be sure they belong to the right cluster. On the other way, this list is not cached... Perhaps we could enhance Provision() call with a node informer?

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

Prevent issues from auto-closing with an /lifecycle frozen comment.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or @fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten
/remove-lifecycle stale

[msau42]

FWIW for gce, we added a call to the cloud provider to list the nodes from the informer and cache the zones.

[msau42]

ref #52322

[gnufied]

/remove-lifecycle stale

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close