Support for Multiple Tenant Access Pods

[caitlinbestler]

The goal of multiple tenant access pods is to enable access to the same backend service, presumably storage, from multiple tenants each with their own tenant scoped network access.

Kubernetes already supports the ability to define a network namespace to which only specified containers have access. This is typically enforced by VLANs or VxLANs.

A typical tenant-scoped virtual network would contain client pods, a pod implementing a user authentication service (usually AD or LDAP) and pods providing access to the storage cluster.

This feature would enable a single pre-existing cluster to provide service to multiple tenants each via a virtual frontend network defined for that client. There may also be a backend storage network defined for the storage cluster, but the backend storage network would pre-exist any client access network.

Providing a shared storage cluster enables economies of scale in providing storage. But there are no economies of scale that would be acceptable if it enabled leakage of content from one tenant to another. Tenant isolation must exist independent of whatever Access Control is provided by the storage cluster itself.

Layering of access control enables re-use of open-source daemons designed for legacy protocols. NFS and iscsi targets are traditionally designed to provide service in a single corporate intranet, not to the world wide internet. Secondly, enforcing tenant isolation in the network itself provides guaranteed isolation that is not
dependent on the storage servers. Tenant isolation is easily understood, ACLs require reading manuals.

The proposed life-cycle is as follows:

1. Provision the storage cluster. Giving it a persistent identity. For this issue we will leave issues on how to identify the hosts for this cluster for the existing multiple storage provider issue to deal with. However the cluster was provisioned there are now a set of hosts that have been marked as eligible for scheduling Tenant Access Pods.
2. Tenant Access networks can then be added (or dropped). Adding them involves creating tenant client pods subject to normal Kubernetes scheduling rules and Tenant Access Pods which could only be scheduled on hosts previously marked as supporting Tenant Access Pods for this specific storage cluster.
3. When the Tenant Access Pod is activated it will establish communication with the storage cluster backend pod already running on the same host. Those pods will continue to communicate based on their own preference. They may continue to use generic IPC, establish message queues in shared memory or even via sharing files in a shared directory (although I wouldn't recommend that approach). If the storage pod is flexible enough, the Tenant Access Pod could even just identify the Virtual Network interface to the backend pod.
4. The backend pod would be responsible for confining all work done for a Tenant Access Pod to resources owned by the matching tenant. This typically means that the set of mount points available is limited to those created for that specific tenant.
5. Each frontend pod would identify, and provide proxy access to tenant-specific authentication services. The backend storage service pod needs to know what role each tenant user has been admitted under so that it can enforce any file/object/volume ACL rules that it is implementing. The syntax of the ACL rules will be tenant independent, but the method of validating identity before the ACL rules can be applied will be tenant specific.

Ideally, no backend storage pod could be deactivated while Tenant Access pods are still active on the same host. In general, storage cluster pods are very poor candidates to migrate because their purpose is to provide persistent storage. Providing persistent storage is incompatible with quick migration. So even without a special hook to enforce this it is probably not a major issue.

[dyusupov]

Currently kubernetes cannot provide "multi-homed" or true "multi-tenant" networking isolation. But good news is that Network plumbers folks working on it:

https://docs.google.com/document/d/1Ny03h6IDVy_e_vmElOqR7UdTPAG_RNydhVE1Kx54kFQ/edit

Meanwhile, multi-homed network can be fairly easily constructed with Intel's "multus" CNI plugin:

https://github.com/Intel-Corp/multus-cni

[caitlinbestler]

Kubernetes does not yet provide "multi-homed" or "multi-tenant" Pods. It is not clear when Intel's Multus project will provide plug-and-play Virtual NICs rather than just multiple Virtual NICs.

To support on-demand creation and deletion of Tenants we need to add and drop the Tenant Access networks. What I am proposing is doing this by having single interface Pods connect with each other through localhost IPC, rather than through Virtual NICs.

Without dynamic creation I am not certain how any number of networks that must be known when the storage cluster is provisioned could support the potentially large number of Tenants that a single storage cluster might have to support across all gateways.

[caitlinbestler]

Clarification:
Where the Intel Multus project is relevant is in provisioning the Backend Pod to use a 2nd NIC for the Backend Network.

Without that it would either:

• Have to be controlled by both NICs, wth Kubernetes still understanding that it was one host.
• Have Kubernetes control if via the front end NIC but only provision the POD to access the backend NIC.
• Provision the backend NIC as an anonymous non-network device that just happens to provide network services.

It would also be ideal if Kubernetes could restrict which Backend Pods any specific frontend Pod could communicate with. This could be solved by only provisioning a single Backend Pod per host and only provisioning frontend pods to that host which are authorized to communicate with it. In other words, if a
Pod does not require access to a Storage Clusgter X host then do not schedule it there.

[caitlinbestler]

The following blogs are relevant for explaining the rationale of this proposal:
https://nexenta.github.io/blog/2018/04/12/BeyondVirtualDisks.html
https://nexenta.github.io/blog/2018/04/12/MultipleTenantAccessToSharedStorage.html

[travisn]

@caitlinbestler could you summarize how Rook can help with this design? There appears to be a number of broader design issues around networking and multi-tenancy than what Rook could help with. Rook is more about orchestrating the needs of the storage backend. But the networking and data path is ultimately outside Rook.

[caitlinbestler]

While network related code would remain a Kubernetes issue, there are a few areas where specific kubernetes strategies can be of benefit to all storage clusters:

• Enabling multiple-tenant isolated access to a shared storage cluster, as opposed to needing to create tenant specific storage clusters.
• Provisioning high-performance low latency backend networks that where switchports can be dynamically assigned to storage networks. Of course "dynamically" here is at the rate where you add/remove a storage server from a storage cluster, which is considerably less often than a compute resource is added/removed from a compute job.
  While the high-performance low-latency backend network may be of use for some compute clusters, there are enough issues that are of common interest to storage clusters that it makes sense to work on a consensus template in a more focused group than doing so in the kubernetes community at large.

Plus, Rook's efforts to define storage clusters in a somewhat vendor neutral way should be aware of these issues, and may even mandate that the previously mentioned templates be followed if you want to be a Rook-compatible storage cluster. Vendors are told that they can schedule their network resources however they want to, but Rook will only understand what they are doing if you do as specified by some Rook published HOW-TO document.

[travisn]

Certainly, wherever there is a common benefit to the storage clusters, Rook either documenting or facilitating the feature is the goal. With multi-tenant access to the storage, however, I am struggling to see where the commonality is. The data protocol is likely where the tenant enforcement comes. For example, object naturally has users that present their creds to the s3 interface, and ceph rgw has a multi-tenant capability. Fundamentally, either the data protocol needs to support multi-tenancy natively, or else the networking needs to keep the tenants isolated.

As mentioned in your design discussion, another option is to have tenant access pods. In that case, Rook would orchestrate starting those access pods. I could see Rook defining a CRD that allows the multi-tenant access via the access pods. Does Nexenta have tenant access pods today? I'm not sure how commonly Ceph users would find the tenant access pods desirable since it would not give full performance access to the OSDs.

[caitlinbestler]

Some of Ceph's storage services are multi-tenant. Their NFS also uses Ganesha, and hence each instance is single-tenant. Multi-tenant protocols, such as Ceph rgw or S3, can isolate users based on tenant-specific authentication but do expose the service to non-tenant DoS attacks. NexentaEdge prefers to use a dedicated backend storage network so that it can manage congestion control itself. Quotas can be enforced in the proxy/gateway machines to preserve long-term fairness while allowing the full resources of the backend network to be allocated to finishing admitted requests as quickly as possible. This would be essential whenever a backend storage network was used (NexentaEdge, FCoE, FC, InfiniBand, NVMe over Fabric, etc.) and typically is beneficial even when using plain TCP/IP networking. Storage and foreground traffic do not intermix that well, which is why backend storage networks have persisted even as the technologies behind them have come and gone. But you are correct that multi-tenant pods are not essential to supporting multiple storage clusters, which is why I proposed this as a separate feature rather than an amendment to the base proposal. Adding support for multi-tenant access pods is something that should be done *after* multiple storage backends is done. Because object protocols support multi-tenant logis the impact will be loss of total opaqueness for object storage protocols (i.e. the list of buckets may be visible to others) and being forced to clone volume and NFS daemon instances. Since the set of mountable volumes (iscsi or NFS) is tenant specific the impact of multiple daemons will mostly be on resource allocation. Per-tenant daemons will waste cache space (RAM and disk), etc.

…

On 4/20/18 2:44 PM, Travis Nielsen wrote: Certainly, wherever there is a common benefit to the storage clusters, Rook either documenting or facilitating the feature is the goal. With multi-tenant access to the storage, however, I am struggling to see where the commonality is. The data protocol is likely where the tenant enforcement comes. For example, object naturally has users that present their creds to the s3 interface, and ceph rgw has a multi-tenant capability <http://docs.ceph.com/docs/master/radosgw/multitenancy/>. Fundamentally, either the data protocol needs to support multi-tenancy natively, or else the networking needs to keep the tenants isolated. As mentioned in your design discussion, another option is to have tenant access pods. In that case, Rook would orchestrate starting those access pods. I could see Rook defining a CRD that allows the multi-tenant access via the access pods. Does Nexenta have tenant access pods today? I'm not sure how commonly Ceph users would find the tenant access pods desirable since it would not give full performance access to the OSDs. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1610 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABwS9mhNnpdlVmKdPZRz4b-9WQJ5mIucks5tqlbTgaJpZM4S9fgt>.

[travisn]

Good point that NFS Ganesha is similar. Wherever there is similarity, that's where we can get the benefits of sharing the platform. And as we add more backends it will become more clear the places where there is overlap that will benefit from common orchestration.

[jbw976]

There has been recent discussion on this (related) topic for providing support for multiple network interfaces, not necessarily from the multi-tenant perspective, but from a perspective of a storage solution being able to separate it's backend (internal) traffic from its frontend (client) traffic. This is related to this issue, but perhaps would fit better in a new issue altogether.

Recent discussion on this topic: #2408 (comment)

[caitlinbestler]

The precise relationship is that there is one internal network and one or more public/client networks (ideally one for each tenant). The reason for keeping them separate is that the lifespans of the networks are different. Internal vs external networks can be specified statically. Multiple client networks have to be created and provisioned as tenants are added or dropped, which requires more configuration sleight of hand than just having internal and external networks.

…

________________________________ From: Jared Watts <notifications@github.com> Sent: Tuesday, February 5, 2019 8:00:15 PM To: rook/rook Cc: Caitlin Bestler; Mention Subject: Re: [rook/rook] Support for Multiple Tenant Access Pods (#1610) There has been recent discussion on this (related) topic for providing support for multiple network interfaces, not necessarily from the multi-tenant perspective, but from a perspective of a storage solution being able to separate it's backend (internal) traffic from its frontend (client) traffic. This is related to this issue, but perhaps would fit better in a new issue altogether. Recent discussion on this topic: #2408 (comment)<#2408 (comment)> — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<#1610 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/ABwS9i8dGUA0DRe9tHGlyJI22tusWFc4ks5vKlNPgaJpZM4S9fgt>.

[jseguillon]

Hi there. Sorry to be late but it is unclear to me if #2048 made this available : choose which interface in Pod should route/listen for Cephs traffic ?

If nothing available "in the box", does anyone know if it's feasible if we tune routes and start command lines or is there no way to achieve this ?

(Interfaces provisioned by multus, one 1G on control plane, the other one on 10 G interface)

[caitlinbestler]

The Pod has a need to connect with an "internal storage network" with a certain QoS, and one or mroe "client networks" which provide service for a set of tenants. Those are matchanable attributes.

…

________________________________ From: Joel Seguillon <notifications@github.com> Sent: Friday, April 19, 2019 11:28:12 AM To: rook/rook Cc: Caitlin Bestler; Mention Subject: Re: [rook/rook] Support for Multiple Tenant Access Pods (#1610) Hi there. Sorry to be late but it is unclear to me if #2048<#2048> made this available : choose which interface in Pod should route/listen for Cephs traffic ? If nothing available "in the box", does anyone know if it's feasible if we tune routes and start command lines or is there no way to achieve this ? (Interfaces provisioned by multus, one 1G on control plane, the other one on 10 G interface) — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<#1610 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AAOBF5WOLMJRN452SHJEQ73PRIFLZANCNFSM4EXV7AWQ>.

[travisn]

It's still not clear to me what Rook can provide here. At the K8s and network layer there are features such as supporting ipv4/v6 dual stack, multus, etc. Each storage provider will need to implement the multi-tenancy. We can discuss again how it can be shared after one storage provider implements something here.