-
Notifications
You must be signed in to change notification settings - Fork 584
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[FEATURE] Support multi-network K8s clusters (storage network) #2285
Comments
Considering moving this back to v1.2.0. |
Move out to v1.3.0. |
cc @jenting |
Interested. As I know, Harvester uses Multus, it'd benefit the Harvester project. Also, make sure the Longhorn engine <-> replica network throughput would not be affected by other applications. |
The components that get affected are the replica process, engine process, and iscsiadm. Currently, the challenge is how to overcome that Multus CNI is not visible on the host network to be used by iscsiadm. |
@joshimoo @keithalucas @derekbit any concerns for what @c3y1huang described? I don't think we need to deal with the communication between iscsi client and server (longhorn engine) over the storage network, but what we do care about are actually below:
|
The iSCSI server ( I agree that we only need to care about using the storage network interface for traffic between the longhorn engine and longhorn replicas for normal read and write operations and the traffic between replicas for syncing. |
Sorry I didn't follow this statement. If I understand correctly, the client and the server are always on the same node but in different network namespaces. And the connection is always a TCP connection. If we need to switch to using the storage network interface totally depends on how we consider the storage network. Is the storage network for data transmission only? Or it's for the whole storage system. There is no extra overhead for the switching. After checking the LEP, I would consider that the network is for the whole storage system. Then I prefer to use the storage network IP only in the engine (including launching tgt target). Otherwise, which IP we should use when invoking the APIs in the engine process would be a confusing part. |
Moved out of sprint to work on the dependent issue #2821 as discussed in #3277 (comment). |
Move this out of the sprint. |
cc @smallteeths |
Pre Ready-For-Testing Checklist
|
verified passed on v1.3.0-rc2 by following the manual test plan. created eth0 (10.0.1.0/24) and eth1 (10.0.2.0/24) for each instance, and let pods communicated with each other on eth1 using multus. |
Is your feature request related to a problem? Please describe.
A typical bare-metal server/network architecture involves servers being attached to multiple networks, including a dedicated storage fabric that provides accelerated throughput and IO separation from the data plane.
Longhorn does not support segregate networks and uses the default CNI network for all the storage traffic. That makes it impossible to guarantee network resources available for storage operations because the same network is shared with all in-cluster application traffic.
Describe the solution you'd like
The canonical approach to provide multi-network capabilities in a K8s cluster is using a CNI multiplexer such as Multus which provides the means to attach pods to different underlay networks.
Longhorn should have an integration with Multus.
That may be implemented by exposing a global setting allowing the admin to specify the Multus network that should be used for any storage-related traffic (e.g. engine -> replica). Longhorn would then configure the pods with the Multus specific annotations, e.g.
k8s.v1.cni.cncf.io/networks: storage-net
.The setup and configuration of the Multus network (e.g. creation of
NetworkAttachmentDefinition
) would be out of the scope of Longhorn.Describe alternatives you've considered
N/A
The text was updated successfully, but these errors were encountered: