[FEATURE] Support multi-network K8s clusters (storage network)

[janeczku]

Is your feature request related to a problem? Please describe.

A typical bare-metal server/network architecture involves servers being attached to multiple networks, including a dedicated storage fabric that provides accelerated throughput and IO separation from the data plane.

Longhorn does not support segregate networks and uses the default CNI network for all the storage traffic. That makes it impossible to guarantee network resources available for storage operations because the same network is shared with all in-cluster application traffic.

Describe the solution you'd like

The canonical approach to provide multi-network capabilities in a K8s cluster is using a CNI multiplexer such as Multus which provides the means to attach pods to different underlay networks.

Longhorn should have an integration with Multus.
That may be implemented by exposing a global setting allowing the admin to specify the Multus network that should be used for any storage-related traffic (e.g. engine -> replica). Longhorn would then configure the pods with the Multus specific annotations, e.g. k8s.v1.cni.cncf.io/networks: storage-net.

The setup and configuration of the Multus network (e.g. creation of NetworkAttachmentDefinition) would be out of the scope of Longhorn.

Describe alternatives you've considered
N/A

[yasker]

Considering moving this back to v1.2.0.

[yasker]

Move out to v1.3.0.

[innobead]

cc @jenting

[jenting]

Interested.

As I know, Harvester uses Multus, it'd benefit the Harvester project.

Also, make sure the Longhorn engine <-> replica network throughput would not be affected by other applications.

[c3y1huang]

For the longhorn-engine replica part, the data server and sync-agent server are the data plane. Having the setting allows the user to specify which network interface is the data plane. At the longhorn engine, it gets the network interface IPs from the Pod and creates the data and sync-agent server with that interface IP address. (Need a way to report error if the network interface not found) Change the setting need to restart the longhorn-engine Pods to recreate the data and sync-agent server (need to consider should the volume need to be detached or not)

The components that get affected are the replica process, engine process, and iscsiadm. Currently, the challenge is how to overcome that Multus CNI is not visible on the host network to be used by iscsiadm.

[innobead]

For the longhorn-engine replica part, the data server and sync-agent server are the data plane. Having the setting allows the user to specify which network interface is the data plane. At the longhorn engine, it gets the network interface IPs from the Pod and creates the data and sync-agent server with that interface IP address. (Need a way to report error if the network interface not found) Change the setting need to restart the longhorn-engine Pods to recreate the data and sync-agent server (need to consider should the volume need to be detached or not)

The components that get affected are the replica process, engine process, and iscsiadm. Currently, the challenge is how to overcome that Multus CNI is not visible on the host network to be used by iscsiadm.

@joshimoo @keithalucas @derekbit any concerns for what @c3y1huang described? I don't think we need to deal with the communication between iscsi client and server (longhorn engine) over the storage network, but what we do care about are actually below:

• data traffic between engine and replicas
• data traffic between replicas

[keithalucas]

For the longhorn-engine replica part, the data server and sync-agent server are the data plane. Having the setting allows the user to specify which network interface is the data plane. At the longhorn engine, it gets the network interface IPs from the Pod and creates the data and sync-agent server with that interface IP address. (Need a way to report error if the network interface not found) Change the setting need to restart the longhorn-engine Pods to recreate the data and sync-agent server (need to consider should the volume need to be detached or not)

The components that get affected are the replica process, engine process, and iscsiadm. Currently, the challenge is how to overcome that Multus CNI is not visible on the host network to be used by iscsiadm.

@joshimoo @keithalucas @derekbit any concerns for what @c3y1huang described? I don't think we need to deal with the communication between iscsi client and server (longhorn engine) over the storage network, but what we do care about are actually below:

• data traffic between engine and replicas
• data traffic between replicas

The iSCSI server (iscsid) runs outside of a pod on a node. iscsiadm instructs iscsid to establish a connection with the tgt process running on the node, but within a pod (the engine instance manager pod). The iscsid to tgt communication is local to that node and shouldn't be transported over the network at all, so we don't need to configure it to use the storage network interface.

I agree that we only need to care about using the storage network interface for traffic between the longhorn engine and longhorn replicas for normal read and write operations and the traffic between replicas for syncing.

[shuo-wu]

The iscsid to tgt communication is local to that node and shouldn't be transported over the network at all

Sorry I didn't follow this statement. If I understand correctly, the client and the server are always on the same node but in different network namespaces. And the connection is always a TCP connection. If we need to switch to using the storage network interface totally depends on how we consider the storage network. Is the storage network for data transmission only? Or it's for the whole storage system. There is no extra overhead for the switching.

After checking the LEP, I would consider that the network is for the whole storage system. Then I prefer to use the storage network IP only in the engine (including launching tgt target). Otherwise, which IP we should use when invoking the APIs in the engine process would be a confusing part.

[c3y1huang]

Moved out of sprint to work on the dependent issue #2821 as discussed in #3277 (comment).

[c3y1huang]

Move this out of the sprint.
#3546 (comment)

[innobead]

cc @smallteeths

[longhorn-io-github-bot]

Pre Ready-For-Testing Checklist

• Where is the reproduce steps/test steps documented?
  The reproduce steps/test steps are at:

• Is there a workaround for the issue? If so, where is it documented?
  The workaround is at:

• Does the PR include the explanation for the fix or the feature?

• Does the PR include deployment change (YAML/Chart)? If so, where are the PRs for both YAML file and Chart?
  The PR for the YAML change is at: Storage network deployment update #3990
  The PR for the chart change is at: Storage network deployment update #3990

• Have the backend code been merged (Manager, Engine, Instance Manager, BackupStore etc) (including backport-needed/*)?
  The PR is at

• Implement storage network through gRPC proxy longhorn-manager#1212
• Implement storage-network backing-image-manager#52
• Implement gRPC proxy longhorn-instance-manager#105

• Which areas/issues this PR might have potential impacts on?
  Area manager, instance-manager, backing-image-manager
  Issues

• If labeled: require/LEP Has the Longhorn Enhancement Proposal PR submitted?
  The LEP PR is at Add LEP for Storage Network Through gRPC Proxy #3277

• If labeled: area/ui Has the UI issue filed or ready to be merged (including backport-needed/*)?
  The UI issue/PR is at

• If labeled: require/doc Has the necessary document PR submitted or merged (including backport-needed/*)?
  The documentation issue/PR is at:

• [DOC] Add storage network setting document #4081
• storage-network: add doc website#509

• If labeled: require/automation-e2e Has the end-to-end test plan been merged? Have QAs agreed on the automation test case? If only test case skeleton w/o implementation, have you created an implementation issue (including backport-needed/*)
  The automation skeleton PR is at
  The automation test case PR is at storage-network: do not reset setting longhorn-tests#984
  The issue of automation test case implementation is at (please create by the template)

• If labeled: require/automation-engine Has the engine integration test been merged (including backport-needed/*)?
  The engine automation PR is at

• If labeled: require/manual-test-plan Has the manual test plan been documented?
  The updated manual test plan is at storage-network: manual setup and test longhorn-tests#980

• If the fix introduces the code for backward compatibility Has a separate issue been filed with the label release/obsolete-compatibility?
  The compatibility issue is filed at

[yangchiu]

verified passed on v1.3.0-rc2 by following the manual test plan.

created eth0 (10.0.1.0/24) and eth1 (10.0.2.0/24) for each instance, and let pods communicated with each other on eth1 using multus.
remember to stop source/destination check else the volume replicas would keep fail.