[FEATURE] Local volume for distributed data workloads

[innobead]

Is your feature request related to a problem? Please describe

Longhorn is a highly available replica-based storage system, and it's good for fault tolerance, read performance, data protection, etc, but on the other side, it also needs some extra costs like requiring more disk paces for replication.

In some cases, especially for distributed data workloads (SS) like databases (ex: Cassandra, Kafka, etc), they already have their own data replication, sharding, etc, so we should provide a better volume type for these use cases but also still support existing specific volume functionalities like snapshotting, backup/restore, etc.

Describe the solution you'd like

• Extend Data Locality with strict or enforced mode to require one replica should be local and next to the workload
• Connecting the local replica via a local socket file instead of a TCP connection

Describe alternatives you've considered

N/A

Additional context

#1965

[derekbit]

As the benchmarking result, if the volume and the single replica are on the same node, the latency and iops can be improved significantly.

In addition, the tcp can be replaced with unix domain socket to gain more performance in this case.

[derekbit]

Replace the tcp connection between the engine and the single replica with unix-domain-socket.

• The IOPS and Latency are improved.
• The bandwidth number is saturated, so the improvement is not seen here.

[joshimoo]

@derekbit good job on the evaluation :)