High performance local storage

[lomori]

HI,

Let's say I have a high-performance storage drive, e.g. NVME, attached to a host. We can mount that somewhere and configure Longhorn to use it. A few questions:

1. Since replicas are done synchronously, to obtain the best performance possible, we have to set the number of replicas to 1, right? Otherwise, network traffic would slow down access.
2. Any particular reason replicas are done synchronously? If Longhorn is RWO, there won't be anybody else in a different node accessing a replica. Or could it?
3. Do you have an idea of the overhead Longhorn imposes on local disk access?

Regards,
Luiz

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[unixfox]

I'm still interested into that feature.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[unixfox]

I'm still interested into that feature.

[yasker]

We recently have published a performance report at https://longhorn.io/blog/performance-scalability-report-aug-2020/

1. Since replicas are done synchronously, to obtain the best performance possible, we have to set the number of replicas to 1, right? Otherwise, network traffic would slow down access.

For the best write performance, yes the number of replicas should be 1. But we can provide a better read performance if there are more replicas.

2. Any particular reason replicas are done synchronously? If Longhorn is RWO, there won't be anybody else in a different node accessing a replica. Or could it?

Because we need to ensure the crash consistency. asynchronously means the software ack the request before it's completely written to the disk, e.g. store the data in the memory or some local cache. So in the case of a node failure, any data in the cache will be lost without the pod knowing because it thought the data has already been written. And replicate synchronously is normally not a problem for the storage system, because Linux is designed to issue multiple requests at the same time, which determined by IO depth.

3. Do you have an idea of the overhead Longhorn imposes on local disk access?

Yes, see https://longhorn.io/blog/performance-scalability-report-aug-2020/.

We're still working on various enhancement for the performance, e.g. #508 . Also, #1045 might have some impact on the performance as well, though it's mainly a stability feature.

[t3hmrman]

Hey @yasker just got a chance to watch your presentation (webinar) earlier this year and took a look at the perf/scalability report released recently as well.

The numbers are pretty staggering -- 20%-30% (so a 1/5th or 1/3rd reduction) in IOPS is a pretty large reduction -- I totally understand the value of crash consistency and the instant migration that it enables (this means longhorn just works for most workloads with no fears when failovers occur), but am wondering if there are any plans to allow looser/configurable consistency levels in the future?

For example, in a situation where I'm running a single Postgres instance with a single "sync" replica (on the local node) for performance and 2 "async" replicas, I might be willing to take the trade-off of potential seconds/minutes of data loss for that 5x increase in IOPS, if entire nodes (especially if they might be dedicated to running databases) going down is relatively rare. Tradeoffs like this get even easier to make in architectures that are fronted by some sort of upstream WAL mechanism (ex. Kafka, etc), where I know that relatively recent state could be recovered, and old state (@ last asynchronous replication) would enable some time to be saved when failing over.

[yasker]

@t3hmrman Yes, we've in fact considered having one local "sync" replica with crash consistency, and multiple async replica to help with the performance. But we haven't brought it up to the schedule yet because:

1. It will be pretty complex from the technical perspective.
2. As you can see from https://longhorn.io/blog/performance-scalability-report-aug-2020/#benchmark-result , the main bottleneck we have right now is not due to multiple replications, but the efficiency of the single replica. That's what we are aiming to improve first via e.g. [FEATURE] Provide a choice to replace revision counter #508 [POC] SPDK #760 . So adding async version of replica won't help much until we improve the sync replica first.

We're aiming to include #508 in v1.1.0. We will publish an updated perf/scalability report then.

[t3hmrman]

@yasker Thanks for the explanation and I appreciate the hard work.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[t3hmrman]

Still interested

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[unixfox]

bump

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[t3hmrman]

bump

[unixfox]

Bump

[joshimoo]

I have added this to planning, to keep the bot from closing it :)
We are definitely interested in improving the IO performance for Longhorn in the future.

[gp187]

Any fixes?

[minhnguyenvan95]

thumbs up

[minhnguyenvan95]

I have tested longhorn vs hostpath vs nfs. WAF -> OCELOT -> .NET APP by upload 133mb file

[with long horn]
waf 20.057172s
direct ocelot -> 18.607835s
direct app -> 15.613427s

[no long horn]
waf 7.311976s
direct ocelot -> 3.974725s
direct app -> 2.080060s

[hostpath]
waf 7.419208s
direct ocelot 04.153185s
direct app 1.802004s

[nfs]
waf 11.251932s
direct ocelot 12.396772s
direct app 8.722210s

[t3hmrman]

It's pretty impressive for Longhorn to be within shouting distance of NFS! NFS has had a lot of man hours poured into it. Excited to see this project get even better and faster!