[FEATURE] Erasure Coding

[michaelpearce-gain]

Is your feature request related to a problem? Please describe.
Currently Longhorn resiliency is based on RF (replication factors), this has a downside of for every TB we store, needing X amount of additional storage based of RF X. It would be advantageous if Erasure Coding (EC) could be added as an option so the underlying storage requirements per used storage can be reduced, some experiences in the past is as much as 50% lower overhead for the same durability guarantee

Describe the solution you'd like
Support of Erasure Coding

Describe alternatives you've considered
Move to another product that supports it.

Additional context
This is something that other HCI solutions / compute and storage systems have had for a while to deal with the above mentioned issue, when using simple RF and since provide EC.

[yasker]

Hi @michaelpearce-gain

We don't plan to do erasure coding, at least for now. The main issue is it's going to complicate things a lot. I will keep this issue open in case we will consider it in the future.

[KyleSanderson]

I too am looking for this. I'm intending on running Rook instead as the 2x storage factor is not ideal.

[darth-veitcher]

It would be great if this was implemented in Longhorn. As others have mentioned this is probably the main reason I'd still consider deploying Rook+Ceph. Would be happy to move fully to Longhorn otherwise.

[KyleSanderson]

@yasker any update on this one now that harvester is GA?

[joshimoo]

Just leaving a note here:
This would require a storage pool / chunk based data format, since we would have conceptually do raid 5 instead of raid 1 like we currently have.
some refs #1061 #3577 #1541

[xeruf]

As much as I love the concept of erasure coding and thought it would be ideal here, I am starting to be unsure whether it makes sense here, reading:

EC has a serious drawback: its effect on performance. Erasure coding is a processing-intensive operation.

Erasure coding is often recommended for storage such as backups or archive -- the types of data sets that are fairly static and not write-intensive.

on https://www.techtarget.com/searchstorage/definition/erasure-coding

One of the problems seems to be that when doing a 2+1 strategy, you gotta distribute the data of a single VM across multiple drives on multiple nodes, which can't be good for performance. It seems we are better off in performance and uptime by accepting the storage overhead, which tends to be much cheaper than the constantly increased processing requirements.

Feel free to convince me otherwise, though!

[KyleSanderson]

As much as I love the concept of erasure coding and thought it would be ideal here, I am starting to be unsure whether it makes sense here, reading:

EC has a serious drawback: its effect on performance. Erasure coding is a processing-intensive operation.

Erasure coding is often recommended for storage such as backups or archive -- the types of data sets that are fairly static and not write-intensive.

on https://www.techtarget.com/searchstorage/definition/erasure-coding

One of the problems seems to be that when doing a 2+1 strategy, you gotta distribute the data of a single VM across multiple drives on multiple nodes, which can't be good for performance. It seems we are better off in performance and uptime by accepting the storage overhead, which tends to be much cheaper than the constantly increased processing requirements.

Feel free to convince me otherwise, though!

I think you've defined the necessity of it quite well, and that longhorn has completely squandered this opportunity.

[xeruf]

Well I use Longhorn because I use Harvester, so most of my data is in active use by kubernetes workloads and thus presumably not qualifying for erasure coding.
I guess it would be useful for backups of volumes, though?

[ggogel]

As much as I love the concept of erasure coding and thought it would be ideal here, I am starting to be unsure whether it makes sense here, reading:

EC has a serious drawback: its effect on performance. Erasure coding is a processing-intensive operation.

Erasure coding is often recommended for storage such as backups or archive -- the types of data sets that are fairly static and not write-intensive.

on https://www.techtarget.com/searchstorage/definition/erasure-coding

One of the problems seems to be that when doing a 2+1 strategy, you gotta distribute the data of a single VM across multiple drives on multiple nodes, which can't be good for performance. It seems we are better off in performance and uptime by accepting the storage overhead, which tends to be much cheaper than the constantly increased processing requirements.

Feel free to convince me otherwise, though!

Storage using erasure coding is not intended for VM workloads, which have high IOPS and random read/write. It is intended for cold storage, meaning any workload with rare sequential writes. This could be data archival or backups for instance.

In my opinion, it would be great if Longhorn had this feature. It would make it much more versatile.

[xeruf]

Oh right! I only used Longhorn for VM storage so I did not consider alternative uses.

[KyleSanderson]

As much as I love the concept of erasure coding and thought it would be ideal here, I am starting to be unsure whether it makes sense here, reading:

EC has a serious drawback: its effect on performance. Erasure coding is a processing-intensive operation.

Erasure coding is often recommended for storage such as backups or archive -- the types of data sets that are fairly static and not write-intensive.

on https://www.techtarget.com/searchstorage/definition/erasure-coding
One of the problems seems to be that when doing a 2+1 strategy, you gotta distribute the data of a single VM across multiple drives on multiple nodes, which can't be good for performance. It seems we are better off in performance and uptime by accepting the storage overhead, which tends to be much cheaper than the constantly increased processing requirements.
Feel free to convince me otherwise, though!

Storage using erasure coding is not intended for VM workloads, which have high IOPS and random read/write. It is intended for cold storage, meaning any workload with rare sequential writes. This could be data archival or backups for instance.

In my opinion, it would be great if Longhorn had this feature. It would make it much more versatile.

It's not really needed anymore, bcachefs handles this out the gate. Give it 6~ months to stabilise and you can forget longhorn was ever a thing.

[ggogel]

It doesn't seem like they are comparable. Longhorn is a distributed filesystem, while bcachefs is not.