- Reed-Solomon Erasure Code engine in pure Go.(Based on intel ISA-L & Klauspost ReedSolomon)
- Fast: more than 10GB/s per physics core
To get the package use the standard:
go get github.com/templexxx/reedsolomon
See the associated GoDoc
- All arch are supported
- Go1.11(for AVX512)
- Coding over in GF(2^8)
- Primitive Polynomial: x^8 + x^4 + x^3 + x^2 + 1 (0x1d)
- mathtool/gentbls.go : generator Primitive Polynomial and it's log table, exp table, multiply table, inverse table etc. We can get more info about how galois field work
- mathtool/cntinverse.go : calculate how many inverse matrix will have in different RS codes config
- Cauchy Matrix is generator matrix
Why so fast?
These three parts will cost too much time:
- lookup galois-field tables
- read/write memory
- calculate inverse matrix in the reconstruct process
SIMD will solve no.1
Cache-friendly codes will help to solve no.2 & no.3, and more, use a sync.Map for cache inverse matrix, it will help to save about 1000ns when we need same matrix.
Performance depends mainly on:
- CPU instruction extension( AVX512 or AVX2)
- number of data/parity vects
- unit size of calculation ( see it in rs.go )
- size of shards
- speed of memory (waste so much time on read/write mem, :D )
- performance of CPU
- the way of using (reuse memory)
And we must know the benchmark test is quite different with encoding/decoding in practice.
Because in benchmark test loops, the CPU Cache will help a lot. In practice, we must reuse the memory to make the performance become as good as the benchmark test.
Example of performance on my AWS c5d.large (Intel(R) Xeon(R) Platinum 8124M CPU @ 3.00GHz.) DataCnt = 10; ParityCnt = 4
|Vector size||AVX512 (MB/S)||AVX2 (MB/S)|