Update lz4 and zstd version?

[mareksimunek]

Hi, I was looking for some backported codecs to hadoop 2.7 and your amazing work just did what I was looking for.

I noticed project versions are:
zstd: 1.0.1 (4 years old)
lz4: 1.3.0 (6 years old)

Would it be please possible to upgrade it?

I would help, but I am no expert in JNI.

[mareksimunek]

Hi @carlomedas I know you are probably focused on different projects. Is there any chance that you will give some hint how to update?

[carlomedas]

I'll see if we can make it become a Fing company github project so we can maintain officially here, as I'm kind of out of personal time for this.
We've been using this in our big-data architecture so it makes a lot of sense.

The problem in updating is not putting latest libraries, which is very fast, unless interface headers changed signatures, but more problematic point is to build the library on all platforms before releasing a new version, as I don't have any more all build VM's.
Let me propose it as Fing project and let you know.

Last point: even by updating libraries, binary compressed data is not going to change, so you can start using it today and when there is updated version you can switch seamlessly.

[mareksimunek]

Thank you very much for effort.

The problem in updating is not putting latest libraries, which is very fast, unless interface headers changed signatures, but more problematic point is to build the library on all platforms before releasing a new version, as I don't have any more all build VM's.

That's what I am afraid of, that interfaces changed after that many years.
As far as platforms support: I will be little selfish and say linux is all I need.

Last point: even by updating libraries, binary compressed data is not going to change, so you can start using it today and when there is updated version you can switch seamlessly.

I tried ZSTD and it needs lot more memory (20-30%) than other compressions (GZIP, LZ4, SNAPPY) so I am not sure if its nature of algorithm or it's just old version where they didn't focus on memory consumption. But compression ratio is the best.

[carlomedas]

Let me confirm we are progressing on that. 4mc has been transferred to Fing company and we will make sure to keep it up to date.

[mareksimunek]

Once again, thanks. :) looking forward for updates.

[carlomedas]

Stay tuned, new release should be coming this week.

[carlomedas]

Done by @noodlesbad !