encode with io.Writer / decode with io.Reader

[lachie]

I'd love to use this code, but my project involves compressing files up to gigabytes in size.

For that use case I really need a streaming version of the algorithm (i.e. using the io.Writer/io.Reader interfaces). I see there's one as part of the reference C implementation, but I really have no idea where to start with porting it to Go.

[dgryski]

The io.Reader/io.Writer interface was removed in 5298eaf due to bugs and slowness.

I agree having a streaming interface would be useful. IIRC, the old C library faked streaming by buffering in (~65k?) blocks and compressing in batches. You could do something similar and construct your own framing, assuming you don't need compatibility with any other implementation.

I see there is now a more complete spec for an LZ4 streaming format: https://docs.google.com/document/d/1gZbUoLw5hRzJ5Q71oPRN6TO4cRMTZur60qip-TE7BhQ/edit . We should probably support this format.

[dgryski]

Looks like the streaming format isn't completely implementing in the C lz4 implementation: https://code.google.com/p/lz4/issues/detail?id=42

[lachie]

ok no worries, that sounds like a good solution

[dgryski]

Leaving this open as a reminder to put back the streaming support.

[dgryski]

http://fastcompression.blogspot.fr/2014/05/streaming-api-for-lz4.html

[vtolstov]

Any progress about steaming support ?

[dgryski]

Spec has been finalised. I'll add it nearer the top of my todo list.

[vtolstov]

@dgryski hello, do you have any news about streaming support?

[vtolstov]

ping...

[dgryski]

I'm unlikely to get to this anytime soon. I will glady review any merge any pull requests for this feature.

Sorry :(

[vtolstov]

2015-01-20 13:39 GMT+03:00 Damian Gryski notifications@github.com:

I'm unlikely to get to this anytime soon. I will glady review any merge
any pull requests for this feature.

Sorry :(

Ok. May be i'm try.

Vasiliy Tolstov,
e-mail: v.tolstov@selfip.ru
jabber: vase@selfip.ru

[xxorde]

ping...

Any progress so far?

Or does anyone has good working example for using blocks and compressing in batches?

[dgryski]

Some of the other Go lz4 ports have implemented the streaming interface: https://github.com/pierrec/lz4 for example

[xxorde]

Thank you for this hint. It was way harder to implement this myself than I thought.