Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add K4os.Compression.LZ4.Buffers prototype #74

Open
wants to merge 17 commits into
base: pipelines-prototype
Choose a base branch
from

Conversation

rmja
Copy link

@rmja rmja commented Sep 14, 2022

This is a working initial prototype of LZ4FrameEncoder and LZ4FrameDecoder that works using PipeReader and PipeWriter.
It is put in a separate K4os.Compression.LZ4.Buffers project as I do not have the correct csproj skillset to get this working in K4os.Compression.LZ4.Streams because of the very old target frameworks. This currently targets .NET 6 - it probably needs to be relaxed, but I need help to accomplish this.

If the target frameworks can be the same as K4os.Compression.LZ4.Streams then I see three proposals:

  1. Add this code directly in K4os.Compression.LZ4.Streams and use it as the fundamentals for the currently exported stream classes.
  2. Let this be in a different assembly/nuget which can be referenced by K4os.Compression.LZ4.Streams to accomplish the same task as 1.
  3. Do nothing with K4os.Compression.LZ4.Streams and have the two different implementations.

This implementation supports almost all features in a LZ4 frame, including ContentSize, block and content checksums, etc. except the dictionary field.

Feel free to make any comments and commit to this pr.

This pr replaces the previous gist https://gist.github.com/rmja/98dc7e0576c933faa0a75629b46af71c.

This pr fixes #14.

cc @MiloszKrajewski @AArnott

@rmja rmja marked this pull request as ready for review September 15, 2022 03:52
@AArnott
Copy link

AArnott commented Sep 15, 2022

I love it. You share nearly all the code between the two modes, and it seems to be truly streaming. Wonderful job, IMO. Thank you.

I left you a present. Happy to talk about it.

@rmja
Copy link
Author

rmja commented Sep 17, 2022

@MiloszKrajewski We now have a reasonably good proposal for a pipe streaming based encoder and decoder, and we now need your inputs to proceed from here. Let us know what you think ! cc @AArnott

@MiloszKrajewski
Copy link
Owner

I am not ignoring you, believe me. I just have my own branch with similar goal (see: abstracted-streams) and I need to wrap my head around it.

@MiloszKrajewski MiloszKrajewski changed the base branch from master to pipelines-prototype March 10, 2023 13:07
@MiloszKrajewski
Copy link
Owner

This PR has a lot of good code and feel bad about not using it.
At the same time, it no longer applies in a sense, and baseline already has this functionality.
So I decided to keep it a pipelines-prototype and look at this later, while deleting the PR because it won't be merged in this form probably ever.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Support for ReadOnlySequence<byte> and IBufferWriter<byte>
3 participants