Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

proposal: support for snappy #82

Closed
matthax opened this issue Mar 8, 2018 · 7 comments
Closed

proposal: support for snappy #82

matthax opened this issue Mar 8, 2018 · 7 comments

Comments

@matthax
Copy link

matthax commented Mar 8, 2018

I've done some digging through the docs, and while I've had better luck with this library than a few others but one thing I've noticed it seems to be lacking is support for snappy.

Most of the Kafka clusters I work with are using snappy by default for compression, so the ability to decompress messages coming from the reader and compress outgoing messages from the writer directly through this library would be great.

I'm just getting started with Go, so I'm not sure I'd be the best person to implement it, but I know the c headers are pretty readily available so it probably should not be terribly difficult...

Thoughts on this?

@achille-roussel
Copy link
Contributor

This is a very good point, and it would be a nice feature to add.

Would you have the cycles to work on this?

@matthax
Copy link
Author

matthax commented Mar 9, 2018

I can give it a shot

@errnoh
Copy link

errnoh commented Mar 14, 2018

I've been using https://github.com/eapache/go-xerial-snappy Decode() for decompressing Message.Value without running into any problems.

@matthax
Copy link
Author

matthax commented Apr 16, 2018

@errnoh When I decode using that library I still seem to get some sort of header in the value. Any idea what might cause that or have you not experienced that problem?

@matthax
Copy link
Author

matthax commented Apr 16, 2018

I noticed these messages also have no Key the nbytes being read for the key is -1. Perhaps related? Not sure.

@Pryz Pryz mentioned this issue Jul 6, 2018
@Pryz
Copy link
Contributor

Pryz commented Jul 19, 2018

Hey @matthax sorry for the late response here. See my last comment here : #100 (comment)

@achille-roussel
Copy link
Contributor

@matthax snappy compression is fully supported in this package and we are using it in production. I'm going to close this issue, feel free to re-open if you have any issues getting snappy compression to work.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

5 participants