Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feature: compression #2082

Closed
sehz opened this issue Jan 7, 2022 · 0 comments
Closed

feature: compression #2082

sehz opened this issue Jan 7, 2022 · 0 comments
Assignees
Labels
Milestone

Comments

@sehz
Copy link
Contributor

sehz commented Jan 7, 2022

With compression on, batch contents are compressed by the producer before sending to SPU. SPU will store and send back compressed batches to consumers. The consumer will decompress batches to the client.

The initial version will use a single algorithm that should be leverage SIMD capability to ensure maximum performance. Later iteration will allow custom compression thru SmartModule.

SmartModule should receive uncompressed content unless custom compression is being used.

@sehz sehz added consumer enhancement New feature or request Performance producer SPU SPU related labels Jan 7, 2022
@sehz sehz added this to the 0.10.0 milestone Jan 8, 2022
@morenol morenol self-assigned this Feb 25, 2022
@nacardin nacardin modified the milestones: 0.10.0, 0.9.21 Mar 7, 2022
bors bot pushed a commit that referenced this issue Mar 11, 2022
@bors bors bot closed this as completed in 2975c47 Mar 11, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants