Skip to content

Compression#1539

Merged
Licenser merged 3 commits intomainfrom
compression
Apr 13, 2022
Merged

Compression#1539
Licenser merged 3 commits intomainfrom
compression

Conversation

@Licenser
Copy link
Copy Markdown
Member

Pull request

Description

This unifies the various compression and decompression pre and postprocessors into a single pre and a single postprocessor.

Checklist

  • The RFC, if required, has been submitted and approved
  • Any user-facing impact of the changes is reflected in docs.tremor.rs
  • The code is tested
  • Use of unsafe code is reasoned about in a comment
  • Update CHANGELOG.md appropriately, recording any changes, bug fixes, or other observable changes in behavior
  • The performance impact of the change is measured (see below)

Performance

Rally just restructuring

Signed-off-by: Heinz N. Gies <heinz@licenser.net>
Signed-off-by: Heinz N. Gies <heinz@licenser.net>
Copy link
Copy Markdown
Member

@mfelsche mfelsche left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

While i like the "unification" aspect, i do not see how it is a win for users to have to type a complex json config structure to chose a compression/decompression algo instead of having handy string shorthands.

Comment thread src/preprocessor/decompress.rs Outdated
Comment thread src/postprocessor/compress.rs Outdated
Comment thread src/postprocessor.rs
"compress" => Ok(Box::new(Compress::from_config(config.config.as_ref())?)),
"separate" => Ok(Box::new(separate::Separate::from_config(&config.config)?)),
"base64" => Ok(Box::new(Base64::default())),
"gzip" => Ok(Box::new(Gzip::default())),
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i don't see how removing all those nice short-hands helps users. Now people need to type the very tedious:

{
  "name": "compress",
  "config": {
    "algorithm": "gzip"
  }
}

instead of just "gzip".

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we want to also consider adding elementwise vs stream compression with making the compress processor require a config?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As we discussed this morning, it reduces the surface we expose as a core, making it simpler, in favor of moving short-hands into userland with definable pre and post processors giving users the option to add syntax sugar to any degree they wish.

Regarding stream compression, so far all compressors have been element wise compression the PR doesn't look at or consider changing this but it'd be a nice ticket to add and consdier

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Got it.

Lets also have a ticket for defining/creating pre/postprocessors in troy and then i'm good :)

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done:
#1541

and for completeness

#1540

Comment thread src/preprocessor.rs
Signed-off-by: Heinz N. Gies <heinz@licenser.net>
@Licenser Licenser merged commit 01fd4ab into main Apr 13, 2022
@Licenser Licenser deleted the compression branch April 13, 2022 14:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants