Skip to content

persist: Wire Parquet dictionary encoding and compression to LaunchDarkly#28678

Merged
danhhz merged 4 commits intoMaterializeInc:mainfrom
ParkMyCar:persist/dict-encoding
Aug 1, 2024
Merged

persist: Wire Parquet dictionary encoding and compression to LaunchDarkly#28678
danhhz merged 4 commits intoMaterializeInc:mainfrom
ParkMyCar:persist/dict-encoding

Conversation

@ParkMyCar
Copy link
Copy Markdown
Contributor

@ParkMyCar ParkMyCar commented Jul 31, 2024

This PR adds two dyncfgs for Parquet encoding, one to enable dictionary encoding and a second to enable compression.

Motivation

  • This PR adds a feature that has not yet been specified.

Parquet supports dictionary encoding and compression which can both seriously reduce the size of the encoded data. We need to run benchmarks to determine if we should enable these features, adding these dyncfgs is the first step in doing that.

Checklist

@ParkMyCar ParkMyCar requested review from a team and benesch as code owners July 31, 2024 21:13
Copy link
Copy Markdown
Contributor

@danhhz danhhz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should circle back and remove some of the compile time deps once we settle on something, but otherwise LGTM

ParkMyCar and others added 4 commits August 1, 2024 08:45
Write up Parquet dictionary encoding and compression to dyncfg
@danhhz danhhz force-pushed the persist/dict-encoding branch from 2ecf471 to 57c87ed Compare August 1, 2024 15:46
@danhhz danhhz enabled auto-merge August 1, 2024 15:56
@danhhz danhhz disabled auto-merge August 1, 2024 15:56
@danhhz danhhz enabled auto-merge (squash) August 1, 2024 15:56
@danhhz danhhz disabled auto-merge August 1, 2024 15:56
@danhhz danhhz enabled auto-merge (squash) August 1, 2024 15:57
@danhhz danhhz merged commit 4df50a4 into MaterializeInc:main Aug 1, 2024
@ParkMyCar ParkMyCar deleted the persist/dict-encoding branch January 13, 2025 00:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants