Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improvement: abandon internal patches of parquet2 #6064

Closed
dantengsky opened this issue Jun 20, 2022 · 3 comments · Fixed by #6067 or #7475
Closed

Improvement: abandon internal patches of parquet2 #6064

dantengsky opened this issue Jun 20, 2022 · 3 comments · Fixed by #6067 or #7475
Labels
C-improvement Category: improvement

Comments

@dantengsky
Copy link
Member

Summary

We have two internal patches of parqeut2, which mainly address the requirement

  • acquire the parquet file meta, right after the parquet file has been written, without re-read the file

It works, but awkwardly: each time we sync with upstream(official parquet2), there are some extra works to do (rebase, resolve potential conflicts...)

Among the new features that parquet2 has introduced recently, the following two seem to be able to resolve the above requirement.

Thus,

  • we should replace our own internal patches using the new APIs that parquet2 exposes.
  • and pin the parquet2 cargo dependency to the rev of the official parquet2 commit
@dantengsky dantengsky added C-feature Category: feature C-improvement Category: improvement labels Jun 20, 2022
@dantengsky dantengsky changed the title Feature: abandon internal patches of parquet2 Improvement: abandon internal patches of parquet2 Jun 20, 2022
@dantengsky dantengsky removed the C-feature Category: feature label Jun 20, 2022
@Xuanwo
Copy link
Member

Xuanwo commented Jul 4, 2022

Let's go upstream first!

@dantengsky
Copy link
Member Author

internal parquet2 patches are not totally abandoned yet ( for data format backward compatibility). after all the old data has been migrated, we should switch to the upstream parquet2,

@BohuTANG
Copy link
Member

BohuTANG commented Sep 6, 2022

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
C-improvement Category: improvement
Projects
None yet
3 participants