Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Implement zstd dictionary compression in Titan #5743

Closed
3 of 5 tasks
Connor1996 opened this issue Oct 28, 2019 · 3 comments · Fixed by #15979
Closed
3 of 5 tasks

Implement zstd dictionary compression in Titan #5743

Connor1996 opened this issue Oct 28, 2019 · 3 comments · Fixed by #15979
Labels
component/titan Component: Titan engine difficulty/medium Difficulty: Medium. You need some kind of understanding of several components to work on this help wanted Help wanted. Contributions are very welcome! sig/engine SIG: Engine type/enhancement Type: Issue - Enhancement
Projects
Challenge Program Engine SIG

Comments

@Connor1996
Copy link
Member

Connor1996 commented Oct 28, 2019 •
edited

Description

Implement zstd dictionary compression in Titan

The task consist of the following steps:

You can take facebook/rocksdb#4952 as reference, and utilities already implemented in rocksdb can all be used in Titan.

Difficulty

  • Hard

Score

  • 3000

Mentor(s)

Recommended Skills

  • Algorithm
@Connor1996 Connor1996 added component/titan Component: Titan engine difficulty/hard Difficulty: Hard. help wanted Help wanted. Contributions are very welcome! T: PCP-S1 labels Oct 28, 2019
@Connor1996 Connor1996 added this to TODO: Easy in Challenge Program via automation Oct 28, 2019
@Connor1996 Connor1996 moved this from TODO: Easy to TODO: Hard in Challenge Program Oct 28, 2019
@breezewish breezewish changed the title PCP: Implement zstd dictionary compression in Titan PCP-23: Implement zstd dictionary compression in Titan Oct 28, 2019
@yiwu-arbug yiwu-arbug added this to To Do in Engine SIG Nov 13, 2019
@yiwu-arbug yiwu-arbug added the sig/engine SIG: Engine label Nov 13, 2019
@foeb
Copy link

foeb commented Nov 14, 2019

Hi! I'd like to work on this.

@sre-bot sre-bot added the PCP-S1 label Nov 15, 2019
This was referenced Nov 15, 2019
Closed
Closed
@yiwu-arbug yiwu-arbug moved this from To Do to In Progress in Engine SIG Nov 22, 2019
@foeb
Copy link

foeb commented Dec 10, 2019

I'm backing out of this story since I don't have enough time. Thanks again, @yiwu-arbug!

@yiwu-arbug yiwu-arbug moved this from In Progress to To Do in Engine SIG Jan 31, 2020
@yiwu-arbug yiwu-arbug added difficulty/medium Difficulty: Medium. You need some kind of understanding of several components to work on this and removed difficulty/hard Difficulty: Hard. labels Jan 31, 2020
@you06 you06 changed the title PCP-23: Implement zstd dictionary compression in Titan Implement zstd dictionary compression in Titan Feb 29, 2020
@yiwu-arbug yiwu-arbug mentioned this issue Sep 9, 2020
Open
3 tasks
@yiwu-arbug
Copy link
Contributor

now part of #8635

@chux0519 chux0519 mentioned this issue Oct 29, 2020
Merged
yiwu-arbug pushed a commit to tikv/titan that referenced this issue Nov 5, 2020
@chux0519
implement dictionary compression decompression (#192)
ab0a8b9 
This is a subtask of issue: tikv/tikv#5743

> Read blob from file with dictionary compression

Mainly made following changes:

- Refactored `BlobDecoder`, add decompression related components in it.
- Refactored `BlobFileReader` and `BlobFileIterator`, initialize the decoder with corresponding data in blob files.
- add testcases

Issue Number: tikv/tikv#8635
@Connor1996 Connor1996 mentioned this issue Nov 13, 2023
Merged
@Connor1996 Connor1996 added the type/enhancement Type: Issue - Enhancement label Nov 13, 2023
Engine SIG automation moved this from To Do to Done Nov 16, 2023
ti-chi-bot bot added a commit that referenced this issue Nov 16, 2023
@Connor1996 @ti-chi-bot @LykxSassinator
titan: Introduce titan zstd dict compression (#15979)
1cd4753 
close #5743

Introduce titan zstd dict compression

Signed-off-by: Connor1996 <zbk602423539@gmail.com>
Signed-off-by: Connor <zbk602423539@gmail.com>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
Co-authored-by: lucasliang <nkcs_lykx@hotmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
component/titan Component: Titan engine difficulty/medium Difficulty: Medium. You need some kind of understanding of several components to work on this help wanted Help wanted. Contributions are very welcome! sig/engine SIG: Engine type/enhancement Type: Issue - Enhancement
Projects
Engine SIG
  
Done
Challenge Program
  
TODO: Hard
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

titan: Introduce titan zstd dict compression Connor1996/tikv
5 participants
@yiwu-arbug @Connor1996 @foeb @tisonkun @sre-bot