Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

LevelCompactionBuilder::GetPathId treats max_bytes_for_level_base as L0 size while it is the size for L1 #2461

Closed
PraveenSinghRao opened this issue Jun 16, 2017 · 2 comments
Closed

LevelCompactionBuilder::GetPathId treats max_bytes_for_level_base as L0 size while it is the size for L1 #2461

PraveenSinghRao opened this issue Jun 16, 2017 · 2 comments
Labels
abandoned-or-aged-out

Comments

@PraveenSinghRao
Copy link
Contributor

While trying to split DB on different media (e.g. SSD/HDD) this causes a problem.

Not sure what L0 size should be, as it is not specified anywhere afaik.
@PraveenSinghRao PraveenSinghRao changed the title LevelCompactionBuilder::GetPathId treats mutable_cf_options.max_bytes_for_level_base as L0 size while it is the size for L1 LevelCompactionBuilder::GetPathId treats max_bytes_for_level_base as L0 size while it is the size for L1 Jun 16, 2017
@PraveenSinghRao
Copy link
Contributor Author

I can create a fix. However what should be L0 size?

We could deduce it from write buffer size * compaction_trigger_for_L0 but what happens when the compaction trigger is turned off.

@PhaniShekhar PhaniShekhar mentioned this issue Sep 19, 2017
Closed
facebook-github-bot pushed a commit that referenced this issue Sep 21, 2017
@PhaniShekhar @facebook-github-bot
Use L1 size as estimate for L0 size in LevelCompactionBuilder::GetPathID
65a9cd6 
Summary:
Fix for [2461](#2461).

Problem: When using multiple db_paths setting with RocksDB, RocksDB incorrectly calculates the size of L1 in LevelCompactionBuilder::GetPathId.

max_bytes_for_level_base is used as L0 size and L1 size is calculated as (L0 size * max_bytes_for_level_multiplier). However, L1 size should be max_bytes_for_level_base.

Solution: Use max_bytes_for_level_base as L1 size. Also, use L1 size as the estimated size of L0.
Closes #2903

Differential Revision: D5885442

Pulled By: maysamyabandeh

fbshipit-source-id: 036da1c9298d173b9b80479cc6661ee4b7a951f6
ifed01 pushed a commit to ifed01/rocksdb that referenced this issue Nov 30, 2017
@PhaniShekhar @ifed01
Use L1 size as estimate for L0 size in LevelCompactionBuilder::GetPathID
62782e7 
Summary:
Fix for [2461](facebook#2461).

Problem: When using multiple db_paths setting with RocksDB, RocksDB incorrectly calculates the size of L1 in LevelCompactionBuilder::GetPathId.

max_bytes_for_level_base is used as L0 size and L1 size is calculated as (L0 size * max_bytes_for_level_multiplier). However, L1 size should be max_bytes_for_level_base.

Solution: Use max_bytes_for_level_base as L1 size. Also, use L1 size as the estimated size of L0.
Closes facebook#2903

Differential Revision: D5885442

Pulled By: maysamyabandeh

fbshipit-source-id: 036da1c9298d173b9b80479cc6661ee4b7a951f6
@gfosco
Copy link
Contributor

gfosco commented Jan 10, 2018

Closing this via automation due to lack of activity. If discussion is still needed here, please re-open or create a new/updated issue.

@gfosco gfosco closed this as completed Jan 10, 2018
tchaikov pushed a commit to ceph/rocksdb that referenced this issue Mar 12, 2018
@PhaniShekhar @tchaikov
Use L1 size as estimate for L0 size in LevelCompactionBuilder::GetPathID
ce64698 
Summary:
Fix for [2461](facebook#2461).

Problem: When using multiple db_paths setting with RocksDB, RocksDB incorrectly calculates the size of L1 in LevelCompactionBuilder::GetPathId.

max_bytes_for_level_base is used as L0 size and L1 size is calculated as (L0 size * max_bytes_for_level_multiplier). However, L1 size should be max_bytes_for_level_base.

Solution: Use max_bytes_for_level_base as L1 size. Also, use L1 size as the estimated size of L0.
Closes facebook#2903

Differential Revision: D5885442

Pulled By: maysamyabandeh

fbshipit-source-id: 036da1c9298d173b9b80479cc6661ee4b7a951f6
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
abandoned-or-aged-out
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@gfosco @PraveenSinghRao