kvdb-rocksdb: configurable memory budget per column #256

ordian · 2019-11-08T11:03:14Z

Closes #135.
This is a breaking change (changes DatabaseConfig, makes memory_budget_per_col private, removed write_rate_limit option from CompactionProfile) .

kvdb-rocksdb/src/lib.rs

grbIzl · 2019-11-08T13:04:33Z

I think, we need to make configurable the following things:

keep_log_file_num (now it's hardcoded as 1 and it makes debug more difficult)
block size (now it's set in Compaction Profile in accordance with disk's type). In https://github.com/paritytech/parity-substrate-rocksdb-tuning was proposed to set block size to 8 Mb. Not it's hardcoded for ssd as 16Kb

grbIzl · 2019-11-08T13:04:51Z

Relates to openethereum/parity-ethereum#11233

ordian · 2019-11-11T10:54:25Z

@grbIzl Added keep_log_file_num in 0ee93d1. Block size is already exposed e.g. via

CompactionProfile {
   block_size: 8 * 1024 * 1024,
   ..CompactionProfile::ssd(),
}

dvdplm

to avoid heavy migration on adding a column due to change in rocksdb config parameters (esp. painful for archive nodes).

Not sure I understand this last part, can you elaborate?

kvdb-rocksdb/src/lib.rs

ordian · 2019-11-12T13:34:05Z

Not sure I understand this last part, can you elaborate?

Each time we add a new column in parity-ethereum, it kicks off a migration process, the budget for each column changes, which in turn may trigger compaction process that might take quite some time on archive nodes, so we would like to avoid that. (cc openethereum/parity-ethereum#11233)

dvdplm

The idea here is to make minimal changes to leave the old API as intact as possible to avoid breakage for consuming code?

Random thoughts: I think using memory_budget instead of columns creates a readability problem.
I wonder if we should try to make the DatabaseConf a richer type with a columns() getter returning all the data we need to manage a column family.
Or maybe DatabaseConf.columns should be an IndexMap so we can access it both by name and index, and have it store a new ColFam type where we store, dunno, name, mem_budget and the cf_handle?

kvdb-rocksdb/src/lib.rs

dvdplm · 2019-11-12T13:20:05Z

kvdb-rocksdb/src/lib.rs

+			// Memtable size is controlled by the option `write_buffer_size`.
+			// If you increase your memtable size, be sure to also increase your L1 size! 
+			// L1 size is controlled by the option `max_bytes_for_level_base`.
+			opts.set_parsed_options(&format!("max_bytes_for_level_base={}", budget)).map_err(other_io_err)?;


So the advice is to set max_bytes_for_level to 2x the memtable size?

I've set it to the same value as write_buffer_size, couldn't find any advice on that.

kvdb-rocksdb/src/lib.rs

dvdplm · 2019-11-12T14:40:58Z

kvdb-rocksdb/src/lib.rs

-	/// Set number of columns
-	pub columns: Option<u32>,
+	/// Specify the maximal number of info log files to be kept.
+	pub keep_log_file_num: i32,
 }

 impl DatabaseConfig {
 	/// Create new `DatabaseConfig` with default parameters and specified set of columns.
 	/// Note that cache sizes must be explicitly set.
 	pub fn with_columns(columns: Option<u32>) -> Self {


Wouldn't new_with_columns be a more idiomatic name for a constructor?

kvdb-rocksdb/src/lib.rs

dvdplm

If changing any of the options we expose requires a migration or risk making the DB inaccessible we need to document that thoroughly. I'm thinking about things like initial_file_size or block_size: can they be changed freely?

kvdb-rocksdb/src/lib.rs

* master: upgrade tiny-keccak to 2.0 (#260)

ordian · 2019-11-18T09:52:12Z

@dvdplm I've changed the impl to keep the columns field and added a HashMap<Column, MemoryBudget>. indexmap doesn't really help here, since it indexes based on insertion order, not the actual keys.

ordian · 2019-11-18T09:53:48Z

The idea here is to make minimal changes to leave the old API as intact as possible to avoid breakage for consuming code?

Yes, but I'm open to ideas how to make the API more ergonomic while keeping the migration overhead low.

dvdplm

lgtm modulo a couple of minor nits. One thing that would be good is having more module level docs explaining how to configure memory.
I'm not sure how important it is to make concessions to keep changes to a minimum. If we need a migration anyway, we might just as well make the API easy to use? Either way, This is better than what we have so imo let's go ahead and merge it.

kvdb-rocksdb/src/lib.rs

dvdplm · 2019-11-18T11:58:49Z

kvdb-rocksdb/src/lib.rs

+		opts.optimize_level_style_compaction(memory_budget_per_col as i32);
+		opts.set_target_file_size_base(self.compaction.initial_file_size);
+
+		opts.set_parsed_options("compression_per_level=").map_err(other_io_err)?;


Not new code but shouldn't there be a value passed in here? (set_parsed_option is pretty terrible imo).

This was added in openethereum/parity-ethereum@3cf1aac by @andresilva with the intention of enabling compression on all levels, otherwise optimize_level_style_compaction will disable compression for the first two levels IIUC. #257 uses https://docs.rs/rocksdb/0.13.0/rocksdb/struct.Options.html#method.set_compression_per_level instead of set_parsed_option.

Co-Authored-By: David <dvdplm@gmail.com>

…ritytech/parity-common into ao-rocksdb-configurable-column-budget * 'ao-rocksdb-configurable-column-budget' of github.com:paritytech/parity-common: Update kvdb-rocksdb/src/lib.rs kvdb-rocksdb: other minor improvements kvdb-rocksdb: make column_config a method on config kvdb-rocksdb: make memory_budget_per_col private kvdb-rocksdb: small cleanup kvdb-rocksdb: less invasive changes kvdb-rocksdb: export keep_log_file_num kvdb-rocksdb: configurable memory budget per column

The API for setting the write limiter is not available upstream and we only want to set it for an HDD, which is a poor target for parity-ethereum-like workload anyway.

ordian · 2019-11-21T16:18:22Z

I've updated the default block size for ssd() CompactionProfile to 8 MiB and removed write_rate_limit in 130ca3e.

grbIzl

Looks good!

* master: kvdb-rocksdb: configurable memory budget per column (#256)

* master: kvdb-rocksdb: configurable memory budget per column (#256) Bump rlp crate version. (#270) Introduce Rlp::at_with_offset method. (#269) Make fixed-hash test structs public (#267) Migrate primitive types to 2018 edition (#262)

* master: Compile triehash for no_std (#280) [kvdb-rocksdb] Use "pinned" gets to avoid allocations (#274) [kvdb-rocksdb] Release 0.2 (#273) [kvdb-rocksdb] switch to upstream (#257) travis: try to fix wasmpack chrome test on macOS (#263) Use 2018 edition for rustfmt (#266) [fixed-hash]: re-export `alloc_` (#268) kvdb-web: async-awaitify (#259) kvdb-rocksdb: configurable memory budget per column (#256) Bump rlp crate version. (#270) Introduce Rlp::at_with_offset method. (#269) Make fixed-hash test structs public (#267) Migrate primitive types to 2018 edition (#262) upgrade tiny-keccak to 2.0 (#260)

kvdb-rocksdb: configurable memory budget per column

f31463a

ordian commented Nov 8, 2019

View reviewed changes

kvdb-rocksdb/src/lib.rs Outdated Show resolved Hide resolved

grbIzl reviewed Nov 8, 2019

View reviewed changes

kvdb-rocksdb/src/lib.rs Show resolved Hide resolved

ordian requested a review from bkchr November 9, 2019 13:04

ordian mentioned this pull request Nov 10, 2019

[kvdb-rocksdb] switch to upstream #257

Merged

4 tasks

kvdb-rocksdb: export keep_log_file_num

0ee93d1

ordian added the A0-pleasereview label Nov 11, 2019

ordian requested a review from dvdplm November 12, 2019 09:36

dvdplm reviewed Nov 12, 2019

View reviewed changes

kvdb-rocksdb/src/lib.rs Outdated Show resolved Hide resolved

dvdplm reviewed Nov 12, 2019

View reviewed changes

kvdb-rocksdb/src/lib.rs Outdated Show resolved Hide resolved

kvdb-rocksdb/src/lib.rs Outdated Show resolved Hide resolved

kvdb-rocksdb/src/lib.rs Outdated Show resolved Hide resolved

ordian added 3 commits November 15, 2019 18:20

kvdb-rocksdb: less invasive changes

06343ed

Merge branch 'master' into ao-rocksdb-configurable-column-budget

dfaaf26

* master: upgrade tiny-keccak to 2.0 (#260)

kvdb-rocksdb: small cleanup

c553a3f

ordian force-pushed the ao-rocksdb-configurable-column-budget branch from 40ac3f7 to c553a3f Compare November 15, 2019 17:25

ordian added 3 commits November 15, 2019 18:28

kvdb-rocksdb: make memory_budget_per_col private

305ee0c

kvdb-rocksdb: make column_config a method on config

72c9097

kvdb-rocksdb: other minor improvements

3474310

dvdplm approved these changes Nov 18, 2019

View reviewed changes

ordian and others added 5 commits November 18, 2019 14:26

Update kvdb-rocksdb/src/lib.rs

584e2d5

Co-Authored-By: David <dvdplm@gmail.com>

kvdb-rocksdb: configurable memory budget per column

4f829f9

kvdb-rocksdb: export keep_log_file_num

f2a996f

kvdb-rocksdb: less invasive changes

edb5111

kvdb-rocksdb: small cleanup

3c42124

ordian added 4 commits November 18, 2019 17:11

kvdb-rocksdb: make memory_budget_per_col private

1dbf5ee

kvdb-rocksdb: make column_config a method on config

05e8203

kvdb-rocksdb: other minor improvements

4933452

dvdplm requested a review from grbIzl November 20, 2019 11:00

ordian added 2 commits November 21, 2019 17:07

kvdb-rocksdb: remove write_limiter

130ca3e

The API for setting the write limiter is not available upstream and we only want to set it for an HDD, which is a poor target for parity-ethereum-like workload anyway.

kvdb-rocksdb: document CompactionProfile better

85f6cdc

grbIzl approved these changes Nov 22, 2019

View reviewed changes

ordian merged commit a2987e8 into master Nov 22, 2019

ordian deleted the ao-rocksdb-configurable-column-budget branch November 22, 2019 11:48

ordian added a commit that referenced this pull request Nov 22, 2019

Merge branch 'master' into ao-rocksdb-switch-to-upstream2

9adccb8

* master: kvdb-rocksdb: configurable memory budget per column (#256)

dvdplm mentioned this pull request Nov 26, 2019

Give state_col 90% of memory budget and fix other col calculation paritytech/substrate#4208

Merged

niklasad1 mentioned this pull request Jan 13, 2020

Wrong RocksDB.Options parameter openethereum/parity-ethereum#11233

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

kvdb-rocksdb: configurable memory budget per column #256

kvdb-rocksdb: configurable memory budget per column #256

ordian commented Nov 8, 2019 •

edited

grbIzl commented Nov 8, 2019

grbIzl commented Nov 8, 2019

ordian commented Nov 11, 2019

dvdplm left a comment

ordian commented Nov 12, 2019 •

edited

dvdplm left a comment

dvdplm Nov 12, 2019

ordian Nov 21, 2019

dvdplm Nov 12, 2019

dvdplm left a comment

ordian commented Nov 18, 2019

ordian commented Nov 18, 2019

dvdplm left a comment

dvdplm Nov 18, 2019

ordian Nov 18, 2019

ordian commented Nov 21, 2019

grbIzl left a comment

kvdb-rocksdb: configurable memory budget per column #256

kvdb-rocksdb: configurable memory budget per column #256

Conversation

ordian commented Nov 8, 2019 • edited

grbIzl commented Nov 8, 2019

grbIzl commented Nov 8, 2019

ordian commented Nov 11, 2019

dvdplm left a comment

Choose a reason for hiding this comment

ordian commented Nov 12, 2019 • edited

dvdplm left a comment

Choose a reason for hiding this comment

dvdplm Nov 12, 2019

Choose a reason for hiding this comment

ordian Nov 21, 2019

Choose a reason for hiding this comment

dvdplm Nov 12, 2019

Choose a reason for hiding this comment

dvdplm left a comment

Choose a reason for hiding this comment

ordian commented Nov 18, 2019

ordian commented Nov 18, 2019

dvdplm left a comment

Choose a reason for hiding this comment

dvdplm Nov 18, 2019

Choose a reason for hiding this comment

ordian Nov 18, 2019

Choose a reason for hiding this comment

ordian commented Nov 21, 2019

grbIzl left a comment

Choose a reason for hiding this comment

ordian commented Nov 8, 2019 •

edited

ordian commented Nov 12, 2019 •

edited