Introduce optimize_filters_for_hits to cf_config #3733

solotzg · 2018-11-03T03:36:00Z

Signed-off-by: Tong Zhigao tongzhigao@pingcap.com

What have you changed? (mandatory)

Introduce the option optimize_filters_for_hits of RocksDB into cf_config.
By now, the default cf is used to store kv data which could be determined whether really exists by upper logic instead of bloom filters.
So, we use this option to optimizes bloom filters. If true, RocksDB won't create bloom filters for the max level of the LSM to reduce metadata that should fit in RAM. The saved memory could be used for more block caches or others.
optimize_filters_for_hits is setted true for default cf by default.

What are the type of the changes? (mandatory)

improvement

How has this PR been tested? (mandatory)

unit test
stress test

Benchmark result if necessary (optional)

When we use cgroup to limit memory or disable block cache, the result show that the improvement is little.
So, we set max-open-files=-1 and set block-cache-size to a large value in order to test the situation that default cf store all data in memory.
- PD x 1, TiKV x 1.
- Setting of go-ycsb:
  - tikv.type=txn
  - operationcount=5000000
  - recordcount=5000000
  - fieldcount=2, fieldlength=10
After several tests, the result show that QPS and Latency of are nearly equal, but the memory usage of TiKV with optimize_filters_for_hits=true is about up to 15% less.
Details are in [Jira TIKV-1809]

siddontang · 2018-11-03T04:10:16Z

Em, if we use Raw mode, we can't disable it. Can we change this dynamically?

huachaohuang · 2018-11-03T05:49:56Z

@solotzg It's not friendly for the community to put an internal link here, please show your result here directly so that everyone can understand it, thanks.

huachaohuang · 2018-11-03T05:55:55Z

Have you observed the cache hit ratio? If you save 15% memory usage, it means that it can cache more data in memory, so the cache hit ratio is expected to increase more or less.

etc/config-template.toml

siddontang · 2018-11-04T03:50:04Z

IMO, maybe we can disable bloom filter for Raft CF and Raft DB. because we can ensure the data can be found. /cc @BusyJay

solotzg · 2018-11-05T02:51:37Z

@huachaohuang I haven't observed the cache hit ratio. I recorded and compared the memory usage of TiKV after job load of go-ycsb finished. I will test cache hit ratio when total memory is limited.

src/config.rs

tests/config/test-custom.toml

Signed-off-by: Tong Zhigao <tongzhigao@pingcap.com>

solotzg · 2018-11-12T08:16:22Z

If the size of value is less than 64B, this kind of short value would be embed into write cf instead of default cf. So, set fieldcount=15, fieldlength=10.
Load Config:

recordcount=4000000
operationcount=4000000
workload=core
fieldcount=15
fieldlength=10
insertorder=hashed
readproportion=1.0
updateproportion=0
insertproportion=0
scanproportion=0
readallfields=false

Read Config:

recordcount=10000000
operationcount=10000000
workload=core
fieldcount=15
fieldlength=10
insertorder=hashed
readproportion=1.0
updateproportion=0
insertproportion=0
scanproportion=0
readallfields=false

TiKV master:

[rocksdb.defaultcf]
optimize-filters-for-hits = false
[rocksdb.lockcf]
optimize-filters-for-hits = false
[rocksdb.writecf]
optimize-filters-for-hits = false
[raftdb]
optimize-filters-for-hits = false
[raftdb.defaultcf]
optimize-filters-for-hits = false

result:

INSERT - Takes(s): 143.5, Count: 3999992, OPS: 27869.3, Avg(us): 3552, Min(us): 1072, Max(us): 253773, 95th(us): 7000, 99th(us): 8000
READ - Takes(s): 122.1, Count: 9999959, OPS: 81896.3, Avg(us): 1213, Min(us): 286, Max(us): 45484, 95th(us): 3000, 99th(us): 5000

TiKV _optimize_filters_for_hits:

[rocksdb.defaultcf]
optimize-filters-for-hits = true
[rocksdb.lockcf]
optimize-filters-for-hits = false
[rocksdb.writecf]
optimize-filters-for-hits = false
[raftdb]
optimize-filters-for-hits = true
[raftdb.defaultcf]
optimize-filters-for-hits = true

result:

INSERT - Takes(s): 136.7, Count: 4000000, OPS: 29252.0, Avg(us): 3381, Min(us): 1009, Max(us): 371471, 95th(us): 7000, 99th(us): 8000
READ - Takes(s): 116.5, Count: 10000000, OPS: 85840.5, Avg(us): 1158, Min(us): 280, Max(us): 49137, 95th(us): 3000, 99th(us): 5000

@huachaohuang

huachaohuang · 2018-11-12T08:44:22Z

What are the size of memory and dataset, how about the cache hit ratio?

solotzg · 2018-11-12T09:51:13Z

TiKV master:

memory: 5353607168
disk:
  12K     ./data/import
  367M    ./data/raft
  3.0G    ./data/db
  4.0K    ./data/snap
  3.4G    ./data
block_cache_data_hit: 71.1K ops

TiKV _optimize_filters_for_hits:

memory: 5177335808
disk:
  12K     ./data/import
  188M    ./data/raft
  3.0G    ./data/db
  4.0K    ./data/snap
  3.2G    ./data
block_cache_data_hit: 72.7K ops

@huachaohuang But the benchmark result is not stable. Sometimes, TiKV master is better.

solotzg · 2018-11-13T08:24:47Z

TiKV master:

TiKV _optimize_filters_for_hits:

Test

Both test cases have loaded 30m records, and benchmark reading under situation

memory limit 2G
max-open-files = 4096
max-write-buffer-number = 2
rocksdb.defaultcf.block-cache-size = "512MB"
rocksdb.lockcf.block-cache-size = "128MB"
rocksdb.writecf.block-cache-size = "256MB"
raftdb.defaultcf.block-cache-size = "128MB"

But performance of the one optimized is not as good as expected. Maybe there are some other operations in txn of go-ycsb.
Further research needs to be done.

solotzg · 2018-11-19T03:07:17Z

Newest Test

TiKV _optimize_filters_for_hits

TiKV Configuration

[rocksdb.defaultcf]
optimize-filters-for-hits = true
level0-file-num-compaction-trigger=2
level0-slowdown-writes-trigger=2
level0-stop-writes-trigger=2
max-write-buffer-number=2
max-bytes-for-level-base="500GB"
block-cache-size="1GB"

[rocksdb.lockcf]
optimize-filters-for-hits = false
max-write-buffer-number=2
block-cache-size="128MB"

[rocksdb.writecf]
optimize-filters-for-hits = false
max-write-buffer-number=2
block-cache-size="256MB"

[raftdb]

[raftdb.defaultcf]
optimize-filters-for-hits = true
max-write-buffer-number=2
block-cache-size="128MB"

Test Result

go-ycsb load data

recordcount=20000000
operationcount=20000000
fieldcount=15
fieldlength=10

go-ycsb read data

READ - Takes(s): 883.8, Count: 10249737, OPS: 11597.1, Avg(us): 8618, Min(us): 311, Max(us): 90707, 95th(us): 12000, 99th(us): 15000

TiKV master

TiKV Configuration

[rocksdb.defaultcf]
optimize-filters-for-hits = false
level0-file-num-compaction-trigger=2
level0-slowdown-writes-trigger=2
level0-stop-writes-trigger=2
max-write-buffer-number=2
max-bytes-for-level-base="500GB"
block-cache-size="1GB"

[rocksdb.lockcf]
optimize-filters-for-hits = false
max-write-buffer-number=2
block-cache-size="128MB"

[rocksdb.writecf]
optimize-filters-for-hits = false
max-write-buffer-number=2
block-cache-size="256MB"

[raftdb]

[raftdb.defaultcf]
optimize-filters-for-hits = false
max-write-buffer-number=2
block-cache-size="128MB"

Test Result

go-ycsb load data

recordcount=20000000
operationcount=20000000
fieldcount=15
fieldlength=10

go-ycsb read data

READ - Takes(s): 1079.1, Count: 12212303, OPS: 11316.7, Avg(us): 8832, Min(us): 551, Max(us): 115857, 95th(us): 12000, 99th(us): 16000

Grafana

The previous is optimized, the next is master.

huachaohuang · 2018-11-26T11:37:40Z

Setup

Hardware

CPU: 40 vCPUs, Intel(R) Xeon(R) CPU E5-2630 v4 @ 2.20GHz
SSD: 1.6T, Intel 3610
MEM: 2GB

Configurations

[readpool.storage]
normal-concurrency = 16
[rocksdb.defaultcf]
block-cache-size = "512MB"
[rocksdb.lockcf]
block-cache-size = "128MB"
[rocksdb.writecf]
block-cache-size = "256MB"
[raftdb.defaultcf]
block-cache-size = "128MB"

Dataset: number of keys = 100,000,000, value size = 64B

Results

optimize filters for hits	data size	index and filter size (default)	index and filter size (write)	block cache hit (data)	read ops
false	11965MB	141.3MB	138.2MB	30.1%	6741.4
true	11820MB	26.9MB	137.7MB	31.9%	7119.8

Conclusion: the optimization in this PR increases the read ops in this PR for 5.6%.

DorianZheng

LGTM

Signed-off-by: Tong Zhigao <tongzhigao@pingcap.com>

siddontang reviewed Nov 4, 2018

View reviewed changes

etc/config-template.toml Show resolved Hide resolved

Connor1996 added the component/rocksdb Component: RocksDB engine label Nov 6, 2018

siddontang reviewed Nov 7, 2018

View reviewed changes

src/config.rs Outdated Show resolved Hide resolved

siddontang reviewed Nov 7, 2018

View reviewed changes

src/config.rs Outdated Show resolved Hide resolved

solotzg force-pushed the _optimize_filters_for_hits branch from 79748de to 3df4d97 Compare November 8, 2018 07:29

zhangjinpeng87 reviewed Nov 9, 2018

View reviewed changes

tests/config/test-custom.toml Outdated Show resolved Hide resolved

Add optimize_filters_for_hits to cf_config;

4dd347e

Signed-off-by: Tong Zhigao <tongzhigao@pingcap.com>

solotzg force-pushed the _optimize_filters_for_hits branch from 657418a to 4dd347e Compare November 12, 2018 07:57

solotzg added the S: DNM label Nov 13, 2018

solotzg changed the title ~~Introduce optimize_filters_for_hits to cf_config~~ [DNM] Introduce optimize_filters_for_hits to cf_config Nov 13, 2018

solotzg removed the S: DNM label Nov 19, 2018

solotzg changed the title ~~[DNM] Introduce optimize_filters_for_hits to cf_config~~ Introduce optimize_filters_for_hits to cf_config Nov 19, 2018

Merge branch 'master' into _optimize_filters_for_hits

d7238a8

Merge branch 'master' into _optimize_filters_for_hits

5a2c77f

huachaohuang approved these changes Nov 26, 2018

View reviewed changes

Merge branch 'master' into _optimize_filters_for_hits

8c0351d

solotzg added the status/LGT1 Indicates that a PR has LGTM 1. label Nov 27, 2018

Merge branch 'master' into _optimize_filters_for_hits

1a0a317

DorianZheng approved these changes Nov 27, 2018

View reviewed changes

DorianZheng merged commit 14812b7 into tikv:master Nov 27, 2018

solotzg deleted the _optimize_filters_for_hits branch October 16, 2019 11:18

sticnarf pushed a commit to sticnarf/tikv that referenced this pull request Oct 27, 2019

Add optimize_filters_for_hits to cf_config; (tikv#3733)

d0e5f78

Signed-off-by: Tong Zhigao <tongzhigao@pingcap.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Introduce optimize_filters_for_hits to cf_config #3733

Introduce optimize_filters_for_hits to cf_config #3733

solotzg commented Nov 3, 2018 •

edited

Loading

siddontang commented Nov 3, 2018

huachaohuang commented Nov 3, 2018

huachaohuang commented Nov 3, 2018

siddontang commented Nov 4, 2018

solotzg commented Nov 5, 2018

solotzg commented Nov 12, 2018 •

edited

Loading

huachaohuang commented Nov 12, 2018

solotzg commented Nov 12, 2018

solotzg commented Nov 13, 2018

solotzg commented Nov 19, 2018

huachaohuang commented Nov 26, 2018

DorianZheng left a comment

Introduce optimize_filters_for_hits to cf_config #3733

Introduce optimize_filters_for_hits to cf_config #3733

Conversation

solotzg commented Nov 3, 2018 • edited Loading

What have you changed? (mandatory)

What are the type of the changes? (mandatory)

How has this PR been tested? (mandatory)

Benchmark result if necessary (optional)

siddontang commented Nov 3, 2018

huachaohuang commented Nov 3, 2018

huachaohuang commented Nov 3, 2018

siddontang commented Nov 4, 2018

solotzg commented Nov 5, 2018

solotzg commented Nov 12, 2018 • edited Loading

huachaohuang commented Nov 12, 2018

solotzg commented Nov 12, 2018

solotzg commented Nov 13, 2018

TiKV master:

TiKV _optimize_filters_for_hits:

Test

solotzg commented Nov 19, 2018

Newest Test

TiKV _optimize_filters_for_hits

TiKV Configuration

Test Result

TiKV master

TiKV Configuration

Test Result

Grafana

huachaohuang commented Nov 26, 2018

Setup

Hardware

Configurations

Results

DorianZheng left a comment

Choose a reason for hiding this comment

solotzg commented Nov 3, 2018 •

edited

Loading

solotzg commented Nov 12, 2018 •

edited

Loading