Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Make fillseq load databases in order when --num_multi_db is used #9650

Closed
mdcallag opened this issue Mar 1, 2022 · 1 comment
Closed

Make fillseq load databases in order when --num_multi_db is used #9650

mdcallag opened this issue Mar 1, 2022 · 1 comment
Assignees
@mdcallag
Labels
enhancement

Comments

@mdcallag
Copy link
Contributor

mdcallag commented Mar 1, 2022

When I run db_bench --benchmarks=fillseq --num_multi_db=X then I want it to run as fast as possible, which means it should minimize the time when IO is not in progress. But today, the DoWrite method does this:

  1. Select an id from 0 to (num_multi_db-1)
  2. Use the key generator for that the db with that id
  3. Repeat

Much of the code is here.

The problem with this is that the write buffers for all databases become full around the same point in time, they will then flush around the same point in time and there will be IO stalls. One way to avoid that is to first load all keys for the db with id 1, then id 2 and so on.

From one example with 256M write buffer and num_multi_db=150 there is no IO for 8 minutes then there is ~30G of IO in a burst, repeat.
@mdcallag mdcallag self-assigned this Mar 1, 2022
@mdcallag
Copy link
Contributor Author

mdcallag commented Mar 1, 2022

A minor issue related to this, after running fillseq with --num_multi_db it is unlikely that all databases get --num keys. Some will have more, some will have less. This can be a source of confusion when you run a test like readrandom and expect all lookups to return a value.

mdcallag added a commit to mdcallag/rocksdb-1 that referenced this issue Mar 18, 2022
@mdcallag
For db_bench --benchmarks=fillseq with --num_multi_db load databases …
684c72e 
…in order

Summary:

This fixes facebook#9650
For db_bench --benchmarks=fillseq --num_multi_db=X it loads databases in sequence
rather than randomly choosing a database per Put. The benefits are:
1) avoids long delays between flushing memtables
2) avoids flushing memtables for all of them at the same point in time
3) puts same number of keys per database so that query tests will find keys as expected

Test Plan:

Using db_bench.1 without the change and db_bench.2 with the change:

for i in 1 2; do rm -rf /data/m/rx/* ; time ./db_bench.$i --db=/data/m/rx --benchmarks=fillseq --num_multi_db=4 --num=10000000; du -hs /data/m/rx ; done

--- without the change
    fillseq      :       3.188 micros/op 313682 ops/sec;   34.7 MB/s
    real    2m7.787s
    user    1m52.776s
    sys     0m46.549s
    2.7G    /data/m/rx

--- with the change

    fillseq      :       3.149 micros/op 317563 ops/sec;   35.1 MB/s
    real    2m6.196s
    user    1m51.482s
    sys     0m46.003s
    2.7G    /data/m/rx

    Also, temporarily added a printf to confirm that the code switches to the next database at the right time
    ZZ switch to db 1 at 10000000
    ZZ switch to db 2 at 20000000
    ZZ switch to db 3 at 30000000

for i in 1 2; do rm -rf /data/m/rx/* ; time ./db_bench.$i --db=/data/m/rx --benchmarks=fillseq,readrandom --num_multi_db=4 --num=100000; du -hs /data/m/rx ; done

--- without the change, smaller database, note that not all keys are found by readrandom because databases have < and > --num keys

    fillseq      :       3.176 micros/op 314805 ops/sec;   34.8 MB/s
    readrandom   :       1.913 micros/op 522616 ops/sec;   57.7 MB/s (99873 of 100000 found)

--- with the change, smaller database, note that all keys are found by readrandom

    fillseq      :       3.110 micros/op 321566 ops/sec;   35.6 MB/s
    readrandom   :       1.714 micros/op 583257 ops/sec;   64.5 MB/s (100000 of 100000 found)

Test Plan:

Reviewers:

Subscribers:

Tasks:

Tags:
@mdcallag mdcallag mentioned this issue Mar 18, 2022
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@mdcallag mdcallag

Labels
enhancement
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

db_bench --benchmarks=fillseq --num_multi_db load databases in order mdcallag/rocksdb-1
1 participant
@mdcallag