Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

mimic: os/bluestore: backport new bitmap allocator #26983

Merged
merged 21 commits into from Apr 1, 2019

Conversation

Projects
None yet
4 participants

ifed01 added some commits May 3, 2018

test/allocator: get rid off bitmap allocator specifics and introduce new
performance test cases

Signed-off-by: Igor Fedotov <ifedotov@suse.com>
(cherry picked from commit dcd3061)
os/bluestore: add new bitmap allocator
Signed-off-by: Igor Fedotov <ifedotov@suse.com>
(cherry picked from commit 631041a)
test/allocator: move bluestore allocator's benchmarks to a standalone UT
Signed-off-by: Igor Fedotov <ifedotov@suse.com>
(cherry picked from commit ac2ec80)
os/bluestore: get rid off allocator's reserve method
Signed-off-by: Igor Fedotov <ifedotov@suse.com>
(cherry picked from commit 88bea46)
os/bluestore: add release(PExtentVector) helper to Allocator class to
free PExtentVector explicitly.

Signed-off-by: Igor Fedotov <ifedotov@suse.com>
(cherry picked from commit 328ea72)
os/bluestore: fix improper access to a BitmapFastAllocator::last_pos
from multiple threads.

Signed-off-by: Igor Fedotov <ifedotov@suse.com>
(cherry picked from commit 0c69672)
os/bluestore: align BitMap allocator's init_rm_free/init_add_free par…
…ameters with min_alloc_size

Signed-off-by: Igor Fedotov <ifedotov@suse.com>
(cherry picked from commit 1b6a56e)
os/bluestore: remove original bitmap allocator
Signed-off-by: Igor Fedotov <ifedotov@suse.com>
(cherry picked from commit 95bd72f)

@ifed01 ifed01 requested a review from liewegas Mar 15, 2019

@smithfarm smithfarm removed the backport label Mar 15, 2019

@smithfarm smithfarm added this to the mimic milestone Mar 15, 2019

ifed01 and others added some commits May 30, 2018

os/bluestore: more verbose logging in new bitmap allocator
Signed-off-by: Igor Fedotov <ifedotov@suse.com>
(cherry picked from commit 160c730)
os/bluestore: cosmetic cleanup in new bitmap allocator.
Signed-off-by: Igor Fedotov <ifedotov@suse.com>
(cherry picked from commit 3db9d9a)
os/bluestore: properly respect min_length when analysing partially free
slots in bitmap allocator.

Signed-off-by: Igor Fedotov <ifedotov@suse.com>
(cherry picked from commit 288bbd4)
os/bluestore: cosmetic new allocator internal method rename.
Signed-off-by: Igor Fedotov <ifedotov@suse.com>
(cherry picked from commit fa06ff2)
os/bluestore: respect min_length as allocation granularity for new
bitmap allocator.
It was used a real minimum threshold before this fix which allowed e.g.
allocated extent length to be equal to min_length + 1.

Signed-off-by: Igor Fedotov <ifedotov@suse.com>
(cherry picked from commit 8a188b2)
test/objectstore/unitetest_fastbmap_allocator: replace ASSERT_TRUE with
ASSERT_EQ

Signed-off-by: Igor Fedotov <ifedotov@suse.com>
(cherry picked from commit 69ccb43)
os/bluestore: perform allocations aligned with min_length in new bitm…
…ap allocator

Signed-off-by: Igor Fedotov <ifedotov@suse.com>
(cherry picked from commit 51974c4)
os/bluestore: rename new bitmap allocator class to BitmapAllocator.
Signed-off-by: Igor Fedotov <ifedotov@suse.com>
(cherry picked from commit a8e8c19)
os/bluestore: provide a rough estimate for bitmap allocator
fragmentation.
The approach is counting 'partial' and 'free' slots at L1 on the fly and
use partial / (partial+free) value as rough fragmentation estimate.

Signed-off-by: Igor Fedotov <ifedotov@suse.com>
(cherry picked from commit dbbe12f)
os/bluestore: fix overlappedd allocation returned by bitmap allocator
Fixes: https://tracker.ceph.com/issues/24598
Signed-off-by: Igor Fedotov <ifedotov@suse.som>
(cherry picked from commit 539af4f)
os/bluestore: fix minor issues in bitmap logging output
Signed-off-by: Igor Fedotov <ifedotov@suse.com>
(cherry picked from commit dc4b5fc)
qa/objectstore: bluestore -> bluestore-{bitmap,stupid}
The new default is bitmap, so we were testing bitmap twice.  Instead,
explicitly call out stupid and bitmap cases so a future default change
won't break coverage.

Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit f390704)

@ifed01 ifed01 force-pushed the ifed01:wip-ifed-bmap-alloc-mimic branch from 9b96378 to cf984c8 Mar 17, 2019

@yuriw

This comment has been minimized.

Copy link
Contributor

commented Mar 20, 2019

@yuriw

This comment has been minimized.

Copy link
Contributor

commented Mar 22, 2019

@ifed01 also broken yaml config:

CEPH_QA_MAIL="ceph-qa@lists.ceph.com"; MACHINE_NAME=smithi; CEPH_BRANCH=wip-yuri-testing-2019-03-20-2018-mimic

teuthology-suite -v --ceph-repo https://github.com/ceph/ceph-ci.git --suite-repo https://github.com/ceph/ceph-ci.git -c $CEPH_BRANCH -m $MACHINE_NAME -s rados --subset 35/2999 -k distro -p 90 -e $CEPH_QA_MAIL --suite-branch $CEPH_BRANCH
2019-03-22 20:18:35,117.117 INFO:teuthology.suite:Using random seed=7853
2019-03-22 20:18:35,118.118 INFO:teuthology.suite.run:kernel sha1: distro
2019-03-22 20:18:35,380.380 DEBUG:teuthology.repo_utils:git ls-remote https://github.com/ceph/ceph-ci wip-yuri-testing-2019-03-20-2018-mimic -> 1467b0d781b1e0fbef738a1f80be854d708681ef
2019-03-22 20:18:35,380.380 INFO:teuthology.suite.run:ceph sha1: 1467b0d781b1e0fbef738a1f80be854d708681ef
2019-03-22 20:18:35,381.381 DEBUG:teuthology.suite.util:Defaults for machine_type smithi distro centos: arch=x86_64, release=centos/7, pkg_type=rpm
2019-03-22 20:18:35,381.381 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F7%2Fx86_64&sha1=1467b0d781b1e0fbef738a1f80be854d708681ef
2019-03-22 20:18:35,548.548 INFO:teuthology.suite.run:ceph version: 13.2.5-98.g1467b0d
2019-03-22 20:18:35,548.548 INFO:teuthology.suite.run:teuthology branch: master
2019-03-22 20:18:35,801.801 DEBUG:teuthology.repo_utils:git ls-remote https://github.com/ceph/ceph-ci.git wip-yuri-testing-2019-03-20-2018-mimic -> 1467b0d781b1e0fbef738a1f80be854d708681ef
2019-03-22 20:18:36,042.042 DEBUG:teuthology.repo_utils:git ls-remote https://github.com/ceph/ceph-ci.git wip-yuri-testing-2019-03-20-2018-mimic -> 1467b0d781b1e0fbef738a1f80be854d708681ef
2019-03-22 20:18:36,042.042 INFO:teuthology.suite.run:ceph-ci branch: wip-yuri-testing-2019-03-20-2018-mimic 1467b0d781b1e0fbef738a1f80be854d708681ef
2019-03-22 20:18:36,046.046 INFO:teuthology.repo_utils:/home/yuriw/src/github.com_ceph_ceph-c_wip-yuri-testing-2019-03-20-2018-mimic was just updated; assuming it is current
2019-03-22 20:18:36,046.046 INFO:teuthology.repo_utils:Resetting repo at /home/yuriw/src/github.com_ceph_ceph-c_wip-yuri-testing-2019-03-20-2018-mimic to branch wip-yuri-testing-2019-03-20-2018-mimic
2019-03-22 20:18:36,134.134 DEBUG:teuthology.suite.run:Suite rados in /home/yuriw/src/github.com_ceph_ceph-c_wip-yuri-testing-2019-03-20-2018-mimic/qa/suites/rados
2019-03-22 20:18:36,134.134 INFO:teuthology.suite.build_matrix:Subset=35/2999
Traceback (most recent call last):
File "/home/yuriw/teuthology/virtualenv/bin/teuthology-suite", line 11, in
load_entry_point('teuthology', 'console_scripts', 'teuthology-suite')()
File "/home/yuriw/teuthology/scripts/suite.py", line 152, in main
return teuthology.suite.main(args)
File "/home/yuriw/teuthology/teuthology/suite/init.py", line 144, in main
run.prepare_and_schedule()
File "/home/yuriw/teuthology/teuthology/suite/run.py", line 343, in prepare_and_schedule
num_jobs = self.schedule_suite()
File "/home/yuriw/teuthology/teuthology/suite/run.py", line 494, in schedule_suite
build_matrix(suite_path, subset=self.args.subset, seed=self.args.seed)
File "/home/yuriw/teuthology/teuthology/suite/build_matrix.py", line 57, in build_matrix
mat, first, matlimit = _get_matrix(path, subset)
File "/home/yuriw/teuthology/teuthology/suite/build_matrix.py", line 67, in _get_matrix
mat = _build_matrix(path, mincyclicity=outof)
File "/home/yuriw/teuthology/teuthology/suite/build_matrix.py", line 146, in _build_matrix
fn)
File "/home/yuriw/teuthology/teuthology/suite/build_matrix.py", line 130, in _build_matrix
item=fn)
File "/home/yuriw/teuthology/teuthology/suite/build_matrix.py", line 146, in _build_matrix
fn)
File "/home/yuriw/teuthology/teuthology/suite/build_matrix.py", line 84, in _build_matrix
raise IOError('%s does not exist (abs %s)' % (path, os.path.abspath(path)))
IOError: /home/yuriw/src/github.com_ceph_ceph-c_wip-yuri-testing-2019-03-20-2018-mimic/qa/suites/rados/singleton-bluestore/objectstore/bluestore.yaml does not exist (abs /home/yuriw/src/github.com_ceph_ceph-c_wip-yuri-testing-2019-03-20-2018-mimic/qa/suites/rados/singleton-bluestore/objectstore/bluestore.yaml)

@ifed01

This comment has been minimized.

Copy link
Contributor Author

commented Mar 22, 2019

@yuriw - yeah, the same here. Already fixed.

@yuriw

This comment has been minimized.

Copy link
Contributor

commented Mar 22, 2019

@yuriw

This comment has been minimized.

Copy link
Contributor

commented Mar 25, 2019

@ifed01 same issue:

CEPH_QA_MAIL="ceph-qa@lists.ceph.com"; MACHINE_NAME=smithi; CEPH_BRANCH=wip-yuri5-testing-2019-03-22-2035-mimic

teuthology-suite -v --ceph-repo https://github.com/ceph/ceph-ci.git --suite-repo https://github.com/ceph/ceph-ci.git -c $CEPH_BRANCH -m $MACHINE_NAME -s rados --subset 31/2999 -k distro -p 90 -e $CEPH_QA_MAIL -l 150
2019-03-25 15:34:28,653.653 INFO:teuthology.suite:Using random seed=4005
2019-03-25 15:34:28,654.654 INFO:teuthology.suite.run:kernel sha1: distro
2019-03-25 15:34:28,924.924 DEBUG:teuthology.repo_utils:git ls-remote https://github.com/ceph/ceph-ci wip-yuri5-testing-2019-03-22-2035-mimic -> bb6749df9c1e0a83d2f41f74bb925d7fafd19dfd
2019-03-25 15:34:28,924.924 INFO:teuthology.suite.run:ceph sha1: bb6749df9c1e0a83d2f41f74bb925d7fafd19dfd
2019-03-25 15:34:28,924.924 DEBUG:teuthology.suite.util:Defaults for machine_type smithi distro centos: arch=x86_64, release=centos/7, pkg_type=rpm
2019-03-25 15:34:28,925.925 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F7%2Fx86_64&sha1=bb6749df9c1e0a83d2f41f74bb925d7fafd19dfd
2019-03-25 15:34:29,135.135 INFO:teuthology.suite.run:ceph version: 13.2.5-70.gbb6749d
2019-03-25 15:34:29,135.135 INFO:teuthology.suite.run:teuthology branch: master
2019-03-25 15:34:29,379.379 DEBUG:teuthology.repo_utils:git ls-remote https://github.com/ceph/ceph-ci.git wip-yuri5-testing-2019-03-22-2035-mimic -> bb6749df9c1e0a83d2f41f74bb925d7fafd19dfd
2019-03-25 15:34:29,615.615 DEBUG:teuthology.repo_utils:git ls-remote https://github.com/ceph/ceph-ci.git wip-yuri5-testing-2019-03-22-2035-mimic -> bb6749df9c1e0a83d2f41f74bb925d7fafd19dfd
2019-03-25 15:34:29,616.616 INFO:teuthology.suite.run:ceph-ci branch: wip-yuri5-testing-2019-03-22-2035-mimic bb6749df9c1e0a83d2f41f74bb925d7fafd19dfd
2019-03-25 15:34:29,629.629 INFO:teuthology.repo_utils:Cloning https://github.com/ceph/ceph-ci.git wip-yuri5-testing-2019-03-22-2035-mimic from upstream
2019-03-25 15:34:34,580.580 INFO:teuthology.repo_utils:Resetting repo at /home/yuriw/src/github.com_ceph_ceph-c_wip-yuri5-testing-2019-03-22-2035-mimic to branch wip-yuri5-testing-2019-03-22-2035-mimic
2019-03-25 15:34:35,048.048 DEBUG:teuthology.suite.run:Suite rados in /home/yuriw/src/github.com_ceph_ceph-c_wip-yuri5-testing-2019-03-22-2035-mimic/qa/suites/rados
2019-03-25 15:34:35,048.048 INFO:teuthology.suite.build_matrix:Subset=31/2999
Traceback (most recent call last):
File "/home/yuriw/teuthology/virtualenv/bin/teuthology-suite", line 11, in
load_entry_point('teuthology', 'console_scripts', 'teuthology-suite')()
File "/home/yuriw/teuthology/scripts/suite.py", line 152, in main
return teuthology.suite.main(args)
File "/home/yuriw/teuthology/teuthology/suite/init.py", line 144, in main
run.prepare_and_schedule()
File "/home/yuriw/teuthology/teuthology/suite/run.py", line 343, in prepare_and_schedule
num_jobs = self.schedule_suite()
File "/home/yuriw/teuthology/teuthology/suite/run.py", line 494, in schedule_suite
build_matrix(suite_path, subset=self.args.subset, seed=self.args.seed)
File "/home/yuriw/teuthology/teuthology/suite/build_matrix.py", line 57, in build_matrix
mat, first, matlimit = _get_matrix(path, subset)
File "/home/yuriw/teuthology/teuthology/suite/build_matrix.py", line 67, in _get_matrix
mat = _build_matrix(path, mincyclicity=outof)
File "/home/yuriw/teuthology/teuthology/suite/build_matrix.py", line 146, in _build_matrix
fn)
File "/home/yuriw/teuthology/teuthology/suite/build_matrix.py", line 130, in _build_matrix
item=fn)
File "/home/yuriw/teuthology/teuthology/suite/build_matrix.py", line 146, in _build_matrix
fn)
File "/home/yuriw/teuthology/teuthology/suite/build_matrix.py", line 84, in _build_matrix
raise IOError('%s does not exist (abs %s)' % (path, os.path.abspath(path)))
IOError: /home/yuriw/src/github.com_ceph_ceph-c_wip-yuri5-testing-2019-03-22-2035-mimic/qa/suites/rados/singleton-bluestore/objectstore/bluestore-bitmap.yaml does not exist (abs /home/yuriw/src/github.com_ceph_ceph-c_wip-yuri5-testing-2019-03-22-2035-mimic/qa/suites/rados/singleton-bluestore/objectstore/bluestore-bitmap.yaml)

@yuriw

This comment has been minimized.

Copy link
Contributor

commented Mar 25, 2019

@ifed01 let me know when I can retry

qa/suites: fix bluestore links
Signed-off-by: Sage Weil <sage@redhat.com>
(cherry picked from commit 269910f)

@ifed01 ifed01 force-pushed the ifed01:wip-ifed-bmap-alloc-mimic branch from ae22fb3 to f26b219 Mar 26, 2019

@ifed01

This comment has been minimized.

Copy link
Contributor Author

commented Mar 26, 2019

@yuriw - thanks a lot for you efforts, hopefully fixed.

@yuriw

This comment has been minimized.

@yuriw

This comment has been minimized.

Copy link
Contributor

commented Mar 29, 2019

tests passed, @liewegas please approve/merge (re: https://trello.com/c/P1591H7t)

@yuriw yuriw merged commit 64524b5 into ceph:mimic Apr 1, 2019

4 checks passed

Docs: build check OK - docs built
Details
Signed-off-by all commits in this PR are signed
Details
Unmodified Submodules submodules for project are unmodified
Details
make check make check succeeded
Details

@ifed01 ifed01 deleted the ifed01:wip-ifed-bmap-alloc-mimic branch Apr 1, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.