Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

mimic: crush: backport recent upmap fixes #27963

Merged
merged 6 commits into from May 9, 2019

Conversation

Projects
None yet
5 participants
@xiexingguo
Copy link
Member

commented May 5, 2019

This pr backports the following fixes:
#26187
#26478
#26605 (included to make backport smooth)
#27142

Related backport tickets:
http://tracker.ceph.com/issues/38966

  • References tracker ticket
  • Updates documentation if necessary
  • Includes tests for new functionality or reproducer for bug

xiexingguo added some commits Jan 26, 2019

osd/OSDMap: more improvements to upmap
- add ability of appending a 2nd, 3rd, etc... pair to existing upmaps
  when possible, rather than just continuing to the next PG
- handle the underfull case: we can rm-pg-upmap-items if there exist
  any upmaps which remapped a PG out from an underfull OSD

See-also: http://tracker.ceph.com/issues/37940
Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>
(cherry picked from commit a7d2adf)

Conflicts:
	- various option changes since luminous
	- s/assert/ceph_assert/
        - s/conf./conf->/
osd/OSDMap: using std::vector::reserve to reduce memory reallocation
In C++ vectors are dynamic arrays.
Vectors are assigned memory in blocks of contiguous locations.
When the memory allocated for the vector falls short of storing
new elements, a new memory block is allocated to vector and all
elements are copied from the old location to the new location.
This reallocation of elements helps vectors to grow when required.
However, it is a costly operation and time complexity is involved
in this step is linear.
Try to use std::vector::reserve whenever possible if performance
matters.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>
(cherry picked from commit 4a0eabb)
crush: fix upmap overkill
It appears that OSDMap::maybe_remove_pg_upmaps's sanity checks
are overzealous. With some customized crush rules it is possible
for osdmaptool to generate valid upmaps, but maybe_remove_pg_upmaps
will cancel them.

Fixes: http://tracker.ceph.com/issues/37968
Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>
(cherry picked from commit 5c4d241)

Conflicts:
	- maybe_remove_pg_upmaps input changing
        - slight c++11 auto conflicts
osd/OSDMap: drop local pool filter in calc_pg_upmaps
The local pre-loaded pool filter is completely redundant since
the below check:

	if (!only_pools.empty() && !only_pools.count(i.first))

could reliably catch both cases - either optimization should be
restricted to specific pools feeded, or all existing pools.

Let's clean it up.

Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>
(cherry picked from commit 3e6bad9)
osd/OSDMap: calc_pg_upmaps - restrict optimization to origin pools only
The current implementation will try to cancel any pg_upmaps that
would otherwise re-map a PG out from an underfull osd, which is wrong,
e.g., because it could reliably fire the following assert:

src/osd/OSDMap.cc: 4405: FAILED assert(osd_weight.count(i.first))

Also it would not match the expectation if automatic balancing
has been strictly restricted to some specific pools by admin.

Fix by excluding any wild PG that does not belong to the origin pools
passed in when trying to do upmap/unmap.

Fixes: http://tracker.ceph.com/issues/38897
Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>
(cherry picked from commit 01e8e94)
osd/OSDMap: add log for better debugging
Signed-off-by: xie xingguo <xie.xingguo@zte.com.cn>
(cherry picked from commit a89281f)

@xiexingguo xiexingguo added this to the mimic milestone May 5, 2019

@xiexingguo

This comment has been minimized.

Copy link
Member Author

commented May 5, 2019

@dvanders FYI

@smithfarm smithfarm requested review from liewegas, badone and neha-ojha May 6, 2019

@smithfarm smithfarm added the core label May 6, 2019

@smithfarm smithfarm requested a review from jan--f May 6, 2019

@smithfarm smithfarm changed the title mimic: backport recent upmap fixes mimic: crush: backport recent upmap fixes May 6, 2019

@yuriw

This comment has been minimized.

Copy link
Contributor

commented May 6, 2019

@jan--f

jan--f approved these changes May 9, 2019

@yuriw yuriw merged commit a2acac3 into ceph:mimic May 9, 2019

4 checks passed

Docs: build check OK - docs built
Details
Signed-off-by all commits in this PR are signed
Details
Unmodified Submodules submodules for project are unmodified
Details
make check make check succeeded
Details

@xiexingguo xiexingguo deleted the xiexingguo:wip-mimic-upmap-fixes branch May 10, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.