New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

osd, os: reduce fiemap burden #14640

Merged
merged 3 commits into from Apr 21, 2017

Conversation

Projects
None yet
2 participants
@branch-predictor
Member

branch-predictor commented Apr 19, 2017

Currently, fiemap gathers file holes into std::map (or interval_set) and packs that into a bufferlist, which is then parsed back into map or interval set which is ineffective and hammers memory manager unnecessarily. This set of patches reduces that burden by moving hole data directly into user-provided map during recovery and/or sparse reads.

Signed-off-by: Piotr Dałek piotr.dalek@corp.ovh.com

@liewegas

this looks good! The interval_set thing in ReplicatedBackend is easy to fix, though.. may as well do it now

::decode(m, iter);
// TODO: bluestore uses interval_set internally, right now
// we move data from interval_set to map, then back from map
// to interval_set...

This comment has been minimized.

@liewegas

liewegas Apr 19, 2017

Member

i'd just make an interval_set ctor that takes the stl map and swap()'s it into place

branch-predictor added some commits Apr 18, 2017

common/interval_set.h: add move_into() and new ctor
move_into() will be used to move data from internal (private)
std::map into other, external std::map, without exposing internal one
to public. New ctor improves moving data from external map to internal
one.

Signed-off-by: Piotr Dałek <piotr.dalek@corp.ovh.com>
ObjectStore, *Store: extend fiemap interface
This extends fiemap interface in objectstore and all data stores so
it's possible to pass in the map used later by caller, instead of
encoding internal map into bufferlist, then parsing it back into the
same type of data container, wasting memory and CPU time in the process.

Signed-off-by: Piotr Dałek <piotr.dalek@corp.ovh.com>
ReplicatedPG: use new fiemap interface
Reduce memory allocator load during sparse reads and recovery by not
encoding sparse file extents from map into bufferlist, then back again
into map.

Signed-off-by: Piotr Dałek <piotr.dalek@corp.ovh.com>
@branch-predictor

This comment has been minimized.

Member

branch-predictor commented Apr 20, 2017

@liewegas done, repushed.

@liewegas liewegas merged commit 2be5b39 into ceph:master Apr 21, 2017

3 checks passed

Signed-off-by all commits in this PR are signed
Details
Unmodifed Submodules submodules for project are unmodified
Details
default Build finished.
Details

@branch-predictor branch-predictor deleted the ovh:bp-fiemap-burden branch May 2, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment