New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fs: Mantle: A Programmable Metadata Load Balancer #10887

Merged
merged 3 commits into from Oct 26, 2016

Conversation

Projects
None yet
5 participants
@michaelsevilla
Contributor

michaelsevilla commented Aug 26, 2016

@smithfarm smithfarm added the feature label Aug 28, 2016

@smithfarm smithfarm changed the title from Mantle: A Programmable Metadata Load Balancer to fs: Mantle: A Programmable Metadata Load Balancer Aug 28, 2016

Show outdated Hide outdated src/mds/MDBalancer.cc Outdated
Show outdated Hide outdated src/mon/MonCommands.h Outdated
Show outdated Hide outdated src/mds/MDBalancer.cc Outdated
Show outdated Hide outdated src/mds/MDBalancer.cc Outdated
Show outdated Hide outdated src/mds/MDBalancer.cc Outdated
Show outdated Hide outdated src/mds/MDBalancer.cc Outdated
Show outdated Hide outdated src/mds/MDBalancer.cc Outdated
Show outdated Hide outdated src/mds/MDSMap.cc Outdated
Show outdated Hide outdated src/mds/MDSMap.cc Outdated
Show outdated Hide outdated src/mds/MDSMap.cc Outdated
Show outdated Hide outdated src/mds/MDSMap.cc Outdated
Show outdated Hide outdated src/mds/MDSMap.cc Outdated
Show outdated Hide outdated src/mds/MDSMap.h Outdated
Show outdated Hide outdated src/mds/Makefile-client.am Outdated
Show outdated Hide outdated src/mds/MDSMap.h Outdated
Show outdated Hide outdated src/mds/Mantle.h Outdated
Show outdated Hide outdated src/mds/Mantle.h Outdated
@@ -0,0 +1,34 @@
#include <lua.hpp>

This comment has been minimized.

@jcsp

jcsp Sep 9, 2016

Contributor

please add #ifdef header guards and the usual modelines+copyright header (copy paste and put your name in). Same in Mantle.cc

@jcsp

jcsp Sep 9, 2016

Contributor

please add #ifdef header guards and the usual modelines+copyright header (copy paste and put your name in). Same in Mantle.cc

@@ -0,0 +1,235 @@
Mantle

This comment has been minimized.

@jcsp

jcsp Sep 9, 2016

Contributor

Wow, I wish every PR came with this much documentation!

@jcsp

jcsp Sep 9, 2016

Contributor

Wow, I wish every PR came with this much documentation!

@jcsp

This comment has been minimized.

Show comment
Hide comment
@jcsp

jcsp Sep 9, 2016

Contributor

This looks to be exactly what was discussed at the last CDM, so all of my comments are just detail things. At a high level: 👍

Does #7338 need poking along somehow?

Contributor

jcsp commented Sep 9, 2016

This looks to be exactly what was discussed at the last CDM, so all of my comments are just detail things. At a high level: 👍

Does #7338 need poking along somehow?

@michaelsevilla

This comment has been minimized.

Show comment
Hide comment
@michaelsevilla

michaelsevilla Sep 9, 2016

Contributor

I already poked #7338 and @noahdesu is going to take care of it.

Dumb question: is it better to rebase (keep history clean but lose your comments) or just add more commits?

Contributor

michaelsevilla commented Sep 9, 2016

I already poked #7338 and @noahdesu is going to take care of it.

Dumb question: is it better to rebase (keep history clean but lose your comments) or just add more commits?

@jcsp

This comment has been minimized.

Show comment
Hide comment
@jcsp

jcsp Sep 9, 2016

Contributor

You can either add more commits (and rebase later before merge) or just do a straight history-squashing rebase.

Usually we leave comments on the "files changed" view (not individual commits) so that they survive a rebase, but I mistakenly put some on commits here -- my fault if they get lost in a rebase.

Contributor

jcsp commented Sep 9, 2016

You can either add more commits (and rebase later before merge) or just do a straight history-squashing rebase.

Usually we leave comments on the "files changed" view (not individual commits) so that they survive a rebase, but I mistakenly put some on commits here -- my fault if they get lost in a rebase.

@gregsfortytwo

This comment has been minimized.

Show comment
Hide comment
@gregsfortytwo

gregsfortytwo Sep 9, 2016

Member

Luckily Github finally fixed that nonsense, at least as long as you navigate from the PR view into the commit (not sure if you just go straight to a commit). You can tell it's working because all the comments are on the diff, not on a commit. ;)

Personally I prefer commits prefixed with "SQUASH" that are intended to be rebase, as it makes life easier on the reviewer when there are follow-up comments, but that may just be me.

Member

gregsfortytwo commented Sep 9, 2016

Luckily Github finally fixed that nonsense, at least as long as you navigate from the PR view into the commit (not sure if you just go straight to a commit). You can tell it's working because all the comments are on the diff, not on a commit. ;)

Personally I prefer commits prefixed with "SQUASH" that are intended to be rebase, as it makes life easier on the reviewer when there are follow-up comments, but that may just be me.

@gregsfortytwo

This comment has been minimized.

Show comment
Hide comment
@gregsfortytwo

gregsfortytwo Oct 18, 2016

Member

I see we did get some updated commits at some point that at least invalidated most comments. @jcsp, want to check this again? Be nice to get it in for @batrick's scaling tests.

Member

gregsfortytwo commented Oct 18, 2016

I see we did get some updated commits at some point that at least invalidated most comments. @jcsp, want to check this again? Be nice to get it in for @batrick's scaling tests.

@jcsp

This comment has been minimized.

Show comment
Hide comment
@jcsp

jcsp Oct 24, 2016

Contributor

This is good to go now from my point of view (apologies @michaelsevilla for delay). Please could you do a quick rebase as something else has touched CMakeLists.txt in the meantime

Contributor

jcsp commented Oct 24, 2016

This is good to go now from my point of view (apologies @michaelsevilla for delay). Please could you do a quick rebase as something else has touched CMakeLists.txt in the meantime

mantle: write balancer policies in Lua
Introduces Mantle, a programmable metadata load balancer. Policies for making
migration decisions are written in Lua but the Migrator and Balancer modules
still do fragmentation and migration. If the Lua balancer fails, control falls
back to the original balancer implementation.

Signed-off-by: Michael Sevilla <mikesevilla3@gmail.com>

michaelsevilla added some commits Feb 2, 2016

mantle: store balancer in RADOS, balancer version in MDSMap
- add docs and sample balancer (greedy-spill)

Signed-off-by: Michael Sevilla <mikesevilla3@gmail.com>
mantle: re-design and fixes
- fix legacy encoding in mds map and add balancer to dumps
- fix blocking rados read and remove temporary files
- fix beacon message spamming

Signed-off-by: Michael Sevilla <mikesevilla3@gmail.com>
@michaelsevilla

This comment has been minimized.

Show comment
Hide comment
@michaelsevilla

michaelsevilla Oct 26, 2016

Contributor

Done -- @batrick: could you send me info on the scaling tests?

Contributor

michaelsevilla commented Oct 26, 2016

Done -- @batrick: could you send me info on the scaling tests?

@batrick

This comment has been minimized.

Show comment
Hide comment
Member

batrick commented Oct 26, 2016

@jcsp jcsp merged commit 32b15e3 into ceph:master Oct 26, 2016

2 checks passed

Signed-off-by all commits in this PR are signed
Details
default Build finished.
Details
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment