mon: restructure prime_pg_temp around a full pg mapping calculated on multiple CPUs #13207

liewegas · 2017-01-31T20:18:56Z

This is removing prime_pg_temps dependency on pg_map, with teh small exception of
checking for creating_pgs (which is being removed in a separate PR).

It also moves processing for calculating PG mappings and checking for needed pg_temp
values to a parallel work queue that runs on multiple cores.

Both work efforts (pg mapping calcs and prime_pg_temp) are time boxed with a graceful fallback.

tchaikov · 2017-02-03T15:40:44Z

i am still reading this.

liewegas · 2017-02-03T22:18:43Z

retest this please

tchaikov

will continue the review tmr.

tchaikov · 2017-02-06T15:00:14Z

src/CMakeLists.txt

@@ -459,6 +459,7 @@ set(libcommon_files
  msg/msg_types.cc
  common/hobject.cc
  osd/OSDMap.cc
+  osd/OSDMapMapping.cc


might want to squash this commit into the previous one.

tchaikov · 2017-02-14T12:46:10Z

src/osd/OSDMap.h

+  void _dump();
+
+public:
+  void get(pg_t pgid,


could pass by the reference.

tchaikov · 2017-02-14T13:07:54Z

src/osd/OSDMapMapping.cc

+  }
+}
+
+void OSDMapMapping::_dump()


could override the operator<<() in case we want to debug it using the dout facility.

tchaikov · 2017-02-14T13:30:06Z

src/mon/OSDMonitor.cc

+    for (auto osd : osds) {
+      const vector<pg_t>& pgs = mapping.get_osd_acting_pgs(osd);
+      dout(20) << __func__ << " osd." << osd << " " << pgs << dendl;
+      for (auto pgid : pgs) {


const auto& as pg_t is not a primitive type.

tchaikov · 2017-02-14T13:31:43Z

src/mon/OSDMonitor.cc

-			       ceph::unordered_map<pg_t, pg_stat_t>::iterator pp)
+void OSDMonitor::prime_pg_temp(
+  OSDMap& next,
+  pg_t pgid)


could pass by const pg_t&.

tchaikov · 2017-02-14T13:36:47Z

src/mon/OSDMonitor.cc

+		     << g_conf->mon_osd_prime_pg_temp_max_time
+		     << " seconds, stopping"
+		     << dendl;
+	    abort = true;


could just return here.

tchaikov · 2017-02-14T13:37:05Z

src/mon/OSDMonitor.cc

+		     << g_conf->mon_osd_prime_pg_temp_max_time
+		     << " seconds, stopping"
+		     << dendl;
+	    abort = true;


could just return here.

tchaikov · 2017-02-14T13:47:22Z

src/mon/OSDMonitor.cc

-    }
-  }
-  return num;
+  pending_inc.new_pg_temp[pgid] = cur_acting;


better off using

pending_inc.new_pg_temp[pgid] = std::move(cur_acting);

so less cpu cycles spent on copy ctor.

tchaikov · 2017-02-14T13:48:45Z

src/mon/OSDMonitor.cc

 	   << dendl;
-  pending_inc.new_pg_temp[pgid] = cur_acting;
+  pending_inc.new_pg_temp[pgid] = acting;


might want to take this chance to use std::move(acting)

liewegas · 2017-02-14T15:26:33Z

simplified the bool abort thing and squashed the first 2 commits.

leaving pg_t args as pass-by-value since it's only a 16 byte struct?

liewegas · 2017-02-15T22:28:23Z

retest this please

tchaikov · 2017-02-16T02:52:48Z

src/osd/OSDMapMapping.cc

+
+void ParallelPGMapper::Job::finish_one()
+{
+  Context *fin = 0;


nit, use nullptr for more consistency.

tchaikov · 2017-02-16T02:55:56Z

src/osd/OSDMapMapping.h

+protected:
+  CephContext *cct;
+
+  struct item {


s/item/Item/

tchaikov · 2017-02-16T05:47:18Z

src/mon/OSDMonitor.cc

-	  break;
-	}
-      }
+    ParallelPGMapper::Job *job = new PrimeTempJob(next, this);


so we can allocate job on stack?

tchaikov · 2017-02-16T08:04:45Z

src/mon/OSDMonitor.cc

-	  break;
+    utime_t stop = ceph_clock_now();
+    stop += g_conf->mon_osd_prime_pg_temp_max_time;
+    int chunk = 1000;


could mark chunk a const or constexpr.

tchaikov · 2017-02-16T08:17:54Z

src/mon/OSDMonitor.h

@@ -111,10 +112,14 @@ struct failure_info_t {

 class OSDMonitor : public PaxosService {
  CephContext *cct;
+
+  ParallelPGMapper mapper;                        ///< for background pg work
+  unique_ptr<OSDMapMapping> mapping;              ///< pg <-> osd mappings


@liewegas i don't understand why we want to use a pointer of OSDMapMapping instead of a plain OSDMapMapping? the same applies to mapping_job.

because they are optional, and because the classes weren't necessarily written with re-use in mind.

tchaikov · 2017-02-16T09:02:20Z

src/mon/PGMap.cc

+  }
+
+  if (check_all) {
+    for (auto p : pg_map->pg_stat) {


might want to use const auto& p, as pair<pg_t,pg_stat_t> is a relatively large struct.

tchaikov · 2017-02-16T09:16:01Z

src/mon/PGMonitor.cc

@@ -67,8 +67,8 @@ void PGMonitor::on_restart()
 void PGMonitor::on_active()
 {
  if (mon->is_leader()) {
+    check_all_pgs = true;


yeah, need_check_down_pgs was misleading before.

tchaikov · 2017-02-16T09:36:45Z

src/osd/OSDMap.h

 class CephContext;
 class CrushWrapper;
+class OSDMap;


this is not necessary any more, as OSDMapMapping is declared in its own header file.

tchaikov · 2017-02-16T09:43:38Z

src/osd/OSDMapMapping.cc

+    pools.emplace(p.first, PoolMapping(p.second.get_size(),
+				       p.second.get_pg_num()));
+  }
+  while (q != pools.end()) {


nit, just put pools.erase(q , pools.end());

tchaikov · 2017-02-16T09:49:08Z

src/osd/OSDMapMapping.cc

+    osdmap.pg_to_up_acting_osds(
+      pg_t(ps, pool),
+      &up, &up_primary, &acting, &acting_primary);
+    i->second.set(ps, up, up_primary, acting, acting_primary);


could use std::move(up) and std::move(acting) for better performance.

liewegas · 2017-02-16T15:21:32Z

updated; will squash once reviewed!

tchaikov · 2017-02-16T16:49:18Z

lgtm.

This is a precalculated table of all pg mappings. Signed-off-by: Sage Weil <sage@redhat.com>

This is temporarily; we'll put this in an async, parallelized thread shortly. Signed-off-by: Sage Weil <sage@redhat.com>

Simplify the code and remove most of the dependency on PGMap. We still need PGMap for the creating_pgs to ignore; that can be dropped later. Signed-off-by: Sage Weil <sage@redhat.com>

Calculate a mapping in parallel over a workqueue + threadpool. Signed-off-by: Sage Weil <sage@redhat.com>

Signed-off-by: Sage Weil <sage@redhat.com>

…t finish Start the background mapping job when we go active. If it has finished by the time we need to prime, use it. If not, skip prime_pg_temp. Signed-off-by: Sage Weil <sage@redhat.com>

The reverse mapping from up osds to pgs is not used; disable it for now. Signed-off-by: Sage Weil <sage@redhat.com>

Signed-off-by: Sage Weil <sage@redhat.com>

…nyway If we have enough noteworthy OSDs it is likely we will touch all or most PGs anyway, at which point it is faster and simpler to check all pgs. Signed-off-by: Sage Weil <sage@redhat.com>

Make Job a parent class that can be specialized for any per-PG batch work. Signed-off-by: Sage Weil <sage@redhat.com>

Do the prime_pg_temp work in parallel using the mapper. Signed-off-by: Sage Weil <sage@redhat.com>

Signed-off-by: Sage Weil <sage@redhat.com>

Two big changes here: 1) Move the updating into PGMapUpdater along with the other related checks. Cleaner! 2) We use the current OSDMap to mark things stale. Before, we had a weird two stage process: first we would apply the OSDMap to the PGMap, and then the *next* time the OSDMap updated we would look for stale PGs. This was silly. Instead, we mark the PGs stale when we process the OSDMap that made them stale. We have a be slightly careful because the PGMap stats primaries might be a bit old and, for example, might have a smaller max_osd than what we have now. This change is motivated right now because the new prime_pg_temp code is smarter about a failed OSD. While previously failing an OSD would install a pg_temp mapping to the old and new OSDs, even though the old OSD is now down, the new code does not. And previously failing both OSDs for a PG would always result in a subsequent OSDMap update removing those bad pg_temp values, giving check_down_pgs() a change to mark the PGs stale, while the new code has not subsequent OSDMap update, which would leave the PG unstale, causing the dump_stuck.py test to fail. Signed-off-by: Sage Weil <sage@redhat.com>

Signed-off-by: Sage Weil <sage@redhat.com>

tchaikov · 2017-02-18T14:55:03Z

retest this please.

liewegas added core feature mon labels Jan 31, 2017

liewegas requested review from tchaikov and jecluis January 31, 2017 20:19

liewegas force-pushed the wip-mapping branch from c1a3445 to ecca454 Compare January 31, 2017 21:36

liewegas added the wip-sage-testing label Feb 3, 2017

liewegas force-pushed the wip-mapping branch 3 times, most recently from c83e936 to 94982cf Compare February 7, 2017 04:34

liewegas removed the wip-sage-testing label Feb 10, 2017

liewegas force-pushed the wip-mapping branch from 2c35725 to 636027b Compare February 10, 2017 14:57

liewegas added needs-qa wip-sage-testing labels Feb 13, 2017

tchaikov reviewed Feb 14, 2017

View reviewed changes

liewegas force-pushed the wip-mapping branch 2 times, most recently from 52d0b1f to 95fd496 Compare February 14, 2017 15:26

liewegas force-pushed the wip-mapping branch from 95fd496 to 4d06121 Compare February 16, 2017 03:57

tchaikov reviewed Feb 16, 2017

View reviewed changes

tchaikov approved these changes Feb 16, 2017

View reviewed changes

liewegas added 4 commits February 16, 2017 12:04

osd/OSDMap: add OSDMapMapping class

5bdf539

This is a precalculated table of all pg mappings. Signed-off-by: Sage Weil <sage@redhat.com>

mon/OSDMonitor: maintain full mapping, synchronously

6af2759

This is temporarily; we'll put this in an async, parallelized thread shortly. Signed-off-by: Sage Weil <sage@redhat.com>

mon/OSDMonitor: prime_pg_temp based on OSDMapMapping

cc12b5f

Simplify the code and remove most of the dependency on PGMap. We still need PGMap for the creating_pgs to ignore; that can be dropped later. Signed-off-by: Sage Weil <sage@redhat.com>

osd/OSDMapMapping: add ParallelMapper

e7bfc65

Calculate a mapping in parallel over a workqueue + threadpool. Signed-off-by: Sage Weil <sage@redhat.com>

liewegas added 13 commits February 16, 2017 12:04

mon: add a threadpool (cpu_tp)

2e2041f

Signed-off-by: Sage Weil <sage@redhat.com>

mon/OSDMonitor: calculate new mapping in parallel

afa11a0

Signed-off-by: Sage Weil <sage@redhat.com>

mon/OSDMonitor: simplify prime_pg_temp slightly

6a8ffa7

Signed-off-by: Sage Weil <sage@redhat.com>

mon/OSDMonitor: run mapping job in background; skip prime if it didn'…

346043b

…t finish Start the background mapping job when we go active. If it has finished by the time we need to prime, use it. If not, skip prime_pg_temp. Signed-off-by: Sage Weil <sage@redhat.com>

osd/OSDMapMapping: disable up_rmap

0d2f9a4

The reverse mapping from up osds to pgs is not used; disable it for now. Signed-off-by: Sage Weil <sage@redhat.com>

mon/OSDMonitor: avoid priming same pgs more than once

e289b47

Signed-off-by: Sage Weil <sage@redhat.com>

osd/OSDMapMapper: count total pgs

3a072e4

Signed-off-by: Sage Weil <sage@redhat.com>

mon/OSDMonitor: prime all pgs if we estimate we're approaching that a…

b3358ee

…nyway If we have enough noteworthy OSDs it is likely we will touch all or most PGs anyway, at which point it is faster and simpler to check all pgs. Signed-off-by: Sage Weil <sage@redhat.com>

osd/OSDMapMapping: restructure mapper as generic PG work running

b1b09e1

Make Job a parent class that can be specialized for any per-PG batch work. Signed-off-by: Sage Weil <sage@redhat.com>

mon/OSDMonitor: use parallel work queue for prime_pg_temp (all)

fa632e6

Do the prime_pg_temp work in parallel using the mapper. Signed-off-by: Sage Weil <sage@redhat.com>

osd/OSDMapMapping: put mappings in a mempool

e193f8b

Signed-off-by: Sage Weil <sage@redhat.com>

mgr: mark stale PGs

3893432

Signed-off-by: Sage Weil <sage@redhat.com>

liewegas force-pushed the wip-mapping branch from 4f57916 to 3893432 Compare February 16, 2017 17:04

liewegas merged commit c1c1f2e into ceph:master Feb 18, 2017

liewegas deleted the wip-mapping branch February 18, 2017 18:58

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

mon: restructure prime_pg_temp around a full pg mapping calculated on multiple CPUs #13207

mon: restructure prime_pg_temp around a full pg mapping calculated on multiple CPUs #13207

liewegas commented Jan 31, 2017

tchaikov commented Feb 3, 2017

liewegas commented Feb 3, 2017

tchaikov left a comment

tchaikov Feb 6, 2017

tchaikov Feb 14, 2017

tchaikov Feb 14, 2017

tchaikov Feb 14, 2017

tchaikov Feb 14, 2017

tchaikov Feb 14, 2017

tchaikov Feb 14, 2017

tchaikov Feb 14, 2017

tchaikov Feb 14, 2017

liewegas commented Feb 14, 2017

liewegas commented Feb 15, 2017

tchaikov Feb 16, 2017

tchaikov Feb 16, 2017

tchaikov Feb 16, 2017

tchaikov Feb 16, 2017

tchaikov Feb 16, 2017

liewegas Feb 16, 2017

tchaikov Feb 16, 2017

tchaikov Feb 16, 2017

tchaikov Feb 16, 2017

tchaikov Feb 16, 2017

tchaikov Feb 16, 2017

liewegas commented Feb 16, 2017

tchaikov commented Feb 16, 2017

tchaikov commented Feb 18, 2017

mon: restructure prime_pg_temp around a full pg mapping calculated on multiple CPUs #13207

mon: restructure prime_pg_temp around a full pg mapping calculated on multiple CPUs #13207

Conversation

liewegas commented Jan 31, 2017

tchaikov commented Feb 3, 2017

liewegas commented Feb 3, 2017

tchaikov left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

liewegas commented Feb 14, 2017

liewegas commented Feb 15, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

liewegas commented Feb 16, 2017

tchaikov commented Feb 16, 2017

tchaikov commented Feb 18, 2017