mgr: skip first non-zero incremental in PGMap::apply_incremental() #18347
Conversation
|
retest this please (jenkins) |
|
Looks sane at a glance but I'd like a second opinion (@tchaikov) |
|
This patch is very welcome, as we are probably suffering the same pain. |
src/mon/PGMap.cc
Outdated
| pg_sum_delta.stats.add(d.stats); | ||
| // if mgr restarts first update of pg_sum is previous pg_sum | ||
| // so skip this huge value | ||
| if (!(pg_sum_old.stats.sum == object_stat_sum_t())) { |
There was a problem hiding this comment.
@agutikov instead, i think it'd be simpler if we just skip this interval if stamp.is_zero() or pg_sum_old.stats.sum.is_zero(). when a ceph-mgr boots, the first interval's stamp is zero, and the so is the pg_sum_old.stats.sum. but the pg stats takes time to pupulate, so the next interval is very likely have a non-zero time stamp, but its pg_sum_old.stats.sum is still zero.
in other words, i'd suggest
- check both
stampandpg_sum_old.stats.sumat the same place - move the
delta_tcalculation down to the if block, - and update the
stampafter checking it
i.e.
if (!stamp.is_zero() && !pg_sum_old.stats.sum.is_zero()) {
utime_t delta_t;
delta_t = inc.stamp;
delta_t -= stamp;
// calculate a delta, and average over the last 2 deltas.
pool_stat_t d = pg_sum;
d.stats.sub(pg_sum_old.stats);
pg_sum_deltas.push_back(make_pair(d, delta_t));
stamp_delta += delta_t;
pg_sum_delta.stats.add(d.stats);
unsigned smooth_intervals = (unsigned)MAX(1, cct ? cct->_conf->mon_stat_smooth_intervals : 1);
if (pg_sum_deltas.size() > smooth_intervals) {
pg_sum_delta.stats.sub(pg_sum_deltas.front().first.stats);
stamp_delta -= pg_sum_deltas.front().second;
pg_sum_deltas.pop_front();
}
}
stamp = inc.stamp;what do you think?
|
@tchaikov |
|
probably will need rebase and resolve the conflict after #18647 |
After initialization of PGMap instance PGMap::stamp is zero and this cause huge first delta. Also after mgr restart first non-zero value coming to PGMap::apply_incremental() is current pg_sum value so it produces unreasonably huge pg_sum_delta. This patch introduces a workaround to save pg_sum and not update pg_sum_delta by first non-zero incremental. Signed-off-by: Aleksei Gutikov <aleksey.gutikov@synesis.ru> Fixes: http://tracker.ceph.com/issues/21773
|
all commits in this PR are signed |
|
submodules for project are unmodified |
|
OK - docs built |
|
make check succeeded |
1 similar comment
|
make check succeeded |
|
Thanks! |
After initialization of PGMap instance PGMap::stamp is zero
and this cause huge first delta.
Also after mgr restart first non-zero value coming to PGMap::apply_incremental()
is current pg_sum value so it produces unreasonably huge pg_sum_delta.
This patch introduces a workaround to save pg_sum and not update pg_sum_delta
by first non-zero incremental.
Signed-off-by: Aleksei Gutikov aleksey.gutikov@synesis.ru
Fixes: http://tracker.ceph.com/issues/21773