mon: Go into ERR state if multiple PGs are stuck inactive

[wido]

If >=X PGs are stuck inactive longer than 'mon_pg_stuck_threshold'
we go into ERR state.

This is useful for situations where one or more PGs stay stuck in
peering or undersized state due to a OSD failure.

RBD volumes can become fully unresponsive if one or more PGs are inactive.

Fixes: #13923

[liewegas]

oops, be careful teh submodule isn't changed

[wido]

Indeed. They always sneak in silently.

[liewegas]

Hmm, is there any case where we woudln't want to go to ERR if there are PGs stuck inactive? Or where only 1 inactive PG is okay, but 2 isn't?

I'd lean toward just adding the err case unconditionally...

[wido]

@liewegas I don't know. But I didn't want to change the current way Ceph behaves. I'm fine with going to ERR state if one or more PGs are inactive. But other usecases which are plain RADOS storage might not?

Maybe clusters with >1000 OSDs don't want this?

We could always make it configurable and set the default to 1.

[liewegas]

We could always make it configurable and set the default to 1.

That sounds good to me!

[theanalyst]

Should the option be mon_pg_min_inactive_num or just mon_pg_min_inactive since this specifies a minimum number?

[wido]

Ok, @liewegas I've set the default to 1.

And good feedback @theanalyst, I changed the config option's name.

[theanalyst]

@wido Thanks! also the second commit is missing a signed-off-by

[wido]

@theanalyst I rebased and force pushed. It's now one commit and also signed off

[theanalyst]

Yep, Looks good