New safety checks on exclusions that could take a database unavailable

[brownleej]

When we're running in a multi-DC configuration, it's possible to run an exclusion that can take the database unavailable. For instance, if a database is configured to run satellite logs, but we exclude all of the satellite processes, the database would go unavailable. Should we add a safety check in the exclusion to prevent this kind of mistake?

On a related note, it's possible that an exclusion could put the database into a configuration that is viable but undesirable. For instance, if a database is configured to run 5 proxies, but we exclude all of the stateless class processes but 1, the database will go down to 1 proxy, and it will put that proxy on that single process along with all of the other stateless roles. Should we add safety checks for this kind of thing as well?

[xumengpanda]

Do we have a formal definition of what a desired database configuration should be?
Based on that, can we measure the "fitness" of a configuration? (Do we have an equation to compute the "fitness" of a database configuration?)

[apkar]

Multi-DC is good example. In general we should add checks in exclude to avoid excluding instances that can make database available.

exclude command has checks to make sure cluster would not end up low disk space problems.

https://github.com/apple/foundationdb/blob/master/fdbcli/fdbcli.actor.cpp#L2427

We don't have any checks to prevent someone excluding too many stateless/tlog processes which can actually prevent cluster controller to recruit enough roles to form a cluster, which would cause immediate outage.

It would be great to have some checks around this.

[sfc-gh-mpilman]

The way we want to do this is by calling into something like BetterMasterExists. So the high level idea is that the management API would ask the ClusterController whether a recovery after an exclude would succeed.