Want ability to take a crucible dataset out of provisioning pool

[askfongjojo]

Similar to #2483 (the ability to take a sled out of provisioning pool), there are multiple circumstatnces (e.g. physical disk failure, crucible-agent failing to come up #3416) that causes a crucible dataset to be non-functional. Regardless of whether the logic for picking crucible is fully randomized, we need a support tool to mark a crucible as "bad".

The tool (probably an API) will be used by Oxide support initially. Eventually, we'd want a heart-beat mechanism in place to make the decision of when to take a crucible out of or back into the provisioning pool.

[askfongjojo]

This API can also be triggered optionally by the "take sled out of provisioning pool" feature since talking a sled out will mean not provisioning to its disks in most cases.

[askfongjojo]

Volume/region background clean-up also needs to skip crucible zones that are already removed - #4331.

[davepacheco]

@sunshowers I think this issue was resolved as of #5032. Is that right?

[davepacheco]

Answering my own question: we discussed this briefly in today's update call. This ticket is different from #5032 in two ways: it covers not just an underlying database facility, but an API and tool that can be used by support (at least) to control this behavior; and it covers dataset-level granularity, whereas #5032 only covers sled-level.