device_health_metrics pool defaults to "host" failure domain

[chinasubbareddym]

Is this a bug report or feature request?

• Bug Report

Deviation from expected behavior:
yes

Expected behavior:

should give us some option to change to different failure domain

How to reproduce it (minimal and precise):

by default device_health_metrics get created with crush rule of host based

File(s) to submit:

• Cluster CR (custom resource), typically called cluster.yaml, if necessary
• Operator's logs, if necessary
• Crashing pod(s) logs, if necessary

To get logs, use kubectl -n <namespace> logs <pod name>
When pasting logs, always surround them with backticks or use the insert code button from the Github UI.
Read Github documentation if you need help.

Environment:

• OS (e.g. from /etc/os-release):
• Kernel (e.g. uname -a):
• Cloud provider or hardware configuration:
• Rook version (use rook version inside of a Rook Pod):
• Storage backend version (e.g. for ceph do ceph -v):
• Kubernetes version (use kubectl version):
• Kubernetes cluster type (e.g. Tectonic, GKE, OpenShift):
• Storage backend status (e.g. for Ceph use ceph health in the Rook Ceph toolbox):

[travisn]

To manage the built-in pool device_health_metrics, how about the following approach?

• Create a CephBlockPool CR that has the desired settings, just like any other pool specification.
• The CR name must be device-health-metrics. K8s does not allow underscores in resource names, so the operator will need to translate this special pool name into the correct pool device_health_metrics that ceph expects.
• Support for changing the failure domain is needed in general for pools, which will be added now with pool: Update the desired failure domain when changed #9214.