Audit rook deployment automated in ocs-ci, focusing on prometheus monitoring setup

[mbukatov]

Right now, when I deploy a cluster via ocs-ci, the ceph mgr prometheus target is not connected to openshift prometheus, and so no ceph metrics are available in prometheus.

Areas to focus on:

• namespace workaround is not tracked, audit ocs-ci for additional workarounds (additional work because of Audit ocs-ci code for hidden workarounds #458 unresolved status): see Remove workaround: switch back to openshift-namespace from rook-ceph #469
• only proper downstream namespace with prefix openshift- will work, so blocked by Remove workaround: switch back to openshift-namespace from rook-ceph #469, proposed fix in WIP: switch back to openshift-storage namespace #470
• cluster.yaml manifest esp. monitoring section there
  • diff on cluster.yaml (rook 81dde4b3 vs ocs-ci ec13283) looks ok (thanks to Change yamls to sync with upstream rook repo changes #460)
  • I need to set monitoring_enabled to True
• servicemonitor manifest (service-monitor.yaml)
  • diff (rook 81dde4b3 vs ocs-ci ec13283) looks ok

[mbukatov]

This effort is related to #460

[mbukatov]

So basically for monitoring to work, I need to:

• use downstream builds
• make sure the namespace is set to openshift-storage (Remove workaround: switch back to openshift-namespace from rook-ceph #469 Remove workaround: Rook mon crashloop and pod disappearance #438)
• set monitoring_enabled to True

Edit: @fbalak this is the only way to setup monitoring properly