Logging OMC Addon - Helm version

[JoaoBraveCoding]

Issue: https://issues.redhat.com/browse/LOG-4288

In this PR we introduce the base code of an OMC AddOn that configures ClusterLogForwarder to push logs to a URL using mTLS authentication. Currently, AddOn assumes that the hub Loki runs on the hub cluster on the openshift-logging namespace. The AddOn is also not responsible for generating any certificates and simply expects certain secrets to already exist, namely, it assumes that there is an mTLS secret created under the namespace of each spoke cluster with the same name as the spoke cluster and that a ConfigMap for the Loki gateway CA bundle exists under openshift-monitoring. The URL will be fetched from an AddOnDeploymentConfig resource that should live under the spoke cluster namespace. All configuration resources for the AddOn can be viewed in the ManagedClusterAddOn and ClusterManagementAddOn. The AddOn will then template a helm chart with these values to generate a set of manifests to be deployed to the spoke cluster, namely, it will deploy a Secret and ClusterLogForwarder

Note to reviewers

Under demo/ there is a README with instructions on how to install/configure RHACM and install and test the AddOn.

How to check if the AddOn is working correctly

Issues

• Currently, we have to configure ClusterLogForwarder.Spec.Outputs.TLS.InsecureSkipVerify to true because of the CommonName in the Loki Gateway Certificate it has the dnsName of the gateway service which then when the spoke collectors are verifying the hub certificate they will find a mismatch in the Hostname resulting in the error Hostname mismatch
• If a user deletes one of the deployed resources on the spoke cluster they will be recreated, however, if a user edits them they will not be overwritten. Further testing would be needed to conclude if this is specific to the helm AddOns (I suspect yes).
• Console LogUI doesn't work out of the box, I manually enable it but then it seems like it doesn't reconcile

Improvements

• The AddOn does some assumptions based on assuming that the hub Loki runs on the Cluster, this might not always be the case so we need to extend configuration to cover for use cases where this is not the case
• (nit) On the hub cluster there is some copying of secrets we currently have to do with bash scripts. I think moving to a fully golang based approach would allow us to move these operations to the AddOn manager
• Currently, the generation of certificates was delegated to cert-manager but it would be awesome if the addon could leverage OCM features for this.