feat: Add support for LoggingStack and ObservabilityUIPlugin resources

[periklis]

⚠️ Experimental: Beware you might be eaten by the sandworm in this dune!!! ⚠️

Description

The following PR is a proof-of-concept implementation to show case how to integrate the OLM-managed Red Hat OpenShift Logging stack into the observability-operator. It leverages the ability of the operator to server-side apply on the minimal configuration to bootstrap a fully-functional Logging Stack. In detail the approach taken:

• Provides a super slim LoggingStack resource to configure the minimum viable aspects to get an in-cluster logging stack.
• Delegates installation of required operators to OLM, i.e. Cluster Logging Operator and Loki Operator. The user is required to pick only a catalog source and a channel names (currently as string, but enums are possible).
• Resolves the necessity to re-package and re-scope CRDs (i.e. ClusterLogging, LokiStack, ClusterLogForwarder) but splitting the operator internally in two operators: a) one for installing leaf operators, b) one for installing the leaf CRs.
• Adds a minimal implemenation of the ObservabilityUIPlugin resource (as proposed in this enhancement proposal) to showcase integration points with the console (currently it requires CLO installing the plugin but this need could eliminated by moving the code from CLO in this operator).

The workflow implemented looks like this (Note: The following diagram shows that the same approach can be expanded to other signals too):

Known Limitations

• Missing a thorough mapping of status conditions from underlying CRDs to improve troubleshooting and visualize progress states in general.
• The split into two separate operators is currently implemented via a simple golang channel messaging approach. Means we start the operator responsible for installing the logging custom resources if and only if the operator responsible installing the logging operators succeeds. Although this two-step approach works, it needs a thorough review on possible error conditions and resolution strategies before adopting. OLM states are incorporated here and this needs careful consideration if we can afford doing so.

[openshift-ci]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: periklis
Once this PR has been reviewed and has the lgtm label, please assign simonpasquier for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[openshift-merge-robot]

PR needs rebase.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[jan--f]

I think this is on the right path. In #434 we started discussing Openshift type dependecies which apply here too I think. I'd be interested in your thoughts re #434 (comment).

[periklis]

Closing for a future implementation based on:

• feat: Add ObservabilityUI plugins API #434
• observability: Add logging-stack with UI and korrel8r integration openshift/enhancements#1566