Investigate memory consumption of che-operator with growing number of namespaces on the cluster

[tolusha]

Is your task related to a problem? Please describe

che-operator pod is OOMKilled when there are a lot of namespaces on the cluster.
It has been fixed by [1] but still I can observer that operator consumes too much memory.
We have to find the reason and fix it.

[1] https://issues.redhat.com/browse/CRW-2383
[2] eclipse-che/che-operator#1146

Describe the solution you'd like

N/A

Describe alternatives you've considered

No response

Additional context

devfile/devworkspace-operator#616
#20529

Release Notes Text

Title: Improved operator memory consumption
Content: Fixed an operator issue that forced to augment the memory requirement based on the number of the namespaces on the cluster.

[ibuziuk]

FYI here is how the memory consumption pattern looks like on the Sandbox clusters:

[tolusha]

Don't close issue until doc is ready

[mmorhun]

The problem was in caches that's managed by Controller Runtime framework used in Che Operator. It caches all objects of the kind if the operator gets at least one object. It had been worked fine, but when Che Operator was switched to watch all namespaces (this PR) it started to cache all objects in cluster. We spot the bug only on a big cluster.
To fix the issue we limit set of objects that the Operator caches. However, it is not possible to control caching of every single object as Controller Runtime framework doesn't allow it. Instead, it is possible to define a selector for each kind of objects that should be cached. To reach that we need a label on each object of kind we limit cache for (we do it for all standard kinds like deployment, service, etc., and do not need the label for namespaces and all CRs).
The label that should be present on all objects Operator interacts with is app.kubernetes.io/part-of=che.eclipse.org.
After applying the fix, the memory usage for ~5000 namespace cluster dropped from ~1.2 GiB (with peaks to ~3.3 GiB) to just ~ 30 MiB:

Also, migration is implemented, so update should be seamless for existing Che installations.

[nickboldt]

sync'd to Red Hat JIRA https://issues.redhat.com/browse/CRW-2549