Proposal: Allow specifying resource requests/limits for critical control plane workloads

[csrwng]

Within a hypershift control plane certain workloads can vary in resource requirements depending on the size/usage of the guest cluster that they are hosting. Particularly, these can be:

1. Etcd
2. Kubernetes API Server
3. Kubernetes Controller Manager

A service provider needs to be able to control resource requests for these workloads (see #239). This is either to pre-allocate a predetermined size based on a cluster t-shirt size (Small, Med, Large) or dynamically to accommodate higher demand.

We can allow specifying these request sizes in the HostedCluster spec.

Either by adding a field that points to a struct that contains all resource requests:

type HostedClusterSpec struct {
  ...
  Resources HostedClusterResources
}

type HostedClusterResources struct {
  Etcd corev1.ResourceRequirements
  KubeAPIServer corev1.ResourceRequirements
  KubeControllerManager corev1.ResourceRequirements
}

Or, adding a field for each component. Here we have the advantage that we already have an Etcd field and a ManagedEtcd struct which can contain the ResourceRequirements, given that if Etcd is not managed, it doesn't make sense to specify them:

type HostedClusterSpec struct {
  Etcd EtcdSpec
  KubeAPIServer ControlPlaneComponentSpec
  KubeControllerManager ControlPlaneComponentSpec
}

type EtcdSpec {
  Managed ManagedEtcdSpec
  Unmanaged UnmanagedEtcdSpec
}

type ManagedEtcdSpec struct {
  Resources corev1.ResourceRequirements
}

type ControlPlaneComponentSpec struct {
  Resources corev1.ResourceRequirements
}

[ironcladlou]

Tracking this here: https://issues.redhat.com/browse/HOSTEDCP-190