Add HPA Support for all Pyroscope components

[weastel]

As Mentioned in #2445 Added Support for HPA for all pyroscope components similar to loki based schema.
Schema for adding HPA to any component is as follows:

  autoscaling:
    # -- Enable autoscaling for the querier, this is only used if `indexGateway.enabled: true`
    enabled: false
    # -- Minimum autoscaling replicas for the querier
    minReplicas: 1
    # -- Maximum autoscaling replicas for the querier
    maxReplicas: 3
    # -- Target CPU utilisation percentage for the querier
    targetCPUUtilizationPercentage: 60
    # -- Target memory utilisation percentage for the querier
    targetMemoryUtilizationPercentage: null
    # -- Allows one to define custom metrics using the HPA/v2 schema (for example, Pods, Object or External metrics)
    customMetrics: []
    # - type: External
    #   external:
    #     metric:
    #       name: loki_inflight_queries
    #     target:
    #       type: AverageValue
    #       averageValue: 12
    behavior:
      # -- Enable autoscaling behaviours
      enabled: false
      # -- define scale down policies, must conform to HPAScalingRules
      scaleDown: {}
      # -- define scale up policies, must conform to HPAScalingRules
      scaleUp: {}

For eg:

distributor:
  kind: Deployment
  replicaCount: 2
  autoscaling:
    enabled: true
    targetCPUUtilizationPercentage: 40
    minReplicas: 1
    maxReplicas: 5
  resources:
    limits:
      memory: 1Gi
    requests:
      memory: 256Mi
      cpu: 500m

results into:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: temp-up-pyroscope-distributor
  namespace: default
  labels:
    helm.sh/chart: pyroscope-1.5.1
    app.kubernetes.io/name: pyroscope
    app.kubernetes.io/instance: temp-up
    app.kubernetes.io/version: "1.5.0"
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/component: "distributor"
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: temp-up-pyroscope-distributor
  minReplicas: 1
  maxReplicas: 5
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 40

[simonswine]

You need to run make helm/check and update the static manifests. Also #3159 is useful

[weastel]

Updated PR, now it will mostly pass on all checks, A Quick question should I add values-micro-services-hpa.json and respective kubeconform. @simonswine

[simonswine]

Updated PR, now it will mostly pass on all checks, A Quick question should I add values-micro-services-hpa.json and respective kubeconform. @simonswine

@weastel I think that would be a nice way to test it yes please!

[weastel]

@simonswine As discussed added tests for new HPA based configuration, Can you please review this addition?

[simonswine]

LGTM

[simonswine]

@weastel thank you for your work on this ❤️