THREESCALE-8754 enhance replica reconciliation

[eguzki]

what

Implements https://issues.redhat.com/browse/THREESCALE-8754

• Remove adding replica values to the CR
• Fixes replicaMutator. It was comparing pointer addresses instead of values.
• Reconciliation (implemented by the mutator) only added when the replicas field is set

Verification steps

run env

make install
make run

Deploy apicast CR with no replica definition

k apply -f -<<EOF
---
apiVersion: v1
kind: Secret
metadata:
  name: supertest
type: Opaque
stringData:
  AdminPortalURL: https://mykey@example.com
---
apiVersion: apps.3scale.net/v1alpha1
kind: APIcast
metadata:
  name: apicast1
spec:
  resources: {}
  logLevel: debug
  deploymentEnvironment: staging
  configurationLoadMode: lazy
  cacheConfigurationSeconds: 0
  adminPortalCredentialsRef:
    name: supertest
EOF

Check that the Apicast CR does not have replicas in the CR

k get apicast apicast1 -o yaml
apiVersion: apps.3scale.net/v1alpha1
kind: APIcast
metadata:
  annotations:
    apicast.apps.3scale.net/operator-version: 0.6.0
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"apps.3scale.net/v1alpha1","kind":"APIcast","metadata":{"annotations":{},"name":"apicast1","namespace":"eguzki"},"spec":{"adminPortalCredentialsRef":{"name":"supertest"},"cacheConfigurationSeconds":0,"configurationLoadMode":"lazy","deploymentEnvironment":"staging","logLevel":"debug","resources":{}}}
  creationTimestamp: "2022-09-28T13:11:32Z"
  generation: 1
  labels:
    secret.apicast.apps.3scale.net/a4d062fb-6d9e-41d1-82e1-33c6355fada4: "true"
  name: apicast1
  namespace: eguzki
  resourceVersion: "108604360"
  uid: 768c8858-43c3-4ca0-abab-afdccf190525
spec:
  adminPortalCredentialsRef:
    name: supertest
  cacheConfigurationSeconds: 0
  configurationLoadMode: lazy
  deploymentEnvironment: staging
  logLevel: debug
  resources: {}
status:
  image: quay.io/3scale/apicast:latest

Wait for deployment to be available

kubectl wait deployment apicast-apicast1 --for condition=Available=True
deployment.apps/apicast-apicast1 condition met

Check the number of pods is 1

k get deployment apicast-apicast1
NAME               READY   UP-TO-DATE   AVAILABLE   AGE
apicast-apicast1   1/1     1            1           2m54s

Scale to 3 via the deployment

kubectl scale --replicas=3 deployment apicast-apicast1

Check the deployment has 3 replicas

k get pods
NAME                                     READY   STATUS    RESTARTS   AGE
apicast-apicast1-6cfd57dc87-bzx8j        1/1     Running   0          6m7s
apicast-apicast1-6cfd57dc87-m6mpm        1/1     Running   0          2m39s
apicast-apicast1-6cfd57dc87-xssj7        1/1     Running   0          2m39s

Add replicas to the Apicast CR with a value of 2

k patch apicast apicast1 --type merge --patch '{"spec":{"replicas": 2}}'

Check the deployment has 2 replicas

k get pods
NAME                                     READY   STATUS    RESTARTS   AGE
apicast-apicast1-6cfd57dc87-bzx8j        1/1     Running   0          12m
apicast-apicast1-6cfd57dc87-xssj7        1/1     Running   0          9m23s

As replicas are set explicitly in the CR, changes in the deployment will be reverted back. If we try to scale to 3 via the deployment

kubectl scale --replicas=3 deployment apicast-apicast1

The number of replicas will still be 2

k get deployment apicast-apicast1
NAME               READY   UP-TO-DATE   AVAILABLE   AGE
apicast-apicast1   2/2     2            2           14m

[MStokluska]

Looks good to me, verified on cluster and code changes make sense as well