kong: increase gateway controlplane resources

[boddumanohar]

There were a lot of restarts for Kong Control Plane. 1091 restarts in 20days. That's a lot. That's 2.3 restarts/hour

kubectl -n kong-system get pods
NAME                                                              READY   STATUS    RESTARTS         AGE
controlplane-vela-public-gateway-lh92v-w8ddf-8c5548d9b-n2vb6      1/1     Running   1091 (22m ago)   20d

when I looked at the reason for restart

Events:
  Type     Reason     Age                   From     Message
  ----     ------     ----                  ----     -------
  Normal   Started    58m (x1085 over 20d)  kubelet  Started container controller
  Warning  Unhealthy  29m (x7652 over 20d)  kubelet  Readiness probe failed: Get "http://10.244.3.175:10254/readyz": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
  Normal   Killing    17m (x899 over 20d)   kubelet  Container controller failed liveness probe, will be restarted
  Normal   Pulled     16m (x1090 over 20d)  kubelet  Container image "kong/kubernetes-ingress-controller:3.4.4" already present on machine
  Normal   Created    16m (x1090 over 20d)  kubelet  Created container: controller
  Warning  Unhealthy  14m (x5703 over 20d)  kubelet  Liveness probe failed: Get "http://10.244.3.175:10254/healthz": context deadline exceeded (Client.Timeout exceeded while awaiting headers)

They are due context deadline exceeded that happens occasionally. This seems like a tail latency. So a general fix for such type of issues is to increase resources.

The current resource allocation for Control Plane is:

    Ready:          True
    Restart Count:  1091
    Limits:
      cpu:     200m
      memory:  100Mi
    Requests:
      cpu:      100m
      memory:   20Mi

As a part of this PR, doubling that to 400m CPU and 200Mi Memory.

The changes here are already applied. This PR is to make sure that those changes are synced to Git.

[mxsrc]

Good catch! Let's let it sit for a few more hours to ensure the change actually solves the issue before merging.

[boddumanohar]

Just checked. It's been around little less around 2 hours. No restarts so far.

kubectl -n kong-system get pods
NAME                                                              READY   STATUS    RESTARTS      AGE
controlplane-vela-public-gateway-lh92v-w8ddf-bb6995756-qxpzz      1/1     Running   0             161m
dataplane-vela-public-gateway-wn57x-bxhj5-85785b688f-pg76c        1/1     Running   0             20d
kong-operator-gateway-operator-controller-manager-85588886dxpbz   1/1     Running   4 (19d ago)   20d

[mxsrc]

Yeah, me too. It definitely doesn't appear in the frequency you describe, but we also don't know whether those restarts were equally distributed, or whether there still might be some lingering ones so I think giving it a workday is sensible.

[boddumanohar]

We monitored this change for 26 hours and this seems like a stable change.

kubectl -n kong-system get pods
NAME                                                              READY   STATUS    RESTARTS      AGE
controlplane-vela-public-gateway-lh92v-w8ddf-bb6995756-qxpzz      1/1     Running   0             26h
dataplane-vela-public-gateway-wn57x-bxhj5-85785b688f-pg76c        1/1     Running   0             21d
kong-operator-gateway-operator-controller-manager-85588886dxpbz   1/1     Running   4 (20d ago)   21d