[Task]: Reduce SPS Operational Costs

[LucaCinquini]

In order to reduce SPS cost, it is suggested we do the following changes:

o Run integration tests every few days, instead of every day
o Do not run the most expensive integration test, which is the L1toL2 end-to-end
o Reduce the retention period of database snapshots from 1 month to 1 week

[LucaCinquini]

Other possible changes to reduce costs:

o Set the database instance_class to db.m5d.large instead of db.m5d.2xlarge
(see terraform-unity/modules/terraform-unity-sps-database/main.tf - it should be set as part of the helm values differently in values.tmpl.yaml or values_high_load.tmpl.yaml)

o Reduce the number of control nodes, keep a higher number for high load deployments? Optimally, all Airflow control nodes would be deployed onto a single EKS node

[nikki-t]

@LucaCinquini - I don't quite understand how to set the instance_class for the aws_db_instance using the Helm charts. I understand the different is for scaling but I don't see a location in the Helm chart that would allow me to set the instance_class. Can you elaborate on what you mean in the first bullet above?

[LucaCinquini]

@nikki-t : if the Helm chart does not support configuring the EC2 instance type (I have not seen such a parameter), why don't we just add a new Terraform variable from the top-level tfvars file, which by default is:

db_instance_class = "db.m5d.xlarge" ?

This way users could overwrite it to something like:

instance_class = "db.m5d.2xlarge"

for a more scalable deployment.

[nikki-t]

I have added a separate db_instance_class_variable. It was harder to track down the reason for the large number of control nodes, but it was related to the OGC API process nodes where there is one node for Airflow components and one node for the OGC API process. Here is what I found:

1. OGC API processes nodepool points to the airflow-core-components selector defined in the kubernetes_deployment.ogc_process_api resource. The instance family did not match other airflow-core-components selectors in the values.tmp.yaml file for the scheduler, triggerer, pgbouncer, webserver, dagProcessor. The OGC API process was updated to match the Airflow components but this still returned two nodes; one for the Airflow components plus the OGC API and one for the OGC API redis pods and statsd.
2. I updated the top-level node affinity defined in values.tmpl.yaml and updated the kubernetes_deployment.redis resource to include the same node affinity as the OGC API process resource. This mean the total nodes deployed by SPS are two: One for the default EKS group and one for Airflow components. Workers and KPO pods will still execution on separate nodes.
3. One other thing we may wish to consider is setting the consolidationPolicy to WhenEmptyOrUnderutilized so that Karpenter consolidates pods on nodes when the node's resources are underutilized as determined by Karpenter rather than`WhenEmpty.