Develop Cost Model

[jordanpadams]

💡 Description

Sub-tasks necessary in order to benchmark and develop cost model for Nucleus deployments per number of apps in the pipeline

[tloubrieu-jpl]

@jimmie and @ramesh-maddegoda will meet end of next week

[tloubrieu-jpl]

meeting scheduled this Friday

[jordanpadams]

📆 January status: In work. Delayed 1 sprint. No impact on delivery

[ramesh-maddegoda]

It was decided to consider multiple approaches to deploy and manage Airflow on AWS.

Following links provide some approaches for this:

Self-managed Apache Airflow deployment for EKS - by AWS Labs
https://awslabs.github.io/data-on-eks/docs/job-schedulers-eks/self-managed-airflow
https://github.com/awslabs/data-on-eks

Amazon EKS Blueprints for Terraform - Apache Airflow add-on
https://aws-ia.github.io/terraform-aws-eks-blueprints/main/add-ons/apache-airflow/

3 Ways to deploy Airflow on AWS
https://thinkport.digital/how-to-deploy-airflow-on-aws/

Deploy Airflow of AWS with Docker Compose
https://github.com/andresionek91/airflow-autoscaling-ecs

How to deploy Apache Airflow with Celery on AWS
https://towardsdatascience.com/how-to-deploy-apache-airflow-with-celery-on-aws-ce2518dbf631

Apache Airflow Deployment on AWS ECS
https://infinitelambda.com/apache-airflow-deployment-on-aws-ecs/

Setup to run Airflow in AWS ECS containers
https://github.com/nicor88/aws-ecs-airflow

How to deploy Airflow on AWS: best practices
https://towardsdatascience.com/how-to-deploy-airflow-on-aws-best-practices-63778d6eab9e

[viviant100]

@ramesh-maddegoda what's the pro/con for each approach?

[ramesh-maddegoda]

@viviant100, I working on analyzing pros and cons of each approach.

[ramesh-maddegoda]

Comparison of Airflow deployment models
https://docs.google.com/document/d/1ko9GxFzBRXbo_hA4_XjDJ3gcAcYBjb4Y3j4EjoQfzjo/edit

[jordanpadams]

📆 February status: Delayed 1 sprint due to comprehensive trade study ongoing per deployment strategies. Awaiting meeting with AWS team to validate decision. No impact on build.

[ramesh-maddegoda]

Had a discussion with Amazon to compare pros and cons of each Airflow deployment approach (MWAA, Airflow on ECS, Airflow on EKS).

Based on that discussion, it was decided to move forward with MWAA for near future.