The same way that we have
Use case / motivation
This will allow airflow to be a general purpose kubernetes orchestrator. Able to do multi-cluster orchestration across multiple clouds.
The text was updated successfully, but these errors were encountered:
I've started putting down the boilerplate code, I think this is how I am going to tackle this. I'm not sure how formal you folks like it yet, but please let me know what you think of the plan.
Add a collection of Apache Airflow Operators which interact with Amazon Elastic Kubernetes Service (EKS) and abstract all Amazon logic from the Airflow user. This will allow Airflow to be a general purpose Kubernetes orchestrator able to do multi-cluster orchestration across multiple clouds.
The proposed solution is a collection of Operators, and their underlying Hooks, which will be added to the Amazon AWS provider package. These Operators will handle creating and deleting clusters, as well as executing tasks using EKS Managed Node Groups.
Assumptions and Prerequisites
Pod - A Kubernetes pod is the way that Kubernetes runs containers on a compute instance and includes containers and specifications for how they should run, networking, and storage. A pod can be a single container or multiple containers that always run together.
Cluster - An Amazon EKS cluster consists of the Amazon EKS control plane, which runs the Kubernetes software and API server, and the pod that is registered with the control plane.
Operator - An operator defines a single task within the workflow.
kubectl - The Kubernetes command-line tool which allows users to run commands against Kubernetes clusters. Uses include deploying applications, inspecting and managing cluster resources, and viewing logs.
eksctl - An open source CLI tool created by the community to create clusters on EKS using CloudFormation.
aws eks (cli tool) - A CLI tool which, among other things, is used to generate the kubeconfig file.
kubeconfig - A config file containing required information about clusters, users, namespaces, and authentication mechanisms. kubectl uses kubeconfig files to find the information it needs to choose a cluster and communicate with the API server of a cluster.
EKS Managed Node Groups (nodegroup) - Infrastructure as a Service - EKS Managed Node Groups create and manage Amazon Elastic Compute Cloud (EC2) instances which host a Kubernetes cluster. This is the default underlying compute platform for EKS clusters.
Task - The process or command being run in a pod.
Context and User Experience
While the basic functions of creating and running pods on EKS can be handled through the existing Cloud Native Computing Foundation (CNCF) Kubernetes Pod Operator, running the pods on EKS introduces pain points to the users, some of which are detailed below, and requires some specific EKS knowledge. By abstracting away some of this Amazon-specific logic, we can automate and streamline the configuration and deployment of new pods.
Currently, in order to deploy a new pod on EKS, the user needs to leverage the kubectl, eksctl, and aws command-line tools and generate config files to manually pass data to the Kubernetes Pod Operator. The current manual process is:
Using the BOTO3 python API, new Operators can automate most or all of those steps and create a more seamless experience for the user.
At a minimum, this solution should offer feature parity with the Google Kubernetes Engine (GKE) Pod Operator functionality.
Create Cluster - Create a Google Kubernetes Engine Cluster of specified dimensions
Delete Cluster - Deletes the cluster, including the Kubernetes endpoint and all worker nodes
Start Pod - Executes a task in a Kubernetes pod in the specified Google Kubernetes Engine cluster
And some sequence diagrams for the Operators for folks who like that kinda thing:
[EDIT: corrected CreateCluster sequence diagram image; initially uploaded the wrong file]
Hi there, will this effort fix the reported problems with receiving logs from EKS pods in MWAA? Not sure if it's not just a general issue with Kubernetes logging mind you, but I had it confirmed by the AWS support team.
Either way, I've made some commits to the EMR operators in the past and would be happy to help out here.
Best I can tell that is an underlying issue which I don't intend to tackle at this time.
Quick update, most of the code for this has been implemented, but moto didn't have any EKS support. So I have a PR in with them to add that, then I'll drop a PR here once they push that to their release.