-
Notifications
You must be signed in to change notification settings - Fork 14.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Create EKS operators #8544
Comments
Happy to lead this effort |
Thank you @john-jac! Please let me know if you need help on the kubernetes or airflow core side of things. I think this would be a great addition to Airflow's k8s story :). |
Hi folks, I'm going to be working with @john-jac on this; the code etc will be in my fork and I'll have a design plan/doc posted here in this ticket in the next day or two. |
I've started putting down the boilerplate code, I think this is how I am going to tackle this. I'm not sure how formal you folks like it yet, but please let me know what you think of the plan. GoalsAdd a collection of Apache Airflow Operators which interact with Amazon Elastic Kubernetes Service (EKS) and abstract all Amazon logic from the Airflow user. This will allow Airflow to be a general purpose Kubernetes orchestrator able to do multi-cluster orchestration across multiple clouds. ProposalThe proposed solution is a collection of Operators, and their underlying Hooks, which will be added to the Amazon AWS provider package. These Operators will handle creating and deleting clusters, as well as executing tasks using EKS Managed Node Groups. Assumptions and Prerequisites
DefinitionsPod - A Kubernetes pod is the way that Kubernetes runs containers on a compute instance and includes containers and specifications for how they should run, networking, and storage. A pod can be a single container or multiple containers that always run together. Cluster - An Amazon EKS cluster consists of the Amazon EKS control plane, which runs the Kubernetes software and API server, and the pod that is registered with the control plane. Operator - An operator defines a single task within the workflow. kubectl - The Kubernetes command-line tool which allows users to run commands against Kubernetes clusters. Uses include deploying applications, inspecting and managing cluster resources, and viewing logs. eksctl - An open source CLI tool created by the community to create clusters on EKS using CloudFormation. aws eks (cli tool) - A CLI tool which, among other things, is used to generate the kubeconfig file. kubeconfig - A config file containing required information about clusters, users, namespaces, and authentication mechanisms. kubectl uses kubeconfig files to find the information it needs to choose a cluster and communicate with the API server of a cluster. EKS Managed Node Groups (nodegroup) - Infrastructure as a Service - EKS Managed Node Groups create and manage Amazon Elastic Compute Cloud (EC2) instances which host a Kubernetes cluster. This is the default underlying compute platform for EKS clusters. Task - The process or command being run in a pod. Context and User ExperienceWhile the basic functions of creating and running pods on EKS can be handled through the existing Cloud Native Computing Foundation (CNCF) Kubernetes Pod Operator, running the pods on EKS introduces pain points to the users, some of which are detailed below, and requires some specific EKS knowledge. By abstracting away some of this Amazon-specific logic, we can automate and streamline the configuration and deployment of new pods. Currently, in order to deploy a new pod on EKS, the user needs to leverage the kubectl, eksctl, and aws command-line tools and generate config files to manually pass data to the Kubernetes Pod Operator. The current manual process is:
Using the BOTO3 python API, new Operators can automate most or all of those steps and create a more seamless experience for the user. Use Cases
Benchmarks At a minimum, this solution should offer feature parity with the Google Kubernetes Engine (GKE) Pod Operator functionality. Create Cluster - Create a Google Kubernetes Engine Cluster of specified dimensions
Delete Cluster - Deletes the cluster, including the Kubernetes endpoint and all worker nodes
Start Pod - Executes a task in a Kubernetes pod in the specified Google Kubernetes Engine cluster
And some sequence diagrams for the Operators for folks who like that kinda thing: [EDIT: corrected CreateCluster sequence diagram image; initially uploaded the wrong file] |
Hi @ferruzzi, this looks great! In general, we don't need anything super formal for creating operators (though this is useful). If you were to make a core change that would involve submitting an AIP, but for this you're good to go! |
Hi there, will this effort fix the reported problems with receiving logs from EKS pods in MWAA? Not sure if it's not just a general issue with Kubernetes logging mind you, but I had it confirmed by the AWS support team. Either way, I've made some commits to the EMR operators in the past and would be happy to help out here. |
Best I can tell that is an underlying issue which I don't intend to tackle at this time. Quick update, most of the code for this has been implemented, but moto didn't have any EKS support. So I have a PR in with them to add that, then I'll drop a PR here once they push that to their release. |
Description
The same way that we have
GKEStartPodOperator
andGKECreateClusterOperator
we should add operators that interact with EKS and abstracts all amazon logic from the airflow user.Use case / motivation
This will allow airflow to be a general purpose kubernetes orchestrator. Able to do multi-cluster orchestration across multiple clouds.
Related Issues
The text was updated successfully, but these errors were encountered: