prefect-agent-on-aks

Prefect on AKS

Deploys Prefect Orion to an AKS Cluster with Azure Blob Storage
Explore the docs »

· Report Bug · Request Feature

Table of Contents

1. About The Project
   • Built With
2. Getting Started
   • Prerequisites
   • Installation
   • Setup
3. Usage
4. Roadmap
5. Contributing
6. License
7. Contact
8. Acknowledgments

About The Project

(back to top)

Built With

• Terraform
• Azure

(back to top)

Getting Started

To begin using this project:

git clone https://github.com/PrefectHQ/prefect-recipes.git

You will additionally need an Azure Service Principal that is configured with the "Contributor" role on your subscription. Steps will be outlined below.

Prerequisites

List of prerequisites and optional packages. Steps are listed in Setup.

• azure-cli
• terraform
• kubectl

(Optional)

• helm
• expect via apt-get, brew, npm, or yum
• lens

Setup

1. Clone the repo

   git clone https://github.com/PrefectHQ/prefect-recipes.git

2. Install required packages

   brew install azure-cli
   brew install terraform
   az aks install-cli --kubelogin-install-location mykubetools/kubelogin

3. Install optional packages - these are used to automate post-config steps, but are not required.

   brew install lens
   brew install expects
   brew install helm

4. Authenticate to ARM az login
5. Retrieve Azure subscription id for the next step. az account show --query "id" --output tsv
6. Only required if one does not exist already. If one already exists, proceed to step 7 with the values. Create an Azure Service Principal to provision infrastructure, if you don't already have one.

   az ad sp create-for-rbac --role="Contributor" --scopes="/subscriptions/< from step 5.>"

7. Move "source_prefect_vars_template.sh", and update with outputs from step 6. source_prefect_vars.sh is sensitive, and is configure to be excluded in .gitignore.

   mv source_prefect_vars_template.sh source_prefect_vars.sh
   
   #!/bin/bash 
   # Values below should be set from the values provided in step 6. 
   export ARM_CLIENT_ID="00000000-0000-0000-0000-000000000000"
   export ARM_CLIENT_SECRET="00000000-0000-0000-0000-000000000000"
   export ARM_SUBSCRIPTION_ID="00000000-0000-0000-0000-000000000000"
   export ARM_TENANT_ID="00000000-0000-0000-0000-000000000000"

8. Source source_prefect_vars.sh to export as environment variables, and validate.

   source ./source_prefect_vars.sh
   echo $ARM_CLIENT_ID

9. Update "local_ip" in aks_main/variables.tf to your local IP address to configure and access the storage container. Your IP can be determined:

   curl ifconfig.me

(back to top)

Usage

With setup of your required binaries, and Authentication to Azure configured, Prefect AKS can be provisioned.

Post-configuration steps are automated in wrap-deploy.sh for development purposes only, and is not intended for production use.

wrap-deploy.sh requires the "expects" binary to be installed, and a valid service principal.

Manual Steps

1. Initialize the providers.

   terraform init

2. Create the plan.

   terraform plan -out=tfplan

3. Execute the plan.

   terraform apply "tfplan"

4. Once terraform completes, retrieve the Resource Group name, cluster name, storage name, and container name for later use.

   export AZ_RESOURCE_GROUP="$(terraform output -raw resource_group_name)"
   export AZ_AKS_CLUSTER_NAME="$(terraform output -raw kubernetes_cluster_name)"
   export STORAGE_NAME="$(terraform output -raw storage_name)"
   export CONTAINER_NAME="$(terraform output -raw container_name)"
   export AZURE_STORAGE_CONNECTION_STRING=$(az storage account show-connection-string --resource-group "$AZ_RESOURCE_GROUP" --name "$STORAGE_NAME" --output tsv)

5. Export your KUBECONFIG to not overwrite any existing kubeconfig you might already have, and retrieve credentials to the cluster.

   export KUBECONFIG="$HOME/.kube/$AZ_AKS_CLUSTER_NAME.yaml"
   az aks get-credentials --resource-group $AZ_RESOURCE_GROUP --name $AZ_AKS_CLUSTER_NAME --file $KUBECONFIG

6. If prefect is already installed locally in your environment, you can generate and deploy the pod-spec:

   prefect kubernetes manifest orion | kubectl apply -f -

   If prefect is not already installed, you can apply the provided prefect.yaml and stop at this step, as the following steps require prefect installed locally first.

   kubectl apply -f prefect.yaml

7. Open a separate terminal session and port forward kubectl traffic to the cluster.

   kubectl port-forward deployment/orion 4200:4200

8. List / display your storage connection string (SENSITIVE), and container name. These are required to connect the Prefect agent to your Blob storage. These were already set in step 4, and will be required for the following step.

   echo $CONTAINER_NAME
   echo $AZURE_STORAGE_CONNECTION_STRING

9. Run prefect config to set the cluster agent settings, API_URL, work-queue, storage, and create a default deployment.

   prefect config set PREFECT_API_URL="http://127.0.0.1:4200/api"
   prefect work-queue create kubernetes
   prefect storage create
   prefect deployment create kubernetes-deployment.py

10. You can launch a browser at http://127.0.0.1:4200/api to see your configuration, or execute the flow manually.

prefect deployment run my-kubernetes-flow/k8s-example

Automated Steps

Requires expects installed, and service principal values exported as env_vars already from Setup.

1. Run wrap-deploy.sh from the root Terraform module directory (aks-prefect).

   ./wrap-deploy.sh

(back to top)

Roadmap

• Secrets Injection
• Ingress Controller for access to Prefect Cloud
• TBD
  • TBD

See the open issues for a full list of proposed features (and known issues).

(back to top)

Contributing

Contributions are what make the open source community such an amazing place to learn, inspire, and create. Any contributions you make are greatly appreciated.

If you have a suggestion that would make this better, please fork the repo and create a pull request. You can also simply open an issue with the tag "enhancement".

Don't forget to give the project a star! Thanks again!

1. Fork the Project
2. Create your Feature Branch (git checkout -b feature/AmazingFeature)
3. Commit your Changes (git commit -m 'Add some AmazingFeature')
4. Push to the Branch (git push origin feature/AmazingFeature)
5. Open a Pull Request

(back to top)

Contact

Your Name - chris.b@prefect.io

Project Link: https://github.com/PrefectHQ/prefect-recipes

(back to top)

Requirements

Name	Version
azurerm	~> 3.10.0

Providers

Name	Version
azurerm	~> 3.10.0
random	n/a

Modules

No modules.

Resources

Name	Type
azurerm_kubernetes_cluster.k8s	resource
azurerm_resource_group.rg	resource
azurerm_storage_account.prefect-logs	resource
azurerm_storage_container.prefect-logs	resource
azurerm_subnet.prefect_node_subnet	resource
azurerm_virtual_network.prefectnetwork	resource
random_id.storage_container_suffix	resource

Inputs

Name	Description	Type	Default	Required
agent_count	Number of AKS nodes to create	number	2	no
cluster_name	n/a	string	"k8stest"	no
container_name	Name of the container created in the storage account	string	"prefect-logs"	no
dns_prefix	n/a	string	"k8stest"	no
env_name	n/a	string	"dev"	no
local_ip	A list of public IP addresses you wish to add to network rules for access	list(string)	[ "131.226.33.86" ]	no
node_subnet_id	IDs of the subnets that will host the aks nodes	list(string)	[ "10.1.0.0/22" ]	no
node_subnet_name	Name of the subnet to create	string	"aks_node_subnet"	no
nodepool_name	n/a	string	"default"	no
pod_subnet_id	IDs of the subnets that will host the aks pods	list(string)	[ "10.1.4.0/22" ]	no
pod_subnet_name	Name of the subnet to create	string	"aks_pod_subnet"	no
resource_group_location	Location of the resource group.	string	"eastus"	no
resource_group_name	Prefix of the resource group name	string	"prefectAKS"	no
storage_account_name	Storage accounts must be globally unique, appended with randomized string	string	"prefectaks"	no
vm_size	Node size for provisioning nodepools	string	"Standard_B2s"	no
vnet_id	IDs of the Vnets that will host the Prefect agent	list(string)	[ "10.1.0.0/16" ]	no
vnet_name	Name of the Vnet to create	string	"prefectVnet"	no

Outputs

Name	Description
container_name	n/a
id	n/a
kubernetes_cluster_name	n/a
resource_group_name	n/a
storage_name	n/a