Home

A template repository with all the fundamentals needed to develop and deploy a Python data-processing routine for Prefect pipelines.

It contains:

• 🌊 A flow.py file with an example function that’s ready to run
• 🚧 A local development configuration that can run independently
• ☁️ A cloud production configuration for a Google Kubernetes Engine executor with Docker storage
• 📈 Easy vertical scaling of production workers via Dask
• 🧪 Unit tests via pytest
• 🚢 Automated deployment via GitHub releases and actions

This wiki contains documentation explaining how to configure and run the code. The repository does not include the necessary configuration of a Prefect Agent to orchestrate flow runs. You can learn how to do that in our post, "How to deploy a Prefect agent to Google Kubernetes Engine"

Table of contents

• Getting Started
• Google Cloud Deployment Configuration
• Releasing updates

About

The project is sponsored by Big Local News, a program at Stanford University that collects data for impactful journalism. The code is maintained by Ben Welsh, a visiting data journalist from the Los Angeles Times.