Astro Databricks provider tutorial

This repository contains the DAG code used in the Astronomer Databricks use case example. The DAG uses both the Astro Databricks provider as well as the Astro Python SDK.

How to use this repository

This section explains how to run this repository with Airflow. Note that you will need to define extra connections (AWS, Databricks and a connection to a relational database). To do so, rename the .env_example file to .env and add your own credentials. The code used in the Databricks notebooks is available in the databricks_notebook_code folder.

Option 1: Use GitHub Codespaces

Run this Airflow project without installing anything locally.

1. Fork this repository.
2. Create a new GitHub codespaces project on your fork. Make sure it uses at least 4 cores!
3. After creating the codespaces project the Astro CLI will automatically start up all necessary Airflow components. This can take a few minutes.
4. Once the Airflow project has started, access the Airflow UI by clicking on the Ports tab and opening the forward URL for port 8080.

Option 2: Use the Astro CLI

Download the Astro CLI to run Airflow locally in Docker. astro is the only package you will need to install.

1. Run git clone https://github.com/astronomer/2-6-example-dags.git on your computer to create a local clone of this repository.
2. Install the Astro CLI by following the steps in the Astro CLI documentation. Docker Desktop/Docker Engine is a prerequisite, but you don't need in-depth Docker knowledge to run Airflow with the Astro CLI.
3. Run astro dev start in your cloned repository.
4. After your Astro project has started. View the Airflow UI at localhost:8080.

Resources

• Orchestrate Databricks jobs with Airflow tutorial.
• Databricks 14 day free trial.
• Astro Databricks provider repository.
• Astro Databricks provider documentation.
• Astro Python SDK tutorial.
• Astro Python SDK documentation.
• Astro Python SDK repository.