Skip to content
Apache Airflow tutorial
Jupyter Notebook Other
Branch: master
Clone or download
tuanavu Revert "Merge pull request #18 from tuanavu/development"
This reverts commit 0950352, reversing
changes made to 1609792.
Latest commit 5bddbf4 Jun 25, 2019

README.md

Airflow tutorial

This is the code for Apache Airflow Tutorials playlist by Tuan Vu on Youtube

Contents

Part Title Git Tag
1 Introduction to Apache Airflow (blog post) v0.1
2 Set up airflow environment with docker (blog post) v0.2
3 Set up airflow environment using Google Cloud Composer (blog post) N/A
4 Writing your first pipeline (blog post) N/A
5 Airflow concept (blog post) N/A
6 Build a data pipeline using Google Cloud Bigquery (blog post) v0.6
7 Airflow variables (blog post) v0.7

Getting Started

These instructions will get you a copy of the project up and running on your local machine for development and testing purposes.

Prerequisites

Usage

Run the web service with docker

docker-compose up -d

# Build the image
# docker-compose up -d --build

Check http://localhost:8080/

  • docker-compose logs - Displays log output
  • docker-compose ps - List containers
  • docker-compose down - Stop containers

Other commands

If you want to run airflow sub-commands, you can do so like this:

  • docker-compose run --rm webserver airflow list_dags - List dags
  • docker-compose run --rm webserver airflow test [DAG_ID] [TASK_ID] [EXECUTION_DATE] - Test specific task

If you want to run/test python script, you can do so like this:

  • docker-compose run --rm webserver python /usr/local/airflow/dags/[PYTHON-FILE].py - Test python script

Connect to database

If you want to use Ad hoc query, make sure you've configured connections: Go to Admin -> Connections and Edit "postgres_default" set this values:

  • Host : postgres
  • Schema : airflow
  • Login : airflow
  • Password : airflow

Credits

You can’t perform that action at this time.