Skip to content

Latest commit

 

History

History

cloud-composer-examples

Folders and files

NameName
Last commit message
Last commit date

parent directory

..

airflow_metadata_export

airflow_metadata_export

 
 

composer_dataflow_examples

composer_dataflow_examples

 
 

composer_http_post_example

composer_http_post_example

 
 

img

img

 
 

tests

tests

 
 

.gitignore

.gitignore

 
 

Dockerfile

Dockerfile

 
 

README.md

README.md

 
 

cloudbuild.yaml

cloudbuild.yaml

 
 

requirements.txt

requirements.txt

 
 

run_tests.sh

run_tests.sh

 
 

README.md

Cloud Composer Examples:

This repo contains the following examples of using Cloud Composer, Google Cloud Platform's managed Apache Airflow service:

  1. Composer Dataflow Examples

    a. Simple Load DAG: provides a common pattern to automatically trigger, via Google Cloud Function, a Dataflow job when a file arrives in Google Cloud Storage, process the data and load it into BigQuery. Alt text

  2. Composer HTTP Example

    a. Ephemeral Dataproc Spark DAG: provides an example of triggering a DAG via HTTP POST to the Airflow API to create a Dataproc cluster, submit a Spark job, and import the newly enhanced GCS files into BigQuery. Alt text

Run Tests

Run this script to automate spin up / tear down of a lightweight airflow environment to run your tests.

./run_tests.sh