Apache-Airflow

data pipeline management framework Airflow and how it can help us solve the problem of the traditional ETL

📗 what is it?

is a platform that lets you build and run and monitor workflows or PipeLines

📗 workflow

squence of scheduled tasks triggered by an event

used to handle big data pipelines

📗 traditional ETL

• write script to pull data from DB & send to HDFS to process

• schedule script as a cronjob

📗 Problems

• Failures: retry if happends(how many times?,how often?)
• Monitoring: pass or fails (how long does it take?)
• Dependencies:
  • Data dependencies: upstream data is missing
  • execution dependencies: job2 runs after job 1 is finished
• Scalability: no centralized schedulerbetween diff cron machines
• Deplyment: deploy smt new constantly
• Process historic data: backfill/rerun historical data

📗 Airflow DAG

🔶 Set up airflow environment with docker

Docker & DockerCompose

allows you to run many containers simultaneously

• container are lightweight no need of of hypervisor
• you can run more container than if tou were using virtual machine

Beneficts:

• no more taks-managing/manteining dependencies, deployment
• easy to share & deploy different version & environments
• keep track through github tags & releases
• ease of deployment from testing to production environment