In this project, you’ll use Apache Airflow to create a data pipeline to extract online stock market data and deliver analytical results. You’ll use Yahoo Finance as the data source. Yahoo Finance provides intra-day market price details down a one-minute interval.
The learning objects of this mini project are:
- Use text processing techniques in Python to make sense of logs
- Learn where logs are located in Airflow
- Learn how to monitor automated Airflow DAGs to ensure they are working properly
The DAGs are located in mnt/airflow/dags directory
Then execute ./start.sh script. This should build and start all the services.
Execute docker-compose ps and you should see below.
Go to localhost:8080 for the airflow UI.
The userid and password is airflow.
After completing you can use ./stop.sh to stop the services
Log analyzer will collect all ERRORs from logs and display them all. It will show the total number of errors, which log the error message is from, and the actual error message.
