About The Project

This is an importer, which get data from Twitter, using it's API and store them in a Mongo db. You can get both user's timeline and different hashtags. The importer will be run daily to get new data.

Built With

• Apachi Airflow
• Mongodb
• Docker

Getting Started

Clone the repo in your local machine.

Prerequisites

• Install Docker Desktop
• Get Twitter Developer API Access

Usage

Run docker-compose up and airflow will be accessible at localhost:8080/admin/

Add Twitter API keys:

twitter-importer/dags/get_hashtags.py

Line 10 in 3ef24a2

def get_api_object():

Modify list of usernames:

twitter-importer/dags/get_user_timeline.py

Line 119 in 3ef24a2

usernames = ['ApacheAirflow', 'Airbnb']

Modify list of hashtags:

twitter-importer/dags/get_hashtags.py

Line 121 in 3ef24a2

'hashtags': ['#unitedAIRLINES', '#Flight', '#beach'],

For more information about the project, please refer to Documentation

For more information about airflow image, please refer to docker-airflow