project_101

Developing a data pipeline for automation in crawling and processing data

Setup

install library

1. open terminal in folder "data_pipeline"
2. cd data_pipeline
3. Run "pip install -r requirement.txt"

create .env file

1. create ".env" file in th path /data_pipeline
2. Copy the structure provided below into the file.
3. USER_NAME= user_name_here
   PASSWORD= password_here
4. replace "user_name_here" with username and "passord_here" with password

Run

Run file "main.py"