A web crawler/scraper to find the broken links in the targeted seed url based on the keywords matched in the broken links contained page .
- Python 2.7+
- First install all dependencies listed in
requirements.txtusing pip package manager :
$ pip install -r requirements.txt
- Set the
(i.e SMTP_USER, SMTP_PASSWORD)in your shell config file(i.e .bashrc , .zshrc or etc)
# your shell config file export DATABASE_PATH='/path/to/database/'
- Also, set the two more environment variables required for
SMTP Serverfor sending email to users in your shell config file.
# your shell config file export SMTP_USER='smtp-username' export SMTP_PASSWORD='smtp-password'
- Also, set the one more environmnet variable to save
Logsof the app in defined location.
# your shell config file export LOGS_DIR='path/to/logs'
Note:- First install
Fabric to run below commands
To run a gui app :
$ fab app
To run a dispatcher :
$ fab dispatcher
To run a worker :
$ fab worker