Scrapy spider that sends a notification email if a URL is down. Can be deployed to the Scrapy cloud using their free plan.
NOTE: I just learned about https://uptimerobot.com/ which has a free plan for 50 monitors .. so this project is pretty much useless in that regards. Other tools are documented here and here
- Install pew
- Initialize
pew new emailifdown-scrapy
pip install scrapy
- Configure SMTP email settings by copying
emailifdown_scrapy/settings-sample.py
toemailifdown_scrapy/settings.py
and editing it (section markedEDIT THIS SECTION
)
- these
MAIL_...
settings are mandatory if-a emailTo=...
below is not skipped - These settings are the scrapy mail settings
- Alternatively, pass the settings directly to scrapy as parameters using
-s
(check below for run example)
- Run:
scrapy runspider emailIfDownSpider.py \
-L WARNING \
-a url="https://duckduckgo.com" \
-a emailTo="my@email.com"
- add the below settings if not put in settings.py (check above note)
...
-s MAIL_USER="another@email.com" \
-s MAIL_HOST="smtp.email.com" \
-s MAIL_PORT=123 \
-s MAIL_PASS="password"
- skip
-a emailTo=....
for no email to be sent
- Follow instructions in
Usage
above - Register on Scrapy cloud
- Start a new project
- Copy the project's "target number" and your API key
- Install shub:
pip install shub
- Deploy:
shub deploy
(will prompt for target number copied above) - Schedule:
shub schedule 12345/emailifdown
where12345
is your target number
- Maybe
shub
supports-a
and-s
parameters similar toscrapy
but it probably didnt work for me
- Navigate to the scrapy cloud / project / periodic jobs dashboard
- Add a new job (should automatically get
emailifdown
in dropdown) - add arguments "url" and "emailTo" and select times at which to run
- Check logs in dashboard