Description:
Create scrapers/scheduler.py to run Scrapy spiders on hourly schedule.
Job:
- Every hour, trigger all 3 spiders
- Log job status
- Catch exceptions
Implementation:
from apscheduler.schedulers.background import BackgroundScheduler
scheduler = BackgroundScheduler()
scheduler.add_job(run_spiders, 'cron', hour='*', minute='0')
scheduler.start()
Acceptance Criteria:
Description:
Create
scrapers/scheduler.pyto run Scrapy spiders on hourly schedule.Job:
Implementation:
Acceptance Criteria: