City Scrapers Template
Template repo for running a City Scrapers project in your area.
Fork this repo to your GitHub account or organization account. Feel free to change the name to something specific to your area (i.e.
city-scrapers-ilfor scrapers in Illinois).
- Enable issues for your fork by going to Settings, and checking the box next to Issues in the Features section.
README.mdwith your organization's information.
Enable Travis CI for running automated tests and builds.
- Setup Azure or S3 for scraper output on your own account, or reach out to us for credentials for storage that we'll host.
- Configure static file storage credentials as environment variables in Travis CI. See
city_scrapers/settings/prod.pyto find out which environment variables need to be set for your storage provider.
- Setup a daily cronjob on Travis CI (see instructions) to run the scrapers.
Setup a free open source account with Sentry for error tracking and notifications. Set necessary environment variables and make sure
scrapy_sentryis enabled in production settings.
Refer to our documentation on getting started with development.