Skip to content

Latest commit

 

History

History
24 lines (21 loc) · 1.2 KB

README.md

File metadata and controls

24 lines (21 loc) · 1.2 KB
Raw

KetoHub Raw Spider

Build Status Coverage Status

To run the spiders:

TIMESTAMP=$(date --iso-8601=seconds | sed -r 's/://g')
OUTPUT_DIR="${HOME}/data/raw/${TIMESTAMP}/"
scrapy crawl hey-keto-mama -s "DOWNLOAD_ROOT=${OUTPUT_DIR}"
scrapy crawl keto-size-me -s "DOWNLOAD_ROOT=${OUTPUT_DIR}"
scrapy crawl ketoconnect -s "DOWNLOAD_ROOT=${OUTPUT_DIR}"
scrapy crawl ketogasm -s "DOWNLOAD_ROOT=${OUTPUT_DIR}"
scrapy crawl ketovale -s "DOWNLOAD_ROOT=${OUTPUT_DIR}"
scrapy crawl ketovangelist-kitchen -s "DOWNLOAD_ROOT=${OUTPUT_DIR}"
scrapy crawl low-carb-yum -s "DOWNLOAD_ROOT=${OUTPUT_DIR}"
scrapy crawl queen-bs -s "DOWNLOAD_ROOT=${OUTPUT_DIR}"
scrapy crawl ruled-me -s "DOWNLOAD_ROOT=${OUTPUT_DIR}"
scrapy crawl skinny-taste -s "DOWNLOAD_ROOT=${OUTPUT_DIR}"
scrapy crawl sugar-free-mom -s "DOWNLOAD_ROOT=${OUTPUT_DIR}"
scrapy crawl wholesome-yum -s "DOWNLOAD_ROOT=${OUTPUT_DIR}"
scrapy crawl your-friends-j -s "DOWNLOAD_ROOT=${OUTPUT_DIR}"