All IT Ebooks.com Scraper

This project has been superceded by my new dockerized Ebook API.

A program that scrapes links, and other data, to your favorite tech ebooks from allitebooks.com and stores them in a MongoDB instance.

Getting Started

Note. One of this projects dependencies, scrapy-mongodb, does not support Python 3 yet. Therefore, this project only supports Python 2

1. Clone this repository.
2. Create a new Python Virtual Environment.
3. pip install -r requirements.txt

Configuration

• Create a fresh MongoDB instance either on our local machine or a free 500 MB remote instance from MLab.

• Replace the following in settings.py with your own credentials:

MONGODB_URI = ''
MONGODB_DATABASE = ''
MONGODB_COLLECTION = ''

• Other MongoDB settings can be found here at the scrapy-mongodb repo.

Contributing

Anyone, regardless of skill level, is encouraged to give feadback and submit pull requests.

Acknowledgments

A special thanks to the following:

• The Scraping Hub Team who maintain the Scrapy project;
• Sebastian Dahlgren maintainer of scrapy-mongodb