A pyLucene-based search module for searching books from goodreads.com
Switch branches/tags
Nothing to show
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Failed to load latest commit information.
JSONs
static
templates
.gitignore
JSON Playground and PageRank.ipynb
JSONs.rar
LICENSE
PageRank.py
README.md
Scraping GoodReads.ipynb
SetQueue.py
booktype.py
crawler.py
elasticsearch-test.py
homepage.png
indexer.py
scraper-demo.py
scraper.py
searcher.py
setqueue-demo.py

README.md

GoodSearcher

A pyLucene-based crawl and search module for searching books from goodreads.com

Image of GoodSearcher homepage

Module Files Description
Crawling crawler.py, scraper.py, SetQueue.py These files contain the implementation of a multi-threaded crawler. Specifically, scraper.py is the implementation for scraping webpages from Goodreads website. crawler.py implements the crawler threads, thread-safe url queues and related stuff. A thread-safe queue with a built-in set for url deduplication is implemented in SetQueue.py
Indexing booktype.py, indexer.py, PageRank.py Indexing using two libraries has been implemented. You can choose between Elasticsearch and Lucene for indexing. For document boosting, PageRank scoring is also implemented.
Searching searcher.py It is a web.py-based minimal web server for searching indexes and displaying results in a web page.

Libraries used

Elasticsearch

Lucene

  • PyLucene: PyLucene is a Python extension for accessing Java Lucene.
  • PyLucene: PyLucene clone on Github

Web

  • Web.py: web.py is a web framework for Python that is as simple as it is powerful.
  • Beautifulsoup4: For web scraping

Scipy/Numpy was also used for calculating PageRank scores.