JavaScript Python
Latest commit 6b48610 Feb 8, 2010 @clofresh update documentation

Couch Crawler

A search engine built on top of couchdb-lucene.




Optionally for Yammer spidering:


Assuming couchdb-lucene was installed to the "_fti" endpoint, you can push Couch Crawler to your CouchDB instance with the command:

cd couchapp
couchapp push

This will create a new CouchDB database called "crawler" on the localhost:5984 CouchDB instance. To change the db, modify couchapp/.couchapprc and do another couchapp push.

To configure the crawler, copy python/couchcrawler-sample.cfg to python/couchcrawler.cfg and fill out the appropriate configuration values.

To start indexing pages, run the crawler script:

cd python
./ crawl

While it's indexing, you can visit the search engine at the following url:



The crawler current has spiders for:

  • MediaWiki
  • Twiki
  • Yammer

It's pretty easy to create your own. See python/couchcrawler/spiders/ for an example, or Scrapy documentation for more a more in-depth explanation.