Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP
Lucandra = Lucene + Cassandra
Java JavaScript Shell

Fetching latest commit…

Cannot retrieve the latest commit at this time

Failed to load latest commit information.
.settings
config
lib
solr-example
src
test
thrift/org/apache/cassandra/thrift
.classpath
.gitignore
.project
LICENSE
README
bookmarks.tsv
build.xml
ivy.xml
pom.xml
run_demo.sh
run_importer.sh

README

Lucandra - A Cassandra Backend for Lucene/Solr

By Jake Luciani - http://twitter.com/tjake
==============================================

Lucandra provides a Lucene IndexReader and IndexWriter that interfaces with Cassandra.
Solr is also supported.

Support
=======
Mailing List:
     http://groups.google.com/group/lucandra-user

Useful articles:
     http://blog.sematext.com/2010/02/09/lucandra-a-cassandra-based-lucene-backend/
     http://anismiles.wordpress.com/2010/05/27/lucandra-an-inside-story/

Slides:
     http://www.slideshare.net/otisg/lucandra

You can see Lucandra in action here:
     http://sparse.ly


Bookmarks demo:
==============
Lucandra includes a delicious like bookmarks demo to get you started. to build run the following:

1. Setup Cassandra 0.6 with storage-conf.xml in config

ANT setup

2. ant lucandra.jar
3. ant test -Dcassandra.host=127.0.0.1 -Dcassandra.port=9160 -Dcassandra.framed=false -Dcassandra.keyspace=Lucandra

Maven

2. mvn package
3. mvn deploy -DrepositoryId=<your repo> -Durl=<your repository repo>

if you perform a release from your own fork, you will need to do so with the following arguments.  Note the need to pass the system properties to the forked release jvm

mvn release:prepare release:perform -Darguments="-DrepositoryId=<your repo> -Durl=<your repository repo>"


#edit run-demo with appropriate settings (delicious clone)
4. run_demo.sh -index bookmarks.tsv
5. run_demo.sh -search title:linu*


Solr example:
=============
Lucandra also supports Solr. to build run the following:

1. Setup Cassandra 0.6 with storage-conf.xml in config
2. ant lucandra.jar
3. cd solr-example; java -jar start.jar
4. cd exampledocs; ./post.sh *.xml
5. surf to http://localhost:8983/solr/admin/


Background
==========

Storing an inverted index in Cassandra was the initial use-case for Cassandra at Facebook.
The Cassandra wiki discusses this:

"You can think of each super column name as a term and the columns within as the docids
with rank info and other attributes being a part of it. If you have keys as the userids
then you can have a per-user index stored in this form. This is how the per user index
 for term search is laid out for Inbox search at Facebook."

Initially we implemented Lucene support with supercolumn as described but ran into
a scaling issue when we tried to index all of wikipedia.
Turns out Cassandra keeps the supercolumn in memory for a given key.

Thankfully Cassandra added support for distributed ordered keys that
allows us to use keys to store index terms without supercolumns.

Implementation Notes
======================

The Lucandra Cassandra config looks like this. 

<Keyspace Name="Lucandra">
    <ColumnFamily Name="TermInfo"
        CompareWith="BytesType"
        ColumnType="Super"
        CompareSubcolumnsWith="BytesType"
        KeysCached="10%" />
    <ColumnFamily Name="Documents"
        CompareWith="BytesType"
        KeysCached="10%" />
</Keyspace>

* To get the most out of Lucandra you should use a OrderedPreservingParitioner.
However if you really want to use the RandomPartitioner you can but you'll loose
some of lucene's most powerful features (Wildcards,Sort,Range Queries).

* Documents Ids are currently random and autogenerated.

* Term keys and Document Keys are encoded as follows (using a random binary delimiter)

      Term Key                     col name         value
      "index_name/field/term" => { documentId , position vector }

      Document Key
      "index_name/documentId" => { fieldName , value }


The IndexReader caches terms aggressively during search and tries to avoid lots of back and forth with Cassandra.


What Works
==========
* Real-Time indexing (documents become available almost immediately)
* No optimizing
* Search
* Sort
* Range Queries
* Delete
* Wildcards and other Lucene magic
* Faceting/Highlighting

What's Missing (for now)
==============
* You can't walk the documents with index reader


What's Slow (for now)
=====================
* Indexes with many documents and very dense terms

Something went wrong with that request. Please try again.