The indexing API in 0.3 will change once again to allow multiple design documents and "views" into Lucene. It will also move much of the Lucene-specific stuff into an options object. Please read the TODO for details.
The indexing API in 0.2 has completely changed, please re-read this document and report any surprises/bugs to the bug tracker;
Issue tracking now available at lighthouseapp.
Sun JDK 5 or higher is necessary. Couchdb-lucene is known to be incompatible with OpenJDK as it includes an earlier, and incompatible, version of the Rhino Javascript library.
- Install Maven 2.
- checkout repository
- type 'mvn'
- configure couchdb (see below)
[couchdb]
os_process_timeout=60000 ; increase the timeout from 5 seconds.
[external]
fti=/usr/bin/java -jar /path/to/couchdb-lucene*-jar-with-dependencies.jar -search
[update_notification]
indexer=/usr/bin/java -jar /path/to/couchdb-lucene*-jar-with-dependencies.jar -index
[httpd_db_handlers]
_fti = {couch_httpd_external, handle_external_req, <<"fti">>}
You must supply a transform function in order to enable couchdb-lucene .
Add a design document called _design/lucene in your database with an attribute called "transform". The value of this attribute is a Javascript function.
The transform function can return null, to prevent indexing, and either a single Document or an array of Documents.
The transform function is called for each document in the database. To pass information to Lucene, you must populate Document instances with data from the original CouchDB document.
You may construct a new Document instance with;
var doc = new Document();
Several functions are available that populate a Document.
// Indexed, analyzed but not stored.
doc.field("name", "value");
// Indexed, analyzed and stored.
doc.field("name", "value", "yes");
// Indexed, stored but not analyzed.
doc.field("name", "value", "yes", "not_analyzed");
// Extract text from the named attachment and index it (but not store it).
doc.attachment("name", "attachment name");
// Interpret "value" as a date using the default date formats.
doc.date("name", "value");
// intrepret "value" as a date using the supplied format string
// (see Java's SimpleDateFormat class for the syntax).
doc.date("name", "value", "format");
function(doc) {
var ret = new Document();
function idx(obj) {
for (var key in obj) {
switch (typeof obj[key]) {
case 'object':
idx(obj[key]);
break;
case 'function':
break;
default:
ret.field(key, obj[key]);
/* Uncomment next line to include
* all attributes into a single field.
*/
// ret.field("all", obj[key]);
break;
}
}
}
// Index all attributes
idx(doc);
// Index all attachments
for(var a in doc._attachments) {
ret.attachment("attachment", a);
}
return ret;
}
function(doc) {
return null;
}
function(doc) {
var result = new Document();
result.field("subject", doc.subject, "yes");
result.field("content", doc.content);
result.date("indexed_at", new Date());
return result;
}
function(doc) {
var result = new Document();
for(var a in doc._attachments) {
result.attachment("attachment", a);
}
return result;
}
function(doc) {
var mk = function(name, value, group) {
var ret = new Document(name, value, "yes");
ret.field("group", group, "yes");
return ret;
};
var ret = [];
if(doc.type != "reference") return null;
for(var g in doc.groups) {
ret.push(mk("library", doc.groups[g].library, g));
ret.push(mk("method", doc.groups[g].method, g));
ret.push(mk("target", doc.groups[g].target, g));
}
return ret;
}
Couchdb-lucene uses Apache Tika to index attachments of the following types, assuming the correct content_type is set in couchdb;
- Excel spreadsheets (application/vnd.ms-excel)
- Word documents (application/msword)
- Powerpoint presentations (application/vnd.ms-powerpoint)
- Visio (application/vnd.visio)
- Outlook (application/vnd.ms-outlook)
- XML (application/xml)
- HTML (text/html)
- Images (image/*)
- Java class files
- Java jar archives
- MP3 (audio/mp3)
- OpenDocument (application/vnd.oasis.opendocument.*)
- Plain text (text/plain)
- PDF (application/pdf)
- RTF (application/rtf)
You can perform all types of queries using Lucene's default query syntax. The _body field is searched by default which will include the extracted text from all attachments. The following parameters can be passed for more sophisticated searches;
- q
- the query to run (e.g, subject:hello)
- sort
- the comma-separated fields to sort on. Prefix with / for ascending order and \ for descending order (ascending is the default if not specified).
- limit
- the maximum number of results to return
- skip
- the number of results to skip
- include_docs
- whether to include the source docs
- stale=ok
- If you set the stale option ok, couchdb-lucene may not perform any refreshing on the index. Searches may be faster as Lucene caches important data (especially for sorting). A query without stale=ok will use the latest data committed to the index.
- debug
- if false, a normal application/json response with results appears. if true, an pretty-printed HTML blob is returned instead.
- rewrite
- (EXPERT) if true, returns a json response with a rewritten query and term frequencies. This allows correct distributed scoring when combining the results from multiple nodes.
All parameters except 'q' are optional.
- _db
- The source database of the document.
- _id
- The _id of the document.
All Dublin Core attributes are indexed and stored if detected in the attachment. Descriptions of the fields come from the Tika javadocs.
- dc.contributor
- An entity responsible for making contributions to the content of the resource.
- dc.coverage
- The extent or scope of the content of the resource.
- dc.creator
- An entity primarily responsible for making the content of the resource.
- dc.date
- A date associated with an event in the life cycle of the resource.
- dc.description
- An account of the content of the resource.
- dc.format
- Typically, Format may include the media-type or dimensions of the resource.
- dc.identifier
- Recommended best practice is to identify the resource by means of a string or number conforming to a formal identification system.
- dc.language
- A language of the intellectual content of the resource.
- dc.modified
- Date on which the resource was changed.
- dc.publisher
- An entity responsible for making the resource available.
- dc.relation
- A reference to a related resource.
- dc.rights
- Information about rights held in and over the resource.
- dc.source
- A reference to a resource from which the present resource is derived.
- dc.subject
- The topic of the content of the resource.
- dc.title
- A name given to the resource.
- dc.type
- The nature or genre of the content of the resource.
http://localhost:5984/dbname/_fti?q=field_name:value http://localhost:5984/dbname/_fti?q=field_name:value&sort=other_field http://localhost:5984/dbname/_fti?debug=true&sort=billing_size&q=body:document AND customer:[A TO C]
Here's an example of a JSON response without sorting;
{
"q": "+_db:enron +content:enron",
"skip": 0,
"limit": 2,
"total_rows": 176852,
"search_duration": 518,
"fetch_duration": 4,
"rows": [
{
"_id": "hain-m-all_documents-257.",
"score": 1.601625680923462
},
{
"_id": "hain-m-notes_inbox-257.",
"score": 1.601625680923462
}
]
}
And the same with sorting;
{
"q": "+_db:enron +content:enron",
"skip": 0,
"limit": 3,
"total_rows": 176852,
"search_duration": 660,
"fetch_duration": 4,
"sort_order": [
{
"field": "source",
"reverse": false,
"type": "string"
},
{
"reverse": false,
"type": "doc"
}
],
"rows": [
{
"_id": "shankman-j-inbox-105.",
"score": 0.6131107211112976,
"sort_order": [
"enron",
6
]
},
{
"_id": "shankman-j-inbox-8.",
"score": 0.7492915391921997,
"sort_order": [
"enron",
7
]
},
{
"_id": "shankman-j-inbox-30.",
"score": 0.507369875907898,
"sort_order": [
"enron",
8
]
}
]
}
Calling couchdb-lucene without arguments returns a JSON object with information about the index.
http://127.0.0.1:5984/enron/_fti
returns;
{"doc_count":517350,"doc_del_count":1,"disk_size":318543045}
To develop "live", type "mvn dependency:unpack-dependencies" and change the external line to something like this;
fti=/usr/bin/java -cp /path/to/couchdb-lucene/target/classes:\ /path/to/couchdb-lucene/target/dependency com.github.rnewson.couchdb.lucene.Main
You will need to restart CouchDB if you change couchdb-lucene source code but this is very fast.
couchdb-lucene respects several system properties;
- couchdb.url
- the url to contact CouchDB with (default is "http://localhost:5984")
- couchdb.lucene.dir
- specify the path to the lucene indexes (the default is to make a directory called 'lucene' relative to couchdb's current working directory.
- couchdb.log.dir
- specify the directory of the log file (which is called couchdb-lucene.log), defaults to the platform-specific temp directory.
You can override these properties like this;
fti=/usr/bin/java -Dcouchdb.lucene.dir=/tmp \ -cp /home/rnewson/Source/couchdb-lucene/target/classes:\ /home/rnewson/Source/couchdb-lucene/target/dependency\ com.github.rnewson.couchdb.lucene.Main
If you put couchdb behind an authenticating proxy you can still configure couchdb-lucene to pull from it by specifying additional system properties. Currently only Basic authentication is supported.
- couchdb.user
- the user to authenticate as.
- couchdb.password
- the password to authenticate with.
The default for couchdb.url is problematic on an IPv6 system. Specify -Dcouchdb.url=http://[::1]:5984 to resolve it.