Skip to content
Enables full-text searching of CouchDB documents using Lucene
Java Shell Python Batchfile
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Failed to load latest commit information.
src
.gitignore
LICENSE
NOTES
README.md
TESTS
THANKS.md
TODO
couchdb-external-hook.py
pom.xml

README.md

WARNING

This branch (master) contains the development version of the next release (0.5). It is not yet stable or feature complete and should not be used for anything but testing. The documentation in this readme is also not yet accurate for 0.5, proceed with caution!

I strongly recommend using the 0.4 release (available at GitHub, please see the README for unpacking details) as the 0.5 work will occur on trunk and will definitely have extended periods of instability and broken-ness.

Version Compatibility

CouchDBcouchdb-lucene
0.9.1, 0.100.4
0.9.1, 0.10, 0.11 (not yet released)0.5 (not yet released)

Issue Tracking

Issue tracking at github.

Minimum System Requirements

Java 1.5 (or above) is required; the Sun version is recommended as it's regularly tested against.

Build couchdb-lucene

  1. Install Maven 2.
  2. checkout repository
  3. type 'mvn'
  4. configure couchdb (see below)

You will now have a zip file in the target/ directory. This contains all the couchdb-lucene code, dependencies, startup scripts and configuration files to run couchdb-lucene.

Configure CouchDB

[couchdb]
os_process_timeout=60000 ; increase the timeout from 5 seconds.

[external]
fti=/path/to/ruby /usr/lib/couchdb/couchdb-lucene/couchdb-external-hook.rb couchdb.log.dir=/tmp couchdb.lucene.host=localhost couchdb.lucene.port=5985

[httpd_db_handlers]
_fti = {couch_httpd_external, handle_external_req, <<"fti">>}

Configure couchdb-lucene

From 0.5 onwards, couchdb-lucene runs in a single, standalone JVM. As such, you can choose to locate your couchdb-lucene server on a different machine to couchdb if you wish, or keep it on the same machine, it's your call.

  1. Unzip the couchdb-lucene zip file.
  2. Open the conf/couchdb-lucene.properties file.
  3. Edit the couchdb.url to point to your couchdb installation.

    Start couchdb-lucene

    To start couchdb-lucene, run;

    bin/run
    

    To stop couchdb-lucene, simply kill the Java process.

    Indexing Strategy

    Document Indexing

    You must supply a index function in order to enable couchdb-lucene as, by default, nothing will be indexed. To suppress a document from the index, return null. It's more typical to return a single Document object which contains everything you'd like to query and retrieve. You may also return an array of Document objects if you wish.

    You may add any number of index views in any number of design documents. All searches will be constrained to documents emitted by the index functions.

    Here's an complete example of a design document with couchdb-lucene features:

    {
        "_id":"_design/a_design_document_with_any_name_you_like",
        "fulltext": {
            "by_subject": {
                "index":"function(doc) { var ret=new Document(); ret.add(doc.subject); return ret }"
            },
            "by_content": {
                "index":"function(doc) { var ret=new Document(); ret.add(doc.content); return ret }"
            }
        }
    }
    

    Here are some example URL's for the given design document;

    http://localhost:5984/database/_fti/lucene/by_subject?q=hello
    http://localhost:5984/database/_fti/lucene/by_content?q=hello
    

    A fulltext object contains multiple index view declarations. An index view consists of;

    analyzer
    (optional) The analyzer to use
    defaults
    (optional) The default for numerous indexing options can be overridden here. A full list of options follows.
    index
    The indexing function itself, documented below.

    The Defaults Object

    The following indexing options can be defaulted;

    name description available options default
    field the field name to index under user-defined default
    type the type of the field date, double, float, integer, long, string string
    store whether the data is stored. The value will be returned in the search result. yes, no no
    index whether (and how) the data is indexed analyzed, analyzed_no_norms, no, not_analyzed, not_analyzed_no_norms analyzed

    The Analyzer Option

    Lucene has numerous ways of converting free-form text into tokens, these classes are called Analyzer's. By default, the StandardAnalyzer is used which lower-cases all text, drops common English words ("the", "and", and so on), among other things. This processing might not always suit you, so you can choose from several others by setting the "analyzer" field to one of the following values;

    • brazilian
    • chinese
    • cjk
    • czech
    • dutch
    • english
    • french
    • german
    • keyword
    • porter
    • russian
    • simple
    • standard
    • thai

    Note: You must also supply analyzer= as a query parameter to ensure that queries are processed correctly.

    The Document class

    You may construct a new Document instance with;

    var doc = new Document();
    

    Data may be added to this document with the add method which takes an optional second object argument that can override any of the above default values.

    // Add with all the defaults.
    doc.add("value");
    
    // Add a numeric field.
    doc.add(35, {"type":"integer"});
    
    // Add a date field.
    doc.add(new Date("2009-01-01"), {"type":"date"});
    
    // Add a date field (object must be a Date object
    
    // Add a subject field.
    doc.add("this is the subject line.", {"field":"subject"});
    
    // Add but ensure it's stored.
    doc.add("value", {"store":"yes"});
    
    // Add but don't analyze.
    doc.add("don't analyze me", {"index":"not_analyzed"});
    
    // Extract text from the named attachment and index it (but not store it).
    doc.attachment("attachment name", {"field":"attachments"});
    

    Example Transforms

    Index Everything

    function(doc) {
        var ret = new Document();
    
        function idx(obj) {
        for (var key in obj) {
            switch (typeof obj[key]) {
            case 'object':
            idx(obj[key]);
            break;
            case 'function':
            break;
            default:
            ret.add(obj[key]);
            break;
            }
        }
        };
    
        idx(doc);
    
        if (doc._attachments) {
        for (var i in doc._attachments) {
            ret.attachment("attachment", i);
        }
        }
    
        return ret;
    }
    

    Index Nothing

    function(doc) {
      return null;
    }
    

    Index Select Fields

    function(doc) {
      var result = new Document();
      result.add(doc.subject, {"field":"subject", "store":"yes"});
      result.add(doc.content, {"field":"subject"});
      result.add({"field":"indexed_at"});
      return result;
    }
    

    Index Attachments

    function(doc) {
      var result = new Document();
      for(var a in doc._attachments) {
        result.add_attachment(a, {"field":"attachment"});
      }
      return result;
    }
    

    A More Complex Example

    function(doc) {
        var mk = function(name, value, group) {
            var ret = new Document();
            ret.add(value, {"field": group, "store":"yes"});
            ret.add(group, {"field":"group", "store":"yes"});
            return ret;
        };
        var ret = [];
        if(doc.type != "reference") return null;
        for(var g in doc.groups) {
            ret.add(mk("library", doc.groups[g].library, g));
            ret.add(mk("method", doc.groups[g].method, g));
            ret.add(mk("target", doc.groups[g].target, g));
        }
        return ret;
    }
    

    Attachment Indexing

    Couchdb-lucene uses Apache Tika to index attachments of the following types, assuming the correct content_type is set in couchdb;

    Supported Formats

    • Excel spreadsheets (application/vnd.ms-excel)
    • HTML (text/html)
    • Images (image/*)
    • Java class files
    • Java jar archives
    • MP3 (audio/mp3)
    • OpenDocument (application/vnd.oasis.opendocument.*)
    • Outlook (application/vnd.ms-outlook)
    • PDF (application/pdf)
    • Plain text (text/plain)
    • Powerpoint presentations (application/vnd.ms-powerpoint)
    • RTF (application/rtf)
    • Visio (application/vnd.visio)
    • Word documents (application/msword)
    • XML (application/xml)

    Searching with couchdb-lucene

    You can perform all types of queries using Lucene's default query syntax. The _body field is searched by default which will include the extracted text from all attachments. The following parameters can be passed for more sophisticated searches;

    analyzer
    The analyzer used to convert the query string into a query object.
    callback
    Specify a JSONP callback wrapper. The full JSON result will be prepended with this parameter and also placed with parentheses."
    debug
    Setting this to true disables response caching (the query is executed every time) and indents the JSON response for readability.
    force_json
    Usually couchdb-lucene determines the Content-Type of its response based on the presence of the Accept header. If Accept contains "application/json", you get "application/json" in the response, otherwise you get "text/plain;charset=utf8". Some tools, like JSONView for FireFox, do not send the Accept header but do render "application/json" responses if received. Setting force_json=true forces all response to "application/json" regardless of the Accept header.
    include_docs
    whether to include the source docs
    limit
    the maximum number of results to return
    q
    the query to run (e.g, subject:hello). If not specified, the default field is searched.
    rewrite
    (EXPERT) if true, returns a json response with a rewritten query and term frequencies. This allows correct distributed scoring when combining the results from multiple nodes.
    skip
    the number of results to skip
    sort
    the comma-separated fields to sort on. Prefix with / for ascending order and \ for descending order (ascending is the default if not specified).
    stale=ok
    If you set the stale option to ok, couchdb-lucene may not perform any refreshing on the index. Searches may be faster as Lucene caches important data (especially for sorting). A query without stale=ok will use the latest data committed to the index.

    All parameters except 'q' are optional.

    Special Fields

    _db
    The source database of the document.
    _id
    The _id of the document.

    Dublin Core

    All Dublin Core attributes are indexed and stored if detected in the attachment. Descriptions of the fields come from the Tika javadocs.

    _dc.contributor
    An entity responsible for making contributions to the content of the resource.
    _dc.coverage
    The extent or scope of the content of the resource.
    _dc.creator
    An entity primarily responsible for making the content of the resource.
    _dc.date
    A date associated with an event in the life cycle of the resource.
    _dc.description
    An account of the content of the resource.
    _dc.format
    Typically, Format may include the media-type or dimensions of the resource.
    _dc.identifier
    Recommended best practice is to identify the resource by means of a string or number conforming to a formal identification system.
    _dc.language
    A language of the intellectual content of the resource.
    _dc.modified
    Date on which the resource was changed.
    _dc.publisher
    An entity responsible for making the resource available.
    _dc.relation
    A reference to a related resource.
    _dc.rights
    Information about rights held in and over the resource.
    _dc.source
    A reference to a resource from which the present resource is derived.
    _dc.subject
    The topic of the content of the resource.
    _dc.title
    A name given to the resource.
    _dc.type
    The nature or genre of the content of the resource.

    Examples

    http://localhost:5984/dbname/_fti/design_doc/view_name?q=field_name:value
    http://localhost:5984/dbname/_fti/design_doc/view_name?q=field_name:value&sort=other_field
    http://localhost:5984/dbname/_fti/design_doc/view_name?debug=true&sort=billing_size&q=body:document AND customer:[A TO C]
    

    Search Results Format

    The search result contains a number of fields at the top level, in addition to your search results.

    etag
    An opaque token that reflects the current version of the index. This value is also returned in an ETag header to facilitate HTTP caching.
    fetch_duration
    The number of milliseconds spent retrieving the documents.
    limit
    The maximum number of results that can appear.
    q
    The query that was executed.
    rows
    The search results array, described below.
    search_duration
    The number of milliseconds spent performing the search.
    skip
    The number of initial matches that was skipped.
    total_rows
    The total number of matches for this query.

    The search results array

    The search results arrays consists of zero, one or more objects with the following fields;

    doc
    The original document from couch, if requested with include_docs=true
    fields
    All the fields that were stored with this match
    id
    The unique identifier for this match.
    score
    The normalized score (0.0-1.0, inclusive) for this match

    Here's an example of a JSON response without sorting;

    {
      "q": "+content:enron",
      "skip": 0,
      "limit": 2,
      "total_rows": 176852,
      "search_duration": 518,
      "fetch_duration": 4,
      "rows":   [
            {
          "id": "hain-m-all_documents-257.",
          "score": 1.601625680923462
        },
            {
          "id": "hain-m-notes_inbox-257.",
          "score": 1.601625680923462
        }
      ]
    }
    

    And the same with sorting;

    {
      "q": "+content:enron",
      "skip": 0,
      "limit": 3,
      "total_rows": 176852,
      "search_duration": 660,
      "fetch_duration": 4,
      "sort_order":   [
            {
          "field": "source",
          "reverse": false,
          "type": "string"
        },
            {
          "reverse": false,
          "type": "doc"
        }
      ],
      "rows":   [
            {
          "id": "shankman-j-inbox-105.",
          "score": 0.6131107211112976,
          "sort_order":       [
            "enron",
            6
          ]
        },
            {
          "id": "shankman-j-inbox-8.",
          "score": 0.7492915391921997,
          "sort_order":       [
            "enron",
            7
          ]
        },
            {
          "id": "shankman-j-inbox-30.",
          "score": 0.507369875907898,
          "sort_order":       [
            "enron",
            8
          ]
        }
      ]
    }
    

    Content-Type of response

    The Content-Type of the response is negotiated via the Accept request header like CouchDB itself. If the Accept header includes "application/json" then that is also the Content-Type of the response. If not, "text/plain;charset=utf-8" is used.

    Fetching information about the index

    Calling couchdb-lucene without arguments returns a JSON object with information about the whole index.

    http://127.0.0.1:5984/enron/_fti
    

    returns;

    {"doc_count":517350,"doc_del_count":1,"disk_size":318543045}
    

    Working With The Source

    To develop "live", type "mvn dependency:unpack-dependencies" and change the external line to something like this;

    fti=/usr/bin/java -server -cp /path/to/couchdb-lucene/target/classes:\
    /path/to/couchdb-lucene/target/dependency com.github.rnewson.couchdb.lucene.Main
    

    You will need to restart CouchDB if you change couchdb-lucene source code but this is very fast.

    Configuration

    couchdb-lucene is configured from the couchdb ini files

    [lucene]
    ; Send lucene indexes elsewhere, defaults to hidden subdir under database_dir
    ;lucene_dir=
    
    ; Send logs elsewhere, defaults to same dir as couchdb log file.
    ;log_dir=
    
    ; alter (soft) maximum ram for indexing.
    ;max_ram=16
    
    ; Specify the default operator for boolean queries. If not specified, or if the setting is not recognized, the defaul is "OR". The other choice is "AND".
    ; default_boolean_operator="OR"
    

    Basic Authentication

    If you put couchdb behind an authenticating proxy you can still configure couchdb-lucene to pull from it by specifying additional system properties. Currently only Basic authentication is supported.

    couchdb.password
    the password to authenticate with.
    couchdb.user
    the user to authenticate as.
Something went wrong with that request. Please try again.