A library enabling easy Lucene indexing of PDF text and metadata
Java Clojure
Latest commit 1d97a87 Nov 26, 2014 @cemerick cemerick derp
Permalink
Failed to load latest commit information.
src add readme Nov 24, 2014
test move PDFxStream lucene integration to separate project Nov 24, 2014
.gitignore
.travis.yml drop testing on JDK 6, its compiler can't consume the lucene v4.x jars Nov 24, 2014
LICENSE
README.md .close() PDF documents in code samples Nov 24, 2014
project.clj

README.md

lucene-pdf Travis CI status

lucene-pdf is a JVM (Java, Scala, Groovy, Clojure, etc) library enabling easy Lucene indexing of PDF text and metadata via integration with PDFxStream.

"Installation"

lucene-pdf is available in Maven central; add it to your Maven project's pom.xml:

<dependency>
  <groupId>com.snowtide</groupId>
  <artifactId>lucene-pdf</artifactId>
  <version>3.0.0</version>
</dependency>

Or, add the above Maven artifact coordinates to your {Gradle, Leiningen, sbt, etc} project file.

lucene-pdf is suitable for use with JDK 1.5+, and is tested against the latest releases of each major revision of Lucene core (1.x, 2.x, 3.x, an 4.x). See the project file for the exact versions used under test.

While lucene-pdf is suitable for many typical Lucene PDF indexing jobs, there may be aspects of your project's requirements that it cannot meet (e.g. taking advantage of some of the more esoteric document indexing parameters available in more recent versions of Lucene). In that case, its source can serve as a useful starting point, exhibiting how PDF data can be extracted using PDFxStream and turned into Lucene Documents; please feel free to import it into your projects and modify it as needed to suit your needs.

Documentation

Example usage

Given a PDF file stored on disk at /tmp/foo.pdf, this Java code will use lucene-pdf to construct a Lucene org.apache.lucene.document.Document populated with fields corresponding to the PDF's main body text and metadata attributes:

import com.snowtide.PDF;
import com.snowtide.pdf.lucene.LucenePDFDocumentFactory;
import org.apache.lucene.document.Document;

// ....

com.snowtide.pdf.Document pdf = PDF.open(new File("/tmp/foo.pdf"));
Document luceneDocument = LucenePDFDocumentFactory.buildPDFDocument(pdf);
pdf.close();

luceneDocument can then be added to a Lucene index.

This is the simplest sample possible, but it uses a default configuration to name the fields in the created Lucene document. You will likely want to provide your own names for:

  • the field containing the source PDF document's main body text
  • fields corresponding to various PDF document metadata attributes

This Java code does just that, using a LucenePDFConfiguration object to control the mapping:

import com.snowtide.PDF;
import com.snowtide.pdf.lucene.LucenePDFDocumentFactory;
import com.snowtide.pdf.lucene.LucenePDFConfiguration;
import org.apache.lucene.document.Document;

// ....

File f = new File("/tmp/foo.pdf");

LucenePDFConfiguration config = new LucenePDFConfiguration();
config.setBodyTextFieldName("mainText");
config.setMetadataFieldMapping("Author", "document_author");
config.setMetadataFieldMapping("Title", "document_title");

com.snowtide.pdf.Document pdf = PDF.open(new File("/tmp/foo.pdf"));
Document luceneDocument = LucenePDFDocumentFactory.buildPDFDocument(pdf, config);
pdf.close();

LucenePDFConfiguration provides a number of additional ways to control how Lucene fields and documents are created, including setting storage, tokenization, and indexing/analysis flags. See Indexing PDF Documents with Lucene and PDFxStream and the lucene-pdf javadoc for details.

License

Copyright © 2004-2014 Snowtide

Distributed under the terms of the MIT License.