Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP
Behemoth is an open source platform for large scale document analysis based on Apache Hadoop.
branch: master

This branch is 132 commits behind DigitalPebble:master

Fetching latest commit…

Cannot retrieve the latest commit at this time

Failed to load latest commit information.
core
gate
io
mahout
solr
tika
uima
.gitignore
LICENSE.txt
README
behemoth-site.xml
eclipse-format.xml
hadoop-job.xml
pom.xml
script.sh

README

Behemoth is an open source platform for large scale document processing based on Apache Hadoop.

It consists of a simple annotation-based implementation of a document and a number of modules operating on these documents.
One of the main aspects of Behemoth is to simplify the deployment of document analysers on a large scale but also to provide reusable modules for :
- ingesting from common data sources (Warc, Nutch, etc...)
- text processing (Tika, UIMA, GATE)
- generating output for external tools (SOLR, Mahout)

Its modular architecture simplifies the development of custom annotators based on MapReduce.

Note that Behemoth does not implement any NLP or Machine Learning components as such but serves as a 'large-scale glueware' for existing resources. Being Hadoop-based, it benefits from all its features, namely scalability, fault-tolerance and most notably the back up of a thriving open source community. 

WIKI : https://github.com/jnioche/behemoth/wiki

Mailing list : http://groups.google.com/group/digitalpebble 
Something went wrong with that request. Please try again.