Skip to content


Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP
branch: master

Add code that will automatically simplify induced schema while parsin…

…g at least X% of the input data. The user indicates what X should be.
latest commit ee99bac813
Mike Cafarella mikecafarella authored
Failed to load latest commit information.
bin Fix new paths for use with maven
src Add code that will automatically simplify induced schema while parsin…
.gitignore Migrated build system from ivy to maven
LICENSE.txt Repair naming error in build file. Add README and LICENSE files
NOTICE.txt Add correct copyright notice to repo and files. Revised README

RecordBreaker - Automatically learn Avro structures from your data


RecordBreaker is a project that automatically turns your text-formatted data (server logs, sensor readings, etc) into structured Avro data, without any need to write parsers or extractors. Its goal is to dramatically reduce the time spent preparing data for analysis, enabling more time for the analysis itself.

You can (and should!) read the full RecordBreaker tutorial here:

The RecordBreaker repository is hosted at GitHub, here:

One interesting part of RecordBreaker is the FishEye system. It's a web-based tool for examining and managing the diverse datasets likely to be found in a typical HDFS installation. It draws features from both filesystem management and database administration tools. Most interestingly, it uses RecordBreaker techniques to automatically figure out the structure of files it finds. You can run it by typing:

bin/learnstructure fisheye -run <portnum> <localstoragedir>

Where portnum is the HTTP port where FishEye will provide data to the user, and localstoreagedir is where it will maintain information about a target filesystem.


$ ant dist


-- Java JDK 1.6 -- Apache ant 1.8.2

Something went wrong with that request. Please try again.