Skip to content
Java Scala Python Shell
Latest commit ee99bac Jul 13, 2014 @mikecafarella mikecafarella Add code that will automatically simplify induced schema while parsin…
…g at least X% of the input data. The user indicates what X should be.
Failed to load latest commit information.
bin
src Add code that will automatically simplify induced schema while parsin… Jul 13, 2014
.gitignore
LICENSE.txt
NOTICE.txt
README.md
pom.xml

README.md

RecordBreaker - Automatically learn Avro structures from your data

Introduction

RecordBreaker is a project that automatically turns your text-formatted data (server logs, sensor readings, etc) into structured Avro data, without any need to write parsers or extractors. Its goal is to dramatically reduce the time spent preparing data for analysis, enabling more time for the analysis itself.

You can (and should!) read the full RecordBreaker tutorial here: http://cloudera.github.com/RecordBreaker/

The RecordBreaker repository is hosted at GitHub, here: https://github.com/cloudera/RecordBreaker

One interesting part of RecordBreaker is the FishEye system. It's a web-based tool for examining and managing the diverse datasets likely to be found in a typical HDFS installation. It draws features from both filesystem management and database administration tools. Most interestingly, it uses RecordBreaker techniques to automatically figure out the structure of files it finds. You can run it by typing:

bin/learnstructure fisheye -run <portnum> <localstoragedir>

Where portnum is the HTTP port where FishEye will provide data to the user, and localstoreagedir is where it will maintain information about a target filesystem.

Installation

$ ant dist

Dependencies

-- Java JDK 1.6 -- Apache ant 1.8.2

Something went wrong with that request. Please try again.