Skip to content
Big Data ETL and Utilities for Hadoop Map Reduce, Spark and Storm
Java Scala Python Shell
Branch: master
Clone or download
Pranab Ghosh
Pranab Ghosh more filter operator
Latest commit de1090a Oct 16, 2019
Type Name Latest commit message Commit time
Failed to load latest commit information.
resource sanity check for record Jul 16, 2019
spark column aggregation Oct 15, 2019
.gitignore toString method Dec 25, 2016 update data transformation tutorial Mar 22, 2019
build.sbt sbt publish local Nov 24, 2015 log2 method Apr 21, 2019 initial commit Jan 4, 2012
pom.xml lucene 7.1.0 Oct 11, 2019


Hadoop based ETL and various utility classes for Hadoop and Storm


  • Simple to use
  • Input output in CSV format
  • Metadata defined in simple JSON file
  • Extremely configurable with many configuration knobs


  • Various relational algebra operation, including Projection, Join etc
  • Data extraction ETL to extract structured record from unstructured data
  • Data extraction ETL to extract structured record from JSON data
  • Data validation ETL with configurable rules and statistical parameters
  • Data profiling ETL with various techniques
  • Data transformation ETL with configurable transformation rules
  • Various statistical data exploration solutions
  • Data normalization
  • Seasonal data analysis
  • Various statistical parameter calculation
  • Various long term statistical parameter calculation with incremental data
  • Bulk inset, update and delete of Hadoop data
  • Bases classes for Storm Spout and Bolt
  • Utility classes for string, configuration
  • Utility classes for Storm and Redis


The following blogs of mine are good source of details. These are the only source of detail documentation. Map reduce jobs in this projec are used in other projects including sifarish, avenir etc. Blogs related to thos projects are also relevant.


For Hadoop 1

  • mvn clean install

For Hadoop 2 (non yarn)

  • git checkout nuovo
  • mvn clean install

For Hadoop 2 (yarn)

  • git checkout nuovo
  • mvn clean install -P yarn

For spark

  • Build chombo first in master branch with
    • mvn clean install
    • sbt publishLocal
  • Build chombo-spark in chombo/spark directory
    • sbt clean package

Need help?

Please feel free to email me at


Contributors are welcome. Please email me at

You can’t perform that action at this time.