A Spark-based data comparison tool at scale which facilitates software development engineers to compare a plethora of pair combinations of possible data sources. Multiple execution modes in multiple environments enable the user to generate a diff report as a Java/Scala-friendly DataFrame or as a file for future use. Comes with out of the box Spa…
Switch branches/tags
Nothing to show
Clone or download
Latest commit 7c045f5 Oct 5, 2018
Failed to load latest commit information.
config Initial Commit Dec 11, 2017
examples/TransformationTesting Version change Oct 2, 2018
.travis.yml added copilot command to run blackduck scan Feb 12, 2018
DCO Initial Commit Dec 11, 2017
LICENSE Initial Commit Dec 11, 2017
README.md Changes version in readme Oct 2, 2018
pom.xml Merge pull request #39 from aosama/develop Oct 6, 2018


Black Duck Security Risk


MegaSparkDiff is an open source tool that helps you compare any pair
combination of data sets that are of the following types: 

(HDFS, JDBC, S3, Hbase, Text Files, Hive). 

MegaSparkDiff can run on Amazon EMR (Elastic Map Reduce),
Amazon EC2 instances and cloud environments
with compatible Spark distributions.

How to Use from Within a Java or Scala Project



It parallelizes source/target data.

The data sources can be in following forms:
    Text File
    HDFS File
    SQL query over a JDBC data source
    Hive Table


Compares pair combinations of supported sources,
Please note in case of comparing a schema-based source to a non-schema based source, the SparkCompare
class will attempt to flatten the schema based source to delimited values and then do the comparison. The delimiter
can be specified while launching the compare job.

How to use via shell script in EMR

There will exist a shell script named a3a.sh that will wrap around
this Java/Scala project.  This script will accept several parameters
related to source definitions, output destination, and run
configurations, as well as which two data sets to compare.

The parameters are as follows:
    -ds=<data_source_folder>: The folder where the database
        connection parameters and data queries reside
    -od=<output_directory>: The directory where MegaSparkDiff will write
        its output
    -rc=<run_config_file_name>: The file that will be used to load
        any special run and Spark configurations.  This parameter is
To specify a data set to compare, pass in the name of one of the
data queries found in a config file inside <data_source_folder>
prepended by "--".  The program will execute the queries assigned to
the names passed into the command line, store them into tables, and
perform the comparison.

Example call:
    ./msd.sh -ds=./data_sources/ -od=output --shraddha --carlos
Additionally, the user will have the option to add JDBC Driver jar
files by including them in the classpath.  This is to enable them to
extract from whichever database they choose.

Run tests on Windows

  1. Download Hadoop winutils
  2. Extract to some path, e.g. C:\Users\MegaSparkDiffFan\bin
  3. Run tests while defining hadoop.home.dir, e.g. mvn test -Dhadoop.home.dir=C:\Users\MegaSparkDiffFan