diff --git a/README.md b/README.md index ce024f3..3e702b8 100644 --- a/README.md +++ b/README.md @@ -1,5 +1,6 @@ - -# DataSqueeze +

+ +

## Overview DataSqueeze performs compaction of files from source directory to target directory maintaining the directory structure of the source. @@ -7,6 +8,14 @@ DataSqueeze performs compaction of files from source directory to target directo ## Documentation This README is intended to provide detailed technical documentation for advanced users. +## Changes since last release + +* Edited the pom file for publishing this project's artifacts to the Maven Central +* Changed namespace to com.expedia.dsp +* Renamed dataSqueeze to datasqueeze +* Refactored code to remove the dataSqueeze-manager layer +* Added DataSqueeze logo + ## General operation DataSqueeze supports two types of compaction @@ -18,19 +27,19 @@ DataSqueeze supports two types of compaction a. Fetch the source file paths to be compacted from the source path provided. b. Perform mapreduce job using the following configuration 1. Mapper maps records together based on same parent directory and emits parent directory as key. - 2. Reducer reduces records based on same key but writes data to the target directory provided by the user, retaining - the directory structure. + 2. Reducer reduces records based on same key but writes data to the target directory provided by the user, + retaining the directory structure. -2. Inplace Compaction - Performs compaction on the source path. This is not recommended on AWS-S3, since the performance will be terrible. +2. In-Place Compaction - Performs compaction on the source path. This is not recommended on AWS-S3, since the performance will be terrible. - Below is a high level summary of the steps that Compaction Utility performs during the course of a typical run for inplace compaction. + Below is a high level summary of the steps that Compaction Utility performs during the course of a typical run for in-place compaction. a. Fetch the file paths to be compacted from the source path provided. b. Perform mapreduce job using the following configuration 1. Mapper maps records together based on same parent directory and emits parent directory as key. - 2. Reducer reduces records based on same key but writes data to the target directory provided by the user, retaining - the directory structure. + 2. Reducer reduces records based on same key but writes data to the target directory provided by the user, + retaining the directory structure. c. Store the compacted files on temp-compacted path. d. Move files from source to temp location. e. Move files from temp-compacted location to source location specified by the user. @@ -49,7 +58,7 @@ DataSqueeze is a standard Maven project. Run the following in the project root f mvn clean package -The compiled JAR can be found at `dataSqueeze-manager/target/dataSqueeze-manager-{VERSION}.jar`. +The compiled JAR can be found at `datasqueeze/target/datasqueeze.jar`. To build an RPM, use the optional Maven profile `-P rpm`: @@ -64,14 +73,14 @@ There are two different ways of running DataSqueeze: 1. CLI - a. For TEXT/ORC/SEQ ```java - hadoop jar dataSqueeze-manager-1.0-SNAPSHOT.jar com.expedia.edw.data.squeeze.Utility + hadoop jar datasqueeze.jar com.expedia.dsp.data.squeeze.Utility -sp s3a://edwprod/user/ysontakke/compactiontest1/ -tp s3a://edwprod/user/ysontakke/compactionoutput_text_yash_1/ -threshold 12345 ``` b. For AVRO ```java - hadoop jar dataSqueeze-manager-1.0-SNAPSHOT.jar com.expedia.edw.data.squeeze.Utility + hadoop jar datasqueeze.jar com.expedia.dsp.data.squeeze.Utility -sp s3a://edwprod/user/ysontakke/compactiontest1/ -tp s3a://edwprod/user/ysontakke/compactionoutput_text_yash_1/ -threshold 12345 -fileType AVRO -schemaPath s3a://edwprod/user/ysontakke/compactionschema_text_yash_1/schema.avsc ``` @@ -87,7 +96,7 @@ There are two different ways of running DataSqueeze: * fileType - Type of file to be compacted (AVRO / TEXT / SEQ / ORC). It is mandatory for AVRO * schemaPath - schema used for compaction (mandatory for AVRO) -2. API - [CompactionManager](dataSqueeze-manager/src/main/java/com/expedia/edw/data/squeeze/CompactionManager.java) +2. API - [CompactionManager](datasqueeze/src/main/java/com/expedia/dsp/data/squeeze/CompactionManager.java) ```java CompactionResponse compact() throws Exception; diff --git a/dataSqueeze-logo.png b/dataSqueeze-logo.png new file mode 100644 index 0000000..74ea46f Binary files /dev/null and b/dataSqueeze-logo.png differ diff --git a/dataSqueeze-manager/pom.xml b/dataSqueeze-manager/pom.xml deleted file mode 100644 index 6f0f4ad..0000000 --- a/dataSqueeze-manager/pom.xml +++ /dev/null @@ -1,212 +0,0 @@ - - - 4.0.0 - - - dataSqueeze-parent - com.expedia.edw.data.squeeze - 1.0-SNAPSHOT - - dataSqueeze-manager - - Hadoop utility to compact small files - https://expediaincecommerceplatform.github.io/dataSqueeze/ - - - - Apache 2.0 - https://www.apache.org/licenses/LICENSE-2.0 - - - - - /opt/dataSqueeze - ${project.basedir}/target - hadoop - hadoop - UTF-8 - - - - - - org.apache.hadoop - hadoop-mapreduce-client-common - provided - - - org.apache.avro - avro - - - - - org.projectlombok - lombok - provided - - - - - commons-cli - commons-cli - compile - - - org.apache.commons - commons-lang3 - compile - - - net.sf.jmimemagic - jmimemagic - compile - - - org.json - json - compile - - - org.apache.httpcomponents - httpcore - compile - - - org.apache.orc - orc-core - compile - - - org.apache.orc - orc-mapreduce - compile - - - xerces - xercesImpl - compile - - - org.apache.avro - avro-mapred - 1.7.4 - compile - hadoop2 - - - - - com.hotels - avro-compatibility - 2.1.1 - - - - - junit - junit - test - - - org.powermock - powermock-module-junit4 - test - - - org.powermock - powermock-api-mockito - test - - - org.apache.mrunit - mrunit - test - hadoop2 - - - org.mockito - mockito-all - test - - - - - - - org.apache.maven.plugins - maven-shade-plugin - 3.0.0 - - - package - - shade - - - - - *:* - - META-INF/*.SF - META-INF/*.DSA - META-INF/*.RSA - - - - - - - - - - - - - rpm - - - - org.codehaus.mojo - rpm-maven-plugin - 2.1.5 - - - attach-rpm - package - - attached-rpm - - - ${project.description} - Applications/Engineering - Linux - redhat - ${rpm.install.user} - ${rpm.install.group} - 775 - 644 - - - ${rpm.install.dir}/lib - - - - ${project.build.directory}/dataSqueeze-manager-${project.version}.jar - - dataSqueeze-manager.jar - - - - - - - - - - - - - diff --git a/dataSqueeze-manager/src/test/java/com/expedia/edw/data/squeeze/models/CompactionCriteriaTest.java b/dataSqueeze-manager/src/test/java/com/expedia/edw/data/squeeze/models/CompactionCriteriaTest.java deleted file mode 100644 index 339e20f..0000000 --- a/dataSqueeze-manager/src/test/java/com/expedia/edw/data/squeeze/models/CompactionCriteriaTest.java +++ /dev/null @@ -1,130 +0,0 @@ -//package com.expedia.edw.data.squeeze.models; -// -//import org.junit.Assert; -//import org.junit.Test; -// -//import java.util.HashMap; -//import java.util.Map; -// -//import static org.junit.Assert.assertEquals; -//import static org.junit.Assert.assertNull; -// -//import com.expedia.edw.data.squeeze.impl.CompactionManagerImpl; -// -///** -// * Tests for {@link CompactionCriteria} -// * -// * @author Yashraj R. Sontakke -// */ -//public class CompactionCriteriaTest { -// -// @Test -// public void testNullThreshold() throws Exception { -// final Map options = retrieveOptions(null, "1000"); -// CompactionCriteria criteria = new CompactionCriteria(options); -// assertNull(criteria.getThresholdInBytes()); -// } -// -// @Test(expected = NumberFormatException.class) -// public void testWrongThreshold() throws Exception { -// final Map options = retrieveOptions("1234a", "1000"); -// CompactionCriteria criteria = new CompactionCriteria(options); -// assertNull(criteria.getThresholdInBytes()); -// } -// -// @Test -// public void testNullMaxReducers() throws Exception { -// final Map options = retrieveOptions("1234", null); -// CompactionCriteria criteria = new CompactionCriteria(options); -// assertNull(criteria.getMaxReducers()); -// } -// -// @Test(expected = NumberFormatException.class) -// public void testWrongMaxReducers() throws Exception { -// final Map options = retrieveOptions("1234", "1000a"); -// CompactionCriteria criteria = new CompactionCriteria(options); -// assertNull(criteria.getMaxReducers()); -// } -// -// @Test(expected = IllegalArgumentException.class) -// public void testNoSchemaPath() throws Exception { -// final Map options = retrieveOptions("1234", "1000"); -// options.put("fileType", "AVRO"); -// CompactionCriteria criteria = new CompactionCriteria(options); -// } -// -// @Test -// public void testCriteria() throws Exception { -//<<<<<<< HEAD -// CompactionCriteria criteria = new CompactionCriteria("source", "target", 12345L, "temp", false, 1000L); -// assertEquals("source", criteria.getSourcePath()); -// assertEquals("target", criteria.getTargetPath()); -// assertEquals(12345L, criteria.getThresholdInBytes(), 0); -// assertEquals("temp", criteria.getTempPath()); -// assertFalse(criteria.isRetainTempFiles()); -// assertEquals(1000L, criteria.getMaxReducers(), 0); -// -// criteria = new CompactionCriteria("source", "target", 12345L, "temp", true, 1000L); -// assertTrue(criteria.isRetainTempFiles()); -// -// Map options = retrieveOptions("1234", "1000"); -// options.put("retainTempFiles", "true"); -// -// criteria = new CompactionCriteria(options); -// assertEquals("sourcePath", criteria.getSourcePath()); -// assertEquals("targetPath", criteria.getTargetPath()); -// assertEquals(1234L, criteria.getThresholdInBytes(), 0); -// assertEquals("tempPath", criteria.getTempPath()); -// assertTrue(criteria.isRetainTempFiles()); -// assertEquals(1000L, criteria.getMaxReducers(), 0); -// -// options.put("retainTempFiles", "false"); -// criteria = new CompactionCriteria(options); -// assertFalse(criteria.isRetainTempFiles()); -// -// options.put("retainTempFiles", "abcd"); -// criteria = new CompactionCriteria(options); -// assertFalse(criteria.isRetainTempFiles()); -// -// assertNull(criteria.getFileType()); -// assertNull(criteria.getSchemaPath()); -// -// options.put("fileType", "AVRO"); -// options.put("schemaPath", "schemaPath"); -// criteria = new CompactionCriteria(options); -// -// assertEquals(FileType.AVRO.getValue(), criteria.getFileType()); -// assertEquals("schemaPath", criteria.getSchemaPath()); -// -// options.put("fileType", "ORC"); -// options.remove("schemaPath"); -// criteria = new CompactionCriteria(options); -// -// assertEquals(FileType.ORC.getValue(), criteria.getFileType()); -// -//======= -// CompactionCriteria criteria = new CompactionCriteria("source", "target", 12345L); -// assertEquals("source", criteria.getSourcePath()); -// assertEquals("target", criteria.getTargetPath()); -// assertEquals(12345L, criteria.getThresholdInBytes(), 0); -//>>>>>>> develop -// } -// -// @Test(expected = IllegalArgumentException.class) -// public void testUnsupportedFileFormat() { -// Map options = retrieveOptions("1234", "1000"); -// options.put("fileType", "AVROO"); -// new CompactionCriteria(options); -// } -// -// private Map retrieveOptions(final String threshold, final String maxReducers) { -// final Map options = new HashMap(); -// options.put("sourcePath", "sourcePath"); -// options.put("targetPath", "targetPath"); -// options.put("thresholdInBytes", threshold); -// options.put("tempPath", "tempPath"); -// options.put("maxReducers", maxReducers); -// return options; -// } -// -//} diff --git a/pom.xml b/pom.xml index f28362a..1bfc37d 100644 --- a/pom.xml +++ b/pom.xml @@ -4,14 +4,13 @@ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"> 4.0.0 - com.expedia.edw.data.squeeze - dataSqueeze-parent - dataSqueeze-parent - Hadoop utility to compact small files - https://expediaincecommerceplatform.github.io/dataSqueeze/ + com.expedia.dsp + datasqueeze + 2.0-SNAPSHOT - pom - 1.0-SNAPSHOT + datasqueeze + Hadoop utility to compact small files + https://expediaincecommerceplatform.github.io/datasqueeze/ @@ -20,15 +19,17 @@ + + scm:git:git://github.com/ExpediaInceCommercePlatform/datasqueeze.git + scm:git:ssh://github.com:ExpediaInceCommercePlatform/datasqueeze.git + https://github.com/ExpediaInceCommercePlatform/datasqueeze/tree/master + + github - https://github.com/ExpediaInceCommercePlatform/dataSqueeze/issues + https://github.com/ExpediaInceCommercePlatform/datasqueeze/issues - - dataSqueeze-manager - - Yashraj R. Sontakke @@ -38,6 +39,8 @@ + UTF-8 + 1.3.2 1.11.98 @@ -45,6 +48,7 @@ 2.4 3.0 + 2.7.4 1.3.2 @@ -74,265 +78,282 @@ 3.3.9 1.10.19 1.3.2 - 1.6.1 1.7.25 2.8.1 + 1.7.4 + + /opt/dataSqueeze + ${project.basedir}/target + hadoop + hadoop + + + + + commons-cli + commons-cli + ${commons-cli.version} + compile + + + org.apache.orc + orc-mapreduce + ${orc.version} + compile + + + org.apache.orc + orc-core + ${orc.version} + compile + + + org.apache.avro + avro-mapred + ${avro.version} + compile + hadoop2 + + + org.apache.commons + commons-lang3 + ${commons-lang3.version} + compile + + + org.apache.httpcomponents + httpcore + ${httpcore.version} + compile + + + org.json + json + ${json.version} + compile + + + xerces + xercesImpl + ${xerces.version} + compile + + + net.sf.jmimemagic + jmimemagic + ${jmimemagic.version} + compile + + + commons-logging + commons-logging + + + commons-io + commons-io + + + log4j + log4j + + + org.codehaus.jackson + jackson-xc + + + org.codehaus.jackson + jackson-jaxrs + + + xerces + xercesImpl + + + + + commons-io + commons-io + ${commons.io.version} + - - - - com.amazonaws - aws-java-sdk-s3 - ${aws.java.sdk.version} - - - commons-cli - commons-cli - ${commons-cli.version} - - - org.apache.commons - commons-io - ${apache.commons.io.version} - - - commons-io - commons-io - ${commons.io.version} - - - org.apache.commons - commons-lang3 - ${commons.lang3.version} - - - org.apache.hadoop - hadoop-aws - ${hadoop.version} - - - org.apache.hadoop - hadoop-common - ${hadoop.version} - - - org.apache.httpcomponents - httpcore - - - - - org.apache.hadoop - hadoop-hdfs - ${hadoop.version} - - - org.apache.hadoop - hadoop-mapreduce-client-jobclient - ${hadoop.version} - - - org.apache.hadoop - hadoop-mapreduce-client-common - ${hadoop.version} - - - org.apache.hive - hive-jdbc - ${hive.jdbc.version} - - - org.apache.hive - hive-metastore - ${hive.metastore.version} - - - org.apache.orc - orc-mapreduce - ${orc.version} - - - org.apache.orc - orc-core - ${orc.version} - - - org.projectlombok - lombok - ${lombok.version} - - - commons-cli - commons-cli - ${commons-cli.version} - - - commons-io - commons-io - ${commons-io.version} - - - org.apache.commons - commons-io - ${commons-io-1x.version} - - - org.apache.commons - commons-lang3 - ${commons-lang3.version} - - - org.apache.httpcomponents - httpclient - ${httpclient.version} - - - org.apache.httpcomponents - httpcore - ${httpcore.version} - - - net.sf.jmimemagic - jmimemagic - ${jmimemagic.version} - - - commons-logging - commons-logging - - - commons-io - commons-io - - - log4j - log4j - - - org.codehaus.jackson - jackson-xc - - - org.codehaus.jackson - jackson-jaxrs - - - xerces - xercesImpl - - - - - org.json - json - ${json.version} - - - xerces - xercesImpl - ${xerces.version} - - - org.apache.maven - maven-model - ${maven.model.version} - + + org.apache.hadoop + hadoop-mapreduce-client-common + ${hadoop.version} + provided + + + org.apache.avro + avro + + + + + com.hotels + avro-compatibility + 2.1.1 + + + org.projectlombok + lombok + ${lombok.version} + provided + - - - junit - junit - ${junit.version} - test - - - org.powermock - powermock-api-mockito - ${powermock.version} - test - - - org.powermock - powermock-core - ${powermock.version} - test - - - org.powermock - powermock-module-junit4 - ${powermock.version} - test - - - org.slf4j - slf4j-log4j12 - ${slf4j.version} - - - org.apache.mrunit - mrunit - 1.1.0 - hadoop2 - test - - - org.mockito - mockito-all - ${mockito.version} - test - - - + + org.apache.maven + maven-model + ${maven.model.version} + - - - central - http://repo1.maven.org/maven2/ - - + + + junit + junit + ${junit.version} + test + + + org.powermock + powermock-api-mockito + ${powermock.version} + test + + + org.apache.mrunit + mrunit + ${mrunit.version} + hadoop2 + test + + + org.mockito + mockito-all + ${mockito.version} + test + + + org.powermock + powermock-module-junit4 + ${powermock.version} + test + + + org.powermock + powermock-core + ${powermock.version} + test + + + org.slf4j + slf4j-log4j12 + ${slf4j.version} + + - - - - org.apache.maven.plugins - maven-surefire-plugin - 2.14.1 - - - org.apache.maven.plugins - maven-failsafe-plugin - 2.15 - - - - integration-tests - verify - - - - - - + ${project.artifactId} + + + org.apache.maven.plugins + maven-compiler-plugin + 3.6.1 + + 1.6 + 1.6 + + + + org.apache.maven.plugins + maven-shade-plugin + 3.0.0 + + + package + + shade + + + + + *:* + + META-INF/*.SF + META-INF/*.DSA + META-INF/*.RSA + + + + + + + + - integration-tests + rpm - org.apache.maven.plugins - maven-surefire-plugin - - true - + org.codehaus.mojo + rpm-maven-plugin + 2.1.5 + + + attach-rpm + package + + attached-rpm + + + ${project.description} + Applications/Engineering + Linux + redhat + ${rpm.install.user} + ${rpm.install.group} + 775 + 644 + + + ${rpm.install.dir}/lib + + + ${project.build.directory}/datasqueeze.jar + datasqueeze.jar + + + + + + + + + + + central + http://repo1.maven.org/maven2/ + + + + + + ossrh + https://oss.sonatype.org/content/repositories/snapshots + + + ossrh + https://oss.sonatype.org/service/local/staging/deploy/maven2/ + + diff --git a/dataSqueeze-manager/src/main/java/com/expedia/edw/data/squeeze/CompactionManager.java b/src/main/java/com/expedia/dsp/data/squeeze/CompactionManager.java similarity index 74% rename from dataSqueeze-manager/src/main/java/com/expedia/edw/data/squeeze/CompactionManager.java rename to src/main/java/com/expedia/dsp/data/squeeze/CompactionManager.java index cae8855..96c0a6d 100644 --- a/dataSqueeze-manager/src/main/java/com/expedia/edw/data/squeeze/CompactionManager.java +++ b/src/main/java/com/expedia/dsp/data/squeeze/CompactionManager.java @@ -1,7 +1,7 @@ -package com.expedia.edw.data.squeeze; +package com.expedia.dsp.data.squeeze; -import com.expedia.edw.data.squeeze.models.CompactionCriteria; -import com.expedia.edw.data.squeeze.models.CompactionResponse; +import com.expedia.dsp.data.squeeze.models.CompactionCriteria; +import com.expedia.dsp.data.squeeze.models.CompactionResponse; /** * Manager that performs compaction. diff --git a/dataSqueeze-manager/src/main/java/com/expedia/edw/data/squeeze/CompactionManagerFactory.java b/src/main/java/com/expedia/dsp/data/squeeze/CompactionManagerFactory.java similarity index 90% rename from dataSqueeze-manager/src/main/java/com/expedia/edw/data/squeeze/CompactionManagerFactory.java rename to src/main/java/com/expedia/dsp/data/squeeze/CompactionManagerFactory.java index b87ab29..291b23e 100644 --- a/dataSqueeze-manager/src/main/java/com/expedia/edw/data/squeeze/CompactionManagerFactory.java +++ b/src/main/java/com/expedia/dsp/data/squeeze/CompactionManagerFactory.java @@ -1,8 +1,8 @@ -package com.expedia.edw.data.squeeze; +package com.expedia.dsp.data.squeeze; -import com.expedia.edw.data.squeeze.impl.CompactionManagerImpl; -import com.expedia.edw.data.squeeze.impl.CompactionManagerInPlaceImpl; -import com.expedia.edw.data.squeeze.models.CompactionCriteria; +import com.expedia.dsp.data.squeeze.impl.CompactionManagerImpl; +import com.expedia.dsp.data.squeeze.impl.CompactionManagerInPlaceImpl; +import com.expedia.dsp.data.squeeze.models.CompactionCriteria; import org.apache.commons.configuration.PropertiesConfiguration; import org.apache.commons.lang3.StringUtils; import org.apache.hadoop.conf.Configuration; diff --git a/dataSqueeze-manager/src/main/java/com/expedia/edw/data/squeeze/SchemaSelector.java b/src/main/java/com/expedia/dsp/data/squeeze/SchemaSelector.java similarity index 65% rename from dataSqueeze-manager/src/main/java/com/expedia/edw/data/squeeze/SchemaSelector.java rename to src/main/java/com/expedia/dsp/data/squeeze/SchemaSelector.java index b3ae25f..1d9b7e0 100644 --- a/dataSqueeze-manager/src/main/java/com/expedia/edw/data/squeeze/SchemaSelector.java +++ b/src/main/java/com/expedia/dsp/data/squeeze/SchemaSelector.java @@ -1,4 +1,4 @@ -package com.expedia.edw.data.squeeze; +package com.expedia.dsp.data.squeeze; public interface SchemaSelector { public String getSchemaJSON(); diff --git a/dataSqueeze-manager/src/main/java/com/expedia/edw/data/squeeze/Utility.java b/src/main/java/com/expedia/dsp/data/squeeze/Utility.java similarity index 96% rename from dataSqueeze-manager/src/main/java/com/expedia/edw/data/squeeze/Utility.java rename to src/main/java/com/expedia/dsp/data/squeeze/Utility.java index ce0d8be..1953c5c 100644 --- a/dataSqueeze-manager/src/main/java/com/expedia/edw/data/squeeze/Utility.java +++ b/src/main/java/com/expedia/dsp/data/squeeze/Utility.java @@ -1,6 +1,6 @@ -package com.expedia.edw.data.squeeze; +package com.expedia.dsp.data.squeeze; -import com.expedia.edw.data.squeeze.models.CompactionResponse; +import com.expedia.dsp.data.squeeze.models.CompactionResponse; import lombok.extern.slf4j.Slf4j; diff --git a/dataSqueeze-manager/src/main/java/com/expedia/edw/data/squeeze/impl/BaseCompactionManagerImpl.java b/src/main/java/com/expedia/dsp/data/squeeze/impl/BaseCompactionManagerImpl.java similarity index 85% rename from dataSqueeze-manager/src/main/java/com/expedia/edw/data/squeeze/impl/BaseCompactionManagerImpl.java rename to src/main/java/com/expedia/dsp/data/squeeze/impl/BaseCompactionManagerImpl.java index b96b637..c335c24 100644 --- a/dataSqueeze-manager/src/main/java/com/expedia/edw/data/squeeze/impl/BaseCompactionManagerImpl.java +++ b/src/main/java/com/expedia/dsp/data/squeeze/impl/BaseCompactionManagerImpl.java @@ -1,7 +1,8 @@ -package com.expedia.edw.data.squeeze.impl; +package com.expedia.dsp.data.squeeze.impl; + +import com.expedia.dsp.data.squeeze.CompactionManager; +import com.expedia.dsp.data.squeeze.models.CompactionCriteria; -import com.expedia.edw.data.squeeze.CompactionManager; -import com.expedia.edw.data.squeeze.models.CompactionCriteria; import org.apache.commons.lang3.Validate; import org.apache.hadoop.conf.Configuration; diff --git a/dataSqueeze-manager/src/main/java/com/expedia/edw/data/squeeze/impl/CombineFileWritable.java b/src/main/java/com/expedia/dsp/data/squeeze/impl/CombineFileWritable.java similarity index 97% rename from dataSqueeze-manager/src/main/java/com/expedia/edw/data/squeeze/impl/CombineFileWritable.java rename to src/main/java/com/expedia/dsp/data/squeeze/impl/CombineFileWritable.java index 07c852f..84f8b1e 100644 --- a/dataSqueeze-manager/src/main/java/com/expedia/edw/data/squeeze/impl/CombineFileWritable.java +++ b/src/main/java/com/expedia/dsp/data/squeeze/impl/CombineFileWritable.java @@ -1,4 +1,4 @@ -package com.expedia.edw.data.squeeze.impl; +package com.expedia.dsp.data.squeeze.impl; import lombok.Getter; import org.apache.hadoop.io.Text; diff --git a/dataSqueeze-manager/src/main/java/com/expedia/edw/data/squeeze/impl/CompactionManagerImpl.java b/src/main/java/com/expedia/dsp/data/squeeze/impl/CompactionManagerImpl.java similarity index 90% rename from dataSqueeze-manager/src/main/java/com/expedia/edw/data/squeeze/impl/CompactionManagerImpl.java rename to src/main/java/com/expedia/dsp/data/squeeze/impl/CompactionManagerImpl.java index 2cc8865..a0f442e 100644 --- a/dataSqueeze-manager/src/main/java/com/expedia/edw/data/squeeze/impl/CompactionManagerImpl.java +++ b/src/main/java/com/expedia/dsp/data/squeeze/impl/CompactionManagerImpl.java @@ -1,23 +1,23 @@ -package com.expedia.edw.data.squeeze.impl; +package com.expedia.dsp.data.squeeze.impl; -import com.expedia.edw.data.squeeze.CompactionManager; -import com.expedia.edw.data.squeeze.CompactionManagerFactory; -import com.expedia.edw.data.squeeze.SchemaSelector; -import com.expedia.edw.data.squeeze.impl.orc.OrcCombineFileInputFormat; -import com.expedia.edw.data.squeeze.impl.text.TextCombineFileInputFormat; -import com.expedia.edw.data.squeeze.mappers.AvroCompactionMapper; -import com.expedia.edw.data.squeeze.mappers.BytesWritableCompactionMapper; -import com.expedia.edw.data.squeeze.mappers.OrcCompactionMapper; -import com.expedia.edw.data.squeeze.mappers.SeqCompactionMapper; -import com.expedia.edw.data.squeeze.mappers.TextCompactionMapper; -import com.expedia.edw.data.squeeze.models.CompactionCriteria; -import com.expedia.edw.data.squeeze.models.CompactionResponse; -import com.expedia.edw.data.squeeze.models.FilePaths; -import com.expedia.edw.data.squeeze.models.FileType; -import com.expedia.edw.data.squeeze.reducers.AvroCompactionReducer; -import com.expedia.edw.data.squeeze.reducers.BytesWritableCompactionReducer; -import com.expedia.edw.data.squeeze.reducers.OrcCompactionReducer; -import com.expedia.edw.data.squeeze.reducers.TextCompactionReducer; +import com.expedia.dsp.data.squeeze.CompactionManager; +import com.expedia.dsp.data.squeeze.CompactionManagerFactory; +import com.expedia.dsp.data.squeeze.impl.orc.OrcCombineFileInputFormat; +import com.expedia.dsp.data.squeeze.mappers.AvroCompactionMapper; +import com.expedia.dsp.data.squeeze.mappers.OrcCompactionMapper; +import com.expedia.dsp.data.squeeze.mappers.SeqCompactionMapper; +import com.expedia.dsp.data.squeeze.mappers.TextCompactionMapper; +import com.expedia.dsp.data.squeeze.models.CompactionCriteria; +import com.expedia.dsp.data.squeeze.models.FilePaths; +import com.expedia.dsp.data.squeeze.models.FileType; +import com.expedia.dsp.data.squeeze.reducers.OrcCompactionReducer; +import com.expedia.dsp.data.squeeze.SchemaSelector; +import com.expedia.dsp.data.squeeze.impl.text.TextCombineFileInputFormat; +import com.expedia.dsp.data.squeeze.mappers.BytesWritableCompactionMapper; +import com.expedia.dsp.data.squeeze.models.CompactionResponse; +import com.expedia.dsp.data.squeeze.reducers.AvroCompactionReducer; +import com.expedia.dsp.data.squeeze.reducers.BytesWritableCompactionReducer; +import com.expedia.dsp.data.squeeze.reducers.TextCompactionReducer; import lombok.extern.slf4j.Slf4j; @@ -56,8 +56,9 @@ import java.util.List; import java.util.UUID; -import static com.expedia.edw.data.squeeze.models.FileType.AVRO; -import static com.expedia.edw.data.squeeze.models.FileType.ORC; +import static com.expedia.dsp.data.squeeze.models.FileType.AVRO; +import static com.expedia.dsp.data.squeeze.models.FileType.ORC; + /** * Implementation for {@link CompactionManager}. diff --git a/dataSqueeze-manager/src/main/java/com/expedia/edw/data/squeeze/impl/CompactionManagerInPlaceImpl.java b/src/main/java/com/expedia/dsp/data/squeeze/impl/CompactionManagerInPlaceImpl.java similarity index 92% rename from dataSqueeze-manager/src/main/java/com/expedia/edw/data/squeeze/impl/CompactionManagerInPlaceImpl.java rename to src/main/java/com/expedia/dsp/data/squeeze/impl/CompactionManagerInPlaceImpl.java index e1fa8e9..c5ce0aa 100644 --- a/dataSqueeze-manager/src/main/java/com/expedia/edw/data/squeeze/impl/CompactionManagerInPlaceImpl.java +++ b/src/main/java/com/expedia/dsp/data/squeeze/impl/CompactionManagerInPlaceImpl.java @@ -1,8 +1,8 @@ -package com.expedia.edw.data.squeeze.impl; +package com.expedia.dsp.data.squeeze.impl; -import com.expedia.edw.data.squeeze.CompactionManager; -import com.expedia.edw.data.squeeze.models.CompactionCriteria; -import com.expedia.edw.data.squeeze.models.CompactionResponse; +import com.expedia.dsp.data.squeeze.CompactionManager; +import com.expedia.dsp.data.squeeze.models.CompactionCriteria; +import com.expedia.dsp.data.squeeze.models.CompactionResponse; import lombok.extern.slf4j.Slf4j; import org.apache.commons.lang3.Validate; import org.apache.hadoop.conf.Configuration; diff --git a/dataSqueeze-manager/src/main/java/com/expedia/edw/data/squeeze/impl/DataSkew.java b/src/main/java/com/expedia/dsp/data/squeeze/impl/DataSkew.java similarity index 96% rename from dataSqueeze-manager/src/main/java/com/expedia/edw/data/squeeze/impl/DataSkew.java rename to src/main/java/com/expedia/dsp/data/squeeze/impl/DataSkew.java index 00e2132..af6f549 100644 --- a/dataSqueeze-manager/src/main/java/com/expedia/edw/data/squeeze/impl/DataSkew.java +++ b/src/main/java/com/expedia/dsp/data/squeeze/impl/DataSkew.java @@ -1,4 +1,4 @@ -package com.expedia.edw.data.squeeze.impl; +package com.expedia.dsp.data.squeeze.impl; import lombok.extern.slf4j.Slf4j; import org.apache.hadoop.io.Text; diff --git a/dataSqueeze-manager/src/main/java/com/expedia/edw/data/squeeze/impl/FileManager.java b/src/main/java/com/expedia/dsp/data/squeeze/impl/FileManager.java similarity index 96% rename from dataSqueeze-manager/src/main/java/com/expedia/edw/data/squeeze/impl/FileManager.java rename to src/main/java/com/expedia/dsp/data/squeeze/impl/FileManager.java index 1809ec9..209e6b0 100644 --- a/dataSqueeze-manager/src/main/java/com/expedia/edw/data/squeeze/impl/FileManager.java +++ b/src/main/java/com/expedia/dsp/data/squeeze/impl/FileManager.java @@ -1,8 +1,9 @@ -package com.expedia.edw.data.squeeze.impl; +package com.expedia.dsp.data.squeeze.impl; + +import com.expedia.dsp.data.squeeze.CompactionManagerFactory; +import com.expedia.dsp.data.squeeze.models.FilePaths; +import com.expedia.dsp.data.squeeze.models.FileType; -import com.expedia.edw.data.squeeze.CompactionManagerFactory; -import com.expedia.edw.data.squeeze.models.FilePaths; -import com.expedia.edw.data.squeeze.models.FileType; import lombok.extern.slf4j.Slf4j; import net.sf.jmimemagic.Magic; import net.sf.jmimemagic.MagicMatchNotFoundException; diff --git a/dataSqueeze-manager/src/main/java/com/expedia/edw/data/squeeze/impl/JobRunner.java b/src/main/java/com/expedia/dsp/data/squeeze/impl/JobRunner.java similarity index 93% rename from dataSqueeze-manager/src/main/java/com/expedia/edw/data/squeeze/impl/JobRunner.java rename to src/main/java/com/expedia/dsp/data/squeeze/impl/JobRunner.java index 031c499..8825b95 100644 --- a/dataSqueeze-manager/src/main/java/com/expedia/edw/data/squeeze/impl/JobRunner.java +++ b/src/main/java/com/expedia/dsp/data/squeeze/impl/JobRunner.java @@ -1,4 +1,4 @@ -package com.expedia.edw.data.squeeze.impl; +package com.expedia.dsp.data.squeeze.impl; import lombok.extern.slf4j.Slf4j; import org.apache.hadoop.conf.Configured; diff --git a/dataSqueeze-manager/src/main/java/com/expedia/edw/data/squeeze/impl/SchemaSelectorImpl.java b/src/main/java/com/expedia/dsp/data/squeeze/impl/SchemaSelectorImpl.java similarity index 87% rename from dataSqueeze-manager/src/main/java/com/expedia/edw/data/squeeze/impl/SchemaSelectorImpl.java rename to src/main/java/com/expedia/dsp/data/squeeze/impl/SchemaSelectorImpl.java index 7ae6281..0386cc9 100644 --- a/dataSqueeze-manager/src/main/java/com/expedia/edw/data/squeeze/impl/SchemaSelectorImpl.java +++ b/src/main/java/com/expedia/dsp/data/squeeze/impl/SchemaSelectorImpl.java @@ -1,4 +1,4 @@ -package com.expedia.edw.data.squeeze.impl; +package com.expedia.dsp.data.squeeze.impl; import java.io.IOException; @@ -7,9 +7,9 @@ import org.apache.hadoop.fs.FileSystem; import org.apache.hadoop.fs.Path; -import com.expedia.edw.data.squeeze.SchemaSelector; -import com.expedia.edw.data.squeeze.models.CompactionCriteria; -import com.expedia.edw.data.squeeze.models.FileType; +import com.expedia.dsp.data.squeeze.SchemaSelector; +import com.expedia.dsp.data.squeeze.models.CompactionCriteria; +import com.expedia.dsp.data.squeeze.models.FileType; import lombok.extern.slf4j.Slf4j; diff --git a/dataSqueeze-manager/src/main/java/com/expedia/edw/data/squeeze/impl/orc/OrcCombineFileInputFormat.java b/src/main/java/com/expedia/dsp/data/squeeze/impl/orc/OrcCombineFileInputFormat.java similarity index 91% rename from dataSqueeze-manager/src/main/java/com/expedia/edw/data/squeeze/impl/orc/OrcCombineFileInputFormat.java rename to src/main/java/com/expedia/dsp/data/squeeze/impl/orc/OrcCombineFileInputFormat.java index b671728..8f763d2 100644 --- a/dataSqueeze-manager/src/main/java/com/expedia/edw/data/squeeze/impl/orc/OrcCombineFileInputFormat.java +++ b/src/main/java/com/expedia/dsp/data/squeeze/impl/orc/OrcCombineFileInputFormat.java @@ -1,6 +1,6 @@ -package com.expedia.edw.data.squeeze.impl.orc; +package com.expedia.dsp.data.squeeze.impl.orc; -import com.expedia.edw.data.squeeze.impl.CombineFileWritable; +import com.expedia.dsp.data.squeeze.impl.CombineFileWritable; import org.apache.hadoop.fs.Path; import org.apache.hadoop.mapreduce.InputSplit; import org.apache.hadoop.mapreduce.JobContext; diff --git a/dataSqueeze-manager/src/main/java/com/expedia/edw/data/squeeze/impl/orc/OrcRecordReader.java b/src/main/java/com/expedia/dsp/data/squeeze/impl/orc/OrcRecordReader.java similarity index 97% rename from dataSqueeze-manager/src/main/java/com/expedia/edw/data/squeeze/impl/orc/OrcRecordReader.java rename to src/main/java/com/expedia/dsp/data/squeeze/impl/orc/OrcRecordReader.java index 0c0769e..7f8cca4 100644 --- a/dataSqueeze-manager/src/main/java/com/expedia/edw/data/squeeze/impl/orc/OrcRecordReader.java +++ b/src/main/java/com/expedia/dsp/data/squeeze/impl/orc/OrcRecordReader.java @@ -1,6 +1,7 @@ -package com.expedia.edw.data.squeeze.impl.orc; +package com.expedia.dsp.data.squeeze.impl.orc; + +import com.expedia.dsp.data.squeeze.impl.CombineFileWritable; -import com.expedia.edw.data.squeeze.impl.CombineFileWritable; import lombok.extern.slf4j.Slf4j; import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.fs.Path; diff --git a/dataSqueeze-manager/src/main/java/com/expedia/edw/data/squeeze/impl/seq/SeqCombineFileInputFormat.java b/src/main/java/com/expedia/dsp/data/squeeze/impl/seq/SeqCombineFileInputFormat.java similarity index 91% rename from dataSqueeze-manager/src/main/java/com/expedia/edw/data/squeeze/impl/seq/SeqCombineFileInputFormat.java rename to src/main/java/com/expedia/dsp/data/squeeze/impl/seq/SeqCombineFileInputFormat.java index efd4269..f135b5f 100644 --- a/dataSqueeze-manager/src/main/java/com/expedia/edw/data/squeeze/impl/seq/SeqCombineFileInputFormat.java +++ b/src/main/java/com/expedia/dsp/data/squeeze/impl/seq/SeqCombineFileInputFormat.java @@ -1,6 +1,7 @@ -package com.expedia.edw.data.squeeze.impl.seq; +package com.expedia.dsp.data.squeeze.impl.seq; + +import com.expedia.dsp.data.squeeze.impl.CombineFileWritable; -import com.expedia.edw.data.squeeze.impl.CombineFileWritable; import org.apache.hadoop.fs.Path; import org.apache.hadoop.io.Text; import org.apache.hadoop.mapreduce.InputSplit; diff --git a/dataSqueeze-manager/src/main/java/com/expedia/edw/data/squeeze/impl/seq/SeqRecordReader.java b/src/main/java/com/expedia/dsp/data/squeeze/impl/seq/SeqRecordReader.java similarity index 97% rename from dataSqueeze-manager/src/main/java/com/expedia/edw/data/squeeze/impl/seq/SeqRecordReader.java rename to src/main/java/com/expedia/dsp/data/squeeze/impl/seq/SeqRecordReader.java index 6b33eb8..cfb613b 100644 --- a/dataSqueeze-manager/src/main/java/com/expedia/edw/data/squeeze/impl/seq/SeqRecordReader.java +++ b/src/main/java/com/expedia/dsp/data/squeeze/impl/seq/SeqRecordReader.java @@ -1,6 +1,6 @@ -package com.expedia.edw.data.squeeze.impl.seq; +package com.expedia.dsp.data.squeeze.impl.seq; -import com.expedia.edw.data.squeeze.impl.CombineFileWritable; +import com.expedia.dsp.data.squeeze.impl.CombineFileWritable; import lombok.extern.slf4j.Slf4j; import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.fs.Path; diff --git a/dataSqueeze-manager/src/main/java/com/expedia/edw/data/squeeze/impl/text/TextCombineFileInputFormat.java b/src/main/java/com/expedia/dsp/data/squeeze/impl/text/TextCombineFileInputFormat.java similarity index 91% rename from dataSqueeze-manager/src/main/java/com/expedia/edw/data/squeeze/impl/text/TextCombineFileInputFormat.java rename to src/main/java/com/expedia/dsp/data/squeeze/impl/text/TextCombineFileInputFormat.java index c3c52ef..8dc3401 100644 --- a/dataSqueeze-manager/src/main/java/com/expedia/edw/data/squeeze/impl/text/TextCombineFileInputFormat.java +++ b/src/main/java/com/expedia/dsp/data/squeeze/impl/text/TextCombineFileInputFormat.java @@ -1,6 +1,7 @@ -package com.expedia.edw.data.squeeze.impl.text; +package com.expedia.dsp.data.squeeze.impl.text; + +import com.expedia.dsp.data.squeeze.impl.CombineFileWritable; -import com.expedia.edw.data.squeeze.impl.CombineFileWritable; import org.apache.hadoop.fs.Path; import org.apache.hadoop.io.Text; import org.apache.hadoop.mapreduce.InputSplit; diff --git a/dataSqueeze-manager/src/main/java/com/expedia/edw/data/squeeze/impl/text/TextRecordReader.java b/src/main/java/com/expedia/dsp/data/squeeze/impl/text/TextRecordReader.java similarity index 97% rename from dataSqueeze-manager/src/main/java/com/expedia/edw/data/squeeze/impl/text/TextRecordReader.java rename to src/main/java/com/expedia/dsp/data/squeeze/impl/text/TextRecordReader.java index 02b976e..ac2c805 100644 --- a/dataSqueeze-manager/src/main/java/com/expedia/edw/data/squeeze/impl/text/TextRecordReader.java +++ b/src/main/java/com/expedia/dsp/data/squeeze/impl/text/TextRecordReader.java @@ -1,6 +1,7 @@ -package com.expedia.edw.data.squeeze.impl.text; +package com.expedia.dsp.data.squeeze.impl.text; + +import com.expedia.dsp.data.squeeze.impl.CombineFileWritable; -import com.expedia.edw.data.squeeze.impl.CombineFileWritable; import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.fs.FSDataInputStream; import org.apache.hadoop.fs.FileSystem; diff --git a/dataSqueeze-manager/src/main/java/com/expedia/edw/data/squeeze/mappers/AvroCompactionMapper.java b/src/main/java/com/expedia/dsp/data/squeeze/mappers/AvroCompactionMapper.java similarity index 98% rename from dataSqueeze-manager/src/main/java/com/expedia/edw/data/squeeze/mappers/AvroCompactionMapper.java rename to src/main/java/com/expedia/dsp/data/squeeze/mappers/AvroCompactionMapper.java index 041083d..4e1a142 100644 --- a/dataSqueeze-manager/src/main/java/com/expedia/edw/data/squeeze/mappers/AvroCompactionMapper.java +++ b/src/main/java/com/expedia/dsp/data/squeeze/mappers/AvroCompactionMapper.java @@ -1,4 +1,4 @@ -package com.expedia.edw.data.squeeze.mappers; +package com.expedia.dsp.data.squeeze.mappers; import java.io.IOException; import java.util.HashMap; diff --git a/dataSqueeze-manager/src/main/java/com/expedia/edw/data/squeeze/mappers/BaseMapper.java b/src/main/java/com/expedia/dsp/data/squeeze/mappers/BaseMapper.java similarity index 97% rename from dataSqueeze-manager/src/main/java/com/expedia/edw/data/squeeze/mappers/BaseMapper.java rename to src/main/java/com/expedia/dsp/data/squeeze/mappers/BaseMapper.java index 7496ef9..6bf3146 100644 --- a/dataSqueeze-manager/src/main/java/com/expedia/edw/data/squeeze/mappers/BaseMapper.java +++ b/src/main/java/com/expedia/dsp/data/squeeze/mappers/BaseMapper.java @@ -1,6 +1,7 @@ -package com.expedia.edw.data.squeeze.mappers; +package com.expedia.dsp.data.squeeze.mappers; + +import com.expedia.dsp.data.squeeze.impl.DataSkew; -import com.expedia.edw.data.squeeze.impl.DataSkew; import lombok.extern.slf4j.Slf4j; import org.apache.commons.lang.StringUtils; import org.apache.hadoop.conf.Configuration; diff --git a/dataSqueeze-manager/src/main/java/com/expedia/edw/data/squeeze/mappers/BytesWritableCompactionMapper.java b/src/main/java/com/expedia/dsp/data/squeeze/mappers/BytesWritableCompactionMapper.java similarity index 98% rename from dataSqueeze-manager/src/main/java/com/expedia/edw/data/squeeze/mappers/BytesWritableCompactionMapper.java rename to src/main/java/com/expedia/dsp/data/squeeze/mappers/BytesWritableCompactionMapper.java index 500a16e..d69c5f3 100644 --- a/dataSqueeze-manager/src/main/java/com/expedia/edw/data/squeeze/mappers/BytesWritableCompactionMapper.java +++ b/src/main/java/com/expedia/dsp/data/squeeze/mappers/BytesWritableCompactionMapper.java @@ -1,4 +1,4 @@ -package com.expedia.edw.data.squeeze.mappers; +package com.expedia.dsp.data.squeeze.mappers; import lombok.extern.slf4j.Slf4j; import org.apache.hadoop.conf.Configuration; diff --git a/dataSqueeze-manager/src/main/java/com/expedia/edw/data/squeeze/mappers/OrcCompactionMapper.java b/src/main/java/com/expedia/dsp/data/squeeze/mappers/OrcCompactionMapper.java similarity index 91% rename from dataSqueeze-manager/src/main/java/com/expedia/edw/data/squeeze/mappers/OrcCompactionMapper.java rename to src/main/java/com/expedia/dsp/data/squeeze/mappers/OrcCompactionMapper.java index 68e43f9..895709d 100644 --- a/dataSqueeze-manager/src/main/java/com/expedia/edw/data/squeeze/mappers/OrcCompactionMapper.java +++ b/src/main/java/com/expedia/dsp/data/squeeze/mappers/OrcCompactionMapper.java @@ -1,6 +1,7 @@ -package com.expedia.edw.data.squeeze.mappers; +package com.expedia.dsp.data.squeeze.mappers; + +import com.expedia.dsp.data.squeeze.impl.CombineFileWritable; -import com.expedia.edw.data.squeeze.impl.CombineFileWritable; import lombok.extern.slf4j.Slf4j; import org.apache.hadoop.io.Text; import org.apache.hadoop.mapreduce.Mapper; diff --git a/dataSqueeze-manager/src/main/java/com/expedia/edw/data/squeeze/mappers/SeqCompactionMapper.java b/src/main/java/com/expedia/dsp/data/squeeze/mappers/SeqCompactionMapper.java similarity index 95% rename from dataSqueeze-manager/src/main/java/com/expedia/edw/data/squeeze/mappers/SeqCompactionMapper.java rename to src/main/java/com/expedia/dsp/data/squeeze/mappers/SeqCompactionMapper.java index 36326ce..f90afd0 100644 --- a/dataSqueeze-manager/src/main/java/com/expedia/edw/data/squeeze/mappers/SeqCompactionMapper.java +++ b/src/main/java/com/expedia/dsp/data/squeeze/mappers/SeqCompactionMapper.java @@ -1,4 +1,4 @@ -package com.expedia.edw.data.squeeze.mappers; +package com.expedia.dsp.data.squeeze.mappers; import lombok.extern.slf4j.Slf4j; import org.apache.hadoop.io.Text; diff --git a/dataSqueeze-manager/src/main/java/com/expedia/edw/data/squeeze/mappers/TextCompactionMapper.java b/src/main/java/com/expedia/dsp/data/squeeze/mappers/TextCompactionMapper.java similarity index 89% rename from dataSqueeze-manager/src/main/java/com/expedia/edw/data/squeeze/mappers/TextCompactionMapper.java rename to src/main/java/com/expedia/dsp/data/squeeze/mappers/TextCompactionMapper.java index 2f4cb1b..7800659 100644 --- a/dataSqueeze-manager/src/main/java/com/expedia/edw/data/squeeze/mappers/TextCompactionMapper.java +++ b/src/main/java/com/expedia/dsp/data/squeeze/mappers/TextCompactionMapper.java @@ -1,6 +1,7 @@ -package com.expedia.edw.data.squeeze.mappers; +package com.expedia.dsp.data.squeeze.mappers; + +import com.expedia.dsp.data.squeeze.impl.CombineFileWritable; -import com.expedia.edw.data.squeeze.impl.CombineFileWritable; import lombok.extern.slf4j.Slf4j; import org.apache.hadoop.io.Text; import org.apache.hadoop.mapreduce.Mapper; diff --git a/dataSqueeze-manager/src/main/java/com/expedia/edw/data/squeeze/models/CompactionCriteria.java b/src/main/java/com/expedia/dsp/data/squeeze/models/CompactionCriteria.java similarity index 99% rename from dataSqueeze-manager/src/main/java/com/expedia/edw/data/squeeze/models/CompactionCriteria.java rename to src/main/java/com/expedia/dsp/data/squeeze/models/CompactionCriteria.java index 22ee1e0..b1904f4 100644 --- a/dataSqueeze-manager/src/main/java/com/expedia/edw/data/squeeze/models/CompactionCriteria.java +++ b/src/main/java/com/expedia/dsp/data/squeeze/models/CompactionCriteria.java @@ -1,4 +1,4 @@ -package com.expedia.edw.data.squeeze.models; +package com.expedia.dsp.data.squeeze.models; import lombok.Getter; import lombok.extern.slf4j.Slf4j; diff --git a/dataSqueeze-manager/src/main/java/com/expedia/edw/data/squeeze/models/CompactionResponse.java b/src/main/java/com/expedia/dsp/data/squeeze/models/CompactionResponse.java similarity index 95% rename from dataSqueeze-manager/src/main/java/com/expedia/edw/data/squeeze/models/CompactionResponse.java rename to src/main/java/com/expedia/dsp/data/squeeze/models/CompactionResponse.java index 6f6d1c2..51c62f7 100644 --- a/dataSqueeze-manager/src/main/java/com/expedia/edw/data/squeeze/models/CompactionResponse.java +++ b/src/main/java/com/expedia/dsp/data/squeeze/models/CompactionResponse.java @@ -1,4 +1,4 @@ -package com.expedia.edw.data.squeeze.models; +package com.expedia.dsp.data.squeeze.models; import lombok.Getter; import lombok.Setter; diff --git a/dataSqueeze-manager/src/main/java/com/expedia/edw/data/squeeze/models/FilePaths.java b/src/main/java/com/expedia/dsp/data/squeeze/models/FilePaths.java similarity index 95% rename from dataSqueeze-manager/src/main/java/com/expedia/edw/data/squeeze/models/FilePaths.java rename to src/main/java/com/expedia/dsp/data/squeeze/models/FilePaths.java index 98ad172..5c66d7f 100644 --- a/dataSqueeze-manager/src/main/java/com/expedia/edw/data/squeeze/models/FilePaths.java +++ b/src/main/java/com/expedia/dsp/data/squeeze/models/FilePaths.java @@ -1,4 +1,4 @@ -package com.expedia.edw.data.squeeze.models; +package com.expedia.dsp.data.squeeze.models; import lombok.Getter; import lombok.extern.slf4j.Slf4j; diff --git a/dataSqueeze-manager/src/main/java/com/expedia/edw/data/squeeze/models/FileType.java b/src/main/java/com/expedia/dsp/data/squeeze/models/FileType.java similarity index 87% rename from dataSqueeze-manager/src/main/java/com/expedia/edw/data/squeeze/models/FileType.java rename to src/main/java/com/expedia/dsp/data/squeeze/models/FileType.java index 9dc590a..bf91b19 100644 --- a/dataSqueeze-manager/src/main/java/com/expedia/edw/data/squeeze/models/FileType.java +++ b/src/main/java/com/expedia/dsp/data/squeeze/models/FileType.java @@ -1,4 +1,4 @@ -package com.expedia.edw.data.squeeze.models; +package com.expedia.dsp.data.squeeze.models; import lombok.Getter; diff --git a/dataSqueeze-manager/src/main/java/com/expedia/edw/data/squeeze/reducers/AvroCompactionReducer.java b/src/main/java/com/expedia/dsp/data/squeeze/reducers/AvroCompactionReducer.java similarity index 96% rename from dataSqueeze-manager/src/main/java/com/expedia/edw/data/squeeze/reducers/AvroCompactionReducer.java rename to src/main/java/com/expedia/dsp/data/squeeze/reducers/AvroCompactionReducer.java index 6be1fc6..84b9a7e 100644 --- a/dataSqueeze-manager/src/main/java/com/expedia/edw/data/squeeze/reducers/AvroCompactionReducer.java +++ b/src/main/java/com/expedia/dsp/data/squeeze/reducers/AvroCompactionReducer.java @@ -1,4 +1,4 @@ -package com.expedia.edw.data.squeeze.reducers; +package com.expedia.dsp.data.squeeze.reducers; import java.io.IOException; import java.net.URI; diff --git a/dataSqueeze-manager/src/main/java/com/expedia/edw/data/squeeze/reducers/BaseReducer.java b/src/main/java/com/expedia/dsp/data/squeeze/reducers/BaseReducer.java similarity index 97% rename from dataSqueeze-manager/src/main/java/com/expedia/edw/data/squeeze/reducers/BaseReducer.java rename to src/main/java/com/expedia/dsp/data/squeeze/reducers/BaseReducer.java index 4d20dba..2ae6381 100644 --- a/dataSqueeze-manager/src/main/java/com/expedia/edw/data/squeeze/reducers/BaseReducer.java +++ b/src/main/java/com/expedia/dsp/data/squeeze/reducers/BaseReducer.java @@ -1,4 +1,4 @@ -package com.expedia.edw.data.squeeze.reducers; +package com.expedia.dsp.data.squeeze.reducers; import lombok.extern.slf4j.Slf4j; import org.apache.hadoop.conf.Configuration; diff --git a/dataSqueeze-manager/src/main/java/com/expedia/edw/data/squeeze/reducers/BytesWritableCompactionReducer.java b/src/main/java/com/expedia/dsp/data/squeeze/reducers/BytesWritableCompactionReducer.java similarity index 97% rename from dataSqueeze-manager/src/main/java/com/expedia/edw/data/squeeze/reducers/BytesWritableCompactionReducer.java rename to src/main/java/com/expedia/dsp/data/squeeze/reducers/BytesWritableCompactionReducer.java index 31054d5..20505d9 100644 --- a/dataSqueeze-manager/src/main/java/com/expedia/edw/data/squeeze/reducers/BytesWritableCompactionReducer.java +++ b/src/main/java/com/expedia/dsp/data/squeeze/reducers/BytesWritableCompactionReducer.java @@ -1,4 +1,4 @@ -package com.expedia.edw.data.squeeze.reducers; +package com.expedia.dsp.data.squeeze.reducers; import lombok.extern.slf4j.Slf4j; import org.apache.hadoop.conf.Configuration; diff --git a/dataSqueeze-manager/src/main/java/com/expedia/edw/data/squeeze/reducers/OrcCompactionReducer.java b/src/main/java/com/expedia/dsp/data/squeeze/reducers/OrcCompactionReducer.java similarity index 94% rename from dataSqueeze-manager/src/main/java/com/expedia/edw/data/squeeze/reducers/OrcCompactionReducer.java rename to src/main/java/com/expedia/dsp/data/squeeze/reducers/OrcCompactionReducer.java index 5121efb..74bfa06 100644 --- a/dataSqueeze-manager/src/main/java/com/expedia/edw/data/squeeze/reducers/OrcCompactionReducer.java +++ b/src/main/java/com/expedia/dsp/data/squeeze/reducers/OrcCompactionReducer.java @@ -1,4 +1,4 @@ -package com.expedia.edw.data.squeeze.reducers; +package com.expedia.dsp.data.squeeze.reducers; import lombok.extern.slf4j.Slf4j; import org.apache.hadoop.io.NullWritable; diff --git a/dataSqueeze-manager/src/main/java/com/expedia/edw/data/squeeze/reducers/TextCompactionReducer.java b/src/main/java/com/expedia/dsp/data/squeeze/reducers/TextCompactionReducer.java similarity index 94% rename from dataSqueeze-manager/src/main/java/com/expedia/edw/data/squeeze/reducers/TextCompactionReducer.java rename to src/main/java/com/expedia/dsp/data/squeeze/reducers/TextCompactionReducer.java index bf6e985..a134b7d 100644 --- a/dataSqueeze-manager/src/main/java/com/expedia/edw/data/squeeze/reducers/TextCompactionReducer.java +++ b/src/main/java/com/expedia/dsp/data/squeeze/reducers/TextCompactionReducer.java @@ -1,4 +1,4 @@ -package com.expedia.edw.data.squeeze.reducers; +package com.expedia.dsp.data.squeeze.reducers; import lombok.extern.slf4j.Slf4j; import org.apache.hadoop.io.NullWritable; diff --git a/dataSqueeze-manager/src/main/resources/compaction.properties b/src/main/resources/compaction.properties similarity index 100% rename from dataSqueeze-manager/src/main/resources/compaction.properties rename to src/main/resources/compaction.properties diff --git a/dataSqueeze-manager/src/main/resources/log4j.properties b/src/main/resources/log4j.properties similarity index 100% rename from dataSqueeze-manager/src/main/resources/log4j.properties rename to src/main/resources/log4j.properties diff --git a/dataSqueeze-manager/src/test/java/com/expedia/edw/data/squeeze/CompactionManagerFactoryTest.java b/src/test/java/com/expedia/dsp/data/squeeze/CompactionManagerFactoryTest.java similarity index 94% rename from dataSqueeze-manager/src/test/java/com/expedia/edw/data/squeeze/CompactionManagerFactoryTest.java rename to src/test/java/com/expedia/dsp/data/squeeze/CompactionManagerFactoryTest.java index 54df008..5d02312 100644 --- a/dataSqueeze-manager/src/test/java/com/expedia/edw/data/squeeze/CompactionManagerFactoryTest.java +++ b/src/test/java/com/expedia/dsp/data/squeeze/CompactionManagerFactoryTest.java @@ -1,7 +1,8 @@ -package com.expedia.edw.data.squeeze; +package com.expedia.dsp.data.squeeze; + +import com.expedia.dsp.data.squeeze.impl.CompactionManagerImpl; +import com.expedia.dsp.data.squeeze.impl.CompactionManagerInPlaceImpl; -import com.expedia.edw.data.squeeze.impl.CompactionManagerImpl; -import com.expedia.edw.data.squeeze.impl.CompactionManagerInPlaceImpl; import org.junit.Test; import java.util.HashMap; diff --git a/dataSqueeze-manager/src/test/java/com/expedia/edw/data/squeeze/DataSkewTest.java b/src/test/java/com/expedia/dsp/data/squeeze/DataSkewTest.java similarity index 90% rename from dataSqueeze-manager/src/test/java/com/expedia/edw/data/squeeze/DataSkewTest.java rename to src/test/java/com/expedia/dsp/data/squeeze/DataSkewTest.java index 3b05d09..06742d5 100644 --- a/dataSqueeze-manager/src/test/java/com/expedia/edw/data/squeeze/DataSkewTest.java +++ b/src/test/java/com/expedia/dsp/data/squeeze/DataSkewTest.java @@ -1,6 +1,7 @@ -package com.expedia.edw.data.squeeze; +package com.expedia.dsp.data.squeeze; + +import com.expedia.dsp.data.squeeze.impl.DataSkew; -import com.expedia.edw.data.squeeze.impl.DataSkew; import org.apache.hadoop.io.Text; import org.json.JSONObject; import org.junit.Test; diff --git a/dataSqueeze-manager/src/test/java/com/expedia/edw/data/squeeze/UtilityTest.java b/src/test/java/com/expedia/dsp/data/squeeze/UtilityTest.java similarity index 93% rename from dataSqueeze-manager/src/test/java/com/expedia/edw/data/squeeze/UtilityTest.java rename to src/test/java/com/expedia/dsp/data/squeeze/UtilityTest.java index 59ea72e..67745fb 100644 --- a/dataSqueeze-manager/src/test/java/com/expedia/edw/data/squeeze/UtilityTest.java +++ b/src/test/java/com/expedia/dsp/data/squeeze/UtilityTest.java @@ -1,7 +1,7 @@ -package com.expedia.edw.data.squeeze; +package com.expedia.dsp.data.squeeze; -import com.expedia.edw.data.squeeze.models.CompactionResponse; -import com.expedia.edw.data.squeeze.models.FileType; +import com.expedia.dsp.data.squeeze.models.CompactionResponse; +import com.expedia.dsp.data.squeeze.models.FileType; import org.junit.Before; import org.junit.Test; import org.junit.runner.RunWith; diff --git a/dataSqueeze-manager/src/test/java/com/expedia/edw/data/squeeze/impl/CompactionManagerImplTest.java b/src/test/java/com/expedia/dsp/data/squeeze/impl/CompactionManagerImplTest.java similarity index 98% rename from dataSqueeze-manager/src/test/java/com/expedia/edw/data/squeeze/impl/CompactionManagerImplTest.java rename to src/test/java/com/expedia/dsp/data/squeeze/impl/CompactionManagerImplTest.java index 2b9da76..7b82350 100644 --- a/dataSqueeze-manager/src/test/java/com/expedia/edw/data/squeeze/impl/CompactionManagerImplTest.java +++ b/src/test/java/com/expedia/dsp/data/squeeze/impl/CompactionManagerImplTest.java @@ -1,10 +1,10 @@ -package com.expedia.edw.data.squeeze.impl; +package com.expedia.dsp.data.squeeze.impl; -import com.expedia.edw.data.squeeze.CompactionManagerFactory; -import com.expedia.edw.data.squeeze.models.CompactionCriteria; -import com.expedia.edw.data.squeeze.models.CompactionResponse; -import com.expedia.edw.data.squeeze.models.FilePaths; -import com.expedia.edw.data.squeeze.models.FileType; +import com.expedia.dsp.data.squeeze.CompactionManagerFactory; +import com.expedia.dsp.data.squeeze.models.CompactionCriteria; +import com.expedia.dsp.data.squeeze.models.CompactionResponse; +import com.expedia.dsp.data.squeeze.models.FilePaths; +import com.expedia.dsp.data.squeeze.models.FileType; import net.sf.jmimemagic.Magic; import net.sf.jmimemagic.MagicMatch; diff --git a/dataSqueeze-manager/src/test/java/com/expedia/edw/data/squeeze/impl/CompactionManagerInPlaceImplTest.java b/src/test/java/com/expedia/dsp/data/squeeze/impl/CompactionManagerInPlaceImplTest.java similarity index 97% rename from dataSqueeze-manager/src/test/java/com/expedia/edw/data/squeeze/impl/CompactionManagerInPlaceImplTest.java rename to src/test/java/com/expedia/dsp/data/squeeze/impl/CompactionManagerInPlaceImplTest.java index 943c79b..3014e04 100644 --- a/dataSqueeze-manager/src/test/java/com/expedia/edw/data/squeeze/impl/CompactionManagerInPlaceImplTest.java +++ b/src/test/java/com/expedia/dsp/data/squeeze/impl/CompactionManagerInPlaceImplTest.java @@ -1,10 +1,10 @@ -package com.expedia.edw.data.squeeze.impl; +package com.expedia.dsp.data.squeeze.impl; -import com.expedia.edw.data.squeeze.CompactionManagerFactory; -import com.expedia.edw.data.squeeze.models.CompactionCriteria; -import com.expedia.edw.data.squeeze.models.CompactionResponse; -import com.expedia.edw.data.squeeze.models.FilePaths; -import com.expedia.edw.data.squeeze.models.FileType; +import com.expedia.dsp.data.squeeze.CompactionManagerFactory; +import com.expedia.dsp.data.squeeze.models.CompactionCriteria; +import com.expedia.dsp.data.squeeze.models.CompactionResponse; +import com.expedia.dsp.data.squeeze.models.FilePaths; +import com.expedia.dsp.data.squeeze.models.FileType; import net.sf.jmimemagic.Magic; import net.sf.jmimemagic.MagicMatch; import org.apache.commons.math3.util.Pair; diff --git a/dataSqueeze-manager/src/test/java/com/expedia/edw/data/squeeze/impl/FileManagerTest.java b/src/test/java/com/expedia/dsp/data/squeeze/impl/FileManagerTest.java similarity index 98% rename from dataSqueeze-manager/src/test/java/com/expedia/edw/data/squeeze/impl/FileManagerTest.java rename to src/test/java/com/expedia/dsp/data/squeeze/impl/FileManagerTest.java index 7b543b5..f4c55d5 100644 --- a/dataSqueeze-manager/src/test/java/com/expedia/edw/data/squeeze/impl/FileManagerTest.java +++ b/src/test/java/com/expedia/dsp/data/squeeze/impl/FileManagerTest.java @@ -1,8 +1,8 @@ -package com.expedia.edw.data.squeeze.impl; +package com.expedia.dsp.data.squeeze.impl; -import com.expedia.edw.data.squeeze.CompactionManagerFactory; -import com.expedia.edw.data.squeeze.models.FilePaths; -import com.expedia.edw.data.squeeze.models.FileType; +import com.expedia.dsp.data.squeeze.CompactionManagerFactory; +import com.expedia.dsp.data.squeeze.models.FilePaths; +import com.expedia.dsp.data.squeeze.models.FileType; import com.google.common.collect.ImmutableList; import net.sf.jmimemagic.Magic; import net.sf.jmimemagic.MagicMatch; diff --git a/dataSqueeze-manager/src/test/java/com/expedia/edw/data/squeeze/impl/JobRunnerTest.java b/src/test/java/com/expedia/dsp/data/squeeze/impl/JobRunnerTest.java similarity index 95% rename from dataSqueeze-manager/src/test/java/com/expedia/edw/data/squeeze/impl/JobRunnerTest.java rename to src/test/java/com/expedia/dsp/data/squeeze/impl/JobRunnerTest.java index 968d43d..0168e19 100644 --- a/dataSqueeze-manager/src/test/java/com/expedia/edw/data/squeeze/impl/JobRunnerTest.java +++ b/src/test/java/com/expedia/dsp/data/squeeze/impl/JobRunnerTest.java @@ -1,4 +1,4 @@ -package com.expedia.edw.data.squeeze.impl; +package com.expedia.dsp.data.squeeze.impl; import org.apache.hadoop.mapreduce.Job; import org.junit.Test; diff --git a/dataSqueeze-manager/src/test/java/com/expedia/edw/data/squeeze/impl/SchemaSelectorImplTest.java b/src/test/java/com/expedia/dsp/data/squeeze/impl/SchemaSelectorImplTest.java similarity index 97% rename from dataSqueeze-manager/src/test/java/com/expedia/edw/data/squeeze/impl/SchemaSelectorImplTest.java rename to src/test/java/com/expedia/dsp/data/squeeze/impl/SchemaSelectorImplTest.java index e8dc9ba..2d78189 100644 --- a/dataSqueeze-manager/src/test/java/com/expedia/edw/data/squeeze/impl/SchemaSelectorImplTest.java +++ b/src/test/java/com/expedia/dsp/data/squeeze/impl/SchemaSelectorImplTest.java @@ -1,4 +1,4 @@ -package com.expedia.edw.data.squeeze.impl; +package com.expedia.dsp.data.squeeze.impl; import static org.mockito.Matchers.any; import static org.powermock.api.mockito.PowerMockito.mock; @@ -22,7 +22,7 @@ import org.powermock.core.classloader.annotations.PrepareForTest; import org.powermock.modules.junit4.PowerMockRunner; -import com.expedia.edw.data.squeeze.models.CompactionCriteria; +import com.expedia.dsp.data.squeeze.models.CompactionCriteria; /** * Tests for {@link SchemaSelectorImpl} diff --git a/dataSqueeze-manager/src/test/java/com/expedia/edw/data/squeeze/impl/orc/OrcCombineFileInputFormatTest.java b/src/test/java/com/expedia/dsp/data/squeeze/impl/orc/OrcCombineFileInputFormatTest.java similarity index 97% rename from dataSqueeze-manager/src/test/java/com/expedia/edw/data/squeeze/impl/orc/OrcCombineFileInputFormatTest.java rename to src/test/java/com/expedia/dsp/data/squeeze/impl/orc/OrcCombineFileInputFormatTest.java index 07663f7..91e0eec 100644 --- a/dataSqueeze-manager/src/test/java/com/expedia/edw/data/squeeze/impl/orc/OrcCombineFileInputFormatTest.java +++ b/src/test/java/com/expedia/dsp/data/squeeze/impl/orc/OrcCombineFileInputFormatTest.java @@ -1,4 +1,4 @@ -package com.expedia.edw.data.squeeze.impl.orc; +package com.expedia.dsp.data.squeeze.impl.orc; import org.apache.hadoop.fs.Path; import org.apache.hadoop.mapreduce.JobContext; diff --git a/dataSqueeze-manager/src/test/java/com/expedia/edw/data/squeeze/impl/orc/OrcRecordReaderTest.java b/src/test/java/com/expedia/dsp/data/squeeze/impl/orc/OrcRecordReaderTest.java similarity index 97% rename from dataSqueeze-manager/src/test/java/com/expedia/edw/data/squeeze/impl/orc/OrcRecordReaderTest.java rename to src/test/java/com/expedia/dsp/data/squeeze/impl/orc/OrcRecordReaderTest.java index 53630fc..663f8b0 100644 --- a/dataSqueeze-manager/src/test/java/com/expedia/edw/data/squeeze/impl/orc/OrcRecordReaderTest.java +++ b/src/test/java/com/expedia/dsp/data/squeeze/impl/orc/OrcRecordReaderTest.java @@ -1,6 +1,7 @@ -package com.expedia.edw.data.squeeze.impl.orc; +package com.expedia.dsp.data.squeeze.impl.orc; + +import com.expedia.dsp.data.squeeze.impl.CombineFileWritable; -import com.expedia.edw.data.squeeze.impl.CombineFileWritable; import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.fs.FileSystem; import org.apache.hadoop.fs.Path; diff --git a/dataSqueeze-manager/src/test/java/com/expedia/edw/data/squeeze/impl/seq/SeqCombineFileInputFormatTest.java b/src/test/java/com/expedia/dsp/data/squeeze/impl/seq/SeqCombineFileInputFormatTest.java similarity index 97% rename from dataSqueeze-manager/src/test/java/com/expedia/edw/data/squeeze/impl/seq/SeqCombineFileInputFormatTest.java rename to src/test/java/com/expedia/dsp/data/squeeze/impl/seq/SeqCombineFileInputFormatTest.java index b1aaef7..7a29618 100644 --- a/dataSqueeze-manager/src/test/java/com/expedia/edw/data/squeeze/impl/seq/SeqCombineFileInputFormatTest.java +++ b/src/test/java/com/expedia/dsp/data/squeeze/impl/seq/SeqCombineFileInputFormatTest.java @@ -1,4 +1,4 @@ -package com.expedia.edw.data.squeeze.impl.seq; +package com.expedia.dsp.data.squeeze.impl.seq; import org.apache.hadoop.fs.Path; import org.apache.hadoop.mapreduce.JobContext; diff --git a/dataSqueeze-manager/src/test/java/com/expedia/edw/data/squeeze/impl/text/TextCombineFileInputFormatTest.java b/src/test/java/com/expedia/dsp/data/squeeze/impl/text/TextCombineFileInputFormatTest.java similarity index 97% rename from dataSqueeze-manager/src/test/java/com/expedia/edw/data/squeeze/impl/text/TextCombineFileInputFormatTest.java rename to src/test/java/com/expedia/dsp/data/squeeze/impl/text/TextCombineFileInputFormatTest.java index 3d9f0ee..6f5104c 100644 --- a/dataSqueeze-manager/src/test/java/com/expedia/edw/data/squeeze/impl/text/TextCombineFileInputFormatTest.java +++ b/src/test/java/com/expedia/dsp/data/squeeze/impl/text/TextCombineFileInputFormatTest.java @@ -1,4 +1,4 @@ -package com.expedia.edw.data.squeeze.impl.text; +package com.expedia.dsp.data.squeeze.impl.text; import org.apache.hadoop.fs.Path; import org.apache.hadoop.mapreduce.JobContext; diff --git a/dataSqueeze-manager/src/test/java/com/expedia/edw/data/squeeze/impl/text/TextRecordReaderTest.java b/src/test/java/com/expedia/dsp/data/squeeze/impl/text/TextRecordReaderTest.java similarity index 98% rename from dataSqueeze-manager/src/test/java/com/expedia/edw/data/squeeze/impl/text/TextRecordReaderTest.java rename to src/test/java/com/expedia/dsp/data/squeeze/impl/text/TextRecordReaderTest.java index 6603543..c634141 100644 --- a/dataSqueeze-manager/src/test/java/com/expedia/edw/data/squeeze/impl/text/TextRecordReaderTest.java +++ b/src/test/java/com/expedia/dsp/data/squeeze/impl/text/TextRecordReaderTest.java @@ -1,4 +1,4 @@ -package com.expedia.edw.data.squeeze.impl.text; +package com.expedia.dsp.data.squeeze.impl.text; import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.fs.FSDataInputStream; diff --git a/dataSqueeze-manager/src/test/java/com/expedia/edw/data/squeeze/mappers/AvroCompactionMapperTest.java b/src/test/java/com/expedia/dsp/data/squeeze/mappers/AvroCompactionMapperTest.java similarity index 99% rename from dataSqueeze-manager/src/test/java/com/expedia/edw/data/squeeze/mappers/AvroCompactionMapperTest.java rename to src/test/java/com/expedia/dsp/data/squeeze/mappers/AvroCompactionMapperTest.java index c1a8458..d92dbb5 100644 --- a/dataSqueeze-manager/src/test/java/com/expedia/edw/data/squeeze/mappers/AvroCompactionMapperTest.java +++ b/src/test/java/com/expedia/dsp/data/squeeze/mappers/AvroCompactionMapperTest.java @@ -1,4 +1,4 @@ -package com.expedia.edw.data.squeeze.mappers; +package com.expedia.dsp.data.squeeze.mappers; import static org.mockito.Matchers.any; import static org.powermock.api.mockito.PowerMockito.mock; diff --git a/dataSqueeze-manager/src/test/java/com/expedia/edw/data/squeeze/mappers/BaseMapperTest.java b/src/test/java/com/expedia/dsp/data/squeeze/mappers/BaseMapperTest.java similarity index 98% rename from dataSqueeze-manager/src/test/java/com/expedia/edw/data/squeeze/mappers/BaseMapperTest.java rename to src/test/java/com/expedia/dsp/data/squeeze/mappers/BaseMapperTest.java index 040e23b..01e541e 100644 --- a/dataSqueeze-manager/src/test/java/com/expedia/edw/data/squeeze/mappers/BaseMapperTest.java +++ b/src/test/java/com/expedia/dsp/data/squeeze/mappers/BaseMapperTest.java @@ -1,4 +1,4 @@ -package com.expedia.edw.data.squeeze.mappers; +package com.expedia.dsp.data.squeeze.mappers; import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.fs.FileStatus; diff --git a/dataSqueeze-manager/src/test/java/com/expedia/edw/data/squeeze/mappers/BytesWritableCompactionMapperTest.java b/src/test/java/com/expedia/dsp/data/squeeze/mappers/BytesWritableCompactionMapperTest.java similarity index 98% rename from dataSqueeze-manager/src/test/java/com/expedia/edw/data/squeeze/mappers/BytesWritableCompactionMapperTest.java rename to src/test/java/com/expedia/dsp/data/squeeze/mappers/BytesWritableCompactionMapperTest.java index 2bd16ed..2304fa8 100644 --- a/dataSqueeze-manager/src/test/java/com/expedia/edw/data/squeeze/mappers/BytesWritableCompactionMapperTest.java +++ b/src/test/java/com/expedia/dsp/data/squeeze/mappers/BytesWritableCompactionMapperTest.java @@ -1,4 +1,4 @@ -package com.expedia.edw.data.squeeze.mappers; +package com.expedia.dsp.data.squeeze.mappers; import org.apache.hadoop.conf.Configuration; diff --git a/dataSqueeze-manager/src/test/java/com/expedia/edw/data/squeeze/mappers/OrcCompactionMapperTest.java b/src/test/java/com/expedia/dsp/data/squeeze/mappers/OrcCompactionMapperTest.java similarity index 97% rename from dataSqueeze-manager/src/test/java/com/expedia/edw/data/squeeze/mappers/OrcCompactionMapperTest.java rename to src/test/java/com/expedia/dsp/data/squeeze/mappers/OrcCompactionMapperTest.java index 612e75d..8a82bb0 100644 --- a/dataSqueeze-manager/src/test/java/com/expedia/edw/data/squeeze/mappers/OrcCompactionMapperTest.java +++ b/src/test/java/com/expedia/dsp/data/squeeze/mappers/OrcCompactionMapperTest.java @@ -1,7 +1,8 @@ -package com.expedia.edw.data.squeeze.mappers; +package com.expedia.dsp.data.squeeze.mappers; -import com.expedia.edw.data.squeeze.impl.CombineFileWritable; +import com.expedia.dsp.data.squeeze.impl.CombineFileWritable; + import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.fs.FileStatus; import org.apache.hadoop.fs.FileSystem; diff --git a/dataSqueeze-manager/src/test/java/com/expedia/edw/data/squeeze/mappers/SeqCompactionMapperTest.java b/src/test/java/com/expedia/dsp/data/squeeze/mappers/SeqCompactionMapperTest.java similarity index 98% rename from dataSqueeze-manager/src/test/java/com/expedia/edw/data/squeeze/mappers/SeqCompactionMapperTest.java rename to src/test/java/com/expedia/dsp/data/squeeze/mappers/SeqCompactionMapperTest.java index 788a0f8..3df22ce 100644 --- a/dataSqueeze-manager/src/test/java/com/expedia/edw/data/squeeze/mappers/SeqCompactionMapperTest.java +++ b/src/test/java/com/expedia/dsp/data/squeeze/mappers/SeqCompactionMapperTest.java @@ -1,4 +1,4 @@ -package com.expedia.edw.data.squeeze.mappers; +package com.expedia.dsp.data.squeeze.mappers; import static org.mockito.Matchers.any; import static org.powermock.api.mockito.PowerMockito.mock; diff --git a/dataSqueeze-manager/src/test/java/com/expedia/edw/data/squeeze/mappers/TextCompactionMapperTest.java b/src/test/java/com/expedia/dsp/data/squeeze/mappers/TextCompactionMapperTest.java similarity index 97% rename from dataSqueeze-manager/src/test/java/com/expedia/edw/data/squeeze/mappers/TextCompactionMapperTest.java rename to src/test/java/com/expedia/dsp/data/squeeze/mappers/TextCompactionMapperTest.java index 986f038..a834013 100644 --- a/dataSqueeze-manager/src/test/java/com/expedia/edw/data/squeeze/mappers/TextCompactionMapperTest.java +++ b/src/test/java/com/expedia/dsp/data/squeeze/mappers/TextCompactionMapperTest.java @@ -1,7 +1,7 @@ -package com.expedia.edw.data.squeeze.mappers; +package com.expedia.dsp.data.squeeze.mappers; -import com.expedia.edw.data.squeeze.impl.CombineFileWritable; +import com.expedia.dsp.data.squeeze.impl.CombineFileWritable; import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.fs.FileStatus; import org.apache.hadoop.fs.FileSystem; diff --git a/src/test/java/com/expedia/dsp/data/squeeze/models/CompactionCriteriaTest.java b/src/test/java/com/expedia/dsp/data/squeeze/models/CompactionCriteriaTest.java new file mode 100644 index 0000000..24ea35e --- /dev/null +++ b/src/test/java/com/expedia/dsp/data/squeeze/models/CompactionCriteriaTest.java @@ -0,0 +1,101 @@ +package com.expedia.dsp.data.squeeze.models; + +import org.junit.Test; + +import java.util.HashMap; +import java.util.Map; + +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertNull; + +/** + * Tests for {@link CompactionCriteria} + * + * @author Yashraj R. Sontakke + */ +public class CompactionCriteriaTest { + + @Test + public void testNullThreshold() throws Exception { + final Map options = retrieveOptions(null, "1000"); + CompactionCriteria criteria = new CompactionCriteria(options); + assertNull(criteria.getThresholdInBytes()); + } + + @Test(expected = NumberFormatException.class) + public void testWrongThreshold() throws Exception { + final Map options = retrieveOptions("1234a", "1000"); + CompactionCriteria criteria = new CompactionCriteria(options); + assertNull(criteria.getThresholdInBytes()); + } + + @Test + public void testNullMaxReducers() throws Exception { + final Map options = retrieveOptions("1234", null); + CompactionCriteria criteria = new CompactionCriteria(options); + assertNull(criteria.getMaxReducers()); + } + + @Test(expected = NumberFormatException.class) + public void testWrongMaxReducers() throws Exception { + final Map options = retrieveOptions("1234", "1000a"); + CompactionCriteria criteria = new CompactionCriteria(options); + assertNull(criteria.getMaxReducers()); + } + + @Test(expected = IllegalArgumentException.class) + public void testNoSchemaPath() throws Exception { + final Map options = retrieveOptions("1234", "1000"); + options.put("fileType", "AVRO"); + options.remove("schemaPath"); + CompactionCriteria criteria = new CompactionCriteria(options); + } + + @Test + public void testCriteria() throws Exception { + Map options = retrieveOptions("1234", "1000"); + + CompactionCriteria criteria = new CompactionCriteria(options); + assertEquals("sourcePath", criteria.getSourcePath()); + assertEquals("targetPath", criteria.getTargetPath()); + assertEquals(1234L, criteria.getThresholdInBytes(), 0); + assertEquals("schemaPath", criteria.getSchemaPath()); + assertEquals(1000L, criteria.getMaxReducers(), 0); + assertNull(criteria.getFileType()); + options.put("fileType", "AVRO"); + + criteria = new CompactionCriteria(options); + assertEquals(FileType.AVRO.getValue(), criteria.getFileType()); + + criteria = new CompactionCriteria("source", "target", 12345L, 1000L); + assertEquals("source", criteria.getSourcePath()); + assertEquals("target", criteria.getTargetPath()); + assertEquals(12345L, criteria.getThresholdInBytes(), 0); + assertEquals(1000L, criteria.getMaxReducers(), 0); + + criteria = new CompactionCriteria("source", "target", 12345L, 1000L, "ORC", "schemaPath"); + assertEquals("source", criteria.getSourcePath()); + assertEquals("target", criteria.getTargetPath()); + assertEquals(12345L, criteria.getThresholdInBytes(), 0); + assertEquals(1000L, criteria.getMaxReducers(), 0); + assertEquals("ORC", criteria.getFileType()); + assertEquals("schemaPath", criteria.getSchemaPath()); + } + + @Test(expected = IllegalArgumentException.class) + public void testUnsupportedFileFormat() { + Map options = retrieveOptions("1234", "1000"); + options.put("fileType", "AVROO"); + new CompactionCriteria(options); + } + + private Map retrieveOptions(final String threshold, final String maxReducers) { + final Map options = new HashMap(); + options.put("sourcePath", "sourcePath"); + options.put("targetPath", "targetPath"); + options.put("thresholdInBytes", threshold); + options.put("schemaPath", "schemaPath"); + options.put("maxReducers", maxReducers); + return options; + } +} diff --git a/dataSqueeze-manager/src/test/java/com/expedia/edw/data/squeeze/models/CompactionResponseTest.java b/src/test/java/com/expedia/dsp/data/squeeze/models/CompactionResponseTest.java similarity index 92% rename from dataSqueeze-manager/src/test/java/com/expedia/edw/data/squeeze/models/CompactionResponseTest.java rename to src/test/java/com/expedia/dsp/data/squeeze/models/CompactionResponseTest.java index 0dc4e80..19ab1c4 100644 --- a/dataSqueeze-manager/src/test/java/com/expedia/edw/data/squeeze/models/CompactionResponseTest.java +++ b/src/test/java/com/expedia/dsp/data/squeeze/models/CompactionResponseTest.java @@ -1,8 +1,8 @@ -package com.expedia.edw.data.squeeze.models; +package com.expedia.dsp.data.squeeze.models; import org.junit.Test; -import static com.expedia.edw.data.squeeze.models.FileType.*; +import static com.expedia.dsp.data.squeeze.models.FileType.*; import static org.junit.Assert.*; /** diff --git a/dataSqueeze-manager/src/test/java/com/expedia/edw/data/squeeze/reducers/AvroCompactionReducerTest.java b/src/test/java/com/expedia/dsp/data/squeeze/reducers/AvroCompactionReducerTest.java similarity index 98% rename from dataSqueeze-manager/src/test/java/com/expedia/edw/data/squeeze/reducers/AvroCompactionReducerTest.java rename to src/test/java/com/expedia/dsp/data/squeeze/reducers/AvroCompactionReducerTest.java index 0806fed..0b9f1b3 100644 --- a/dataSqueeze-manager/src/test/java/com/expedia/edw/data/squeeze/reducers/AvroCompactionReducerTest.java +++ b/src/test/java/com/expedia/dsp/data/squeeze/reducers/AvroCompactionReducerTest.java @@ -1,4 +1,4 @@ -package com.expedia.edw.data.squeeze.reducers; +package com.expedia.dsp.data.squeeze.reducers; import static org.powermock.api.mockito.PowerMockito.mock; import static org.powermock.api.mockito.PowerMockito.when; diff --git a/dataSqueeze-manager/src/test/java/com/expedia/edw/data/squeeze/reducers/BaseReducerTest.java b/src/test/java/com/expedia/dsp/data/squeeze/reducers/BaseReducerTest.java similarity index 98% rename from dataSqueeze-manager/src/test/java/com/expedia/edw/data/squeeze/reducers/BaseReducerTest.java rename to src/test/java/com/expedia/dsp/data/squeeze/reducers/BaseReducerTest.java index 938a8c8..f065829 100644 --- a/dataSqueeze-manager/src/test/java/com/expedia/edw/data/squeeze/reducers/BaseReducerTest.java +++ b/src/test/java/com/expedia/dsp/data/squeeze/reducers/BaseReducerTest.java @@ -1,4 +1,4 @@ -package com.expedia.edw.data.squeeze.reducers; +package com.expedia.dsp.data.squeeze.reducers; import com.google.common.collect.ImmutableList; import org.apache.hadoop.conf.Configuration; diff --git a/dataSqueeze-manager/src/test/java/com/expedia/edw/data/squeeze/reducers/BytesWritableCompactionReducerTest.java b/src/test/java/com/expedia/dsp/data/squeeze/reducers/BytesWritableCompactionReducerTest.java similarity index 98% rename from dataSqueeze-manager/src/test/java/com/expedia/edw/data/squeeze/reducers/BytesWritableCompactionReducerTest.java rename to src/test/java/com/expedia/dsp/data/squeeze/reducers/BytesWritableCompactionReducerTest.java index 8f8de08..98491e6 100644 --- a/dataSqueeze-manager/src/test/java/com/expedia/edw/data/squeeze/reducers/BytesWritableCompactionReducerTest.java +++ b/src/test/java/com/expedia/dsp/data/squeeze/reducers/BytesWritableCompactionReducerTest.java @@ -1,4 +1,4 @@ -package com.expedia.edw.data.squeeze.reducers; +package com.expedia.dsp.data.squeeze.reducers; import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.io.BytesWritable; diff --git a/dataSqueeze-manager/src/test/java/com/expedia/edw/data/squeeze/reducers/OrcCompactionReducerTest.java b/src/test/java/com/expedia/dsp/data/squeeze/reducers/OrcCompactionReducerTest.java similarity index 98% rename from dataSqueeze-manager/src/test/java/com/expedia/edw/data/squeeze/reducers/OrcCompactionReducerTest.java rename to src/test/java/com/expedia/dsp/data/squeeze/reducers/OrcCompactionReducerTest.java index 3e0d4f7..b2bd492 100644 --- a/dataSqueeze-manager/src/test/java/com/expedia/edw/data/squeeze/reducers/OrcCompactionReducerTest.java +++ b/src/test/java/com/expedia/dsp/data/squeeze/reducers/OrcCompactionReducerTest.java @@ -1,4 +1,4 @@ -package com.expedia.edw.data.squeeze.reducers; +package com.expedia.dsp.data.squeeze.reducers; import org.apache.hadoop.io.Text; import org.apache.hadoop.mapreduce.Reducer; diff --git a/dataSqueeze-manager/src/test/java/com/expedia/edw/data/squeeze/reducers/TextCompactionReducerTest.java b/src/test/java/com/expedia/dsp/data/squeeze/reducers/TextCompactionReducerTest.java similarity index 98% rename from dataSqueeze-manager/src/test/java/com/expedia/edw/data/squeeze/reducers/TextCompactionReducerTest.java rename to src/test/java/com/expedia/dsp/data/squeeze/reducers/TextCompactionReducerTest.java index ecb8b6e..7dd0921 100644 --- a/dataSqueeze-manager/src/test/java/com/expedia/edw/data/squeeze/reducers/TextCompactionReducerTest.java +++ b/src/test/java/com/expedia/dsp/data/squeeze/reducers/TextCompactionReducerTest.java @@ -1,4 +1,4 @@ -package com.expedia.edw.data.squeeze.reducers; +package com.expedia.dsp.data.squeeze.reducers; import org.apache.hadoop.io.Text; import org.apache.hadoop.mapreduce.Reducer; diff --git a/dataSqueeze-manager/src/test/resources/compaction.properties b/src/test/resources/compaction.properties similarity index 100% rename from dataSqueeze-manager/src/test/resources/compaction.properties rename to src/test/resources/compaction.properties diff --git a/dataSqueeze-manager/src/test/resources/log4j.properties b/src/test/resources/log4j.properties similarity index 100% rename from dataSqueeze-manager/src/test/resources/log4j.properties rename to src/test/resources/log4j.properties diff --git a/dataSqueeze-manager/src/test/resources/source/path/text/textfile.txt b/src/test/resources/source/path/text/textfile.txt similarity index 100% rename from dataSqueeze-manager/src/test/resources/source/path/text/textfile.txt rename to src/test/resources/source/path/text/textfile.txt