Refactor datasqueeze code structure and namespace as well as add publishing to maven central #4

michaelzhu2003 · 2018-11-26T23:42:30Z

No description provided.

yashrajrs · 2018-11-26T23:53:43Z

Please update the readme to include all the changes. From what I see there are more changes than just publishing to maven central

yashrajrs · 2018-11-26T23:59:56Z

Can we add the DataSqueeze logo commit from our internal code base for readMe as a part of this pull request?

yashrajrs · 2018-11-27T00:02:51Z

pom.xml

+        <repository>
+            <id>central</id>
+            <name>public-maven-virtual</name>
+            <url>https://artylab.expedia.biz:443/public-maven-virtual</url>


This is internal to Expedia.

Good catch. Thanks.

yashrajrs · 2018-11-27T00:03:16Z

The branch fails to build locally

michaelzhu2003 · 2018-11-27T00:09:00Z

It builds on my local from both IntelliJ and maven command line.
What is the failure on your local?

yashrajrs · 2018-11-27T03:53:46Z

README.md


 ## Documentation
 This README is intended to provide detailed technical documentation for advanced users.

+## Changes since last release
+
+* Edided the pom file for publishing this project's artifacts to the Maven Central


Type - Edited

Thanks so much.

yashrajrs · 2018-11-27T03:55:00Z

README.md


 1. CLI -
    a. For TEXT/ORC/SEQ
    ```java
-        hadoop jar dataSqueeze-manager-1.0-SNAPSHOT.jar com.expedia.edw.data.squeeze.Utility
+        hadoop jar Datasqueeze.jar com.expedia.dsp.data.squeeze.Utility


typo - "datasqueeze"

Great catch. Thanks so much.

yashrajrs · 2018-11-27T03:55:36Z

README.md

        -sp s3a://edwprod/user/ysontakke/compactiontest1/ -tp s3a://edwprod/user/ysontakke/compactionoutput_text_yash_1/
        -threshold 12345
    ```

    b. For AVRO
    ```java
-        hadoop jar dataSqueeze-manager-1.0-SNAPSHOT.jar com.expedia.edw.data.squeeze.Utility
+        hadoop jar Datasqueeze.jar com.expedia.dsp.data.squeeze.Utility


typo - "datasqueeze"

yashrajrs · 2018-11-27T03:55:46Z

README.md

@@ -87,22 +95,22 @@ There are two different ways of running DataSqueeze:
       * fileType - Type of file to be compacted (AVRO / TEXT / SEQ / ORC). It is mandatory for AVRO
       * schemaPath - schema used for compaction (mandatory for AVRO)

-2. API - [CompactionManager](dataSqueeze-manager/src/main/java/com/expedia/edw/data/squeeze/CompactionManager.java)
+2. API - [CompactionManager](Datasqueeze/src/main/java/com/expedia/dsp/data/squeeze/CompactionManager.java)


typo - "datasqueeze"

yashrajrs · 2018-11-27T03:58:14Z

pom.xml

        <apache.commons.io.version>1.3.2</apache.commons.io.version>
        <aws.java.sdk.version>1.11.98</aws.java.sdk.version>

        <commons-cli.version>1.2</commons-cli.version>
        <commons.io.version>2.4</commons.io.version>
        <commons.lang3.version>3.0</commons.lang3.version>

-        <hadoop.version>2.7.4</hadoop.version>
+        <!-- Use 2.7.4 for github-->
+        <hadoop.version>2.7.1.2.4.3.0-227</hadoop.version>


Please revert this change

yashrajrs · 2018-11-27T04:00:40Z

pom.xml


+    <version>2.0-SNAPSHOT</version>


This needs to move right below artifactId

yashrajrs · 2018-11-27T21:31:13Z

README.md



        a. Fetch the file paths to be compacted from the source path provided.
        b. Perform mapreduce job using the following configuration
            1. Mapper maps records together based on same parent directory and emits parent directory as key.
-            2. Reducer reduces records based on same key but writes data to the target directory provided by the user, retaining
-                the directory structure.
+            2. Reducer reduces records based on same key but writes data to the target directory provided by the user, retaining the directory structure.


Format looks off

I think the format looks better now than before?

I would prefer not adding this in the commit range if the text is not updated.
Essentially, keep the master copy for this as is

yashrajrs · 2018-11-27T21:33:04Z

Update Pull request description. Looks good otherwise

yashrajrs · 2018-11-27T21:35:28Z

can you also update readMe to say where the jars can be found

baumandm · 2018-11-27T21:41:58Z

can you also update readMe to say where the jars can be found

This will be easier to update after we actually publish the 2.0 release. So I would suggest we wait on this.

yashrajrs

LGTM

Refactor datasqueeze for publishing to maven central.

4a08fed

michaelzhu2003 requested review from baumandm and yashrajrs November 26, 2018 23:42

yashrajrs suggested changes Nov 27, 2018

View reviewed changes

YUQING ZHU added 2 commits November 26, 2018 16:47

Changes of pr comments.

a733c46

Correct a copy mistake.

5a0ba4a

yashrajrs reviewed Nov 27, 2018

View reviewed changes

YUQING ZHU and others added 2 commits November 27, 2018 10:10

Correct some typo like mistakes.

a1ff03c

Fix DataSqueeze capitalization in README

4ee0091

baumandm previously approved these changes Nov 27, 2018

View reviewed changes

yashrajrs reviewed Nov 27, 2018

View reviewed changes

Rollback format changes

6daafd1

baumandm dismissed their stale review via 6daafd1 November 27, 2018 21:53

baumandm approved these changes Nov 27, 2018

View reviewed changes

michaelzhu2003 changed the title ~~Refactor datasqueeze for publishing to maven central.~~ Refactor datasqueeze code structure and namespace as well as add publishing to maven central Nov 27, 2018

yashrajrs approved these changes Nov 27, 2018

View reviewed changes

yashrajrs merged commit 42d886a into master Nov 27, 2018

yashrajrs deleted the feature/RefactForPublishToMavenCentral branch November 27, 2018 22:02

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Refactor datasqueeze code structure and namespace as well as add publishing to maven central #4

Refactor datasqueeze code structure and namespace as well as add publishing to maven central #4

michaelzhu2003 commented Nov 26, 2018

yashrajrs commented Nov 26, 2018

yashrajrs commented Nov 26, 2018

yashrajrs Nov 27, 2018

michaelzhu2003 Nov 27, 2018

yashrajrs commented Nov 27, 2018

michaelzhu2003 commented Nov 27, 2018

yashrajrs Nov 27, 2018

michaelzhu2003 Nov 27, 2018

yashrajrs Nov 27, 2018

michaelzhu2003 Nov 27, 2018

yashrajrs Nov 27, 2018

yashrajrs Nov 27, 2018

yashrajrs Nov 27, 2018

yashrajrs Nov 27, 2018

yashrajrs Nov 27, 2018

baumandm Nov 27, 2018

yashrajrs Nov 27, 2018

yashrajrs commented Nov 27, 2018

yashrajrs commented Nov 27, 2018

baumandm commented Nov 27, 2018

yashrajrs left a comment

Refactor datasqueeze code structure and namespace as well as add publishing to maven central #4

Refactor datasqueeze code structure and namespace as well as add publishing to maven central #4

Conversation

michaelzhu2003 commented Nov 26, 2018

yashrajrs commented Nov 26, 2018

yashrajrs commented Nov 26, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yashrajrs commented Nov 27, 2018

michaelzhu2003 commented Nov 27, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yashrajrs commented Nov 27, 2018

yashrajrs commented Nov 27, 2018

baumandm commented Nov 27, 2018

yashrajrs left a comment

Choose a reason for hiding this comment