fixes #980 #995

cjmctague · 2017-12-31T06:35:20Z

-Shade libthrift into fluo-core jar

ctubbsii

I double checked that there shouldn't be any need to update the LICENSE and/or NOTICE file with this change... as long as upstream Thrift has kept their LICENSE and NOTICE files accurate for the libthrift jar.

But, we still need to do relocations of the libthrift classes, and fix up the any class path issues with the naming of the final jar with the shaded content.

It couldn't hurt to generate a diff of the resulting jar file listings before and after this change. I'm mentioning that here so I don't forget to do it in a subsequent review.

ctubbsii · 2017-12-31T07:01:19Z

modules/core/pom.xml

+              <minimizeJar>true</minimizeJar>
+              <artifactSet>
+                <includes>
+                  <include>org.apache.thrift:libthrift</include>


Perhaps the original issue should have been more specific, but the intent was to relocate the libthrift classes. If we don't do the relocation, then the original conflict with Accumulo's libthrift dependency will not be resolved.

ctubbsii · 2017-12-31T07:03:54Z

modules/core/pom.xml

+              <shadedClassifierName>shaded</shadedClassifierName>
+              <filters>
+                <filter>
+                  <artifact>org.apache.thrift:libthrift</artifact>


I don't think this filter is doing anything, since it's not actually inluding/excluding any resource patterns.

ctubbsii · 2017-12-31T07:10:46Z

modules/core/pom.xml

+                </includes>
+              </artifactSet>
+              <shadedArtifactAttached>true</shadedArtifactAttached>
+              <shadedClassifierName>shaded</shadedClassifierName>


Do we want to modify our original jar, or have the shaded one separate? I'm thinking we probably want the shaded one to be the default artifact, for this module, rather than a separate artifact with a classifier.

I also think it would be best to modify the original jar instead of create a separate shaded jar.

@cjmctague are you still working on this? I was asking because I was thinking it may be nice to get this in 1.2.0.

Also, feel free to ask any quesitons I will help the best I can. I have used the shade plugin a few times, but not recently and not for this specific case.

@keith-turner Yes I was still working on this. Currently in Hawaii kinda off the grid for most of the day for class. Survived the "Incoming ballistic missile" false alarm though.

I have it shaded and relocated but it is creating an uber-jar in fluo-core that has all the dependancies from the entire project and not just just that module so I want to fix that at some point.

I could probably vpn into my workstation later and push what I have changed.

mikewalch

The issue number in the commit is wrong. It should be #980 instead of #890

kennethmcfarland · 2018-01-02T20:45:37Z

@mikewalch nice eye catching the lysdexic mistakes lately, glad I'm not the only one now!

…

On Jan 2, 2018 9:10 AM, "Keith Turner" ***@***.***> wrote: ***@***.**** commented on this pull request. ------------------------------ In modules/core/pom.xml <#995 (comment)>: > + <artifactId>maven-shade-plugin</artifactId> + <executions> + <execution> + <goals> + <goal>shade</goal> + </goals> + <phase>package</phase> + <configuration> + <minimizeJar>true</minimizeJar> + <artifactSet> + <includes> + <include>org.apache.thrift:libthrift</include> + </includes> + </artifactSet> + <shadedArtifactAttached>true</shadedArtifactAttached> + <shadedClassifierName>shaded</shadedClassifierName> I also think it would be best to modify the original jar instead of create a separate shaded jar. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#995 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/Acg-LMGxpkaots9siyyHv8X6QZgmx-Xkks5tGmLvgaJpZM4RPyG3> .

mikewalch · 2018-01-02T21:36:01Z

@kpm1985, I guess I am the only one that clicks on the issue number in commit to read the issue ;-) 🥇

-Shade libthrift into fluo-core jar

- Shade libthrift into fluo-core jar

cjmctague · 2018-01-17T18:44:09Z

@ctubbsii @keith-turner
Sorry for the force push but I rebased out of habit and it didn't like the fast-forward.

Currently traveling back from Hawaii so my response are going to be delayed. Just a heads up.

ctubbsii · 2018-01-17T19:03:13Z

modules/core/pom.xml

@@ -73,6 +73,10 @@
      <groupId>org.apache.hadoop</groupId>
      <artifactId>hadoop-client</artifactId>
    </dependency>
+    <dependency>
+      <groupId>org.apache.thrift</groupId>
+      <artifactId>libthrift</artifactId>


I think this still needs to be <scope>provided</scope>, or perhaps <optional>true</optional> in order to prevent transitive dependency resolution by consumers of the fluo-core jar. That is, unless the shade plugin also modifies the POM to exclude libthrift... and I'm not sure it does.

I found that <scope>provided</scope> would exclude it from the shaded jar so I went with <optional>true</optional>

Also the shaded plugin doesn't modify the original POM

This works, and it probably avoids us having to do anything with the dependency reduced POM... although I would do a comparison with the dependency reduced version manually... just to make sure nothing strange was overlooked.

I didn't see thrift in the reduced POM and it changed the checkstyle plugin config

It changed a few things... interesting things... but not important things. The reduced POM looks like our POM, so that's good. I'll check the jar contents manually just to do a quick sanity check, but so far, everything looks the way I'd expect.

ctubbsii · 2018-01-17T19:04:12Z

modules/core/pom.xml

+              <relocations>
+                <relocation>
+                  <pattern>org.apache.thrift</pattern>
+                  <shadedPattern>org.shaded.thrfit</shadedPattern>


Typo here... but also, the new package should be a subpackage of org.apache.fluo.core (or whatever the base package is for this fluo module).

ctubbsii · 2018-01-17T19:07:00Z

modules/core/pom.xml

+                <includes>
+                  <include>org.apache.fluo.core.*</include>
+                  <include>org.shaded.thrift:*</include>
+                  <include>org.apache.thrift:*</include>


I'm not sure these includes are correct. The first one looks like a package pattern. The other two look like maven artifacts, but only the groupId. But, there's no such thing as a groupId named org.shaded.thrift.

I changed them up. The org.apache.fluo.core.* include is needed so the plugin doesn't pull from the entire project but rather just this module.

The <include>org.shaded.thrift:*</include> was needed to include the new shaded artifact and the <include>org.apache.thrift:*</include> was needed for the plugin to relocate thrift.

ctubbsii · 2018-01-17T19:08:49Z

modules/core/pom.xml

+                  <include>org.apache.thrift:*</include>
+                </includes>
+              </artifactSet>
+              <createDependencyReducedPom>false</createDependencyReducedPom>


If this POM is created, I wonder if it would replace the original POM when installed to the local maven repository or deployed to the remote maven repository. If so, it may be useful to let the plugin create this. See comment above about libthrift dependency being resolved transitively.

I excluded in because I was running into failed maven builds for invalid license header on that generated file.

keith-turner · 2018-01-30T18:10:38Z

This seems to be working well. I installed the fluo jars locally and was able to do the following.

Ran mvn dependency:tree on stresso and did not see thrift as a dep of fluo core

Used javap to inspeact a fluo class that uses thrift

$ javap -cp ~/.m2/repository/org/apache/fluo/fluo-core/1.2.0-SNAPSHOT/fluo-core-1.2.0-SNAPSHOT.jar -p org.apache.fluo.core.oracle.OracleServer | grep shaded.thrift
private org.apache.fluo.core.shaded.thrift.server.THsHaServer server;
public org.apache.fluo.core.thrift.Stamps getTimestamps(java.lang.String, int) throws org.apache.fluo.core.shaded.thrift.TException;
private synchronized long getTimestampsImpl(java.lang.String, int) throws org.apache.fluo.core.shaded.thrift.TException;
public boolean isLeader() throws org.apache.fluo.core.shaded.thrift.TException;
private java.net.InetSocketAddress startServer() throws org.apache.fluo.core.shaded.thrift.transport.TTransportException;
static org.apache.fluo.core.shaded.thrift.server.THsHaServer access$600(org.apache.fluo.core.oracle.OracleServer);

Used jar -tf to list files in fluo core jar and saw the shaded and relocated thrift jars
Successfully ran mvn verify on stresso (against Accumulo 1.7 and 1.8) using jars created by this PR.

@ctubbsii do you think this is ready to merge?

keith-turner · 2018-01-30T18:14:01Z

I ran the following to create the source jars. When I looked at the source jars for Fluo I did not see the shaded source code and the relocation was not done in the source. I am not sure if we want to do anything about this. I think we can open a follow on issue to explore this.

mvn clean source:jar install -PskipQA

ctubbsii

I did see one problem when manually inspecting the content of the shaded jar:

It seems to include the LICENSE.txt and NOTICE.txt files from the libthrift. I checked, and there's nothing we need to preserve from these files that isn't already in our own LICENSE and NOTICE file (also included in the shaded jar). So, these just need to be excluded from the libthrift jar when it is shaded in.

keith-turner · 2018-02-08T19:13:29Z

I built this locally and verified LICENSE.txt and NOTICE.txt were absent.

ctubbsii requested changes Dec 31, 2017

View reviewed changes

mikewalch reviewed Jan 2, 2018

View reviewed changes

cjmctague added 2 commits January 17, 2018 13:33

fixes apache#890

f1b8724

-Shade libthrift into fluo-core jar

fixes - apache#980

74ec120

- Shade libthrift into fluo-core jar

cjmctague force-pushed the fluo-980 branch from 078df36 to 74ec120 Compare January 17, 2018 18:36

cjmctague changed the title ~~fixes #890~~ fixes #980 Jan 17, 2018

ctubbsii reviewed Jan 17, 2018

View reviewed changes

review work

9a37c4a

ctubbsii requested changes Feb 2, 2018

View reviewed changes

review

324430e

keith-turner approved these changes Feb 8, 2018

View reviewed changes

keith-turner merged commit 082ee8e into apache:master Feb 8, 2018

keith-turner mentioned this pull request Feb 20, 2018

fixes #1016 Build w/ any Accumulo w/o setting thrift ver #1017

Merged

cjmctague deleted the fluo-980 branch June 12, 2018 18:57

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fixes #980 #995

fixes #980 #995

cjmctague commented Dec 31, 2017

ctubbsii left a comment

ctubbsii Dec 31, 2017

ctubbsii Dec 31, 2017

ctubbsii Dec 31, 2017

keith-turner Jan 2, 2018

keith-turner Jan 12, 2018

keith-turner Jan 12, 2018

cjmctague Jan 13, 2018

mikewalch left a comment

kennethmcfarland commented Jan 2, 2018 via email

mikewalch commented Jan 2, 2018

cjmctague commented Jan 17, 2018

ctubbsii Jan 17, 2018

cjmctague Jan 22, 2018

ctubbsii Jan 22, 2018

cjmctague Jan 28, 2018

ctubbsii Feb 2, 2018

ctubbsii Jan 17, 2018

ctubbsii Jan 17, 2018

cjmctague Jan 22, 2018 •

edited

Loading

ctubbsii Jan 17, 2018

cjmctague Jan 19, 2018

keith-turner commented Jan 30, 2018

keith-turner commented Jan 30, 2018

ctubbsii left a comment

keith-turner commented Feb 8, 2018

fixes #980 #995

fixes #980 #995

Conversation

cjmctague commented Dec 31, 2017

ctubbsii left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mikewalch left a comment

Choose a reason for hiding this comment

kennethmcfarland commented Jan 2, 2018 via email

mikewalch commented Jan 2, 2018

cjmctague commented Jan 17, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cjmctague Jan 22, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

keith-turner commented Jan 30, 2018

keith-turner commented Jan 30, 2018

ctubbsii left a comment

Choose a reason for hiding this comment

keith-turner commented Feb 8, 2018

cjmctague Jan 22, 2018 •

edited

Loading