Split elephant-bird into modules #201

johanoskarsson · 2012-05-16T04:48:18Z

I split the project into core, pig, hive, cascading2 and rcfile modules. This will make it easier to depend on only the parts you need without getting a metric ton of extra jar files. The rcfile module was only since it needs both hive and pig.

I switched the project to Maven in order to easier modularize it. I don't really care what is used though, this just seemed easier. Added two plugins for thrift and protobuf code gen from Maven.

This branch is almost fully baked, but there are a couple of nits. I probably won't have time to get this all the way to the finish line, but it's almost there. Hopefully someone else can help out with these.

testStableHashcodeAcrossJVMs in TestProtobufWritable is commented out. This was using some Ant dependent way of launching another JVM. Should be doable to make this work without Ant though.
Maven complains about directly using the artifacts in lib/, but most of those have been discussed in another issue (frankenjar etc). It seems to work anyway.
All tests pass, but I have not tested any of the artifacts with a real Hadoop job to see if it actually works

…his is the one used in the wild

…nto modules

…. Move to Maven to make this easier.

sagemintblue · 2012-05-16T15:35:23Z

Raghu, the maven-release-plugin can take care of version ids in the parent and all sub modules if you go that route:

http://maven.apache.org/plugins/maven-release-plugin/prepare-mojo.html#autoVersionSubmodules

rangadi · 2012-05-16T15:35:58Z

huge thanks Johan. will go through more thoroughly.
yeah, I think we should be able to take this further.. it might even be EB-3.0.0 (doesn't mean a big delay :).

sagemintblue · 2012-05-16T15:39:09Z

elephant-bird-rcfile/pom.xml

+<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+    <modelVersion>4.0.0</modelVersion>
+    <groupId>com.twitter</groupId>


Add parent info? We can configure dependency management in parent pom.xml to simplify dependency version management and other build plugins across all sub modules that way.

I'm not very well versed in the Maven world. What exactly is the parent info you are referring to?

Think I found it. Adding now.

traviscrawford · 2012-05-16T15:51:27Z

Can you update the readme with a section about building, testing and deploying to the in-git-repo repository? This is a pretty major build change and we should point users in the right direction.

traviscrawford · 2012-05-16T15:52:47Z

elephant-bird-cascading2/pom.xml

+        <dependency>
+            <groupId>com.twitter</groupId>
+            <artifactId>elephant-bird-core</artifactId>
+            <version>2.2.3-SNAPSHOT</version>


Is there a way to set the project version in one place and have all sub projects use it? I could see these version strings getting out of date, not just for the EB release, but for stuff like protobuf. Can we reference properties or something instead?

Yes, we can specify all dependency version info in parent pom.xml within dependencyManagement section. Also, if each sub module pom.xml contains parent section, we can leave off the sub module's groupId and version elements.

johanoskarsson · 2012-05-21T20:15:44Z

I've updated the readme and fixed the broken test. I also added the two plugins to our maven repo so they won't disappear. Using the hadoop-lzo from the local dir until we can get that in a repo. I believe this covers all of the issues pointed out above.

sagemintblue · 2012-05-21T20:18:54Z

elephant-bird-core/pom.xml

+        <groupId>com.twitter</groupId>
+        <artifactId>elephant-bird</artifactId>
+        <version>2.2.3-SNAPSHOT</version>
+    </parent>


May also want element in here and in other submodule pom.xmls:

http://maven.apache.org/pom.html#Inheritance

Hmm, not following unfortunately. What element? What does it do?

Looks like github filtered out the html element from my earlier comment :(

I was referring to parent/relativePath element, which just points submodule to directory containing parent pom.xml if it's likely to be close by. Generally, this is just "..".

sagemintblue · 2012-05-27T21:32:58Z

When I push the hcatalog dep config down into eb hive module, I get this error:

[INFO] Scanning for projects...
[WARNING] 
[WARNING] Some problems were encountered while building the effective model for com.twitter.elephantbird:elephant-bird-hive:jar:3.0.0-SNAPSHOT
[WARNING] 'dependencies.dependency.systemPath' for org.apache.hcatalog:hcatalog:jar should not point at files within the project directory, ${basedir}/../lib/hcatalog-0.5.0-dev.jar will be unresolvable by dependent projects @ line 33, column 19
[WARNING] 
[WARNING] It is highly recommended to fix these problems because they threaten the stability of your build.
[WARNING] 
[WARNING] For this reason, future Maven versions might no longer support building such malformed projects.

Not sure if we should worry about this one, but we could alleviate it by publishing this jar into some publicly accessibly maven repo and updating repository config for EB hive module.

dvryaboy · 2012-05-27T21:41:23Z

https://issues.apache.org/jira/browse/HCATALOG-132 -- looks like Travis got the pom generated, but auto-publishing isn't happening yet.

johanoskarsson · 2012-05-27T21:41:41Z

Does the hcatalog jar contain modified code or can we simply change that to one of their released artifacts?

johanoskarsson · 2012-05-27T21:44:03Z

If we just need one of their artifacts but it's not in a maven repo yet we can simply put it in maven.twittr.com till they do.

dvryaboy · 2012-05-27T21:47:37Z

We need the 0.5 snapshot (which is a moving target, naturally). But afaik there are no custom patches. Travis is the authority on this one.

- Removes yamlbeans from dependency management as it's not used by any modules. - Moves relative system path of hcatalog dep from parent pom to hive pom. - Adds slf4j impl jar dep to submodules. This dep is optional by default so each submodule has to specify what slf4j impl jar they want to use. - Forces junit47 provider within default surefire configuration. I tested parallel test execution, but some tests fail in that mode.

sagemintblue · 2012-05-28T00:07:45Z

I've made a few small updates to the poms. Please see https://github.com/johanoskarsson/elephant-bird/pull/2

Question: The pig and mahout module tests produce a lot of logging output. We can add src/test/resources/log4j.properties configurations to reduce the console output to WARN / ERROR messages. Sound good, or should we just leave all output there?

sagemintblue · 2012-05-28T00:09:33Z

Seems like there are a few places in core module which still use log4j api's directly, instead of slf4j. Anyone mind if I submit a pull req to migrate to slf4j completely?

Tweaks to poms

sagemintblue · 2012-05-28T01:18:44Z

Looks like there's also a path for deployment of 3rd-party artifacts to Sonatype-- another option for the hcatalog jar:

https://docs.sonatype.org/display/Repository/Uploading+3rd-party+Artifacts+to+The+Central+Repository

I've been working more on release process and configuration for deployment to Sonatype. Hoping to have another pull request ready later tonight.

…o modules

dvryaboy · 2012-05-28T21:54:48Z

Ok the latest seems to have worked for me ("seems" because I am now working through other issues. But at least EB was fetched and the compile is no longer complaining about transitive dependencies like Hadoop).

- Adds Johan to contributors section of Readme - Adds developers section - Adds parent section to parent pom which references Sonatype's root pom - Removes conflicting configuration from pluginManagement section

…bird into modules

sagemintblue · 2012-05-29T05:49:12Z

Some small updates to support deployment to Sonatype OSS repos:

https://github.com/johanoskarsson/elephant-bird/pull/3

Updates parent pom.xml to include required Sonatype OSS config

sagemintblue · 2012-05-29T17:57:32Z

Here's my +1 on this branch. The one remaining thing I'd wanted to test is full staging release to Sonatype repo, but this is awkward till after this branch has been merged.

I'd also like to revisit name of submodule directories, in case the maven release plugin is more forgiving of those now.. I prefer simpler names i.e. elephant-bird/elephant-bird-pig becomes elephant-bird/pig.

rangadi · 2012-05-30T00:52:47Z

the output from tests does not go console using ant.
I prefer the simpler names for top level directories. I am not how much more maven work not use the release plugin..

sagemintblue · 2012-05-30T21:06:45Z

@rangadi I had tested adding a test-specific log4j.properties configuration which pushes test logging output to a file. This is a viable solution, but we'll have to add it to path src/test/resources/ within every submodule.

In terms of release process, I'd like to try and stick with the release plugin if at all possible. The only issue with using submodule directory names which don't match the module names is that SCM paths must be explicitly defined within submodule poms. Otherwise, they default to SCM paths based on module names, and not the actual directory names.

rangadi · 2012-05-30T21:13:18Z

explicitly defining paths does not seem that bad. it is done only once. The directory names are the ones we come across all the time.

adding log4j.properties to each directory is not bad either..

sagemintblue · 2012-05-30T21:18:37Z

I'll make those two things happen and request another pull from Johan into this branch.

rangadi · 2012-05-30T21:22:36Z

Thanks guys. I will make a branch-0.3 later today and you can even publish the jars. will merge into master in a few days..

sagemintblue · 2012-05-30T21:25:04Z

branch-0.3 or branch-3.0?

rangadi · 2012-05-30T21:34:57Z

3.0 of course, or we could merge directly the master. What do you guys think?
Only 'beta' part I see are the things like directory names etc.

Also updates .gitignore to reflect new module paths.

Renames all submodule directories, removing "elephant-bird-" prefix

rangadi · 2012-05-31T00:41:50Z

thanks guys.
merged to https://github.com/kevinweil/elephant-bird/tree/branch-3.0

johanoskarsson added 7 commits May 14, 2012 18:03

Change the org for protobuf from com.google to com.google.protobuf, t…

1ed2e07

…his is the one used in the wild

Enable travis ci build

943d5b6

Install protobuf compiler

1758397

Merge branch 'master' of https://github.com/kevinweil/elephant-bird i…

7217b97

…nto modules

Split the project into core, pig, hive, cascading2 and rcfile modules…

69ab8ed

…. Move to Maven to make this easier.

Don't need the travis-ci config in this branch

0b10cab

Put back Hadoop proto code gen

93b186f

sagemintblue reviewed May 16, 2012
View reviewed changes

traviscrawford reviewed May 16, 2012
View reviewed changes

johanoskarsson added 8 commits May 18, 2012 15:06

Merge branch 'master' of https://github.com/kevinweil/elephant-bird

6054733

Merge with master

23722b5

Extract mahout module out of pig

c3e0a58

Remove out of date TODO

164de1b

Update readme with Maven changes. Add javadoc plugin

baa33cd

Added the two plugins to the public Twitter repo

6c5a070

Fix the testStableHashcodeAcrossJVMs test

2b8dc23

Pick up hadoop-lzo from local dir

0cb8afc

sagemintblue reviewed May 21, 2012
View reviewed changes

Merge pull request #2 from sagemintblue/modules

016ef35

Tweaks to poms

johanoskarsson added 2 commits May 28, 2012 10:43

Merge with kevinweil/master

c26c008

Merge branch 'modules' of github.com:johanoskarsson/elephant-bird int…

bc47950

…o modules

sagemintblue added 2 commits May 28, 2012 22:42

Updates parent pom.xml to include required Sonatype OSS config

d5af77a

- Adds Johan to contributors section of Readme - Adds developers section - Adds parent section to parent pom which references Sonatype's root pom - Removes conflicting configuration from pluginManagement section

Merge branch 'modules' of https://github.com/johanoskarsson/elephant-…

aed9132

…bird into modules

Merge pull request #3 from sagemintblue/modules

1f972b2

Updates parent pom.xml to include required Sonatype OSS config

sagemintblue and others added 3 commits May 30, 2012 14:53

Renames all submodule directories, removing "elephant-bird-" prefix

6e645c4

Orders submodules by alpha within parent pom.xml

8118384

Also updates .gitignore to reflect new module paths.

Merge pull request #4 from sagemintblue/modules

418225a

Renames all submodule directories, removing "elephant-bird-" prefix

rangadi closed this May 31, 2012

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Split elephant-bird into modules #201

Split elephant-bird into modules #201

johanoskarsson commented May 16, 2012

sagemintblue commented May 16, 2012

rangadi commented May 16, 2012

sagemintblue May 16, 2012

johanoskarsson May 18, 2012

johanoskarsson May 18, 2012

traviscrawford commented May 16, 2012

traviscrawford May 16, 2012

sagemintblue May 16, 2012

johanoskarsson commented May 21, 2012

sagemintblue May 21, 2012

johanoskarsson May 21, 2012

sagemintblue May 21, 2012

sagemintblue commented May 27, 2012

dvryaboy commented May 27, 2012

johanoskarsson commented May 27, 2012

johanoskarsson commented May 27, 2012

dvryaboy commented May 27, 2012

sagemintblue commented May 28, 2012

sagemintblue commented May 28, 2012

sagemintblue commented May 28, 2012

dvryaboy commented May 28, 2012

sagemintblue commented May 29, 2012

sagemintblue commented May 29, 2012

rangadi commented May 30, 2012

sagemintblue commented May 30, 2012

rangadi commented May 30, 2012

sagemintblue commented May 30, 2012

rangadi commented May 30, 2012

sagemintblue commented May 30, 2012

rangadi commented May 30, 2012

rangadi commented May 31, 2012

Split elephant-bird into modules #201

Split elephant-bird into modules #201

Conversation

johanoskarsson commented May 16, 2012

sagemintblue commented May 16, 2012

rangadi commented May 16, 2012

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

traviscrawford commented May 16, 2012

Choose a reason for hiding this comment

Choose a reason for hiding this comment

johanoskarsson commented May 21, 2012

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sagemintblue commented May 27, 2012

dvryaboy commented May 27, 2012

johanoskarsson commented May 27, 2012

johanoskarsson commented May 27, 2012

dvryaboy commented May 27, 2012

sagemintblue commented May 28, 2012

sagemintblue commented May 28, 2012

sagemintblue commented May 28, 2012

dvryaboy commented May 28, 2012

sagemintblue commented May 29, 2012

sagemintblue commented May 29, 2012

rangadi commented May 30, 2012

sagemintblue commented May 30, 2012

rangadi commented May 30, 2012

sagemintblue commented May 30, 2012

rangadi commented May 30, 2012

sagemintblue commented May 30, 2012

rangadi commented May 30, 2012

rangadi commented May 31, 2012