MAHOUT-1999 Multi artifact build #350

rawkintrevo · 2017-11-05T23:41:32Z

Purpose of PR:

Build multiple binaries of spark in one maven shot.

The issues surrounding SBT wrt MAHOUT-2020 continue, however the produced artifacts work fine for maven builds, as well as console/Zeppelin integrations.

In short- the distribution module executes mvn package twice more, with Scala-2.11/SPark-2.0 and Scala-2.11/SPark-2.1 switches.

Build both scalas
Build ViennaCL / OMP for both scalas

Important ToDos

Please mark each with an "x"

A JIRA ticket exists (if not, please create this first)[https://issues.apache.org/jira/browse/ZEPPELIN/]
Title of PR is "MAHOUT-XXXX Brief Description of Changes" where XXXX is the JIRA number.
Created unit tests where appropriate
Added licenses correct on newly added files
Assigned JIRA to self
Added documentation in scala docs/java docs, and to website
Successfully built and ran all unit tests, verified that all tests pass locally.

If all of these things aren't complete, but you still feel it is
appropriate to open a PR, please add [WIP] after MAHOUT-XXXX before the
descriptions- e.g. "MAHOUT-XXXX [WIP] Description of Change"

Does this change break earlier versions?

Is this the beginning of a larger project for which a feature branch should be made?

andrewpalumbo · 2017-11-08T02:44:57Z

@rawkintrevo Which build command should we be testing?

andrewpalumbo · 2017-11-08T03:05:32Z

distribution/pom.xml

@@ -39,6 +39,131 @@
        <artifactId>exec-maven-plugin</artifactId>
        <version>1.1.1</version>
        <executions>
+          <!-- Various Scalas -->
+          <execution>


not getting viennacl(S) with mvn clean install -Pdistribution (though maybe this is not needed? looks like its is only in the package arg. Doesn't seem like an issue- but raising point.

Checked again with mvn clean package -Pdistribution:

mvn clean package -Pdistribution [INFO] Scanning for projects... {...} [WARNING] The expression ${version} is deprecated. Please use ${project.version} instead. [WARNING] The expression ${version} is deprecated. Please use ${project.version} instead. [WARNING] [WARNING] It is highly recommended to fix these problems because they threaten the stability of your build. [WARNING] [WARNING] For this reason, future Maven versions might no longer support building such malformed projects. [WARNING] [INFO] ------------------------------------------------------------------------ [INFO] Reactor Build Order: [INFO] [INFO] Mahout Build Tools [INFO] Apache Mahout [INFO] Mahout Math [INFO] Mahout HDFS [INFO] Mahout Map-Reduce [INFO] Mahout Integration [INFO] Mahout Examples [INFO] Mahout Math Scala bindings [INFO] Mahout Spark bindings [INFO] Mahout H2O backend [INFO] Mahout Release Package

No viennacl

builds cleanly for scala-2.11.6:
mvn clean install -Dscala.compat.version=2.11 -Dscala.version=2.11.6 -Pviennacl -DskipTests running tests now..

all tests pass for me with mvn clean install -Dscala.compat.version=2.11 -Dscala.version=2.11.6 -Pviennacl -DskipTests && mvn test -Dscala.compat.version=2.11 -Dscala.version=2.11.6 -Pviennacl

mvn clean install -Pviennacl,viennacl-omp,distribution

There are executions to pick up multiple scala versions of both in Pdistribution

andrewpalumbo · 2017-11-08T03:18:33Z

distribution/pom.xml

@@ -39,6 +39,131 @@
        <artifactId>exec-maven-plugin</artifactId>
        <version>1.1.1</version>
        <executions>
+          <!-- Various Scalas -->
+          <execution>


builds cleanly for scala-2.11.6:
mvn clean install -Dscala.compat.version=2.11 -Dscala.version=2.11.6 -Pviennacl -DskipTests running tests now..

andrewpalumbo · 2017-11-08T03:21:32Z

pom.xml

@@ -586,7 +586,7 @@
          <version>2.5.2</version>
          <configuration>
            <useReleaseProfile>true</useReleaseProfile>
-            <releaseProfiles>mahout-release,apache-release,hadoop2</releaseProfiles>
+            <releaseProfiles>distribution,mahout-release,apache-release,hadoop2</releaseProfiles>


Add viennacl-omp here? this would kick off bothviennacl, and viennacl-omp

yes I think it's reasonable to do that.

andrewpalumbo · 2017-11-08T03:23:32Z

pom.xml

+    <profile>
+      <id>distribution</id>
+      <activation>
+        <activeByDefault>true</activeByDefault>


this, I would think should default to false.

agreed updated

andrewpalumbo · 2017-11-08T04:23:26Z

distribution/pom.xml

@@ -39,6 +39,131 @@
        <artifactId>exec-maven-plugin</artifactId>
        <version>1.1.1</version>
        <executions>
+          <!-- Various Scalas -->
+          <execution>


all tests pass for me with mvn clean install -Dscala.compat.version=2.11 -Dscala.version=2.11.6 -Pviennacl -DskipTests && mvn test -Dscala.compat.version=2.11 -Dscala.version=2.11.6 -Pviennacl

pferrel · 2017-11-15T00:32:57Z

We need a matrix if profiles and important -D options, also what is the default, IMO is should be Spark 2.1 at least and Scala 2.11, these are reasonably recent CDH distros and are forward compatible with Spark 2.3 with Hadoop 2.8, which is what I'm running (as opposed to building for).

mvn clean install -Dspark.version=spark-2.2.0 -Dspark.compat.version=2.0 -Dscala.compat.version=2.11 -Dscala.version=2.11.11

tests running...

pferrel · 2017-11-15T00:35:53Z

BTW if this works I'll test the speedup PR too and merge it. Only my test app needed Scala 2.11, the speedup doesn't care.

pferrel · 2017-11-15T01:03:24Z

-Dspark.version=spark-2.2.0 failed [WARNING] The POM for org.apache.spark:spark-core_2.11:jar:spark-2.2.0 is missing, no dependency information available

2.2.0 is the default download from the Spark site.

Trying Dspark.version=spark-2.1.0 Do any intermediate ones build or are the versions hard coded not based on the passed in string?

pferrel · 2017-11-15T01:50:13Z

Hmm mvn clean install -Dspark.version=spark-2.1.0 -Dspark.compat.version=2.0 -Dscala.compat.version=2.11 -Dscala.version=2.11.11 also fails [WARNING] The POM for org.apache.spark:spark-core_2.11:jar:spark-2.1.0 is missing, no dependency information available

pferrel · 2017-11-15T01:54:20Z

Hmm got the wrong branch, missed "-b"
nevermind

pferrel · 2017-11-17T19:14:44Z

This tests out for me. Merge away

all scalas, viennacls, distribution optional responded to apalumbo's comments on PR350 bad scala versions spark-style scala switch, docs, etc change scala version back to 2.10 travis scala version

rawkintrevo mentioned this pull request Nov 6, 2017

Mahout 1999- Automating Multi-Artifact Build Process #331

Closed

7 tasks

rawkintrevo changed the title ~~[WIP] MAHOUT-1999 Multi artifact build~~ MAHOUT-1999 Multi artifact build Nov 6, 2017

andrewpalumbo reviewed Nov 8, 2017

View reviewed changes

MAHOUT-1999 Multi artifact build

e7d6ee6

all scalas, viennacls, distribution optional responded to apalumbo's comments on PR350 bad scala versions spark-style scala switch, docs, etc change scala version back to 2.10 travis scala version

rawkintrevo force-pushed the mahout-1999-b branch from 22a0113 to e7d6ee6 Compare November 17, 2017 23:02

asfgit closed this in d9b32f3 Nov 17, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

MAHOUT-1999 Multi artifact build #350

MAHOUT-1999 Multi artifact build #350

rawkintrevo commented Nov 5, 2017 •

edited

Loading

andrewpalumbo commented Nov 8, 2017

andrewpalumbo Nov 8, 2017 •

edited

Loading

andrewpalumbo Nov 8, 2017 •

edited

Loading

andrewpalumbo Nov 8, 2017

andrewpalumbo Nov 8, 2017 •

edited

Loading

rawkintrevo Nov 15, 2017

andrewpalumbo Nov 15, 2017

andrewpalumbo Nov 8, 2017

andrewpalumbo Nov 8, 2017

rawkintrevo Nov 15, 2017

andrewpalumbo Nov 8, 2017

rawkintrevo Nov 15, 2017

andrewpalumbo Nov 8, 2017 •

edited

Loading

pferrel commented Nov 15, 2017 •

edited

Loading

pferrel commented Nov 15, 2017

pferrel commented Nov 15, 2017

pferrel commented Nov 15, 2017

pferrel commented Nov 15, 2017

pferrel commented Nov 17, 2017

MAHOUT-1999 Multi artifact build #350

MAHOUT-1999 Multi artifact build #350

Conversation

rawkintrevo commented Nov 5, 2017 • edited Loading

Purpose of PR:

Important ToDos

andrewpalumbo commented Nov 8, 2017

andrewpalumbo Nov 8, 2017 • edited Loading

Choose a reason for hiding this comment

andrewpalumbo Nov 8, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

andrewpalumbo Nov 8, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

andrewpalumbo Nov 8, 2017 • edited Loading

Choose a reason for hiding this comment

pferrel commented Nov 15, 2017 • edited Loading

pferrel commented Nov 15, 2017

pferrel commented Nov 15, 2017

pferrel commented Nov 15, 2017

pferrel commented Nov 15, 2017

pferrel commented Nov 17, 2017

rawkintrevo commented Nov 5, 2017 •

edited

Loading

andrewpalumbo Nov 8, 2017 •

edited

Loading

andrewpalumbo Nov 8, 2017 •

edited

Loading

andrewpalumbo Nov 8, 2017 •

edited

Loading

andrewpalumbo Nov 8, 2017 •

edited

Loading

pferrel commented Nov 15, 2017 •

edited

Loading