Add move_to_spark_1.sh script and update README to mention #1307

Closed
heuermh opened this Issue Dec 9, 2016 · 3 comments

Comments

Projects
1 participant
@heuermh
Member

heuermh commented Dec 9, 2016

For building from source, it is useful to go back and forth between building against Spark 1.x/2.x. We have a move_to_spark_2.sh script that is called as part of the release process but no corresponding move_to_spark_1.sh to reverse those changes.

Then as mentioned in #1306 the building from source section of README should mention these scripts.

@heuermh

This comment has been minimized.

Show comment
Hide comment
@heuermh

heuermh Dec 19, 2016

Member

Then as mentioned in #1306 the building from source section of README should mention these scripts.

I'll update README after #1319.

Member

heuermh commented Dec 19, 2016

Then as mentioned in #1306 the building from source section of README should mention these scripts.

I'll update README after #1319.

@heuermh heuermh reopened this Dec 27, 2016

@heuermh

This comment has been minimized.

Show comment
Hide comment
@heuermh

heuermh Dec 27, 2016

Member

On a new machine, on git HEAD I tried

$ mvn clean install
$ ./scripts/move_to_scala_2.11.sh 
$ mvn clean install
$ ./scripts/move_to_spark_2.sh 
$ mvn clean install
$ ./scripts/move_to_scala_2.10.sh 
$ mvn clean install

and at this point the build fails

...
[INFO] --- maven-install-plugin:2.5.2:install (default-install) @ adam-core_2.10 ---
[INFO] Installing /Users/mheuer2/working/adam/adam-core/target/adam-core_2.10-0.20.1-SNAPSHOT.jar to /Users/mheuer2/.m2/repository/org/bdgenomics/adam/adam-core_2.10/0.20.1-SNAPSHOT/adam-core_2.10-0.20.1-SNAPSHOT.jar
[INFO] Installing /Users/mheuer2/working/adam/adam-core/pom.xml to /Users/mheuer2/.m2/repository/org/bdgenomics/adam/adam-core_2.10/0.20.1-SNAPSHOT/adam-core_2.10-0.20.1-SNAPSHOT.pom
[INFO] Installing /Users/mheuer2/working/adam/adam-core/target/adam-core_2.10-0.20.1-SNAPSHOT-tests.jar to /Users/mheuer2/.m2/repository/org/bdgenomics/adam/adam-core_2.10/0.20.1-SNAPSHOT/adam-core_2.10-0.20.1-SNAPSHOT-tests.jar
[INFO] Installing /Users/mheuer2/working/adam/adam-core/target/adam-core_2.10-0.20.1-SNAPSHOT-javadoc.jar to /Users/mheuer2/.m2/repository/org/bdgenomics/adam/adam-core_2.10/0.20.1-SNAPSHOT/adam-core_2.10-0.20.1-SNAPSHOT-javadoc.jar
[INFO]                                                                         
[INFO] ------------------------------------------------------------------------
[INFO] Building ADAM_2.10: APIs for Java 0.20.1-SNAPSHOT
[INFO] ------------------------------------------------------------------------
Downloading: https://oss.sonatype.org/content/repositories/snapshots/org/bdgenomics/adam/adam-core-spark2_2.10/0.20.1-SNAPSHOT/maven-metadata.xml
Downloading: https://oss.sonatype.org/content/repositories/snapshots/org/bdgenomics/adam/adam-core-spark2_2.10/0.20.1-SNAPSHOT/adam-core-spark2_2.10-0.20.1-SNAPSHOT.pom
[WARNING] The POM for org.bdgenomics.adam:adam-core-spark2_2.10:jar:0.20.1-SNAPSHOT is missing, no dependency information available
[WARNING] The POM for org.bdgenomics.adam:adam-core-spark2_2.10:jar:tests:0.20.1-SNAPSHOT is missing, no dependency information available
Downloading: https://oss.sonatype.org/content/repositories/snapshots/org/bdgenomics/adam/adam-core-spark2_2.10/0.20.1-SNAPSHOT/adam-core-spark2_2.10-0.20.1-SNAPSHOT.jar
Downloading: https://oss.sonatype.org/content/repositories/snapshots/org/bdgenomics/adam/adam-core-spark2_2.10/0.20.1-SNAPSHOT/adam-core-spark2_2.10-0.20.1-SNAPSHOT-tests.jar
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO] 
[INFO] ADAM_2.10 .......................................... SUCCESS [  4.373 s]
[INFO] ADAM_2.10: Core .................................... SUCCESS [02:32 min]
[INFO] ADAM_2.10: APIs for Java ........................... FAILURE [  0.905 s]
[INFO] ADAM_2.10: CLI ..................................... SKIPPED
[INFO] ADAM_2.10: Assembly ................................ SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 02:37 min
[INFO] Finished at: 2016-12-27T10:53:53-06:00
[INFO] Final Memory: 64M/1027M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal on project adam-apis_2.10: Could not resolve dependencies for project org.bdgenomics.adam:adam-apis_2.10:jar:0.20.1-SNAPSHOT: The following artifacts could not be resolved: org.bdgenomics.adam:adam-core-spark2_2.10:jar:0.20.1-SNAPSHOT, org.bdgenomics.adam:adam-core-spark2_2.10:jar:tests:0.20.1-SNAPSHOT: Could not find artifact org.bdgenomics.adam:adam-core-spark2_2.10:jar:0.20.1-SNAPSHOT in sonatype-nexus-snapshots (https://oss.sonatype.org/content/repositories/snapshots) -> [Help 1]

Note that the jar being installed is adam-core_2.10-0.20.1-SNAPSHOT.jar and not adam-core-spark2_2.10-0.20.1-SNAPSHOT.jar.

There must be something wrong with the move_to_spark_2.sh script.

Member

heuermh commented Dec 27, 2016

On a new machine, on git HEAD I tried

$ mvn clean install
$ ./scripts/move_to_scala_2.11.sh 
$ mvn clean install
$ ./scripts/move_to_spark_2.sh 
$ mvn clean install
$ ./scripts/move_to_scala_2.10.sh 
$ mvn clean install

and at this point the build fails

...
[INFO] --- maven-install-plugin:2.5.2:install (default-install) @ adam-core_2.10 ---
[INFO] Installing /Users/mheuer2/working/adam/adam-core/target/adam-core_2.10-0.20.1-SNAPSHOT.jar to /Users/mheuer2/.m2/repository/org/bdgenomics/adam/adam-core_2.10/0.20.1-SNAPSHOT/adam-core_2.10-0.20.1-SNAPSHOT.jar
[INFO] Installing /Users/mheuer2/working/adam/adam-core/pom.xml to /Users/mheuer2/.m2/repository/org/bdgenomics/adam/adam-core_2.10/0.20.1-SNAPSHOT/adam-core_2.10-0.20.1-SNAPSHOT.pom
[INFO] Installing /Users/mheuer2/working/adam/adam-core/target/adam-core_2.10-0.20.1-SNAPSHOT-tests.jar to /Users/mheuer2/.m2/repository/org/bdgenomics/adam/adam-core_2.10/0.20.1-SNAPSHOT/adam-core_2.10-0.20.1-SNAPSHOT-tests.jar
[INFO] Installing /Users/mheuer2/working/adam/adam-core/target/adam-core_2.10-0.20.1-SNAPSHOT-javadoc.jar to /Users/mheuer2/.m2/repository/org/bdgenomics/adam/adam-core_2.10/0.20.1-SNAPSHOT/adam-core_2.10-0.20.1-SNAPSHOT-javadoc.jar
[INFO]                                                                         
[INFO] ------------------------------------------------------------------------
[INFO] Building ADAM_2.10: APIs for Java 0.20.1-SNAPSHOT
[INFO] ------------------------------------------------------------------------
Downloading: https://oss.sonatype.org/content/repositories/snapshots/org/bdgenomics/adam/adam-core-spark2_2.10/0.20.1-SNAPSHOT/maven-metadata.xml
Downloading: https://oss.sonatype.org/content/repositories/snapshots/org/bdgenomics/adam/adam-core-spark2_2.10/0.20.1-SNAPSHOT/adam-core-spark2_2.10-0.20.1-SNAPSHOT.pom
[WARNING] The POM for org.bdgenomics.adam:adam-core-spark2_2.10:jar:0.20.1-SNAPSHOT is missing, no dependency information available
[WARNING] The POM for org.bdgenomics.adam:adam-core-spark2_2.10:jar:tests:0.20.1-SNAPSHOT is missing, no dependency information available
Downloading: https://oss.sonatype.org/content/repositories/snapshots/org/bdgenomics/adam/adam-core-spark2_2.10/0.20.1-SNAPSHOT/adam-core-spark2_2.10-0.20.1-SNAPSHOT.jar
Downloading: https://oss.sonatype.org/content/repositories/snapshots/org/bdgenomics/adam/adam-core-spark2_2.10/0.20.1-SNAPSHOT/adam-core-spark2_2.10-0.20.1-SNAPSHOT-tests.jar
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO] 
[INFO] ADAM_2.10 .......................................... SUCCESS [  4.373 s]
[INFO] ADAM_2.10: Core .................................... SUCCESS [02:32 min]
[INFO] ADAM_2.10: APIs for Java ........................... FAILURE [  0.905 s]
[INFO] ADAM_2.10: CLI ..................................... SKIPPED
[INFO] ADAM_2.10: Assembly ................................ SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 02:37 min
[INFO] Finished at: 2016-12-27T10:53:53-06:00
[INFO] Final Memory: 64M/1027M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal on project adam-apis_2.10: Could not resolve dependencies for project org.bdgenomics.adam:adam-apis_2.10:jar:0.20.1-SNAPSHOT: The following artifacts could not be resolved: org.bdgenomics.adam:adam-core-spark2_2.10:jar:0.20.1-SNAPSHOT, org.bdgenomics.adam:adam-core-spark2_2.10:jar:tests:0.20.1-SNAPSHOT: Could not find artifact org.bdgenomics.adam:adam-core-spark2_2.10:jar:0.20.1-SNAPSHOT in sonatype-nexus-snapshots (https://oss.sonatype.org/content/repositories/snapshots) -> [Help 1]

Note that the jar being installed is adam-core_2.10-0.20.1-SNAPSHOT.jar and not adam-core-spark2_2.10-0.20.1-SNAPSHOT.jar.

There must be something wrong with the move_to_spark_2.sh script.

@heuermh

This comment has been minimized.

Show comment
Hide comment
@heuermh

heuermh Dec 27, 2016

Member

If from a clean working directory move_to_spark_2.sh is run before move_to_scala_2.11.sh, then the build succeeds

$ ./scripts/move_to_spark_2.sh 
$ mvn clean install
...
[INFO] --- maven-install-plugin:2.5.2:install (default-install) @ adam-core-spark2_2.10 ---
[INFO] Installing /Users/mheuer2/working/adam/adam-core/target/adam-core-spark2_2.10-0.20.1-SNAPSHOT.jar to /Users/mheuer2/.m2/repository/org/bdgenomics/adam/adam-core-spark2_2.10/0.20.1-SNAPSHOT/adam-core-spark2_2.10-0.20.1-SNAPSHOT.jar
[INFO] Installing /Users/mheuer2/working/adam/adam-core/pom.xml to /Users/mheuer2/.m2/repository/org/bdgenomics/adam/adam-core-spark2_2.10/0.20.1-SNAPSHOT/adam-core-spark2_2.10-0.20.1-SNAPSHOT.pom
[INFO] Installing /Users/mheuer2/working/adam/adam-core/target/adam-core-spark2_2.10-0.20.1-SNAPSHOT-tests.jar to /Users/mheuer2/.m2/repository/org/bdgenomics/adam/adam-core-spark2_2.10/0.20.1-SNAPSHOT/adam-core-spark2_2.10-0.20.1-SNAPSHOT-tests.jar
[INFO] Installing /Users/mheuer2/working/adam/adam-core/target/adam-core-spark2_2.10-0.20.1-SNAPSHOT-javadoc.jar to /Users/mheuer2/.m2/repository/org/bdgenomics/adam/adam-core-spark2_2.10/0.20.1-SNAPSHOT/adam-core-spark2_2.10-0.20.1-SNAPSHOT-javadoc.jar
[INFO]                                                                         
...
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO] 
[INFO] ADAM_2.10 .......................................... SUCCESS [  3.605 s]
[INFO] ADAM_2.10: Core .................................... SUCCESS [02:23 min]
[INFO] ADAM_2.10: APIs for Java ........................... SUCCESS [ 16.678 s]
[INFO] ADAM_2.10: CLI ..................................... SUCCESS [ 34.241 s]
[INFO] ADAM_2.10: Assembly ................................ SUCCESS [  7.412 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
Member

heuermh commented Dec 27, 2016

If from a clean working directory move_to_spark_2.sh is run before move_to_scala_2.11.sh, then the build succeeds

$ ./scripts/move_to_spark_2.sh 
$ mvn clean install
...
[INFO] --- maven-install-plugin:2.5.2:install (default-install) @ adam-core-spark2_2.10 ---
[INFO] Installing /Users/mheuer2/working/adam/adam-core/target/adam-core-spark2_2.10-0.20.1-SNAPSHOT.jar to /Users/mheuer2/.m2/repository/org/bdgenomics/adam/adam-core-spark2_2.10/0.20.1-SNAPSHOT/adam-core-spark2_2.10-0.20.1-SNAPSHOT.jar
[INFO] Installing /Users/mheuer2/working/adam/adam-core/pom.xml to /Users/mheuer2/.m2/repository/org/bdgenomics/adam/adam-core-spark2_2.10/0.20.1-SNAPSHOT/adam-core-spark2_2.10-0.20.1-SNAPSHOT.pom
[INFO] Installing /Users/mheuer2/working/adam/adam-core/target/adam-core-spark2_2.10-0.20.1-SNAPSHOT-tests.jar to /Users/mheuer2/.m2/repository/org/bdgenomics/adam/adam-core-spark2_2.10/0.20.1-SNAPSHOT/adam-core-spark2_2.10-0.20.1-SNAPSHOT-tests.jar
[INFO] Installing /Users/mheuer2/working/adam/adam-core/target/adam-core-spark2_2.10-0.20.1-SNAPSHOT-javadoc.jar to /Users/mheuer2/.m2/repository/org/bdgenomics/adam/adam-core-spark2_2.10/0.20.1-SNAPSHOT/adam-core-spark2_2.10-0.20.1-SNAPSHOT-javadoc.jar
[INFO]                                                                         
...
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO] 
[INFO] ADAM_2.10 .......................................... SUCCESS [  3.605 s]
[INFO] ADAM_2.10: Core .................................... SUCCESS [02:23 min]
[INFO] ADAM_2.10: APIs for Java ........................... SUCCESS [ 16.678 s]
[INFO] ADAM_2.10: CLI ..................................... SUCCESS [ 34.241 s]
[INFO] ADAM_2.10: Assembly ................................ SUCCESS [  7.412 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------

fnothaft added a commit to fnothaft/adam that referenced this issue Dec 27, 2016

[ADAM-1307] move_to_spark_2 fails after moving to scala 2.11.
The sed command that moved to Spark 2 looked for _2.10 in the
ADAM pom artifact names. This commit fixes that issue by looking
for _2.1 instead, which matches both Scala 2.10 and 2.11, as well
as the eventual Scala 2.12. Resolves #1307.

@heuermh heuermh closed this in #1329 Dec 27, 2016

heuermh added a commit that referenced this issue Dec 27, 2016

[ADAM-1307] move_to_spark_2 fails after moving to scala 2.11.
The sed command that moved to Spark 2 looked for _2.10 in the
ADAM pom artifact names. This commit fixes that issue by looking
for _2.1 instead, which matches both Scala 2.10 and 2.11, as well
as the eventual Scala 2.12. Resolves #1307.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment