Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add move_to_spark_1.sh script and update README to mention #1307

Closed
heuermh opened this issue Dec 9, 2016 · 3 comments
Closed

Add move_to_spark_1.sh script and update README to mention #1307

heuermh opened this issue Dec 9, 2016 · 3 comments
Assignees
Labels
Milestone

Comments

@heuermh
Copy link
Member

@heuermh heuermh commented Dec 9, 2016

For building from source, it is useful to go back and forth between building against Spark 1.x/2.x. We have a move_to_spark_2.sh script that is called as part of the release process but no corresponding move_to_spark_1.sh to reverse those changes.

Then as mentioned in #1306 the building from source section of README should mention these scripts.

@heuermh
Copy link
Member Author

@heuermh heuermh commented Dec 19, 2016

Then as mentioned in #1306 the building from source section of README should mention these scripts.

I'll update README after #1319.

@heuermh heuermh reopened this Dec 27, 2016
@heuermh
Copy link
Member Author

@heuermh heuermh commented Dec 27, 2016

On a new machine, on git HEAD I tried

$ mvn clean install
$ ./scripts/move_to_scala_2.11.sh 
$ mvn clean install
$ ./scripts/move_to_spark_2.sh 
$ mvn clean install
$ ./scripts/move_to_scala_2.10.sh 
$ mvn clean install

and at this point the build fails

...
[INFO] --- maven-install-plugin:2.5.2:install (default-install) @ adam-core_2.10 ---
[INFO] Installing /Users/mheuer2/working/adam/adam-core/target/adam-core_2.10-0.20.1-SNAPSHOT.jar to /Users/mheuer2/.m2/repository/org/bdgenomics/adam/adam-core_2.10/0.20.1-SNAPSHOT/adam-core_2.10-0.20.1-SNAPSHOT.jar
[INFO] Installing /Users/mheuer2/working/adam/adam-core/pom.xml to /Users/mheuer2/.m2/repository/org/bdgenomics/adam/adam-core_2.10/0.20.1-SNAPSHOT/adam-core_2.10-0.20.1-SNAPSHOT.pom
[INFO] Installing /Users/mheuer2/working/adam/adam-core/target/adam-core_2.10-0.20.1-SNAPSHOT-tests.jar to /Users/mheuer2/.m2/repository/org/bdgenomics/adam/adam-core_2.10/0.20.1-SNAPSHOT/adam-core_2.10-0.20.1-SNAPSHOT-tests.jar
[INFO] Installing /Users/mheuer2/working/adam/adam-core/target/adam-core_2.10-0.20.1-SNAPSHOT-javadoc.jar to /Users/mheuer2/.m2/repository/org/bdgenomics/adam/adam-core_2.10/0.20.1-SNAPSHOT/adam-core_2.10-0.20.1-SNAPSHOT-javadoc.jar
[INFO]                                                                         
[INFO] ------------------------------------------------------------------------
[INFO] Building ADAM_2.10: APIs for Java 0.20.1-SNAPSHOT
[INFO] ------------------------------------------------------------------------
Downloading: https://oss.sonatype.org/content/repositories/snapshots/org/bdgenomics/adam/adam-core-spark2_2.10/0.20.1-SNAPSHOT/maven-metadata.xml
Downloading: https://oss.sonatype.org/content/repositories/snapshots/org/bdgenomics/adam/adam-core-spark2_2.10/0.20.1-SNAPSHOT/adam-core-spark2_2.10-0.20.1-SNAPSHOT.pom
[WARNING] The POM for org.bdgenomics.adam:adam-core-spark2_2.10:jar:0.20.1-SNAPSHOT is missing, no dependency information available
[WARNING] The POM for org.bdgenomics.adam:adam-core-spark2_2.10:jar:tests:0.20.1-SNAPSHOT is missing, no dependency information available
Downloading: https://oss.sonatype.org/content/repositories/snapshots/org/bdgenomics/adam/adam-core-spark2_2.10/0.20.1-SNAPSHOT/adam-core-spark2_2.10-0.20.1-SNAPSHOT.jar
Downloading: https://oss.sonatype.org/content/repositories/snapshots/org/bdgenomics/adam/adam-core-spark2_2.10/0.20.1-SNAPSHOT/adam-core-spark2_2.10-0.20.1-SNAPSHOT-tests.jar
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO] 
[INFO] ADAM_2.10 .......................................... SUCCESS [  4.373 s]
[INFO] ADAM_2.10: Core .................................... SUCCESS [02:32 min]
[INFO] ADAM_2.10: APIs for Java ........................... FAILURE [  0.905 s]
[INFO] ADAM_2.10: CLI ..................................... SKIPPED
[INFO] ADAM_2.10: Assembly ................................ SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 02:37 min
[INFO] Finished at: 2016-12-27T10:53:53-06:00
[INFO] Final Memory: 64M/1027M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal on project adam-apis_2.10: Could not resolve dependencies for project org.bdgenomics.adam:adam-apis_2.10:jar:0.20.1-SNAPSHOT: The following artifacts could not be resolved: org.bdgenomics.adam:adam-core-spark2_2.10:jar:0.20.1-SNAPSHOT, org.bdgenomics.adam:adam-core-spark2_2.10:jar:tests:0.20.1-SNAPSHOT: Could not find artifact org.bdgenomics.adam:adam-core-spark2_2.10:jar:0.20.1-SNAPSHOT in sonatype-nexus-snapshots (https://oss.sonatype.org/content/repositories/snapshots) -> [Help 1]

Note that the jar being installed is adam-core_2.10-0.20.1-SNAPSHOT.jar and not adam-core-spark2_2.10-0.20.1-SNAPSHOT.jar.

There must be something wrong with the move_to_spark_2.sh script.

@heuermh
Copy link
Member Author

@heuermh heuermh commented Dec 27, 2016

If from a clean working directory move_to_spark_2.sh is run before move_to_scala_2.11.sh, then the build succeeds

$ ./scripts/move_to_spark_2.sh 
$ mvn clean install
...
[INFO] --- maven-install-plugin:2.5.2:install (default-install) @ adam-core-spark2_2.10 ---
[INFO] Installing /Users/mheuer2/working/adam/adam-core/target/adam-core-spark2_2.10-0.20.1-SNAPSHOT.jar to /Users/mheuer2/.m2/repository/org/bdgenomics/adam/adam-core-spark2_2.10/0.20.1-SNAPSHOT/adam-core-spark2_2.10-0.20.1-SNAPSHOT.jar
[INFO] Installing /Users/mheuer2/working/adam/adam-core/pom.xml to /Users/mheuer2/.m2/repository/org/bdgenomics/adam/adam-core-spark2_2.10/0.20.1-SNAPSHOT/adam-core-spark2_2.10-0.20.1-SNAPSHOT.pom
[INFO] Installing /Users/mheuer2/working/adam/adam-core/target/adam-core-spark2_2.10-0.20.1-SNAPSHOT-tests.jar to /Users/mheuer2/.m2/repository/org/bdgenomics/adam/adam-core-spark2_2.10/0.20.1-SNAPSHOT/adam-core-spark2_2.10-0.20.1-SNAPSHOT-tests.jar
[INFO] Installing /Users/mheuer2/working/adam/adam-core/target/adam-core-spark2_2.10-0.20.1-SNAPSHOT-javadoc.jar to /Users/mheuer2/.m2/repository/org/bdgenomics/adam/adam-core-spark2_2.10/0.20.1-SNAPSHOT/adam-core-spark2_2.10-0.20.1-SNAPSHOT-javadoc.jar
[INFO]                                                                         
...
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO] 
[INFO] ADAM_2.10 .......................................... SUCCESS [  3.605 s]
[INFO] ADAM_2.10: Core .................................... SUCCESS [02:23 min]
[INFO] ADAM_2.10: APIs for Java ........................... SUCCESS [ 16.678 s]
[INFO] ADAM_2.10: CLI ..................................... SUCCESS [ 34.241 s]
[INFO] ADAM_2.10: Assembly ................................ SUCCESS [  7.412 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
fnothaft added a commit to fnothaft/adam that referenced this issue Dec 27, 2016
The sed command that moved to Spark 2 looked for _2.10 in the
ADAM pom artifact names. This commit fixes that issue by looking
for _2.1 instead, which matches both Scala 2.10 and 2.11, as well
as the eventual Scala 2.12. Resolves bigdatagenomics#1307.
heuermh added a commit that referenced this issue Dec 27, 2016
The sed command that moved to Spark 2 looked for _2.10 in the
ADAM pom artifact names. This commit fixes that issue by looking
for _2.1 instead, which matches both Scala 2.10 and 2.11, as well
as the eventual Scala 2.12. Resolves #1307.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Linked pull requests

Successfully merging a pull request may close this issue.

1 participant
You can’t perform that action at this time.