Update move_to_spark_2.sh #1316

Closed
wants to merge 1 commit into
from

Conversation

Projects
None yet
5 participants
@creggian

Fix adam-parent_2.11 SNAPSHOT version

In maven repository, adam-parent-spark2_2.11 is available only in version 0.20.0 and not in version 0.20.1 as coded in the pom.xml

Link:
https://mvnrepository.com/artifact/org.bdgenomics.adam/adam-parent-spark2_2.11

Update move_to_spark_2.sh
fix adam-parent_2.11 SNAPSHOT version
@AmplabJenkins

This comment has been minimized.

Show comment
Hide comment
@AmplabJenkins

AmplabJenkins Dec 14, 2016

Can one of the admins verify this patch?

Can one of the admins verify this patch?

@fnothaft

This comment has been minimized.

Show comment
Hide comment
@fnothaft

fnothaft Dec 14, 2016

Member

Thank you very much for opening the issue, and the patch @creggian! I will take a look over the patch today.

Perhaps I'm missing the issue here?

@heuermh This PR addresses SNAPSHOTs, not pushed releases:

https://oss.sonatype.org/content/repositories/snapshots/org/bdgenomics/adam/

Member

fnothaft commented Dec 14, 2016

Thank you very much for opening the issue, and the patch @creggian! I will take a look over the patch today.

Perhaps I'm missing the issue here?

@heuermh This PR addresses SNAPSHOTs, not pushed releases:

https://oss.sonatype.org/content/repositories/snapshots/org/bdgenomics/adam/

@creggian

This comment has been minimized.

Show comment
Hide comment
@creggian

creggian Dec 14, 2016

In the default pom.xml, adam-parent-spark2_2.11 version is set to "0.20.1" (line 13). I was not able to install ADAM (git clone & mvn clean package -DskipTests), because maven is not able to retrieve 0.20.1 version of the package from the onlire repository. The only version available is 0.20.0, as you can see form the link provided. So I changed version to 0.20.0 so to successfully install it.

*https://mvnrepository.com/artifact/org.bdgenomics.adam/adam-parent-spark2_2.11

Given the links you provided, version 0.20.1 is not provided in any of the artifacts, so there is actually a better fix. Update pom.xml instead of scripts/move_to_spark_2.sh.

I'll make another pull request with the updated pom.xml, and you can close/reject this one

In the default pom.xml, adam-parent-spark2_2.11 version is set to "0.20.1" (line 13). I was not able to install ADAM (git clone & mvn clean package -DskipTests), because maven is not able to retrieve 0.20.1 version of the package from the onlire repository. The only version available is 0.20.0, as you can see form the link provided. So I changed version to 0.20.0 so to successfully install it.

*https://mvnrepository.com/artifact/org.bdgenomics.adam/adam-parent-spark2_2.11

Given the links you provided, version 0.20.1 is not provided in any of the artifacts, so there is actually a better fix. Update pom.xml instead of scripts/move_to_spark_2.sh.

I'll make another pull request with the updated pom.xml, and you can close/reject this one

@heuermh

This comment has been minimized.

Show comment
Hide comment
@fnothaft

This comment has been minimized.

Show comment
Hide comment
@fnothaft

fnothaft Dec 14, 2016

Member

Er, actually, I see what you're saying @heuermh. @creggian are you looking for the 0.20.1-SNAPSHOT artifacts, or a 0.20.1 release? To @heuermh's point, we haven't made a 0.20.1 release yet. We are working on a 0.21.0 release which should drop very soon. If you want the latest SNAPSHOT artifacts (i.e., adam version = 0.20.1-SNAPSHOT), you'll need to tell Maven to look at the oss.sonatype.org snapshot repository. You can do this by adding the following configuration to your pom.xml:

  <repositories>
    <repository>
      <id>Sonatype</id>
      <url>http://oss.sonatype.org/content/repositories/snapshots/</url>
    </repository>
  </repositories>
Member

fnothaft commented Dec 14, 2016

Er, actually, I see what you're saying @heuermh. @creggian are you looking for the 0.20.1-SNAPSHOT artifacts, or a 0.20.1 release? To @heuermh's point, we haven't made a 0.20.1 release yet. We are working on a 0.21.0 release which should drop very soon. If you want the latest SNAPSHOT artifacts (i.e., adam version = 0.20.1-SNAPSHOT), you'll need to tell Maven to look at the oss.sonatype.org snapshot repository. You can do this by adding the following configuration to your pom.xml:

  <repositories>
    <repository>
      <id>Sonatype</id>
      <url>http://oss.sonatype.org/content/repositories/snapshots/</url>
    </repository>
  </repositories>
@fnothaft

This comment has been minimized.

Show comment
Hide comment
@fnothaft

fnothaft Dec 14, 2016

Member

There are no spark2_2.10 versions however.

Yeah, I was just noticing that...

Member

fnothaft commented Dec 14, 2016

There are no spark2_2.10 versions however.

Yeah, I was just noticing that...

@heuermh

This comment has been minimized.

Show comment
Hide comment
@heuermh

heuermh Dec 14, 2016

Member

@creggian Sorry, are you saying if you

$ git clone https://github.com/bigdatagenomics/adam.git
$ cd adam
$ ./scripts/move_to_spark_2.sh
$ mvn install

the build complains about missing version 0.20.1 dependencies? If so maybe something is wrong with the move_to_spark_2.sh script. In any case, that script shouldn't change the artifact versions, that is handled by the release script.

Member

heuermh commented Dec 14, 2016

@creggian Sorry, are you saying if you

$ git clone https://github.com/bigdatagenomics/adam.git
$ cd adam
$ ./scripts/move_to_spark_2.sh
$ mvn install

the build complains about missing version 0.20.1 dependencies? If so maybe something is wrong with the move_to_spark_2.sh script. In any case, that script shouldn't change the artifact versions, that is handled by the release script.

@creggian

This comment has been minimized.

Show comment
Hide comment
@creggian

creggian Dec 14, 2016

@fnothaft, I was discussing 0.20.1-SNAPSHOT.

By the way, I'm not able to reproduce the error. With

$ git clone https://github.com/bigdatagenomics/adam.git
$ cd adam
$ ./scripts/move_to_spark_2.sh
$ export MAVEN_OPTS="-Xmx512m -XX:MaxPermSize=256m"
$ mvn clean package -DskipTests

and

$ git clone https://github.com/bigdatagenomics/adam.git
$ cd adam
$ ./scripts/move_to_spark_2.sh
$ ./scripts/move_to_scala_2.11.sh
$ export MAVEN_OPTS="-Xmx512m -XX:MaxPermSize=256m"
$ mvn clean package -DskipTests

the build is successful in both cases.

So, sorry for the pull request, it is not meaningful anymore

@fnothaft, I was discussing 0.20.1-SNAPSHOT.

By the way, I'm not able to reproduce the error. With

$ git clone https://github.com/bigdatagenomics/adam.git
$ cd adam
$ ./scripts/move_to_spark_2.sh
$ export MAVEN_OPTS="-Xmx512m -XX:MaxPermSize=256m"
$ mvn clean package -DskipTests

and

$ git clone https://github.com/bigdatagenomics/adam.git
$ cd adam
$ ./scripts/move_to_spark_2.sh
$ ./scripts/move_to_scala_2.11.sh
$ export MAVEN_OPTS="-Xmx512m -XX:MaxPermSize=256m"
$ mvn clean package -DskipTests

the build is successful in both cases.

So, sorry for the pull request, it is not meaningful anymore

@creggian creggian closed this Dec 14, 2016

@heuermh

This comment has been minimized.

Show comment
Hide comment
@heuermh

heuermh Dec 14, 2016

Member

No problem! Thank you for looking into this.

We find the rewriting pom files in place rather sketchy, but couldn't come up with a better alternative. If interested, you might be able to find a thread on spark-dev@ from last summer.

Member

heuermh commented Dec 14, 2016

No problem! Thank you for looking into this.

We find the rewriting pom files in place rather sketchy, but couldn't come up with a better alternative. If interested, you might be able to find a thread on spark-dev@ from last summer.

@ryan-williams

This comment has been minimized.

Show comment
Hide comment
@ryan-williams

ryan-williams Dec 14, 2016

Member

I have an alternative in my SBT fork / using hammerlab/sbt-parent, where you use -Dspark1 to build against Spark 1.x (default builds against Spark 2.x):

sbt +publishSigned sonatypeRelease           // release {Spark 2.x} ⅹ {2.10,2.11} artifacts.
sbt -Dspark1 +publishSigned sonatypeRelease  // release {Spark 1.x} ⅹ {2.10,2.11} artifacts.

Hopefully I'll have that in evangelization-worthy shape soon.

Member

ryan-williams commented Dec 14, 2016

I have an alternative in my SBT fork / using hammerlab/sbt-parent, where you use -Dspark1 to build against Spark 1.x (default builds against Spark 2.x):

sbt +publishSigned sonatypeRelease           // release {Spark 2.x} ⅹ {2.10,2.11} artifacts.
sbt -Dspark1 +publishSigned sonatypeRelease  // release {Spark 1.x} ⅹ {2.10,2.11} artifacts.

Hopefully I'll have that in evangelization-worthy shape soon.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment