Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update move_to_spark_2.sh #1316

Closed
wants to merge 1 commit into from
Closed

Conversation

@creggian
Copy link

creggian commented Dec 14, 2016

Fix adam-parent_2.11 SNAPSHOT version

In maven repository, adam-parent-spark2_2.11 is available only in version 0.20.0 and not in version 0.20.1 as coded in the pom.xml

Link:
https://mvnrepository.com/artifact/org.bdgenomics.adam/adam-parent-spark2_2.11

fix adam-parent_2.11 SNAPSHOT version
@AmplabJenkins
Copy link

AmplabJenkins commented Dec 14, 2016

Can one of the admins verify this patch?

@fnothaft
Copy link
Member

fnothaft commented Dec 14, 2016

Thank you very much for opening the issue, and the patch @creggian! I will take a look over the patch today.

Perhaps I'm missing the issue here?

@heuermh This PR addresses SNAPSHOTs, not pushed releases:

https://oss.sonatype.org/content/repositories/snapshots/org/bdgenomics/adam/

@creggian
Copy link
Author

creggian commented Dec 14, 2016

In the default pom.xml, adam-parent-spark2_2.11 version is set to "0.20.1" (line 13). I was not able to install ADAM (git clone & mvn clean package -DskipTests), because maven is not able to retrieve 0.20.1 version of the package from the onlire repository. The only version available is 0.20.0, as you can see form the link provided. So I changed version to 0.20.0 so to successfully install it.

*https://mvnrepository.com/artifact/org.bdgenomics.adam/adam-parent-spark2_2.11

Given the links you provided, version 0.20.1 is not provided in any of the artifacts, so there is actually a better fix. Update pom.xml instead of scripts/move_to_spark_2.sh.

I'll make another pull request with the updated pom.xml, and you can close/reject this one

@heuermh
Copy link
Member

heuermh commented Dec 14, 2016

@fnothaft
Copy link
Member

fnothaft commented Dec 14, 2016

Er, actually, I see what you're saying @heuermh. @creggian are you looking for the 0.20.1-SNAPSHOT artifacts, or a 0.20.1 release? To @heuermh's point, we haven't made a 0.20.1 release yet. We are working on a 0.21.0 release which should drop very soon. If you want the latest SNAPSHOT artifacts (i.e., adam version = 0.20.1-SNAPSHOT), you'll need to tell Maven to look at the oss.sonatype.org snapshot repository. You can do this by adding the following configuration to your pom.xml:

  <repositories>
    <repository>
      <id>Sonatype</id>
      <url>http://oss.sonatype.org/content/repositories/snapshots/</url>
    </repository>
  </repositories>
@fnothaft
Copy link
Member

fnothaft commented Dec 14, 2016

There are no spark2_2.10 versions however.

Yeah, I was just noticing that...

@heuermh
Copy link
Member

heuermh commented Dec 14, 2016

@creggian Sorry, are you saying if you

$ git clone https://github.com/bigdatagenomics/adam.git
$ cd adam
$ ./scripts/move_to_spark_2.sh
$ mvn install

the build complains about missing version 0.20.1 dependencies? If so maybe something is wrong with the move_to_spark_2.sh script. In any case, that script shouldn't change the artifact versions, that is handled by the release script.

@creggian
Copy link
Author

creggian commented Dec 14, 2016

@fnothaft, I was discussing 0.20.1-SNAPSHOT.

By the way, I'm not able to reproduce the error. With

$ git clone https://github.com/bigdatagenomics/adam.git
$ cd adam
$ ./scripts/move_to_spark_2.sh
$ export MAVEN_OPTS="-Xmx512m -XX:MaxPermSize=256m"
$ mvn clean package -DskipTests

and

$ git clone https://github.com/bigdatagenomics/adam.git
$ cd adam
$ ./scripts/move_to_spark_2.sh
$ ./scripts/move_to_scala_2.11.sh
$ export MAVEN_OPTS="-Xmx512m -XX:MaxPermSize=256m"
$ mvn clean package -DskipTests

the build is successful in both cases.

So, sorry for the pull request, it is not meaningful anymore

@creggian creggian closed this Dec 14, 2016
@heuermh
Copy link
Member

heuermh commented Dec 14, 2016

No problem! Thank you for looking into this.

We find the rewriting pom files in place rather sketchy, but couldn't come up with a better alternative. If interested, you might be able to find a thread on spark-dev@ from last summer.

@ryan-williams
Copy link
Member

ryan-williams commented Dec 14, 2016

I have an alternative in my SBT fork / using hammerlab/sbt-parent, where you use -Dspark1 to build against Spark 1.x (default builds against Spark 2.x):

sbt +publishSigned sonatypeRelease           // release {Spark 2.x} ⅹ {2.10,2.11} artifacts.
sbt -Dspark1 +publishSigned sonatypeRelease  // release {Spark 1.x} ⅹ {2.10,2.11} artifacts.

Hopefully I'll have that in evangelization-worthy shape soon.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

5 participants
You can’t perform that action at this time.