Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Rework build to allow publishing core to maven #3069

Merged
merged 15 commits into from Oct 29, 2013

Conversation

adriaanm
Copy link
Contributor

ant publish-core-signed-opt-nodocs creates and publishes the smallest
possible set of jars (library/reflect/compiler) we can send to maven.
It uploads jars with class files and sources, but no docs
(so that we don't need to build scaladoc or any other modules).

ant publish-core-signed-opt is similar, but also publishes docs.

For local experimentation: ant publish-core-local-nodocs.
With a little bit more work, this can be used instead of replacestarr.

This should allow breaking cycles in a principled way in dbuild:

  1. build the Scala core (library/reflect/compiler), publish locally
    Building the core doesn't have any dependencies on the new modules (it's similar to locker.done).
    (Don't test --> no need for partest, skip docs -> ignore soon-to-be-module scaladoc.)
  2. Use this Scala core to build all modules (especially partest and scaladoc and all its dependencies)
  3. Use partest and scaladoc from step 2. to test and document the core and all modules
  4. Publish tested, documented Scala core and modules to maven.

The next step is to build and publish a modularized scaladoc in this build,
and support using a maven-resolved scaladoc to generate docs.

Review by everyone: @gkossakowski, @retronym, @cunei, @jsuereth, @xeno-by, @paulp.

It tended too easily to get out of synch with build.xml.
Publish to maven with `ant publish.local`,
`ant publish`, or `ant publish.signed`.

For now, we must keep copying `src/build/maven/maven-deploy.xml`
to `dists/maven/latest.build.xml`, along with its dependencies,
as it's used by jenkins jobs and PR validation.

TODO: `inline src/build/maven/maven-deploy.xml` into `build.xml`.
Use `unless` attribute in targets so that `docs.skip` influences dependency graph.

Create `scaladoc` task in `pack.core` so that we don't need `pack.done`
for publishing the core.

Move source bundle creation from `dist.src` to `osgi.core`/`osgi.done`.

Split dependencies of maven publishing into core/all variants:
`osgi.core` creates osgi bundles for just lib/reflect/comp.
`ant publish-core-signed-opt-nodocs` creates and publishes the smallest
possible set of jars (library/reflect/compiler) we can send to maven.
It uploads jars with class files and sources, but no docs
(so that we don't need to build scaladoc or any other modules).

`ant publish-core-signed-opt` is similar, but also publishes docs.

For local experimentation: `ant publish-core-local-nodocs`.
With a little bit more work, this can be used instead of `replacestarr`.
@retronym
Copy link
Member

Perhaps you could trigger a few builds that aren't part of the PR validation and link the results here, e.g.

@retronym
Copy link
Member

I just tried this, and the artifact:mvn task logged errors (I think due to GPG, I got a popup window to enter my passphrase for my RSA key during the process.)

But, the overall build claimed to be successful.

% ant publish-core-signed-opt-nodocs
...
[artifact:mvn] 35K downloaded
[artifact:mvn] [INFO] ------------------------------------------------------------------------
[artifact:mvn] [INFO] Building Maven Default Project
[artifact:mvn] [INFO]    task-segment: [org.apache.maven.plugins:maven-gpg-plugin:1.3:sign-and-deploy-file] (aggregator-style)
[artifact:mvn] [INFO] ------------------------------------------------------------------------
[artifact:mvn] [INFO] [gpg:sign-and-deploy-file]
[artifact:mvn] [INFO] Retrieving previous build number from sonatype-nexus
[artifact:mvn] [INFO] ------------------------------------------------------------------------
[artifact:mvn] [ERROR] BUILD ERROR
[artifact:mvn] [INFO] ------------------------------------------------------------------------
[artifact:mvn] [INFO] Error retrieving previous build number for artifact 'org.scala-lang:scala-compiler:jar': repository metadata for: 'snapshot org.scala-lang:scala-compiler:2.11.0-SNAPSHOT' could not be retrieved from repository: sonatype-nexus due to an error: Error transferring file
[artifact:mvn]
[artifact:mvn] the trustAnchors parameter must be non-empty
[artifact:mvn] [INFO] ------------------------------------------------------------------------
[artifact:mvn] [INFO] For more information, run Maven with the -e switch
[artifact:mvn] [INFO] ------------------------------------------------------------------------
[artifact:mvn] [INFO] Total time: 44 seconds
[artifact:mvn] [INFO] Finished at: Wed Oct 23 23:37:00 CEST 2013
[artifact:mvn] [INFO] Final Memory: 42M/549M
[artifact:mvn] [INFO] ------------------------------------------------------------------------
[artifact:mvn] Java Result: 1
     [echo] Deploying /Users/jason/code/scala3/dists/maven/2.11.0-20131022-170325-d925c2155d/scala-library/scala-library-[pom.xml|src.jar|docs.jar].
     [copy] Copying 1 file to /Users/jason/code/scala3/dists/maven/2.11.0-20131022-170325-d925c2155d/scala-library
[artifact:mvn] [INFO] Scanning for projects...
[artifact:mvn] [INFO] ------------------------------------------------------------------------
[artifact:mvn] [INFO] Building Maven Default Project
[artifact:mvn] [INFO]    task-segment: [org.apache.maven.plugins:maven-gpg-plugin:1.3:sign-and-deploy-file] (aggregator-style)
[artifact:mvn] [INFO] ------------------------------------------------------------------------
[artifact:mvn] [INFO] [gpg:sign-and-deploy-file]
[artifact:mvn] [INFO] Retrieving previous build number from sonatype-nexus
[artifact:mvn] [INFO] ------------------------------------------------------------------------
[artifact:mvn] [ERROR] BUILD ERROR
[artifact:mvn] [INFO] ------------------------------------------------------------------------
[artifact:mvn] [INFO] Error retrieving previous build number for artifact 'org.scala-lang:scala-library:jar': repository metadata for: 'snapshot org.scala-lang:scala-library:2.11.0-SNAPSHOT' could not be retrieved from repository: sonatype-nexus due to an error: Error transferring file
[artifact:mvn]
[artifact:mvn] the trustAnchors parameter must be non-empty
[artifact:mvn] [INFO] ------------------------------------------------------------------------
[artifact:mvn] [INFO] For more information, run Maven with the -e switch
[artifact:mvn] [INFO] ------------------------------------------------------------------------
[artifact:mvn] [INFO] Total time: < 1 second
[artifact:mvn] [INFO] Finished at: Wed Oct 23 23:37:01 CEST 2013
[artifact:mvn] [INFO] Final Memory: 46M/550M
[artifact:mvn] [INFO] ------------------------------------------------------------------------
[artifact:mvn] Java Result: 1
     [echo] Deploying /Users/jason/code/scala3/dists/maven/2.11.0-20131022-170325-d925c2155d/scala-reflect/scala-reflect-[pom.xml|src.jar|docs.jar].
     [copy] Copying 1 file to /Users/jason/code/scala3/dists/maven/2.11.0-20131022-170325-d925c2155d/scala-reflect
[artifact:mvn] [INFO] Scanning for projects...
[artifact:mvn] [INFO] ------------------------------------------------------------------------
[artifact:mvn] [INFO] Building Maven Default Project
[artifact:mvn] [INFO]    task-segment: [org.apache.maven.plugins:maven-gpg-plugin:1.3:sign-and-deploy-file] (aggregator-style)
[artifact:mvn] [INFO] ------------------------------------------------------------------------
[artifact:mvn] [INFO] [gpg:sign-and-deploy-file]
[artifact:mvn] [INFO] Retrieving previous build number from sonatype-nexus
[artifact:mvn] [INFO] ------------------------------------------------------------------------
[artifact:mvn] [ERROR] BUILD ERROR
[artifact:mvn] [INFO] ------------------------------------------------------------------------
[artifact:mvn] [INFO] Error retrieving previous build number for artifact 'org.scala-lang:scala-reflect:jar': repository metadata for: 'snapshot org.scala-lang:scala-reflect:2.11.0-SNAPSHOT' could not be retrieved from repository: sonatype-nexus due to an error: Error transferring file
[artifact:mvn]
[artifact:mvn] the trustAnchors parameter must be non-empty
[artifact:mvn] [INFO] ------------------------------------------------------------------------
[artifact:mvn] [INFO] For more information, run Maven with the -e switch
[artifact:mvn] [INFO] ------------------------------------------------------------------------
[artifact:mvn] [INFO] Total time: 1 second
[artifact:mvn] [INFO] Finished at: Wed Oct 23 23:37:02 CEST 2013
[artifact:mvn] [INFO] Final Memory: 50M/550M
[artifact:mvn] [INFO] ------------------------------------------------------------------------
[artifact:mvn] Java Result: 1

BUILD SUCCESSFUL
Total time: 2 minutes 5 seconds
pr/3069 ~/code/scala3

@retronym
Copy link
Member

  • We should use:
diff --git a/src/build/maven/maven-deploy.xml b/src/build/maven/maven-deploy.xml
index 90bfaa4..9e4215d 100644
--- a/src/build/maven/maven-deploy.xml
+++ b/src/build/maven/maven-deploy.xml
@@ -63,7 +63,7 @@
         </then><else>
           <property name="repo" value="${local.repository}"/>
         </else></if>
-        <artifact:mvn>
+        <artifact:mvn failonerror="true">
           <arg value="org.apache.maven.plugins:maven-gpg-plugin:1.3:sign-and-deploy-file" />
           <arg value="-Durl=${repo}" />
           <arg value="-DrepositoryId=${repository.credentials.id}" />

@retronym
Copy link
Member

Incidentally, the certificate problem could be resolved on my machine with:

      <arg value="-Djavax.net.ssl.trustStore=/Library/Java/JavaVirtualMachines/jdk1.7.0_17.jdk/Contents/Home/jre/lib/security/cacerts"/>

That gives, as expected, a 401 uploading the new snapshot to Sonatype.

[artifact:mvn] [INFO] [gpg:sign-and-deploy-file]
[artifact:mvn] [INFO] Retrieving previous build number from sonatype-nexus
[artifact:mvn] Uploading: https://oss.sonatype.org/content/repositories/snapshots/org/scala-lang/scala-compiler/2.11.0-SNAPSHOT/scala-compiler-2.11.0-20131023.215222-435.jar
[artifact:mvn] 14537K uploaded
[artifact:mvn] [INFO] ------------------------------------------------------------------------
[artifact:mvn] [ERROR] BUILD ERROR
[artifact:mvn] [INFO] ------------------------------------------------------------------------
[artifact:mvn] [INFO] Error deploying artifact: Failed to transfer file: https://oss.sonatype.org/content/repositories/snapshots/org/scala-lang/scala-compiler/2.11.0-SNAPSHOT/scala-compiler-2.11.0-20131023.215222-435.jar. Return code is: 401

@ghost ghost assigned cunei Oct 25, 2013
@adriaanm
Copy link
Contributor Author

For the record, here's the diff to maven-deploy, modulo moving stuff around.
The only observable change is that docs are no longer published to the local publishing step when docs.skip is set.

diff --git 1/var/folders/xk/z9mv3_2d68l15z0lnqcnsgs40000gn/T/partXXX.k3PJVdGv 2/var/folders/xk/z9mv3_2d68l15z0lnqcnsgs40000gn/T/partXXX.k3PJVdGv2
index 8860897e55..1d3bf1e85c 100644
--- 1/var/folders/xk/z9mv3_2d68l15z0lnqcnsgs40000gn/T/partXXX.k3PJVdGv
+++ 2/var/folders/xk/z9mv3_2d68l15z0lnqcnsgs40000gn/T/partXXX.k3PJVdGv2
@@ -1,8 +1,9 @@
 <macrodef name="deploy-one">
   <attribute name="dir" default=""/>
   <attribute name="name" />
-  <attribute name="local"  default="false"/>
-  <attribute name="signed" default="false"/>
+  <attribute name="version" />
+  <attribute name="local" />
+  <attribute name="signed" />

   <sequential>
     <local name="path"/> <property name="path" value="@{dir}@{name}/@{name}"/>
@@ -11,7 +12,7 @@

     <copy file="${path}-pom.xml" tofile="${path}-pom-filtered.xml" overwrite="true">
       <filterset>
-        <filter token="VERSION" value="${maven.version.number}" />
+        <filter token="VERSION" value="@{version}" />
         <filter token="SCALA_BINARY_VERSION" value="${scala.binary.version}" />
         <filter token="XML_VERSION" value="${scala-xml.version.number}" />
         <filter token="PARSER_COMBINATORS_VERSION" value="${scala-parser-combinators.version.number}" />
@@ -43,7 +44,6 @@
             <artifact:localRepository path="${local.repository}"  id="${repository.credentials.id}" />
             <artifact:pom refid="@{name}.pom" />
             <artifact:attach type="jar" file="${path}-src.jar"  classifier="sources" />
-            <artifact:attach type="jar" file="${path}-docs.jar" classifier="javadoc" />
           </artifact:install>
         </else>
         </if>
@@ -71,17 +71,16 @@
 </macrodef>

 <macrodef name="deploy">
-  <attribute name="dir"    default=""/>
   <attribute name="local"  default="false"/>
   <attribute name="signed" default="false"/>

   <sequential>
-    <deploy-one dir="@{dir}"          name="scala-library"     local="@{local}" signed="@{signed}"/>
-    <deploy-one dir="@{dir}"          name="scala-reflect"     local="@{local}" signed="@{signed}"/>
-    <deploy-one dir="@{dir}"          name="scala-compiler"    local="@{local}" signed="@{signed}"/>
-    <deploy-one dir="@{dir}"          name="scala-actors"      local="@{local}" signed="@{signed}"/>
-    <deploy-one dir="@{dir}"          name="scala-swing"       local="@{local}" signed="@{signed}"/>
-    <deploy-one dir="@{dir}"          name="scalap"            local="@{local}" signed="@{signed}"/>
-    <deploy-one dir="@{dir}plugins/"  name="continuations"     local="@{local}" signed="@{signed}"/>
+    <deploy-one                name="scala-actors"   version="${maven.version.number}" local="@{local}" signed="@{signed}"/>
+    <deploy-one                name="scala-compiler" version="${maven.version.number}" local="@{local}" signed="@{signed}"/>
+    <deploy-one                name="scala-library"  version="${maven.version.number}" local="@{local}" signed="@{signed}"/>
+    <deploy-one                name="scala-reflect"  version="${maven.version.number}" local="@{local}" signed="@{signed}"/>
+    <deploy-one                name="scala-swing"    version="${maven.version.number}" local="@{local}" signed="@{signed}"/>
+    <deploy-one                name="scalap"         version="${maven.version.number}" local="@{local}" signed="@{signed}"/>
+    <deploy-one dir="plugins/" name="continuations"  version="${maven.version.number}" local="@{local}" signed="@{signed}"/>
   </sequential>
 </macrodef>

@retronym
Copy link
Member

LGTM.

Is tools/deploy-local-maven-snapshot now out of date or obsolete? I'm doing a cleanup job in there at the moment and noticed that one.

@paulp
Copy link
Contributor

paulp commented Oct 28, 2013

@retronym I expect it can go.

adriaanm added a commit that referenced this pull request Oct 29, 2013
Rework build to allow publishing core to maven
@adriaanm adriaanm merged commit 6d4b08b into scala:master Oct 29, 2013
@adriaanm adriaanm deleted the build-maven-publish branch October 29, 2013 02:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
5 participants