Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use Scala modules for 2.11 #3

Closed
dragos opened this issue Aug 29, 2013 · 14 comments
Closed

Use Scala modules for 2.11 #3

dragos opened this issue Aug 29, 2013 · 14 comments

Comments

@dragos
Copy link
Contributor

dragos commented Aug 29, 2013

Since 2.11.0-M4, Scala cross-publishes scala-xml and scala-combinator-parsers as separate modules under org.scala-lang.modules. The dbuild script should reflect that.

@jsuereth
Copy link
Contributor

Hmm, I may need some more info. What does the cross versioning look like for the snapshot scala?

@adriaanm
Copy link
Contributor

as soon as scala/scala#2855 is merged, scala snapshots will no longer contain scala-xml nor scala-parser-combinators

@jsuereth
Copy link
Contributor

right, how/where can I get them?

@adriaanm
Copy link
Contributor

libraryDependencies += "org.scala-lang.modules" %% "scala-xml"                     % "1.0-RC3"
libraryDependencies += "org.scala-lang.modules" %% "scala-parser-combinators"      % "1.0-RC1"

@jsuereth
Copy link
Contributor

it's the %% that has me worried for these integration builds. I assume everything is fine in testing, so I'll try to update this in the dbuild scripts.

@adriaanm
Copy link
Contributor

Well, the scripts override the binary version of org.scala-lang.modules to be the one they're published under.
Have a look at scala/jenkins-scripts#23 for the specifics.

@jsuereth
Copy link
Contributor

Yeah.... this is quite the rube goldberg device. I may need you to sit with my via skype sometime and walk me through what we're doing here. It seems I've been too far removed, and this hackery has advanced sufficiently beyond my ability to reason through it quickly. Either that or I need more coffee.

@adriaanm
Copy link
Contributor

I agree we must simplify these scripts. If the the sbt experts can't parse them, we're in trouble.

@jsuereth
Copy link
Contributor

@adriaanm

[warn]      ::::::::::::::::::::::::::::::::::::::::::::::
[warn]      ::          UNRESOLVED DEPENDENCIES         ::
[warn]      ::::::::::::::::::::::::::::::::::::::::::::::
[warn]      :: org.scala-lang#scala-parser-combinators_2.11.0-SNAPSHOT;1.0-RC1: not found```

Combined with:

[warn]      ::::::::::::::::::::::::::::::::::::::::::::::
[warn]      ::          UNRESOLVED DEPENDENCIES         ::
[warn]      ::::::::::::::::::::::::::::::::::::::::::::::
[warn]      :: org.scala-lang#scala-parser-combinators_2.11.0-SNAPSHOT;1.0-SNAPSHOT: not found
[warn]      ::::::::::::::::::::::::::::::::::::::::::::::```

When building, if I have to jump through loops to figure out which versions you're publishing and which you aren't, we're in trouble. Again, if everything were in dbuild, great. Since the IDE team doesn't want us to rebuild Scala + Modules in our build (which would keep our build safer), I need to know how you plan to publish-release these modules.

MAYBE, (if you're not already doing it), could we include the sbt building + publishing in your current dbuild?

If not, maybe we can switch to using dbuilds cache between builds and we can alter our build to "build scala" unless it's found in dbuild's "is the SHA already build?" cache. Then for PR validation we won't rebuild Scala, just grab the existing JARs from your other job.

In either case, the current solution leaves a lot to be desired. I do not want to have this issue reopen on every scala Milestone because we can't abstract out these module versions in some sane fashion.

@adriaanm
Copy link
Contributor

Just stay calm and embrace the modules.

Here's a list of the current versions we're using for each module (scala/scala#2855 will update them):
https://github.com/scala/scala/blob/master/versions.properties

A dbuild is planned for these modules soon. Going as fast as I can here.

We already build Scala only once per commit and reuse the artifacts.

We don't build the external modules during Scala PR validation yet, though. There's no need, as we're trying to experience what full binary compatibility would entail by building modules once every milestone, and enforcing binary compatibility between each release.

I've booked a meeting in your calendar next Tuesday to discuss this.

@jsuereth
Copy link
Contributor

Cool! Again, as long as I have a reasonable way to consume them.

Ideally, we can make dbuild aware of the results without having to use either Ivy or rebuild things, if we're using it initially. Look forward to the meeting.

@adriaanm
Copy link
Contributor

adriaanm commented Sep 5, 2013

Ugh! I accidentally deleted my comment. Could you paste it from your email? The gist:

I've started fixing this issue at https://github.com/adriaanm/sbt-builds-for-ide/blob/master/sbt-on-2.11.x by porting my old solution from scala/jenkins-scripts#23, so that I can merge scala/scala#2855.

It fails because the doc task can't find xml.

repro:

cd ~/git && git clone https://github.com/scala/jenkins-scripts.git
mkdir  /tmp/pr2855
cd /tmp/pr2855/
curl -sO https://scala-webapps.epfl.ch/jenkins/job/pr-scala-distpack/1464/artifact/scala/build.number
curl -sO https://scala-webapps.epfl.ch/jenkins/job/pr-scala-distpack/1464/artifact/scala/versions.properties
curl -#O https://scala-webapps.epfl.ch/jenkins/job/pr-scala-distpack/1464/artifact/jenkins-artifacts/maven.tgz
export sha=ce5c506ec3b815b8c97a2b938371154ab7dc668d
export DBUILDURL="https://github.com/adriaanm/sbt-builds-for-ide.git"
~/git/jenkins-scripts/job/pr-scala-integrate-ide 

@jsuereth
Copy link
Contributor

jsuereth commented Sep 5, 2013

Ok, got it:
@jsuereth, I've started porting scala/jenkins-scripts#23 to the dbuild approach in my fork (adriaanm/sbt-builds-for-ide) in anticipation of scala/scala#2855, but I'm stuck at an error I didn't see using the plain sbt approach from scala/jenkins-scripts#23:

To repro:

cd ~/git && git clone https://github.com/scala/jenkins-scripts.git
cd /tmp
mkdir pr2855
cd pr2855/
curl -sO https://scala-webapps.epfl.ch/jenkins/job/pr-scala-distpack/1464/artifact/scala/build.number
curl -sO https://scala-webapps.epfl.ch/jenkins/job/pr-scala-distpack/1464/artifact/scala/versions.properties
curl -#O https://scala-webapps.epfl.ch/jenkins/job/pr-scala-distpack/1464/artifact/jenkins-artifacts/maven.tgz
export sha=ce5c506ec3b815b8c97a2b938371154ab7dc668d
export DBUILDURL="https://github.com/adriaanm/sbt-builds-for-ide.git"
~/git/jenkins-scripts/job/pr-scala-integrate-ide
Fails as follows:

[info] [info] Set current project to SBinary Parent (in build file:/Users/adriaan/jenkins-staging/pr2855/sbt-builds-for-ide/target-0.6.4-josh-hack-1/project-builds/sbinary-74fbba46b1cd830f4f9280457afe5e36536b34bd/)
[info] [info] Defining core/:library-dependencies
[info] [info] The new value will be used by core/
:all-dependencies
[info] [info] Reapplying settings...
[info] [info] Set current project to SBinary Parent (in build file:/Users/adriaan/jenkins-staging/pr2855/sbt-builds-for-ide/target-0.6.4-josh-hack-1/project-builds/sbinary-74fbba46b1cd830f4f9280457afe5e36536b34bd/)
[info] [info] Updating dependencies...
[info] [info] Updating publishTo repo in one scope
[info] [info] Found publishMavenStyle in one scope; changing publishTo settings accordingly.
[info] [info] Updating version strings in 4 scopes
[info] [info] Adding resolvers to retrieve build artifacts in one scope
[info] [info] Updating dependencies in 3 scopes
[info] [info] Preparing Scala binaries: scala-library version 2.11.0-ce5c506-SNAPSHOT
[info] [info] Setting Scala version to: 2.11.0-ce5c506-SNAPSHOT in 4 scopes
[info] [info] Setting Scala instance in 3 scopes
[info] [info] Patching the inter-project resolver in 3 scopes
[info] [info] Disabling Scala binary checking in 3 scopes
[info] [info] Set current project to SBinary Parent (in build file:/Users/adriaan/jenkins-staging/pr2855/sbt-builds-for-ide/target-0.6.4-josh-hack-1/project-builds/sbinary-74fbba46b1cd830f4f9280457afe5e36536b34bd/)
[info] These subprojects will be built: core
[info] All Dependencies for subproject core:
[info] org.scala-lang:scala-library:2.11.0-ce5c506-SNAPSHOT
[info] net.sourceforge.fmpp:fmpp:0.9.14:fmpp
[info] org.scala-lang.modules:scala-xml:1.0-RC2
[info] Running build: core
[info] [info] Updating {file:/Users/adriaan/jenkins-staging/pr2855/sbt-builds-for-ide/target-0.6.4-josh-hack-1/project-builds/sbinary-74fbba46b1cd830f4f9280457afe5e36536b34bd/}core...
[info] [info] Wrote /Users/adriaan/jenkins-staging/pr2855/sbt-builds-for-ide/target-0.6.4-josh-hack-1/project-builds/sbinary-74fbba46b1cd830f4f9280457afe5e36536b34bd/core/target/scala-2.11/sbinary_2.11.0-ce5c506-SNAPSHOT-0.4.2-dbuildx74fbba46b1cd830f4f9280457afe5e36536b34bd.pom
[info] [info] Resolving org.scala-lang#scala-library;2.11.0-ce5c506-SNAPSHOT ...
[info] [info] Resolving org.scala-lang.modules#scala-xml_2.11.0-M4;1.0-RC2 ...
[info] [info] Resolving org.apache.ant#ant;1.8.4 ...
[info] [info] Resolving org.apache.ant#ant-parent;1.8.4 ...
[info] [info] Resolving org.apache.ant#ant-launcher;1.8.4 ...
[info] [info] Resolving org.apache.ant#ant-parent;1.8.4 ...
[info] [info] Resolving com.googlecode.java-diff-utils#diffutils;1.3.0 ...
[info] [info] Resolving org.sonatype.oss#oss-parent;7 ...
[info] [info] Resolving org.scala-lang#scalap;2.11.0-M4 ...
[info] [info] Resolving org.scala-lang#scala-compiler;2.11.0-ce5c506-SNAPSHOT ...
[info] [info] Resolving org.scala-lang.modules#scala-xml_2.11.0-M4;1.0-RC3 ...
[info] [info] Resolving org.scala-lang.modules#scala-parser-combinators_2.11.0-M4;1.0-RC1 ...
[info] [info] Resolving org.scala-lang#scala-reflect;2.11.0-ce5c506-SNAPSHOT ...
[info] [info] Resolving jline#jline;2.11 ...
[info] [info] Resolving org.sonatype.oss#oss-parent;7 ...
[info] [info] Resolving org.scalacheck#scalacheck_2.11.0-M4;1.10.1 ...
[info] [info] Resolving org.scala-tools.testing#test-interface;0.5 ...
[info] [info] Resolving org.scala-lang#scala-actors;2.11.0-M4 ...
[info] [info] Resolving org.scala-lang#scala-parser-combinators;2.11.0-M4 ...
[info] [info] Resolving org.scala-sbt#test-interface;1.0 ...
[info] [info] Resolving net.sourceforge.fmpp#fmpp;0.9.14 ...
[info] [info] Resolving org.freemarker#freemarker;[2.3.15,2.4) ...
[info] [info] Resolving org.sonatype.oss#oss-parent;7 ...
[info] [info] Resolving oro#oro;[2.0.8,) ...
[info] [info] Resolving org.beanshell#bsh;[1.3,2.1) ...
[info] [info] Resolving xml-resolver#xml-resolver;[1.1,) ...
[info] [info] Resolving org.apache#apache;3 ...
[info] [info] downloading http://repo1.maven.org/maven2/org/scala-lang/modules/scala-xml_2.11.0-M4/1.0-RC3/scala-xml_2.11.0-M4-1.0-RC3.jar ...
[info] [info] [SUCCESSFUL ] org.scala-lang.modules#scala-xml_2.11.0-M4;1.0-RC3!scala-xml_2.11.0-M4.jar (248ms)
[info] [info] downloading http://repo1.maven.org/maven2/org/scala-lang/modules/scala-parser-combinators_2.11.0-M4/1.0-RC1/scala-parser-combinators_2.11.0-M4-1.0-RC1.jar ...
[info] [info] [SUCCESSFUL ] org.scala-lang.modules#scala-parser-combinators_2.11.0-M4;1.0-RC1!scala-parser-combinators_2.11.0-M4.jar (109ms)
[info] [info] downloading file:/Users/adriaan/jenkins-staging/pr2855/sbt-builds-for-ide/target-0.6.4-josh-hack-1/project-builds/sbinary-74fbba46b1cd830f4f9280457afe5e36536b34bd/.dbuild/local-repo/org.scala-lang/scala-reflect/2.11.0-ce5c506-SNAPSHOT/jars/scala-reflect.jar ...
[info] [info] [SUCCESSFUL ] org.scala-lang#scala-reflect;2.11.0-ce5c506-SNAPSHOT!scala-reflect.jar (5ms)
[info] [info] downloading http://repo1.maven.org/maven2/jline/jline/2.11/jline-2.11.jar ...
[info] [info] [SUCCESSFUL ] jline#jline;2.11!jline.jar (59ms)
[info] [info] downloading http://repo1.maven.org/maven2/net/sourceforge/fmpp/fmpp/0.9.14/fmpp-0.9.14.jar ...
[info] [info] [SUCCESSFUL ] net.sourceforge.fmpp#fmpp;0.9.14!fmpp.jar (81ms)
[info] [info] downloading http://repo1.maven.org/maven2/org/freemarker/freemarker/2.3.20/freemarker-2.3.20.jar ...
[info] [info] [SUCCESSFUL ] org.freemarker#freemarker;2.3.20!freemarker.jar (219ms)
[info] [info] downloading http://repo1.maven.org/maven2/oro/oro/2.0.8/oro-2.0.8.jar ...
[info] [info] [SUCCESSFUL ] oro#oro;2.0.8!oro.jar (39ms)
[info] [info] downloading http://repo1.maven.org/maven2/org/beanshell/bsh/2.0b5/bsh-2.0b5.jar ...
[info] [info] [SUCCESSFUL ] org.beanshell#bsh;2.0b5!bsh.jar (102ms)
[info] [info] downloading http://repo1.maven.org/maven2/xml-resolver/xml-resolver/1.2/xml-resolver-1.2.jar ...
[info] [info] [SUCCESSFUL ] xml-resolver#xml-resolver;1.2!xml-resolver.jar (42ms)
[info] [info] Done updating.
[info] [info] Running fmpp.tools.CommandLine -U all -S /Users/adriaan/jenkins-staging/pr2855/sbt-builds-for-ide/target-0.6.4-josh-hack-1/project-builds/sbinary-74fbba46b1cd830f4f9280457afe5e36536b34bd/core/src -O /Users/adriaan/jenkins-staging/pr2855/sbt-builds-for-ide/target-0.6.4-josh-hack-1/project-builds/sbinary-74fbba46b1cd830f4f9280457afe5e36536b34bd/core/target/scala-2.11/src_managed/main --ignore-temporary-files /Users/adriaan/jenkins-staging/pr2855/sbt-builds-for-ide/target-0.6.4-josh-hack-1/project-builds/sbinary-74fbba46b1cd830f4f9280457afe5e36536b34bd/core/src/defaultprotocol.scala /Users/adriaan/jenkins-staging/pr2855/sbt-builds-for-ide/target-0.6.4-josh-hack-1/project-builds/sbinary-74fbba46b1cd830f4f9280457afe5e36536b34bd/core/src/generic.scala /Users/adriaan/jenkins-staging/pr2855/sbt-builds-for-ide/target-0.6.4-josh-hack-1/project-builds/sbinary-74fbba46b1cd830f4f9280457afe5e36536b34bd/core/src/io.scala /Users/adriaan/jenkins-staging/pr2855/sbt-builds-for-ide
/target-
0.6.4-josh-hack-1/project-builds/sbinary-74fbba46b1cd830f4f9280457afe5e36536b34bd/core/src/javaprotocol.scala /Users/adriaan/jenkins-staging/pr2855/sbt-builds-for-ide/target-0.6.4-josh-hack-1/project-builds/sbinary-74fbba46b1cd830f4f9280457afe5e36536b34bd/core/src/operations.scala /Users/adriaan/jenkins-staging/pr2855/sbt-builds-for-ide/target-0.6.4-josh-hack-1/project-builds/sbinary-74fbba46b1cd830f4f9280457afe5e36536b34bd/core/src/protocol.scala /Users/adriaan/jenkins-staging/pr2855/sbt-builds-for-ide/target-0.6.4-josh-hack-1/project-builds/sbinary-74fbba46b1cd830f4f9280457afe5e36536b34bd/core/src/standardtypes.scala
[info] - Executing: defaultprotocol.scala
[info] - Executing: generic.scala
[info] - Executing: io.scala
[info] - Executing: javaprotocol.scala
[info] - Executing: operations.scala
[info] - Executing: protocol.scala
[info] - Executing: standardtypes.scala
[info]
[info] *** DONE ***
[info]
[info] 7 executed + 0 rendered + 0 copied = 7 successfully processed
[info] 0 failed, 0 warning(s)
[info] Time elapsed: 0.151 seconds
[info]
[info] [info] Compiling 7 Scala sources to /Users/adriaan/jenkins-staging/pr2855/sbt-builds-for-ide/target-0.6.4-josh-hack-1/project-builds/sbinary-74fbba46b1cd830f4f9280457afe5e36536b34bd/core/target/scala-2.11/classes...
[info] [info] Packaging /Users/adriaan/jenkins-staging/pr2855/sbt-builds-for-ide/target-0.6.4-josh-hack-1/project-builds/sbinary-74fbba46b1cd830f4f9280457afe5e36536b34bd/core/target/scala-2.11/sbinary_2.11.0-ce5c506-SNAPSHOT-0.4.2-dbuildx74fbba46b1cd830f4f9280457afe5e36536b34bd-sources.jar ...
[info] [info] Generating Scala API documentation for main sources to /Users/adriaan/jenkins-staging/pr2855/sbt-builds-for-ide/target-0.6.4-josh-hack-1/project-builds/sbinary-74fbba46b1cd830f4f9280457afe5e36536b34bd/core/target/scala-2.11/api...
[info] [info] 'compiler-interface' not yet compiled for Scala 2.11.0-20130829-200905-ce5c506ec3. Compiling...
[info] [info] Done packaging.
[info] [info] Compilation completed in 12.602 s
[info] [warn] there were 7 feature warning(s); re-run with -feature for details
[info] model contains 43 documentable templates
[info] [warn] there were 8 deprecation warning(s); re-run with -deprecation for details
[info] [warn] there were 7 feature warning(s); re-run with -feature for details
[info] [warn] two warnings found
[info] [info] Packaging /Users/adriaan/jenkins-staging/pr2855/sbt-builds-for-ide/target-0.6.4-josh-hack-1/project-builds/sbinary-74fbba46b1cd830f4f9280457afe5e36536b34bd/core/target/scala-2.11/sbinary_2.11.0-ce5c506-SNAPSHOT-0.4.2-dbuildx74fbba46b1cd830f4f9280457afe5e36536b34bd.jar ...
[info] [info] Done packaging.
[info] java.lang.NoClassDefFoundError: scala/xml/NamespaceBinding
[info] at scala.tools.nsc.doc.html.HtmlFactory.generate(HtmlFactory.scala:126)
[info] at scala.tools.nsc.doc.html.Doclet.generateImpl(Doclet.scala:16)
[info] at scala.tools.nsc.doc.doclet.Generator.generate(Generator.scala:24)
[info] at scala.tools.nsc.doc.DocFactory.generate$1(DocFactory.scala:121)
[info] at scala.tools.nsc.doc.DocFactory.document(DocFactory.scala:124)
[info] at xsbt.Runner.run(ScaladocInterface.scala:30)
[info] at xsbt.ScaladocInterface.run(ScaladocInterface.scala:11)
[info] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[info] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
[info] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
[info] at java.lang.reflect.Method.invoke(Method.java:597)
[info] at sbt.compiler.AnalyzingCompiler.call(AnalyzingCompiler.scala:73)
[info] at sbt.compiler.AnalyzingCompiler.doc(AnalyzingCompiler.scala:54)
[info] at sbt.compiler.AnalyzingCompiler.doc(AnalyzingCompiler.scala:50)
[info] at sbt.Scaladoc$$anonfun$apply$3.apply(Doc.scala:62)
[info] at sbt.Scaladoc$$anonfun$apply$3.apply(Doc.scala:62)
[info] at sbt.Doc$class.generate(Doc.scala:36)

@jsuereth
Copy link
Contributor

jsuereth commented Sep 9, 2013

Ok, this should be fixed in the new 2.11 nightlies.

@jsuereth jsuereth closed this as completed Sep 9, 2013
johan-bjareholt added a commit to qvantel/orcd-dbconnector that referenced this issue Feb 28, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants