Upgrade tot Scala 2.11.0-M5 #3

Merged
merged 4 commits into from Sep 19, 2013

4 participants

@adriaanm
The Scala Programming Language member

review by @gkossakowski, /cc @jsuereth

note the

+// scalap depends on scala-compiler, which depends (for the scaladoc part) on scala-xml and scala-parser-combinators
+// more precisely, scala-compiler_2.11.0-M5 depends on
+//                      scala-xml_2.11.0-M4 and
+//       scala-parser-combinators_2.11.0-M4,
+// so that we get a binary version incompatibility warning
+// it isn't really a problem, but we should consider doing something about this
+// my preference: modularize scaladoc
+conflictWarning ~= { _.copy(failOnConflict = false) }
adriaanm added some commits Sep 7, 2013
@adriaanm adriaanm Use scalaVersion for scalap dependency version 43a377a
@adriaanm adriaanm Upgrade to Scala 2.11.0-M5 aadb0ef
@adriaanm adriaanm Upgrade to scala-xml 1.0-RC4 c9812f1
@adriaanm adriaanm update shouldn't fail on conflicting binary versions
scalap depends on scala-compiler, which depends (for the scaladoc part)
on scala-xml and scala-parser-combinators

More precisely, scala-compiler_2.11.0-M5 depends on
                     scala-xml_2.11.0-M4 and
      scala-parser-combinators_2.11.0-M4,
so that we get a binary version incompatibility warning.

It isn't really a problem, but we should consider doing something about this.

My preference: modularize scaladoc.
3facfba
@adriaanm
The Scala Programming Language member

once rickynils/scalacheck#53 is merged and scalacheck_2.11.0-M5%2.10.1 is published, this PR can be merged and a partest release cut

@jsuereth
The Scala Programming Language member

Yeah, no matter what we do here, we should coordinate scala modularization with sbt so that whatever semantics we need for BC are supported. I think the actual reality of how things were modularized is a bit unexpected (at least from my perspective). We may need some time to fix our tooling. However, as you mention, the error is innocuous for now.

@gkossakowski
The Scala Programming Language member

@jsuereth:
We do not work according to any master plan because we really do not know what kind of obstacles we are going to face. We discover them as we go.
Can you point out which specific aspect of modularization is unexpected?

@jsuereth
The Scala Programming Language member

Right, I'm not saying there is any master plan. Just pointing out that the reality of how the modules got moved out will require tooling support, or we have to deal with annoyances.

Primarily, we have an odd scenario: Primarily this Dependency chain:

scala-compiler M5 -> (scala-library M5, scala-xml_M4 RC, scala-parser-comibator_M4 RC)
scala-xml_M4 RC* -> scala-library M4

The entire "binary compatibility" hack we've encoded into Maven/sbt assumes that a single scala version can encode binary compatibility. It assumes (finally) that

  • release version numbers are numbers without additional gunk after X.Y.Z.
  • Major releases versions are compatible (i.e. X.Y)
  • Anything else is not.

SO, the way we've encoded modules (on top of utter hackery for encoding binary compatibility as a thing in maven) will confuse the hell out of existing tools. They're just not sophisticated enough to represent what you want to represent right now.

My point is that now that we know what you want the modules to be, we can agree the tooling is not ready to support this particular case. We have two options:

1) We deal with poor experience and confusing error messages. In this case, I think only people working on the modules, or during RC/M releases will notice.
2) We re-adjust our priorities and start coordinating with the tools teams so that the functionality we need is ready ASAP.

Nothing you guys have done has been outside what we discussed, but it has been outside expectations. This isn't unusual, happens every time. However, when people have different expectations, we need to take the time to re-synch on what we need, and we need to pull the tools along. Honestly, I was not expecting to externalize modules for which this circular dependency would exist. I thought they'd be stand-alone and scala's circular building would remain internal to the scala build. As that is not the case, we have some fallout. This is a bit of it. Let's make it happen.

@gkossakowski
The Scala Programming Language member

@jsuereth: I agree that dependencies of scala-compiler needs fixing. They are that way because of scaladoc which hasn't been split (yet) into a separate jar. We plan to do that for M6.

Is there anything else unexpected or potentially troublesome?

@jsuereth
The Scala Programming Language member

Yes. right now tools are unaware that scaladoc will be split out. let's do some cross-team coordination for the next release to make sure that the tools are ready for M6, otherwise there will be issues.

I'll set it up with Adriaan/Derek/etc.

@jrudolph

From the ML:

On Thu, Sep 12, 2013 at 11:08 PM, Paul Phillips paulp@improving.org wrote:
I have to dispute the "it isn't really a problem" characterization.

I want to add that this comment also doesn't come with an explanation why it isn't thought to be a problem. The explanation probably is that you usually 1) don't use the scaladoc component of the scala compiler and 2) scaladoc classes also aren't accidentally touched on classloading which could trigger incompatible class change errors on transitive classloading.

However, those are the dependencies of @paulp's example project:

[info]   +-org.scala-lang:scala-compiler:2.11.0-M5
[info]   | +-org.scala-lang.modules:scala-parser-combinators_2.11.0-M4:1.0-RC1
[info]   | | +-org.scala-lang:scala-library:2.11.0-M4 (evicted by: 2.11.0-M5)
[info]   | | +-org.scala-lang:scala-library:2.11.0-M5
[info]   | | 
[info]   | +-org.scala-lang.modules:scala-xml_2.11.0-M4:1.0-RC3
[info]   | | +-org.scala-lang:scala-library:2.11.0-M4 (evicted by: 2.11.0-M5)
[info]   | | +-org.scala-lang:scala-library:2.11.0-M5
[info]   | | 
[info]   | +-org.scala-lang:scala-library:2.11.0-M5
[info]   | +-org.scala-lang:scala-reflect:2.11.0-M5
[info]   |   +-org.scala-lang:scala-library:2.11.0-M5
[info]   |   
[info]   +-org.scala-lang:scala-library:2.11.0-M5
[info]   +-org.scalacheck:scalacheck_2.11.0-M5:1.10.1
[info]     +-org.scala-lang.modules:scala-parser-combinators_2.11.0-M5:1.0-RC2
[info]     | +-org.scala-lang:scala-library:2.11.0-M5
[info]     | 
[info]     +-org.scala-lang:scala-actors:2.11.0-M5
[info]     | +-org.scala-lang:scala-library:2.11.0-M5
[info]     | 
[info]     +-org.scala-lang:scala-library:2.11.0-M5
[info]     +-org.scala-tools.testing:test-interface:0.5
[info]     

See how ivy's default eviction policy (always take the latest version) works for scala-library because it doesn't use cross-built versioning but not for scala-xml and scala-parser-combinators. That means that in effect you now got two versions of parser-combinators and xml on the classpath. If they accidentally stayed completely unchanged between M4 and M5 everything will probably be fine, indeed. Otherwise, you will get strange results depending on the exact ordering of those libraries on the classpath and the exact compatibility of those libraries between M4 and M5.

So, what you actually need, in general, in addition to this fix is another fix which explicitly excludes the accidental versions of the components so that they cannot interfere which the ones you really need.

@jsuereth
The Scala Programming Language member

So I think perhaps the longer term fix is to ensure these two things:

  1. Nothing in the scala core libraries ever depends on a module
  2. No module can depend on any other module that is not in the same "binary compatibility generation" (i.e. 2.10.0-M5 vs 2.10.0-M4)

IIRC - The Scala team is aiming to pull scaladoc out of the scala-compiler which would remove this circular dependnecy between scala project and external modules which could help with (1), but (2) would still need "manual" enforcement.

Otherwise, you need to start listing excludes everywhere. Sounds pretty fun.

@gkossakowski
The Scala Programming Language member

@jsuereth: I'd like to point out that strictly speaking there are no circular dependencies because they are of different version. However, that's nitpicking: there's a real problem that @jrudolph explains very well.

We don't pretend there's no problem. All we say is that this is not problem big enough to block the release of the milestone so we went ahead and released it (or we are in the process of releasing it). We are well aware of the problem and we are going to mention it in release notes for M5. Also, @adriaanm wants to fix it in M6 by pulling out scaladoc.

@jsuereth: I agree with your analysis. I don't think the second bullet will be that hard to enforce manually. We really don't have that many artifacts and dependencies between them to check that manually. Also, if we stick to sbt's defaults then everything is fine because it does the right thing by default. We arrived at current situation because we (consciously) overrode sbt's defaults in order to be able to modularize.

Some intermittent breakage is expected in milestone releases and that's what we are observing here. Thanks to everyone for paying attention to this, though.

@adriaanm
The Scala Programming Language member

Thanks for the great point about cross-versioning messing up eviction of old versions, @jrudolph!
By "no problem" I meant it's not a fundamental problem with modularization: we can work around it by replicating the current build, which uses the current compiler to rebuild xml and parser-combinators, instead of using an old version, so the problem trivially disappears as all dependencies are on the same binary version of the compiler. We're currently working on setting up dbuild to do this in time for M6.

@adriaanm
The Scala Programming Language member

Going to merge this and release partest.

@adriaanm adriaanm merged commit 9869b5c into scala:master Sep 19, 2013
@jrudolph

we can work around it by replicating the current build, which uses the current compiler to rebuild xml and parser-combinators, instead of using an old version

I must admit I don't fully understand this explanation (probably because I don't no what the "current build" is). Wasn't the problem that the compiler itself depended on old versions of some of the modules? So, AFAIU the solution would have to be that the compiler has to be rebuilt to use the current versions of its dependencies in the same way as it always depends on the current version of scala-library, no? Or, maybe it is (already?) solved by separating scaladoc because that was the only part depending on modules from a previous version?

As a corollary, I guess the guideline should be that the compiler sources themselves must not depend on anything else than scala-library, then the problem should disappear (and maybe that's already accomplished).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment