Meta build resolves to unused Scala 2.10.2 #1439
Comments
jsuereth
changed the title
|
It's not quite that bad. When downloading sbt itself, we include some "default" precompiled libraries. These really should only be downloaded after we discover a project that needs them, but the hook we're using pre-downloads them every time. We have two options here:
@gkossakowski and I have talked about doing (2) for various other reasons, but I don't see that likely to happen soon (next few months) |
|
Ok, well you guys have a lot to do in the shared server work, and rewriting ivy, so please do focus on those other high priority areas :-) |
|
we're still seeing crazy things like various sbt plugins depending on different versions of scala-library or scala-compiler and sbt is pulling them all in before realising what should be ignored. That seems terribly wasteful for both resolution and network usage. It increases the probability of CI failures, e.g. we keep getting a failure getting 2.10.2 of |
|
@fommil Your report was not understood. Josh is referring to some precompiled jars of trivial size. You are talking about it downloading scala-compiler 2.10.2, 2.10.3, 2.10.4, scala-reflect 2.10.0, 2.10.2, 2.10.4, and so on, which is indeed what it does if you have a couple plugins. Good news, I convinced sbt not to do that. With the following in project/plugins.sbt, it only downloads 2.10.4 scala artifacts. |
|
@paulp This has some a lot of merit. With the exclusion rule optimisations @eed3si9n has recently added, it may even dramatically improve resolution performance for plugins. I may even be more aggressive and alter the rule to be NOTE: Edited org.scala-lang => org.scala-sbt |
|
Isn't |
|
@fommil Also, with regards to downloading those libraries, ivy will downloads all ivy.xml/pom.xml files during resolution. This isn't supposed to include JARs, but who knows what happens. I spent 2 days inside the cache implementation and I'm not sure I trust it. In any case, Paul's solution has a lot of merit to help with the situation. @paulp In terms of Precompiled, sbt will ALWAYS resolve the full scala JARs for each precompiled interface, given how we've defined the ivy, not just the ivy.xml but the full JARs. SO, scala 2.9.3, Scala 2.10.4 and I think Scala 2.8.2. You know, for giggles. Also, see my edited comment. Was typing too fast. |
|
Right, it's just that the full jars are 250K. Compared to the 14+ MB per scala-compiler jar, it's not significant. |
|
@paulp hmm, it's not pulling in the full scala-compiler as well? AFAIK our ivy config, it should be doing that too (which we'd also need to fix). i..e I see this as two issues:
I think we can mitigate both of these if not negate them completely. |
|
Nope, it's not downloading any compiler but 2.10.4. Here's the complete list of what's downloaded from a cold start (this is easy to reproduce with my-sbt -no-share) sorted by size. This is with the plugins.sbt posted above, but that shouldn't be especially relevant to the contents of this list. |
|
Ah, that's good to hear. I need look into the hackery used for the precompiler jars more it seems... |
|
That's great! The precompiled binaries are a pain also, especially since their classifiers can never be resolved, it results in errors anytime updateClassifiers is called (on an SBT plugin) |
|
Thanks @paulp ... that appears to be working for me. |
eed3si9n
changed the title
Scala instance is added to the Ivy graph via autoLibraryDependency. For metabuilds, scala-library is scoped under “provided” configuration, which does not seem to evict modules on “compiled” configuration. This commit turns overrideScalaVersion flag to true for the metabuilds, so override rules are added for the following modules: - scala-library - scala-compiler - scala-reflect
Fixes #1439. Fixes metabuild downloading unused Scala 2.10.2
This is a workaround for sbt/sbt#1439 and lack of Nexus proxy hosted by Travis infrastructure.
|
Beside working on the introduction of a Nexus proxy, the Travis team will soon ship new VM images. As a faster workaround to this issue, I'd like to pre-install these "2.10.2" jars (see travis-ci/travis-cookbooks#362) I can upload the resulting Compared to current VM image, the "2.10.2" jars will be present. Please tell me if anything else should be preinstalled to solve this problem. |
|
@gildegoma Thanks a lot for working on this. Does preloaded cache include all other text files like |
|
@gildegoma If you want to compare with what I'm using, here it is: https://www.dropbox.com/s/zorqzskm17f4dgy/cache.tar.gz I recommend all those files for inclusion (plus the 2.10.2 version of all the 2.10.4 files - I didn't need them personally since I've applied the sbt fix which limits it to 2.10.4.) |
|
great! I recommend these files plus whatever else gets put into |
|
Thanks everyone for the fast feedbacks! @eed3si9n No worries, all these @paulp I've compared, and here are the directories from your cache tarball that wouldn't be included in Travis VM image (in Kbytes): Do you see any problem if these parts are not preinstalled ? @fommil wrote:
So as workaround, I added to 2.10.2 to the pre-installation list, so we'll have: 2.11.2, 2.10.4, 2.10.2 and 2.9.3. Should I really add 2.9.2 as well, or you meant 2.10.2? So far, we populate |
|
No, those are mostly the transitive dependencies of sbt-pgp. But it's not going to be a problem regardless of what's included, because it's only optimization except for the 2.10.2-related breakage. |
|
@gildegoma there are two issues here. One is dealing with the extreme situation of 2.10.2 being removed from central, the other is concerned with ensuring that all the deps of And actually 2.9.2 would be good, as it is used for 2.9 cross builds (2.9.2 and 2.9.3 are not binary compatible, so you may as well think of them as different scala versions entirely) |
Different 2.9.x versions of Scala are binary incompatible. Happily, Scala patch versions from 2.10.0 on have become binary compatible, so we won't have to worry about this point in the future. See sbt/sbt#1439 (comment)
I think we are good now, but please tell me if you see anything that should be improved or fixed. |
|
😀 |
|
If the 2.10.2 jar is never used, why bother prepopulating it via dropbox etc. -- can't you just |
|
sha/md5 |
|
Could that also be faked? |
|
sure: if you want to become the world's most awesome code breaker |
|
My point is, can I pre-populate travis with some dummy files so it doesn't bother downloading 2.10.2 which I'm not going to use? |
steps
project/plugins.sbt:build.sbt:Run:
problem
The meta build resolves to Scala 2.10.2 and downloads it.
expectation
The meta build uses Scala 2.10.4 (or which ever Scala version the sbt 0.13.x uses)
notes (workaround posted by @eed3si9n)
Put this in
project/plugins.sbt:Meta build's Compile configuration seems to be unaware of the ultimate Scala version the build uses (2.10.4 for sbt 0.13.5) and uses the Scala version that the plugins used to publish. The above workaround uses Ivy eviction to bump up scala-compiler and scala-library version.
original report by @fommil (Sbt downloads precompiled incremental compiler interfaces that users never plan to use)
in one of the projects I contribute to http://github.com/ensime/ensime-server and starting with a fresh ivy cache, type
sbt compilefor this scala 2.10.4 project and see several versions of the scala-compiler and sbt itself being downloaded.Are all dependencies aggressively downloaded before working out what is evicted?
The text was updated successfully, but these errors were encountered: