Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

SBT reload fails on SNAPSHOT which are republished locally *while SBT runs* #892

Closed
Blaisorblade opened this issue Sep 29, 2013 · 8 comments

Comments

@Blaisorblade
Copy link
Contributor

I'm developing a SBT plugin and testing it by reloading in a project using it. The reload however fails, complaining about corruption of the plugin JAR file, and I must exit and re-enter the client SBT project; this happened with SBT 0.13.0 in the client, and 0.12.4 in the SBT used for development.

The first time, the client SBT complained that it can't find sbt/sbt.plugins within the JAR. But unzip confirms the entry is there; exiting and re-entering SBT in the client project fixes the problem. The second time, the error was different, but again unzip -t confirms that the JAR is fine. The third time, the error was yet different, but again about ZIP corruption.

Note that before the failed reload, lsof reveals that SBT has two file descriptors pointing at the JAR of the plugin, while afterwards it has three or four file descriptors. My suspect is that it does not reopen the JAR file from scratch, but tries to reuse some old info about it.

To support this hypothesis, this only happens if settings from the plugin are actually used (by putting seq($thisPluginSettings: _*) in build.sbt).

Steps to reproduce in my case:

git clone git@github.com:softprops/np.git
cd np
# edit build.sbt, set version to 0.2.1-SNAPSHOT
# edit plugins/plugins.sbt, comment out gpg plugin
sbt
> publish-local
# Close this or leave it open, we'll need it later.
# Go to another project, even a fresh one
# in project/plugins.sbt, add:
addSbtPlugin("me.lessis" % "np" % "0.2.1-SNAPSHOT")
# in build.sbt, add:
seq(npSettings: _*)
# Start sbt
# Go back to first console, reopen SBT or reuse the existing session (it doesn't matter), modify some source to force recompilation, perform publish-local
# Go back to second console
# Try using the plugin, it still works if used before, otherwise gives ClassNotFoundException.
# Try a reload, get some error about corruption of the ZIP file, for instance:

> reload
[info] Loading global plugins from /Users/pgiarrusso/.sbt/0.13/plugins
[info] Updating {file:/Users/pgiarrusso/.sbt/0.13/plugins/}global-plugins...
[info] Resolving org.fusesource.jansi#jansi;1.4 ...
[info] Done updating.
[error] java.io.FileNotFoundException: JAR entry sbt/sbt.plugins not found in /Users/pgiarrusso/.ivy2/local/me.lessis/np/scala_2.10/sbt_0.13/0.2.1-SNAPSHOT/jars/np.jar
[error] Use 'last' for the full log.
Project loading failed: (r)etry, (q)uit, (l)ast, or (i)gnore?
# Press i to ignore
# Try using the plugin, get ClassNotFoundException:
> np::scout
[error] java.lang.NoClassDefFoundError: np/Plugin$$anonfun$npSettings0$2$$anonfun$apply$2$$anonfun$apply$3
[error] Use 'last' for the full log.

Error message 1:

java.io.FileNotFoundException: JAR entry sbt/sbt.plugins not found in /Users/pgiarrusso/.ivy2/local/me.lessis/np/scala_2.10/sbt_0.13/0.2.1-SNAPSHOT/jars/np.jar
    at sun.net.www.protocol.jar.JarURLConnection.connect(JarURLConnection.java:140)
    at sun.net.www.protocol.jar.JarURLConnection.getInputStream(JarURLConnection.java:150)
    at java.net.URL.openStream(URL.java:1037)
    at sbt.Using$$anonfun$urlReader$1.apply(Using.scala:87)
    at sbt.Using$$anonfun$urlReader$1.apply(Using.scala:87)
    at sbt.Using$$anon$3.open(Using.scala:65)
    at sbt.Using.apply(Using.scala:24)
    at sbt.IO$.readLinesURL(IO.scala:669)
    at sbt.Load$$anonfun$binaryPlugins$2.apply(Load.scala:630)
    at sbt.Load$$anonfun$binaryPlugins$2.apply(Load.scala:629)
    at scala.collection.immutable.Stream$$anonfun$flatMap$1.apply(Stream.scala:450)
    at scala.collection.immutable.Stream$$anonfun$flatMap$1.apply(Stream.scala:450)
    at scala.collection.immutable.Stream.append(Stream.scala:237)
    at scala.collection.immutable.Stream$$anonfun$append$1.apply(Stream.scala:237)
    at scala.collection.immutable.Stream$$anonfun$append$1.apply(Stream.scala:237)
    at scala.collection.immutable.Stream$Cons.tail(Stream.scala:1085)
    at scala.collection.immutable.Stream$Cons.tail(Stream.scala:1077)
    at scala.collection.immutable.Stream$$anonfun$$plus$plus$1.apply(Stream.scala:330)
    at scala.collection.immutable.Stream$$anonfun$$plus$plus$1.apply(Stream.scala:330)
    at scala.collection.immutable.Stream$Cons.tail(Stream.scala:1085)
    at scala.collection.immutable.Stream$Cons.tail(Stream.scala:1077)
    at scala.collection.immutable.Stream$$anonfun$scala$collection$immutable$Stream$$loop$4$1.apply(Stream.scala:850)
    at scala.collection.immutable.Stream$$anonfun$scala$collection$immutable$Stream$$loop$4$1.apply(Stream.scala:850)
    at scala.collection.immutable.Stream$Cons.tail(Stream.scala:1085)
    at scala.collection.immutable.Stream$Cons.tail(Stream.scala:1077)
    at scala.collection.immutable.Stream$$anonfun$map$1.apply(Stream.scala:376)
    at scala.collection.immutable.Stream$$anonfun$map$1.apply(Stream.scala:376)
    at scala.collection.immutable.Stream$Cons.tail(Stream.scala:1085)
    at scala.collection.immutable.Stream$Cons.tail(Stream.scala:1077)
    at scala.collection.immutable.Stream$$anonfun$map$1.apply(Stream.scala:376)
    at scala.collection.immutable.Stream$$anonfun$map$1.apply(Stream.scala:376)
    at scala.collection.immutable.Stream$Cons.tail(Stream.scala:1085)
    at scala.collection.immutable.Stream$Cons.tail(Stream.scala:1077)
    at scala.collection.immutable.Stream.force(Stream.scala:249)
    at scala.collection.immutable.Stream.mkString(Stream.scala:702)
    at sbt.BuildUtil$.importAll(BuildUtil.scala:65)
    at sbt.BuildUtil$.importAllRoot(BuildUtil.scala:66)
    at sbt.Load$.buildGlobalSettings(Load.scala:80)
    at sbt.Load$.loadGlobalSettings(Load.scala:74)
    at sbt.Load$.defaultWithGlobal(Load.scala:68)
    at sbt.Load$.defaultLoad(Load.scala:38)
    at sbt.BuiltinCommands$.doLoadProject(Main.scala:434)
    at sbt.BuiltinCommands$$anonfun$loadProjectImpl$2.apply(Main.scala:428)
    at sbt.BuiltinCommands$$anonfun$loadProjectImpl$2.apply(Main.scala:428)
    at sbt.Command$$anonfun$applyEffect$1$$anonfun$apply$2.apply(Command.scala:60)
    at sbt.Command$$anonfun$applyEffect$1$$anonfun$apply$2.apply(Command.scala:60)
    at sbt.Command$$anonfun$applyEffect$2$$anonfun$apply$3.apply(Command.scala:62)
    at sbt.Command$$anonfun$applyEffect$2$$anonfun$apply$3.apply(Command.scala:62)
    at sbt.Command$.process(Command.scala:95)
    at sbt.MainLoop$$anonfun$1$$anonfun$apply$1.apply(MainLoop.scala:87)
    at sbt.MainLoop$$anonfun$1$$anonfun$apply$1.apply(MainLoop.scala:87)
    at sbt.State$$anon$1.process(State.scala:176)
    at sbt.MainLoop$$anonfun$1.apply(MainLoop.scala:87)
    at sbt.MainLoop$$anonfun$1.apply(MainLoop.scala:87)
    at sbt.ErrorHandling$.wideConvert(ErrorHandling.scala:18)
    at sbt.MainLoop$.next(MainLoop.scala:87)
    at sbt.MainLoop$.run(MainLoop.scala:80)
    at sbt.MainLoop$$anonfun$runWithNewLog$1.apply(MainLoop.scala:69)
    at sbt.MainLoop$$anonfun$runWithNewLog$1.apply(MainLoop.scala:66)
    at sbt.Using.apply(Using.scala:25)
    at sbt.MainLoop$.runWithNewLog(MainLoop.scala:66)
    at sbt.MainLoop$.runAndClearLast(MainLoop.scala:49)
    at sbt.MainLoop$.runLoggedLoop(MainLoop.scala:33)
    at sbt.MainLoop$.runLogged(MainLoop.scala:25)
    at sbt.xMain.run(Main.scala:26)
    at xsbt.boot.Launch$$anonfun$run$1.apply(Launch.scala:57)
    at xsbt.boot.Launch$.withContextLoader(Launch.scala:77)
    at xsbt.boot.Launch$.run(Launch.scala:57)
    at xsbt.boot.Launch$$anonfun$explicit$1.apply(Launch.scala:45)
    at xsbt.boot.Launch$.launch(Launch.scala:65)
    at xsbt.boot.Launch$.apply(Launch.scala:16)
    at xsbt.boot.Boot$.runImpl(Boot.scala:32)
    at xsbt.boot.Boot$.main(Boot.scala:21)
    at xsbt.boot.Boot.main(Boot.scala)
[error] java.io.FileNotFoundException: JAR entry sbt/sbt.plugins not found in /Users/pgiarrusso/.ivy2/local/me.lessis/np/scala_2.10/sbt_0.13/0.2.1-SNAPSHOT/jars/np.jar
[error] Use 'last' for the full log.

Error message 2:

java.util.zip.ZipException: invalid stored block lengths
    at java.util.zip.InflaterInputStream.read(InflaterInputStream.java:164)
    at java.io.FilterInputStream.read(FilterInputStream.java:133)
    at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:283)
    at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:325)
    at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:177)
    at java.io.InputStreamReader.read(InputStreamReader.java:184)
    at java.io.BufferedReader.fill(BufferedReader.java:154)
    at java.io.BufferedReader.readLine(BufferedReader.java:317)
    at java.io.BufferedReader.readLine(BufferedReader.java:382)
    at sbt.IO$.readLine$1(IO.scala:689)
    at sbt.IO$.foldLines(IO.scala:692)
    at sbt.IO$.readLines(IO.scala:677)
    at sbt.IO$$anonfun$readLinesURL$1.apply(IO.scala:669)
    at sbt.IO$$anonfun$readLinesURL$1.apply(IO.scala:669)
    at sbt.Using.apply(Using.scala:25)
    at sbt.IO$.readLinesURL(IO.scala:669)
    at sbt.Load$$anonfun$binaryPlugins$2.apply(Load.scala:630)
    at sbt.Load$$anonfun$binaryPlugins$2.apply(Load.scala:629)
    at scala.collection.immutable.Stream$$anonfun$flatMap$1.apply(Stream.scala:450)
    at scala.collection.immutable.Stream$$anonfun$flatMap$1.apply(Stream.scala:450)
    at scala.collection.immutable.Stream.append(Stream.scala:237)
    at scala.collection.immutable.Stream$$anonfun$append$1.apply(Stream.scala:237)
    at scala.collection.immutable.Stream$$anonfun$append$1.apply(Stream.scala:237)
    at scala.collection.immutable.Stream$Cons.tail(Stream.scala:1085)
    at scala.collection.immutable.Stream$Cons.tail(Stream.scala:1077)
    at scala.collection.immutable.Stream$$anonfun$$plus$plus$1.apply(Stream.scala:330)
    at scala.collection.immutable.Stream$$anonfun$$plus$plus$1.apply(Stream.scala:330)
    at scala.collection.immutable.Stream$Cons.tail(Stream.scala:1085)
    at scala.collection.immutable.Stream$Cons.tail(Stream.scala:1077)
    at scala.collection.immutable.Stream$$anonfun$scala$collection$immutable$Stream$$loop$4$1.apply(Stream.scala:850)
    at scala.collection.immutable.Stream$$anonfun$scala$collection$immutable$Stream$$loop$4$1.apply(Stream.scala:850)
    at scala.collection.immutable.Stream$Cons.tail(Stream.scala:1085)
    at scala.collection.immutable.Stream$Cons.tail(Stream.scala:1077)
    at scala.collection.immutable.Stream$$anonfun$map$1.apply(Stream.scala:376)
    at scala.collection.immutable.Stream$$anonfun$map$1.apply(Stream.scala:376)
    at scala.collection.immutable.Stream$Cons.tail(Stream.scala:1085)
    at scala.collection.immutable.Stream$Cons.tail(Stream.scala:1077)
    at scala.collection.immutable.Stream$$anonfun$map$1.apply(Stream.scala:376)
    at scala.collection.immutable.Stream$$anonfun$map$1.apply(Stream.scala:376)
    at scala.collection.immutable.Stream$Cons.tail(Stream.scala:1085)
    at scala.collection.immutable.Stream$Cons.tail(Stream.scala:1077)
    at scala.collection.immutable.Stream.force(Stream.scala:249)
    at scala.collection.immutable.Stream.mkString(Stream.scala:702)
    at sbt.BuildUtil$.importAll(BuildUtil.scala:65)
    at sbt.BuildUtil$.importAllRoot(BuildUtil.scala:66)
    at sbt.Load$.buildGlobalSettings(Load.scala:80)
    at sbt.Load$.loadGlobalSettings(Load.scala:74)
    at sbt.Load$.defaultWithGlobal(Load.scala:68)
    at sbt.Load$.defaultLoad(Load.scala:38)
    at sbt.BuiltinCommands$.doLoadProject(Main.scala:434)
    at sbt.BuiltinCommands$$anonfun$loadProjectImpl$2.apply(Main.scala:428)
    at sbt.BuiltinCommands$$anonfun$loadProjectImpl$2.apply(Main.scala:428)
    at sbt.Command$$anonfun$applyEffect$1$$anonfun$apply$2.apply(Command.scala:60)
    at sbt.Command$$anonfun$applyEffect$1$$anonfun$apply$2.apply(Command.scala:60)
    at sbt.Command$$anonfun$applyEffect$2$$anonfun$apply$3.apply(Command.scala:62)
    at sbt.Command$$anonfun$applyEffect$2$$anonfun$apply$3.apply(Command.scala:62)
    at sbt.Command$.process(Command.scala:95)
    at sbt.MainLoop$$anonfun$1$$anonfun$apply$1.apply(MainLoop.scala:87)
    at sbt.MainLoop$$anonfun$1$$anonfun$apply$1.apply(MainLoop.scala:87)
    at sbt.State$$anon$1.process(State.scala:176)
    at sbt.MainLoop$$anonfun$1.apply(MainLoop.scala:87)
    at sbt.MainLoop$$anonfun$1.apply(MainLoop.scala:87)
    at sbt.ErrorHandling$.wideConvert(ErrorHandling.scala:18)
    at sbt.MainLoop$.next(MainLoop.scala:87)
    at sbt.MainLoop$.run(MainLoop.scala:80)
    at sbt.MainLoop$$anonfun$runWithNewLog$1.apply(MainLoop.scala:69)
    at sbt.MainLoop$$anonfun$runWithNewLog$1.apply(MainLoop.scala:66)
    at sbt.Using.apply(Using.scala:25)
    at sbt.MainLoop$.runWithNewLog(MainLoop.scala:66)
    at sbt.MainLoop$.runAndClearLast(MainLoop.scala:49)
    at sbt.MainLoop$.runLoggedLoop(MainLoop.scala:33)
    at sbt.MainLoop$.runLogged(MainLoop.scala:25)
    at sbt.xMain.run(Main.scala:26)
    at xsbt.boot.Launch$$anonfun$run$1.apply(Launch.scala:57)
    at xsbt.boot.Launch$.withContextLoader(Launch.scala:77)
    at xsbt.boot.Launch$.run(Launch.scala:57)
    at xsbt.boot.Launch$$anonfun$explicit$1.apply(Launch.scala:45)
    at xsbt.boot.Launch$.launch(Launch.scala:65)
    at xsbt.boot.Launch$.apply(Launch.scala:16)
    at xsbt.boot.Boot$.runImpl(Boot.scala:32)
    at xsbt.boot.Boot$.main(Boot.scala:21)
    at xsbt.boot.Boot.main(Boot.scala)
@harrah
Copy link
Member

harrah commented Sep 30, 2013

sbt is getting a URL from ClassLoader.getResource and then opening the stream for the returned URL. So, this is problem with the Java standard library. Directly overwriting jars in use by a class loader seems to cause issues in some cases.

@harrah harrah closed this as completed Sep 30, 2013
@Blaisorblade
Copy link
Contributor Author

I see — no workaround possible for SBT then? Since you closed, I assume not.

@ritschwumm
Copy link

@Blaisorblade
Copy link
Contributor Author

@ritschwumm, thanks! One answer suggests that throwing away the classloader would help, which would make sense.
http://stackoverflow.com/a/17060732/53974

Is SBT reusing classloaders? Could they be thrown away on reload, at least if the underlying files changed timestamp (and/or it's a SNAPSHOT)?

@harrah
Copy link
Member

harrah commented Sep 30, 2013

sbt doesn't reuse the plugin classloader, but it doesn't discard the old project until the new one has successfully loaded. It will fall back on the old one if the new one fails and you select (i)gnore.

In general, sbt can cache/reuse class loaders, but it only currently does this for Scala jars, the loaders are behind SoftReferences, and it checks the last modified time before reusing the class loader.

@dwijnand
Copy link
Member

Ouch, this sucks :(

@reid-spencer
Copy link

+1
I know this has been ignored mostly but has anyone found a workaround?
The only workaround I have today is to restart sbt

@rbellamy
Copy link

This makes testing newly published plugins locally, in a "real" SBT project (e.g. not a sbt-scripted test project) very difficult.

jvican added a commit to scalacenter/bloop that referenced this issue Nov 3, 2019
See details in sbt/sbt#892. We were getting
same error because the plugin comes from a jar. Now, we lie to sbt, we
present the plugin as a project with classes directory which just
happens to have a dependency that brings in the shaded sbt plugin we
want.
jvican added a commit to scalacenter/compiler-benchmark that referenced this issue Nov 4, 2019
jvican added a commit to scalacenter/bloop that referenced this issue Nov 4, 2019
jvican added a commit to scalacenter/bloop that referenced this issue Nov 4, 2019
jvican added a commit to scalacenter/bloop that referenced this issue Nov 4, 2019
jvican added a commit to scalacenter/bloop that referenced this issue Nov 5, 2019
Including a simplification of the work around for sbt/sbt#892
jvican added a commit to scalacenter/bloop that referenced this issue Nov 6, 2019
Including a simplification of the work around for sbt/sbt#892
jvican added a commit to scalacenter/bloop that referenced this issue Nov 7, 2019
Including a simplification of the work around for sbt/sbt#892
ckipp01 added a commit to scalacenter/bloop that referenced this issue Dec 21, 2022
Alright, so I'm pretty confident with this at the moment, and quite
happy with it. Much of it was mimicking what Alex did in
bloop-core by just ripping it all out.

As I stated in the initial description, it seems that the entire notion
of having the sbt-bloop-build-naked and the bloop-shaded-plugin is
primarily to dogfood itself in a way that doesn't conflict with the
issue that is outlined in sbt/sbt#892. Maybe at one point in time when
we were rapidly developing the sbt plugin and build and always wanting
to dogfood those changes this may have been worth it, but I don't think
it is anymore. The entire meta-meta build can just be thrown away if we
get rid of the idea of wanting to do this. Instead, we just ensure that
we publish sbt-bloop before situations where we want to ensure we're
using that. This is primarily when we're running the frontend tests and
we an the Bloop definitions of the projects in the frontend resources
and when we're running buildpress. In both of these scenarios we just
publish sbt-bloop first, and it works the same way. Benchmarks will also
work exactly the same as the version is passed directly to it that we're
using. We included the sbt-bloop-build-naked there before, but from my
understanding it was again only to dogfood the bloop generation when we
can just use the one we just published.

So to outline the changes, this

- gets rid of the sbt-bloop-build-naked
- gets rid of the bloop-build-shaded-plugin
- benchpress is still able to run fine
- frontend test resources are still able to be generated fine
- tests all still pass
- the build is way less crazy

The main thing we lose out on is:

We no longer dogfood the absolute latest changes This is a tradeoff I'm
100% ok with.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

6 participants