NullPointerException in compile after upgrade to sbt 1.0.2 #3623
Assignees
Labels
Milestone
Comments
|
I have been upgrading a service from sbt 0.13.15 to sbt 1.0.2, and have also been seeing this error about a fraction of the times. |
|
|
We also noticed this, on our build servers, when we upgraded to sbt 1.0.2 recently. Locally it seems to work fine, and was not able to reproduce the issue locally so far. Here's an example failed build https://jenkins.akka.io:8498/job/pr-validator-per-commit-jenkins/10520/console |
This was referenced Oct 30, 2017
|
@ktoso Just noticed in your Jenkins console output that your code uses Protobuf. Our service does too. In fact another service of ours that uses Protobuf also has this error, and they are ONLY ones. Maybe this is due to sbt and Protobuf somehow not playing nicely? |
|
Could be, I wonder. |
|
I've managed to recreate with this example https://github.com/benblack86/sbt-protobuf-test It is still random, but if you run |
|
@ktoso FYI, it also fails in local builds (sometimes). Example - |
|
I was not able to reproduce it using https://github.com/benblack86/sbt-protobuf-test neither on my machine, where I have never seen this error before, neither on our CI server where this error was spontaneously happening while building Akka. I was also using sbt-repeat and doing |
|
Yes we are also using protobuf
|
|
@2m yes, I don't think sbt-repeat will help recreate since it seems to be something random linked to when sbt starts up. |
|
I also tried |
|
Another report on NPE with the same line number: facebook/buck#403 |
|
@2m the buck stack trace seems to be different. It's only the entry point |
|
@2m did you try just doing |
|
Yup. Quite a few times with no NPE. :\ |
|
Happening to me as well when compiling akka incrementally with sbt 1.0.2 |
|
This still happens regularly during Akka's CI builds. I had a look to find the (JDK) sources that fail. I didn't find any version of OpenJDK where all the line numbers would match the openjdk sources (but maybe I just missed something). From looking at the sources it's unclear where the null dereference would happen at all. The most likely place (that's where the line number points to in most revisions) would be the array access at |
when building Akka as part of the Scala community build, too, e.g. at https://scala-ci.typesafe.com/view/scala-2.13.x/job/scala-2.13.x-integrate-community-build/791/ |
|
@jrudolph what's |
|
Perhaps this is a stupid and obvious question, but anyway. What is changed in sbt 1.0.2 with regards to concurrency? The stack trace hints at |
|
I instrumented tools.jar and found out that indeed several threads access the problematic method. The interesting one is this: There are several things at play here:
|
|
A first workaround could be to make sure that PositionImpl is thread-safe (by initializing fields eagerly and not holding to |
|
For reference here is the stack trace of the concurrent access: |
|
fwiw, a note about async logging. I understand, that it is compelling to do as much work as possible asynchronously when you already have an async logger. However, the ultimate reason to do async logging is to avoid I/O in the main worker threads. Shaving off a bit of CPU time from the main process by doing the string processing in the logger thread will only work on under-saturated machines anyway. In many other cases, it's probably even faster to do the inevitable string processing as soon as possible on the worker threads (better parallelism) and execute only the actual recording of the log events asynchronously. |
|
Thanks for the investigation @jrudolph! |
eed3si9n
added a commit
to eed3si9n/zinc
that referenced
this issue
Dec 6, 2017
Fixes sbt#464 sbt/sbt#3623 Currently compiling a lot of Java code in CI environment causes NullPointerException, which is suspected of race condition around `javax.tools.Diagnostics[S]`, which is held by `PositionImpl` and then later accessed by async logging. This change makes the `PositionImpl` strict and immutable, and extracts it from a Java Diagnostics object. No other observable changes are introduced besides, hopefully the lack of NPE. Credit on the detective work goes to Lightbend Akka team ktoso, 2m, and jrudolph.
eed3si9n
added a commit
to eed3si9n/zinc
that referenced
this issue
Dec 6, 2017
Fixes sbt#464 sbt/sbt#3623 Currently compiling a lot of Java code in CI environment causes NullPointerException, which is suspected of race condition around `javax.tools.Diagnostics[S]`, which is held by `PositionImpl` and then later accessed by async logging. This change makes the `PositionImpl` strict and immutable, and extracts it from a Java Diagnostics object. No other observable changes are introduced besides, hopefully the lack of NPE. Credit on the detective work goes to Lightbend Akka team ktoso, 2m, and jrudolph.
eed3si9n
added a commit
to eed3si9n/zinc
that referenced
this issue
Dec 6, 2017
Fixes sbt#464 sbt/sbt#3623 Currently compiling a lot of Java code in CI environment causes NullPointerException, which is suspected of race condition around `javax.tools.Diagnostics[S]`, which is held by `PositionImpl` and then later accessed by async logging. This change makes the `PositionImpl` strict and immutable, and extracts it from a Java Diagnostics object. No other observable changes are introduced besides, hopefully the lack of NPE. Credit on the detective work goes to Lightbend Akka team ktoso, 2m, and jrudolph.
eed3si9n
added a commit
to eed3si9n/zinc
that referenced
this issue
Dec 8, 2017
Fixes sbt#464 sbt/sbt#3623 Currently compiling a lot of Java code in CI environment causes NullPointerException, which is suspected of race condition around `javax.tools.Diagnostics[S]`, which is held by `PositionImpl` and then later accessed by async logging. This change makes the `PositionImpl` strict and immutable, and extracts it from a Java Diagnostics object. No other observable changes are introduced besides, hopefully the lack of NPE. Credit on the detective work goes to Lightbend Akka team ktoso, 2m, and jrudolph.
|
New Zinc was merged in #3816 with the fix. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
(See the guidelines for contributing, linked above)
steps
problem
expectation
This happens only in 30% of the builds, so it's really hard to reproduce but with sbt 0.13.x it never happened
notes
sbt version: 1.0.2
The text was updated successfully, but these errors were encountered: