Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Lambda in REPL (using object-wrappers) + concurrency = deadlock #9076

Closed
scabug opened this issue Jan 11, 2015 · 12 comments
Closed

Lambda in REPL (using object-wrappers) + concurrency = deadlock #9076

scabug opened this issue Jan 11, 2015 · 12 comments

Comments

@scabug
Copy link

@scabug scabug commented Jan 11, 2015

The following code is a test of generating and using java.util.Spliterator to create a java.util.stream.Stream:

package xp

import java.util.Spliterator
import java.util.function.Consumer

class Aspi(a: Array[String], private val i0: Int, private var i1: Int) extends Spliterator[String] {
  private var i = i0
  def characteristics = Spliterator.SIZED | Spliterator.SUBSIZED | Spliterator.IMMUTABLE
  def estimateSize = i1-i
  override def getExactSizeIfKnown = i1-i
  def tryAdvance(f: Consumer[_ >: String]): Boolean = {
    println(s"Trying at $i [$i0, $i1)")
    if (i >= i1) false
    else { f.accept(a(i)); i += 1; true }
  }
  def trySplit(): Aspi = {
    if (i1-i < 4) {
      println(s"Could not split [$i $i1)")
      null
    }
    else {
      println(s"Splitting from [$i $i1)")
      val child = new Aspi(a, (i+i1+1)/2, i1)
      i1 = child.i0
      child
    }
  }
}

object Test { def main(args: Array[String]) {
  def nu = new xp.Aspi(Array("one", "two", "three", "four", "five"), 0, 5)
  val st = java.util.stream.StreamSupport.stream(nu, true)
  val c = new java.util.Comparator[String]{ def compare(s: String, t: String) = s compareTo t }
  println(st.map[String]((s: String) => s + s).max(c))

This compiles and runs successfully. However, when the contents of main are entered in the REPL, the last line hangs. Stack trace:

"ForkJoinPool.commonPool-worker-1" #14 daemon prio=5 os_prio=0 tid=0x00007f966855f800 nid=0x721f in Object.wait() [0x00007f964a18b000]
   java.lang.Thread.State: RUNNABLE
	at $line6.$read$$iw$$iw$$anonfun$1.apply(<console>:11)
	at $line6.$read$$iw$$iw$$anonfun$1.apply(<console>:11)
	at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
	at xp.Aspi.tryAdvance(Test.scala:14)
	at java.util.Spliterator.forEachRemaining(Spliterator.java:326)
        ...

Explicitly creating the SAM like so:

println(st.map[String](new java.util.function.Function[String, String] {
  def apply(s: String) = s + s
}).max(c))

solves the issue. (Note: I have not tried this on 2.11.5; I assumed 2.12.x was up to date in anything relevant.)

@scabug
Copy link
Author

@scabug scabug commented Jan 11, 2015

Imported From: https://issues.scala-lang.org/browse/SI-9076?orig=1
Reporter: @Ichoran
Affected Versions: 2.12.0

@scabug
Copy link
Author

@scabug scabug commented Jan 13, 2015

@retronym said:
Enabling -Xprint:all in the REPL, shows the difference between your manually expanded version and the SAM-lambda version:

  object iw extends Object {
    private[this] val res0: scala.runtime.BoxedUnit = _;
    <stable> <accessor> def res0(): Unit = ();
    final def apply$body$1(s: String): String = s.+(s);
    def <init>(): type = {
      iw.super.<init>();
      iw.this.res0 = {
        Test.test({
          new <$anon: java.util.function.Function>()
        });
        scala.runtime.BoxedUnit.UNIT
      };
      ()
    }
  };
  @SerialVersionUID(value = 0) final class anonfun$1 extends Object with java.util.function.Function with Serializable {
    final override <synthetic> def apply(s: String): String = $line3.iw.apply$body$1(s);
    final override <synthetic> <bridge> <artifact> def apply(x$1: Object): Object = anonfun$1.this.apply(x$1.$asInstanceOf[String]());
    def <init>(): <$anon: java.util.function.Function> = {
      anonfun$1.super.<init>();
      ()
    }
  }

The apply method delegates to iw.apply$body$1. This blocks when called form the threadpool because the static initializer of iw$ has not completed, it is blocking on the entire operation to complete.

See http://stackoverflow.com/questions/23108731/why-is-scala-await-result-timing-out-in-repl-when-passed-the-same-future-twice#comment35339401_23111645

Indeed this terminates again under scala -Yrepl-class-based. But we can't switch to that as a default until we sort out problems like:

% qscala -Yrepl-class-based

scala> class C(val a: Any) extends AnyVal
<console>:7: error: value class may not be a member of another class
       class C(val a: Any) extends AnyVal
             ^
@scabug
Copy link
Author

@scabug scabug commented Jan 13, 2015

@retronym said:
BTW, a workaround is to indirect through a lambda:

scala> () => Test.test(s => s + s)
res0: () => Unit = <function0>

scala> res0()
Splitting from [0 5)
Could not split [0 3)
Could not split [3 5)
Trying at 3 [3, 5)
Trying at 0 [0, 3)
Trying at 1 [0, 3)
Trying at 4 [3, 5)
Trying at 2 [0, 3)
Trying at 5 [3, 5)
Trying at 3 [0, 3)
Optional[twotwo]
@scabug
Copy link
Author

@scabug scabug commented Jan 13, 2015

@retronym said:
See also: https://groups.google.com/forum/#!topic/scala-user/fljtxY9bJV0

/cc @som-snytt Do you think we could encode my workaround above into the object based wrappers to make the default REPL less prone to this problem?

@scabug
Copy link
Author

@scabug scabug commented Jan 13, 2015

@som-snytt said:
How about making it an App?

scala/scala#4246

@scabug
Copy link
Author

@scabug scabug commented Aug 19, 2016

@SethTisue said:
@retronym should this be retargeted for 2.12.1 or Backlog?

@scabug
Copy link
Author

@scabug scabug commented Mar 1, 2017

Dave Gurnell (davegurnell) said (edited on Mar 1, 2017 10:35:25 AM UTC):
Seth suggested I post this recreation here. Try this on a 2.12.1 REPL:

import scala.concurrent._
import scala.concurrent.duration._
import scala.concurrent.ExecutionContext.Implicits.global

// This works fine:
val x = Future(42)
Await.result(x, 1.second)

// This throws a TimeoutException:
Await.result(Future(42), 1.second)

Adding scalacOptions += "-Ydelambdafy:inline" to build.sbt makes the problem go away.

The problem can also result in a NoClassDefFoundError as shown here (running Tut on an ebook project):

[tut] compiling: /Users/dave/dev/projects/advanced-scala/src/pages/test.md
java.lang.NoClassDefFoundError: Could not initialize class $line4.$read$$iw$$iw$$iw$$iw$$iw$$iw$
    at scala.runtime.java8.JFunction0$mcI$sp.apply(JFunction0$mcI$sp.java:12)
    at scala.concurrent.Future$.$anonfun$apply$1(Future.scala:653)
    at scala.util.Success.$anonfun$map$1(Try.scala:251)
    at scala.util.Success.map(Try.scala:209)
    at scala.concurrent.Future.$anonfun$map$1(Future.scala:287)
    ... manually elided for brevity
[tut] *** Error reported at /Users/dave/dev/projects/advanced-scala/src/pages/monads/test.md:6
java.util.concurrent.TimeoutException: Futures timed out after [1 second]
  at scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:255)
  at scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:259)
  at scala.concurrent.Await$.$anonfun$result$1(package.scala:190)
  at scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53)
  at scala.concurrent.Await$.result(package.scala:123)
  ... 88 elided
[error] (run-main-0) java.lang.Exception: Tut execution failed.
[trace] Stack trace suppressed: run last *:streams for the full output.
[trace] Stack trace suppressed: run last *:tutOnly for the full output.
[error] (*:tutOnly) Nonzero exit code: 1
[error] Total time: 5 s, completed 27-Feb-2017 09:38:19
@adriaanm
Copy link
Member

@adriaanm adriaanm commented Jun 26, 2017

Probably a wontfix in 2.12, but should consider this as we improve the repl in 2.13 (where we'll standardize on some form of class-based wrapping).

@SethTisue
Copy link
Member

@SethTisue SethTisue commented Sep 11, 2018

should this still be on the 2.13.0-RC1 milestone, or shall we push it out to 2.13.x or 2.14?

@smarter
Copy link

@smarter smarter commented Sep 12, 2018

It's indeed fixed in Dotty: scala/scala-dev#195 :)

@adriaanm adriaanm modified the milestones: 2.13.0-RC1, 2.14.0-M1 Jan 8, 2019
@adriaanm adriaanm modified the milestones: 2.14.0-M1, 2.13.2 Nov 26, 2019
@adriaanm adriaanm modified the milestones: 2.13.2, 2.13.3 Nov 26, 2019
@adriaanm
Copy link
Member

@adriaanm adriaanm commented Nov 26, 2019

Re-scheduling for 2.13 in the context of scala/scala-dev#325 and a commercial support request.

First we should finish the jline 3 upgrade, but then let's look into improving the encoding.

@adriaanm
Copy link
Member

@adriaanm adriaanm commented Dec 3, 2019

@retronym suggested a few alternative solutions -- one is to type check using the current object wrappers, and then have a repl phase that does the class-based encoding? This would avoid breaking anyone that relies on the current repl tree shapes during typer. Have to check how this affects serialization in Spark.

@SethTisue SethTisue changed the title Lambda in REPL (using object-wrappers) gives non-termination of Java 8 Stream Lambda in REPL (using object-wrappers) + concurrency = deadlock Dec 24, 2019
@dwijnand dwijnand assigned dwijnand and unassigned adriaanm Feb 12, 2020
@dwijnand dwijnand added the has PR label Feb 24, 2020
@dwijnand dwijnand modified the milestones: 2.13.3, 2.13.2 Feb 24, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

6 participants
You can’t perform that action at this time.