New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Lambda in REPL (using object-wrappers) gives non-termination of Java 8 Stream #9076

Open
scabug opened this Issue Jan 11, 2015 · 11 comments

Comments

Projects
None yet
6 participants
@scabug

scabug commented Jan 11, 2015

The following code is a test of generating and using java.util.Spliterator to create a java.util.stream.Stream:

package xp

import java.util.Spliterator
import java.util.function.Consumer

class Aspi(a: Array[String], private val i0: Int, private var i1: Int) extends Spliterator[String] {
  private var i = i0
  def characteristics = Spliterator.SIZED | Spliterator.SUBSIZED | Spliterator.IMMUTABLE
  def estimateSize = i1-i
  override def getExactSizeIfKnown = i1-i
  def tryAdvance(f: Consumer[_ >: String]): Boolean = {
    println(s"Trying at $i [$i0, $i1)")
    if (i >= i1) false
    else { f.accept(a(i)); i += 1; true }
  }
  def trySplit(): Aspi = {
    if (i1-i < 4) {
      println(s"Could not split [$i $i1)")
      null
    }
    else {
      println(s"Splitting from [$i $i1)")
      val child = new Aspi(a, (i+i1+1)/2, i1)
      i1 = child.i0
      child
    }
  }
}

object Test { def main(args: Array[String]) {
  def nu = new xp.Aspi(Array("one", "two", "three", "four", "five"), 0, 5)
  val st = java.util.stream.StreamSupport.stream(nu, true)
  val c = new java.util.Comparator[String]{ def compare(s: String, t: String) = s compareTo t }
  println(st.map[String]((s: String) => s + s).max(c))

This compiles and runs successfully. However, when the contents of main are entered in the REPL, the last line hangs. Stack trace:

"ForkJoinPool.commonPool-worker-1" #14 daemon prio=5 os_prio=0 tid=0x00007f966855f800 nid=0x721f in Object.wait() [0x00007f964a18b000]
   java.lang.Thread.State: RUNNABLE
	at $line6.$read$$iw$$iw$$anonfun$1.apply(<console>:11)
	at $line6.$read$$iw$$iw$$anonfun$1.apply(<console>:11)
	at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
	at xp.Aspi.tryAdvance(Test.scala:14)
	at java.util.Spliterator.forEachRemaining(Spliterator.java:326)
        ...

Explicitly creating the SAM like so:

println(st.map[String](new java.util.function.Function[String, String] {
  def apply(s: String) = s + s
}).max(c))

solves the issue. (Note: I have not tried this on 2.11.5; I assumed 2.12.x was up to date in anything relevant.)

@scabug

This comment has been minimized.

Show comment
Hide comment
@scabug

scabug Jan 11, 2015

Imported From: https://issues.scala-lang.org/browse/SI-9076?orig=1
Reporter: @Ichoran
Affected Versions: 2.12.0

scabug commented Jan 11, 2015

Imported From: https://issues.scala-lang.org/browse/SI-9076?orig=1
Reporter: @Ichoran
Affected Versions: 2.12.0

@scabug

This comment has been minimized.

Show comment
Hide comment
@scabug

scabug Jan 13, 2015

@retronym said:
Enabling -Xprint:all in the REPL, shows the difference between your manually expanded version and the SAM-lambda version:

  object iw extends Object {
    private[this] val res0: scala.runtime.BoxedUnit = _;
    <stable> <accessor> def res0(): Unit = ();
    final def apply$body$1(s: String): String = s.+(s);
    def <init>(): type = {
      iw.super.<init>();
      iw.this.res0 = {
        Test.test({
          new <$anon: java.util.function.Function>()
        });
        scala.runtime.BoxedUnit.UNIT
      };
      ()
    }
  };
  @SerialVersionUID(value = 0) final class anonfun$1 extends Object with java.util.function.Function with Serializable {
    final override <synthetic> def apply(s: String): String = $line3.iw.apply$body$1(s);
    final override <synthetic> <bridge> <artifact> def apply(x$1: Object): Object = anonfun$1.this.apply(x$1.$asInstanceOf[String]());
    def <init>(): <$anon: java.util.function.Function> = {
      anonfun$1.super.<init>();
      ()
    }
  }

The apply method delegates to iw.apply$body$1. This blocks when called form the threadpool because the static initializer of iw$ has not completed, it is blocking on the entire operation to complete.

See http://stackoverflow.com/questions/23108731/why-is-scala-await-result-timing-out-in-repl-when-passed-the-same-future-twice#comment35339401_23111645

Indeed this terminates again under scala -Yrepl-class-based. But we can't switch to that as a default until we sort out problems like:

% qscala -Yrepl-class-based

scala> class C(val a: Any) extends AnyVal
<console>:7: error: value class may not be a member of another class
       class C(val a: Any) extends AnyVal
             ^

scabug commented Jan 13, 2015

@retronym said:
Enabling -Xprint:all in the REPL, shows the difference between your manually expanded version and the SAM-lambda version:

  object iw extends Object {
    private[this] val res0: scala.runtime.BoxedUnit = _;
    <stable> <accessor> def res0(): Unit = ();
    final def apply$body$1(s: String): String = s.+(s);
    def <init>(): type = {
      iw.super.<init>();
      iw.this.res0 = {
        Test.test({
          new <$anon: java.util.function.Function>()
        });
        scala.runtime.BoxedUnit.UNIT
      };
      ()
    }
  };
  @SerialVersionUID(value = 0) final class anonfun$1 extends Object with java.util.function.Function with Serializable {
    final override <synthetic> def apply(s: String): String = $line3.iw.apply$body$1(s);
    final override <synthetic> <bridge> <artifact> def apply(x$1: Object): Object = anonfun$1.this.apply(x$1.$asInstanceOf[String]());
    def <init>(): <$anon: java.util.function.Function> = {
      anonfun$1.super.<init>();
      ()
    }
  }

The apply method delegates to iw.apply$body$1. This blocks when called form the threadpool because the static initializer of iw$ has not completed, it is blocking on the entire operation to complete.

See http://stackoverflow.com/questions/23108731/why-is-scala-await-result-timing-out-in-repl-when-passed-the-same-future-twice#comment35339401_23111645

Indeed this terminates again under scala -Yrepl-class-based. But we can't switch to that as a default until we sort out problems like:

% qscala -Yrepl-class-based

scala> class C(val a: Any) extends AnyVal
<console>:7: error: value class may not be a member of another class
       class C(val a: Any) extends AnyVal
             ^
@scabug

This comment has been minimized.

Show comment
Hide comment
@scabug

scabug Jan 13, 2015

@retronym said:
BTW, a workaround is to indirect through a lambda:

scala> () => Test.test(s => s + s)
res0: () => Unit = <function0>

scala> res0()
Splitting from [0 5)
Could not split [0 3)
Could not split [3 5)
Trying at 3 [3, 5)
Trying at 0 [0, 3)
Trying at 1 [0, 3)
Trying at 4 [3, 5)
Trying at 2 [0, 3)
Trying at 5 [3, 5)
Trying at 3 [0, 3)
Optional[twotwo]

scabug commented Jan 13, 2015

@retronym said:
BTW, a workaround is to indirect through a lambda:

scala> () => Test.test(s => s + s)
res0: () => Unit = <function0>

scala> res0()
Splitting from [0 5)
Could not split [0 3)
Could not split [3 5)
Trying at 3 [3, 5)
Trying at 0 [0, 3)
Trying at 1 [0, 3)
Trying at 4 [3, 5)
Trying at 2 [0, 3)
Trying at 5 [3, 5)
Trying at 3 [0, 3)
Optional[twotwo]
@scabug

This comment has been minimized.

Show comment
Hide comment
@scabug

scabug Jan 13, 2015

@retronym said:
See also: https://groups.google.com/forum/#!topic/scala-user/fljtxY9bJV0

/cc @som-snytt Do you think we could encode my workaround above into the object based wrappers to make the default REPL less prone to this problem?

scabug commented Jan 13, 2015

@retronym said:
See also: https://groups.google.com/forum/#!topic/scala-user/fljtxY9bJV0

/cc @som-snytt Do you think we could encode my workaround above into the object based wrappers to make the default REPL less prone to this problem?

@scabug

This comment has been minimized.

Show comment
Hide comment
@scabug

scabug Jan 13, 2015

@som-snytt said:
How about making it an App?

scala/scala#4246

scabug commented Jan 13, 2015

@som-snytt said:
How about making it an App?

scala/scala#4246

@scabug

This comment has been minimized.

Show comment
Hide comment
@scabug

scabug Aug 19, 2016

@SethTisue said:
@retronym should this be retargeted for 2.12.1 or Backlog?

scabug commented Aug 19, 2016

@SethTisue said:
@retronym should this be retargeted for 2.12.1 or Backlog?

@scabug

This comment has been minimized.

Show comment
Hide comment
@scabug

scabug Mar 1, 2017

Dave Gurnell (davegurnell) said (edited on Mar 1, 2017 10:35:25 AM UTC):
Seth suggested I post this recreation here. Try this on a 2.12.1 REPL:

import scala.concurrent._
import scala.concurrent.duration._
import scala.concurrent.ExecutionContext.Implicits.global

// This works fine:
val x = Future(42)
Await.result(x, 1.second)

// This throws a TimeoutException:
Await.result(Future(42), 1.second)

Adding scalacOptions += "-Ydelambdafy:inline" to build.sbt makes the problem go away.

The problem can also result in a NoClassDefFoundError as shown here (running Tut on an ebook project):

[tut] compiling: /Users/dave/dev/projects/advanced-scala/src/pages/test.md
java.lang.NoClassDefFoundError: Could not initialize class $line4.$read$$iw$$iw$$iw$$iw$$iw$$iw$
    at scala.runtime.java8.JFunction0$mcI$sp.apply(JFunction0$mcI$sp.java:12)
    at scala.concurrent.Future$.$anonfun$apply$1(Future.scala:653)
    at scala.util.Success.$anonfun$map$1(Try.scala:251)
    at scala.util.Success.map(Try.scala:209)
    at scala.concurrent.Future.$anonfun$map$1(Future.scala:287)
    ... manually elided for brevity
[tut] *** Error reported at /Users/dave/dev/projects/advanced-scala/src/pages/monads/test.md:6
java.util.concurrent.TimeoutException: Futures timed out after [1 second]
  at scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:255)
  at scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:259)
  at scala.concurrent.Await$.$anonfun$result$1(package.scala:190)
  at scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53)
  at scala.concurrent.Await$.result(package.scala:123)
  ... 88 elided
[error] (run-main-0) java.lang.Exception: Tut execution failed.
[trace] Stack trace suppressed: run last *:streams for the full output.
[trace] Stack trace suppressed: run last *:tutOnly for the full output.
[error] (*:tutOnly) Nonzero exit code: 1
[error] Total time: 5 s, completed 27-Feb-2017 09:38:19

scabug commented Mar 1, 2017

Dave Gurnell (davegurnell) said (edited on Mar 1, 2017 10:35:25 AM UTC):
Seth suggested I post this recreation here. Try this on a 2.12.1 REPL:

import scala.concurrent._
import scala.concurrent.duration._
import scala.concurrent.ExecutionContext.Implicits.global

// This works fine:
val x = Future(42)
Await.result(x, 1.second)

// This throws a TimeoutException:
Await.result(Future(42), 1.second)

Adding scalacOptions += "-Ydelambdafy:inline" to build.sbt makes the problem go away.

The problem can also result in a NoClassDefFoundError as shown here (running Tut on an ebook project):

[tut] compiling: /Users/dave/dev/projects/advanced-scala/src/pages/test.md
java.lang.NoClassDefFoundError: Could not initialize class $line4.$read$$iw$$iw$$iw$$iw$$iw$$iw$
    at scala.runtime.java8.JFunction0$mcI$sp.apply(JFunction0$mcI$sp.java:12)
    at scala.concurrent.Future$.$anonfun$apply$1(Future.scala:653)
    at scala.util.Success.$anonfun$map$1(Try.scala:251)
    at scala.util.Success.map(Try.scala:209)
    at scala.concurrent.Future.$anonfun$map$1(Future.scala:287)
    ... manually elided for brevity
[tut] *** Error reported at /Users/dave/dev/projects/advanced-scala/src/pages/monads/test.md:6
java.util.concurrent.TimeoutException: Futures timed out after [1 second]
  at scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:255)
  at scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:259)
  at scala.concurrent.Await$.$anonfun$result$1(package.scala:190)
  at scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53)
  at scala.concurrent.Await$.result(package.scala:123)
  ... 88 elided
[error] (run-main-0) java.lang.Exception: Tut execution failed.
[trace] Stack trace suppressed: run last *:streams for the full output.
[trace] Stack trace suppressed: run last *:tutOnly for the full output.
[error] (*:tutOnly) Nonzero exit code: 1
[error] Total time: 5 s, completed 27-Feb-2017 09:38:19
@adriaanm

This comment has been minimized.

Show comment
Hide comment
@adriaanm

adriaanm Jun 26, 2017

Member

Probably a wontfix in 2.12, but should consider this as we improve the repl in 2.13 (where we'll standardize on some form of class-based wrapping).

Member

adriaanm commented Jun 26, 2017

Probably a wontfix in 2.12, but should consider this as we improve the repl in 2.13 (where we'll standardize on some form of class-based wrapping).

@SethTisue

This comment has been minimized.

Show comment
Hide comment
@SethTisue

SethTisue Sep 11, 2018

Member

should this still be on the 2.13.0-RC1 milestone, or shall we push it out to 2.13.x or 2.14?

Member

SethTisue commented Sep 11, 2018

should this still be on the 2.13.0-RC1 milestone, or shall we push it out to 2.13.x or 2.14?

@som-snytt

This comment has been minimized.

Show comment
Hide comment
@som-snytt

som-snytt Sep 11, 2018

Basically anything from 2015 onward has to be 2.13. Anything earlier is so intractable it must wait for the Scala 3 re-write, when all bets are off. Unless someone is betting on Scala 3? Sorry if I spoke out of turn.

som-snytt commented Sep 11, 2018

Basically anything from 2015 onward has to be 2.13. Anything earlier is so intractable it must wait for the Scala 3 re-write, when all bets are off. Unless someone is betting on Scala 3? Sorry if I spoke out of turn.

@smarter

This comment has been minimized.

Show comment
Hide comment
@smarter

smarter Sep 12, 2018

It's indeed fixed in Dotty: scala/scala-dev#195 :)

smarter commented Sep 12, 2018

It's indeed fixed in Dotty: scala/scala-dev#195 :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment