Scala 2.13.0-M4: StackoverflowError during compilation

[xerial]

Scala 2.13.0-M4 shows the following StackOverflowError, which never happened in Scala 2.11, 2.12:

https://travis-ci.org/wvlet/airframe/builds/405575941 (Only Scala 2.13.0-M4 build fails)
https://travis-ci.org/wvlet/airframe/jobs/405575946 (StackOverflowError in Scala 2.13.0-M4):

[error] java.lang.StackOverflowError
[error] 	at scala.reflect.internal.tpe.TypeMaps$SubstMap.apply(TypeMaps.scala:674)
[error] 	at scala.reflect.internal.tpe.TypeMaps$SubstSymMap.apply(TypeMaps.scala:735)
[error] 	at scala.reflect.internal.Types$TypeVar.mapOver(Types.scala:3505)
[error] 	at scala.reflect.internal.tpe.TypeMaps$SubstMap.apply(TypeMaps.scala:674)
[error] 	at scala.reflect.internal.tpe.TypeMaps$SubstSymMap.apply(TypeMaps.scala:735)
[error] 	at scala.reflect.internal.tpe.TypeMaps$SubstSymMap.apply(TypeMaps.scala:708)
[error] 	at scala.reflect.internal.Types$RefinedType.mapOver(Types.scala:1687)
[error] 	at scala.reflect.internal.tpe.TypeMaps$SubstMap.apply(TypeMaps.scala:674)
[error] 	at scala.reflect.internal.tpe.TypeMaps$SubstSymMap.apply(TypeMaps.scala:735)
[error] 	at scala.reflect.internal.Types$TypeVar.mapOver(Types.scala:3505)
[error] 	at scala.reflect.internal.tpe.TypeMaps$SubstMap.apply(TypeMaps.scala:674)
[error] 	at scala.reflect.internal.tpe.TypeMaps$SubstSymMap.apply(TypeMaps.scala:735)
[error] 	at scala.reflect.internal.tpe.TypeMaps$SubstSymMap.apply(TypeMaps.scala:708)
[error] 	at scala.reflect.internal.Types$RefinedType.mapOver(Types.scala:1687)

How to reproduce:

$ git clone git@github.com:wvlet/airframe
$ cd airframe
$ ./sbt 
> ++ 2.13.0-M4
>  ~codec/compile

The cause of the error:
If I remove the following lines from the code, the error disappearred:
https://github.com/wvlet/airframe/blob/f16ea745e545b82d4b47e5725600ff325e2c6d44/airframe-codec/src/main/scala/wvlet/airframe/codec/MessageCodecFactory.scala#L60-L69

It seems there might be some cyclic references in the code, or already traversed references are not properly cached in the compiler side.

[SethTisue]

does increasing the JVM stack size setting make the problem go away? we need to distinguish whether it's actually consuming unbounded stack (bug), or just is consuming a little more stack than it used to and happened to exceed the limit on this particular codebase with your particular JVM settings (not a bug)

[xerial]

@SethTisue I tried -Xss128m (from 2m), but the problem still occurred.

It seems using two classes extending MessageCodec[Seq[A]] and MessageCodec[IndexSeq[A]] at the same time causes this issue:
https://github.com/wvlet/airframe/blob/32cf6f4b7e112e6fb4b9cd21cd2614542db2a5e9/airframe-codec/src/main/scala/wvlet/airframe/codec/CollectionCodec.scala#L59-L77

If I use only one of them, this error doesn't occur.

[SethTisue]

okay. are you able to reproduce it in a self-contained piece of code that we could incorporate into our regression tests...?

[xerial]

ok. I'll try.

[SethTisue]

note that even on 2.12.6, compiling with -Ytyper-debug induces compilation to fail:

[error] /Users/tisue/airframe/airframe-log/jvm/src/main/scala/wvlet/log/LogLevelScanner.scala:146:12: recursive method copy needs result type
[error] case class LogLevelScannerConfig(logLevelFileCandidates: Seq[String],
[error]            ^
[error] /Users/tisue/airframe/airframe-log/jvm/src/main/scala/wvlet/log/io/Resource.scala:185:14: recursive method copy needs result type
[error]   case class SystemFile(file: java.io.File, logicalPath: String) extends VirtualFile {
[error]              ^
[error] /Users/tisue/airframe/airframe-log/jvm/src/main/scala/wvlet/log/io/Resource.scala:197:14: recursive method copy needs result type
[error]   case class FileInJar(resourceURL: URL, logicalPath: String, isDirectory: Boolean) extends VirtualFile {
[error]              ^
[error] /Users/tisue/airframe/airframe-log/shared/src/main/scala/wvlet/log/LogRecord.scala:51:12: recursive method copy needs result type
[error] case class LogRecord(level: LogLevel, source: Option[LogSource], message: String, cause: Option[Throwable])
[error]            ^
[error] /Users/tisue/airframe/airframe-log/shared/src/main/scala/wvlet/log/LogRecord.scala:26:12: recursive method copy needs result type
[error] case class LogSource(path: String, fileName: String, line: Int, col: Int) {
[error]            ^
[error] /Users/tisue/airframe/airframe-log/shared/src/main/scala/wvlet/log/LogSupport.scala:45:51: illegal cyclic reference involving macro method error
[error]   protected def error(message: Any): Unit = macro errorLog
[error]                                                   ^
[error] /Users/tisue/airframe/airframe-log/shared/src/main/scala/wvlet/log/LogSupport.scala:46:69: illegal cyclic reference involving macro method error|    |-- def warn BYVALmode-EXPRmode (site: trait LoggingMethods) 
[error]   protected def error(message: Any, cause: Throwable): Unit = macro errorLogWithCause
[error]                                                                     ^
[error] /Users/tisue/airframe/airframe-log/shared/src/main/scala/wvlet/log/LogSupport.scala:48:50: illegal cyclic reference involving macro method warn|    |-- def warn BYVALmode-EXPRmode (site: trait LoggingMethods) 
[error]   protected def warn(message: Any): Unit = macro warnLog
[error]                                                  ^
[error] /Users/tisue/airframe/airframe-log/shared/src/main/scala/wvlet/log/LogSupport.scala:49:68: illegal cyclic reference involving macro method warn
[error]   protected def warn(message: Any, cause: Throwable): Unit = macro warnLogWithCause
[error]                                                                    ^
[error] /Users/tisue/airframe/airframe-log/shared/src/main/scala/wvlet/log/LogSupport.scala:51:50: illegal cyclic reference involving macro method info
[error]   protected def info(message: Any): Unit = macro infoLog
[error]                                                  ^
[error] /Users/tisue/airframe/airframe-log/shared/src/main/scala/wvlet/log/LogSupport.scala:52:68: illegal cyclic reference involving macro method info
[error]   protected def info(message: Any, cause: Throwable): Unit = macro infoLogWithCause|    |-- def debug BYVALmode-EXPRmode (site: trait LoggingMethods) 
[error]                                                                    ^
[error] /Users/tisue/airframe/airframe-log/shared/src/main/scala/wvlet/log/LogSupport.scala:54:51: illegal cyclic reference involving macro method debug|    |-- def debug BYVALmode-EXPRmode (site: trait LoggingMethods) 
[error]   protected def debug(message: Any): Unit = macro debugLog
[error]                                                   ^|    |    |-- Unit TYPEmode (site: macro method debug in LoggingMethods) 
[error] /Users/tisue/airframe/airframe-log/shared/src/main/scala/wvlet/log/LogSupport.scala:55:69: illegal cyclic reference involving macro method debug
[error]   protected def debug(message: Any, cause: Throwable): Unit = macro debugLogWithCause
[error]                                                                     ^
[error] /Users/tisue/airframe/airframe-log/shared/src/main/scala/wvlet/log/LogSupport.scala:57:51: illegal cyclic reference involving macro method trace
[error]   protected def trace(message: Any): Unit = macro traceLog
[error]                                                   ^
[error] /Users/tisue/airframe/airframe-log/shared/src/main/scala/wvlet/log/LogSupport.scala:58:69: illegal cyclic reference involving macro method trace
[error]   protected def trace(message: Any, cause: Throwable): Unit = macro traceLogWithCause
[error]                                                                     ^
[error] /Users/tisue/airframe/airframe-log/shared/src/main/scala/wvlet/log/Logger.scala:48:41: illegal cyclic reference involving macro method error|    |    \-> <error>
[error]   def error(message: Any): Unit = macro errorLogMethod
[error]                                         ^
[error] /Users/tisue/airframe/airframe-log/shared/src/main/scala/wvlet/log/Logger.scala:49:59: illegal cyclic reference involving macro method error
[error]   def error(message: Any, cause: Throwable): Unit = macro errorLogMethodWithCause
[error]                                                           ^
[error] /Users/tisue/airframe/airframe-log/shared/src/main/scala/wvlet/log/Logger.scala:51:40: illegal cyclic reference involving macro method warn
[error]   def warn(message: Any): Unit = macro warnLogMethod
[error]                                        ^
[error] /Users/tisue/airframe/airframe-log/shared/src/main/scala/wvlet/log/Logger.scala:52:58: illegal cyclic reference involving macro method warn
[error]   def warn(message: Any, cause: Throwable): Unit = macro warnLogMethodWithCause
[error]                                                          ^
[error] /Users/tisue/airframe/airframe-log/shared/src/main/scala/wvlet/log/Logger.scala:54:40: illegal cyclic reference involving macro method info
[error]   def info(message: Any): Unit = macro infoLogMethod
[error]                                        ^
[error] /Users/tisue/airframe/airframe-log/shared/src/main/scala/wvlet/log/Logger.scala:55:58: illegal cyclic reference involving macro method info
[error]   def info(message: Any, cause: Throwable): Unit = macro infoLogMethodWithCause
[error]                                                          ^|    |-- def debug BYVALmode-EXPRmode (site: class Logger) 
[error] /Users/tisue/airframe/airframe-log/shared/src/main/scala/wvlet/log/Logger.scala:57:41: illegal cyclic reference involving macro method debug
[error]   def debug(message: Any): Unit = macro debugLogMethod
[error]                                         ^
[error] /Users/tisue/airframe/airframe-log/shared/src/main/scala/wvlet/log/Logger.scala:58:59: illegal cyclic reference involving macro method debug
[error]   def debug(message: Any, cause: Throwable): Unit = macro debugLogMethodWithCause|    |-- def trace BYVALmode-EXPRmode (site: class Logger) 
[error]                                                           ^
[error] /Users/tisue/airframe/airframe-log/shared/src/main/scala/wvlet/log/Logger.scala:60:41: illegal cyclic reference involving macro method trace
[error]   def trace(message: Any): Unit = macro traceLogMethod
[error]                                         ^
[error] /Users/tisue/airframe/airframe-log/shared/src/main/scala/wvlet/log/Logger.scala:61:59: illegal cyclic reference involving macro method trace
[error]   def trace(message: Any, cause: Throwable): Unit = macro traceLogMethodWithCause
[error]                                                           ^
[error] 25 errors found
[error] (logJVM / Compile / compileIncremental) Compilation failed

[xerial]

@SethTisue interesting, but i guess it's unrelated to this issue because even if I remove Logger macro code, the problem still happens. Could you try 'codec/compile' ?

[xerial]

OK. It seems I need to fix the above cyclic reference first to see the typer debug log in codec project.

[SethTisue]

codec/compile is what I already tried, but I had enabled -Ytyper-debug build-wide. so yes, it's a good suggestion to enable it in the codec project only, using set codec / Compile / scalacOptions += "-Ytyper-debug". when I do that, I still get errors on 2.12.6:

[error] /Users/tisue/airframe/airframe-codec/src/main/scala/wvlet/airframe/codec/CollectionCodec.scala:62:14: recursive method copy needs result type
[error]   case class SeqCodec[A](surface: Surface, elementCodec: MessageCodec[A]) extends MessageCodec[Seq[A]] {
[error]              ^
[error] /Users/tisue/airframe/airframe-codec/src/main/scala/wvlet/airframe/codec/CollectionCodec.scala:72:14: recursive method copy needs result type
[error]   case class IndexedSeqCodec[A](surface: Surface, elementCodec: MessageCodec[A]) extends MessageCodec[IndexedSeq[A]] {
[error]              ^
[error] /Users/tisue/airframe/airframe-codec/src/main/scala/wvlet/airframe/codec/CollectionCodec.scala:91:14: recursive method copy needs result type
[error]   case class ListCodec[A](surface: Surface, elementCodec: MessageCodec[A]) extends MessageCodec[Seq[A]] {
[error]              ^
[error] /Users/tisue/airframe/airframe-codec/src/main/scala/wvlet/airframe/codec/CollectionCodec.scala:102:14: recursive method copy needs result type
[error]   case class JavaListCodec[A](elementCodec: MessageCodec[A]) extends MessageCodec[java.util.List[A]] {
[error]              ^
[error] /Users/tisue/airframe/airframe-codec/src/main/scala/wvlet/airframe/codec/CollectionCodec.scala:123:14: recursive method copy needs result type
[error]   case class MapCodec[A, B](keyCodec: MessageCodec[A], valueCodec: MessageCodec[B]) extends MessageCodec[Map[A, B]] {
[error]              ^
[error] /Users/tisue/airframe/airframe-codec/src/main/scala/wvlet/airframe/codec/CollectionCodec.scala:147:14: recursive method copy needs result type
[error]   case class JavaMapCodec[A, B](keyCodec: MessageCodec[A], valueCodec: MessageCodec[B])
[error]              ^
[error] /Users/tisue/airframe/airframe-codec/src/main/scala/wvlet/airframe/codec/DataType.scala:57:14: recursive method copy needs result type
[error]   case class ARRAY(elementType: DataType) extends StructuredType {
[error]              ^
[error] /Users/tisue/airframe/airframe-codec/src/main/scala/wvlet/airframe/codec/DataType.scala:63:14: recursive method copy needs result type
[error]   case class MAP(keyType: DataType, valueType: DataType) extends StructuredType {
[error]              ^
[error] /Users/tisue/airframe/airframe-codec/src/main/scala/wvlet/airframe/codec/DataType.scala:74:14: recursive method copy needs result type
[error]   case class UNION(types: Seq[RecordType]) extends StructuredType {
[error]              ^
[error] /Users/tisue/airframe/airframe-codec/src/main/scala/wvlet/airframe/codec/DataType.scala:83:14: recursive method copy needs result type
[error]   case class RecordType(typeName: String, column: Seq[Column]) extends DataType {
[error]              ^
[error] /Users/tisue/airframe/airframe-codec/src/main/scala/wvlet/airframe/codec/DataType.scala:113:14: recursive method copy needs result type
[error]   case class Column(name: String, columnType: DataType) {
[error]              ^
[error] /Users/tisue/airframe/airframe-codec/src/main/scala/wvlet/airframe/codec/ObjectCodec.scala:27:12: recursive method copy needs result type
[error] case class ObjectCodec[A](surface: Surface, paramCodec: Seq[MessageCodec[_]]) extends MessageCodec[A] with LogSupport {
[error]            ^
[error] /Users/tisue/airframe/airframe-codec/src/main/scala/wvlet/airframe/codec/ScalaStandardCodec.scala:23:14: recursive method copy needs result type
[error]   case class OptionCodec[A](elementCodec: MessageCodec[A]) extends MessageCodec[Option[A]] {
[error]              ^
[error] /Users/tisue/airframe/airframe-codec/src/main/scala/wvlet/airframe/codec/ScalaStandardCodec.scala:44:14: recursive method copy needs result type
[error]   case class TupleCodec(elementCodec: Seq[MessageCodec[_]]) extends MessageCodec[Product] {
[error]              ^
[error] /Users/tisue/airframe/airframe-codec/src/main/scala/wvlet/airframe/codec/StandardCodec.scala:133:14: recursive method copy needs result type
[error]   case class EnumCodec[A](enumType: Class[A]) extends MessageCodec[A] with LogSupport {
[error]              ^
[error] 15 errors found
[error] (codec / Compile / compileIncremental) Compilation failed
[error] Total time: 9 s, completed Jul 18, 2018 4:44:34 PM

whether it's related or not, I don't know (especially since we don't have a minimization yet)

[SethTisue]

OK. It seems I need to fix the above cyclic reference first to see the typer debug log in codec project.

(see my comment, which I wrote before I'd seen yours, even though it ended up showing up after it on GitHub)

[xerial]

@SethTisue
Here is the minimized code to reproduce recursive type error:
build.sbt

scalaVersion := "2.13.0-M4"
scalacOptions ++= Seq("-Ytyper-debug")

Sample.scala

package sample

trait Expr[A]
case class Elem[A](e:A) extends Expr[A]

object Sample {
   def main(args:Array[String]): Unit = {
     val e = Elem(1)
  }
}

[error] /Users/leo/work/tmp/scala-bug-2_13_0_M4/src/main/scala/Sample.scala:4:12: recursive method copy needs result type
[error] case class Elem[A](e:A) extends Expr[A]
[error]            ^
[error] one error found

[xerial]

• Scala 2.13.0-M4, 2.12.6 <- failed
• Scala 2.11.11 <- passed

[xerial]

It seems this two lines are good enough to reproduce this recursive type error:

trait Expr[A]
case class Elem[A](e:A) extends Expr[A]

[xerial]

This error didn't happen if the case class is changed to a regular class.

[xerial]

At least I've found a workaround for my project (using regular classes instead of case classes) to avoid this stackoverflow error.