JLine 3: make sure it works with `sbt console` as well as standalone #678

SethTisue · 2020-03-05T10:51:31Z

sbt uses JLine 2, so it's not exactly difficult to imagine things going wrong

the primary testing should be against sbt 1, but we also shouldn't break sbt 0.13 unless it turns out to be a too-deep rabbithole

cc @eed3si9n

SethTisue · 2020-03-05T12:16:59Z

at scala/scala#8036 (comment) @dwijnand mentioned "source-breakage of zinc's "console" abstraction of the REPL, which we'd have to address" — what does that mean?

dwijnand · 2020-03-05T12:24:25Z

I'm pretty sure what I meant was "I'm only likely to complain if I see a source-breakage that impacts sbt's console". In the hypothetical.

SethTisue · 2020-03-05T12:30:04Z

some further details in scala/scala#8036 (comment)

eed3si9n · 2020-03-05T14:46:59Z

I just tested again using scala/scala@2ef554a on Windows 7 VM, and it's still broken.

It says "WARNING: Unable to create a system terminal, creating a dumb terminal (enable debug logging for more information)".

More importantly, the prompt doesn't accept "1" or enter key. Nothing shows up. Ctrl-C does work, to exit out. I checked to see Scala 2.13.1 REPL using the same setup and that works fine.

lrytz · 2020-03-09T10:31:13Z

Summarizing what we discussed on Friday

JLine jars

JLine 2 (Scala 2.13.1 uses version 2.14.6) has jansi embedded in the jar (not shaded, org/fusecode classfiles, binaries in META-INF/native/). No dependency on jansi or jna.
JLine 3 (3.14.0 is the latest) no longer embeds jansi, the jar only has JLine classfiles, no native binaires. There are optional dependencies on jansi and jna.

2.13 jars

2.13.1: pom depends on jline 2.14.6 and jansi 1.12 (both not optional). The jansi dependency was added as a side-effect of add sbt-whitesource scala#8209 (comment).
2.13.0: pom depends on jline 2.14.6 (not optional)
2.13 compiler jar ships without any integrated / shaded jline or jansi (scala/scala@3de08541f2)
the "distribution" (zip archive from the website) ships with the dependencies in the lib folder (jline, jansi for 2.13.1)

2.12 jars

2.12.9 / 2.12.10 have optional dependencies on jline 2.14.6 and jansi 1.12 (jansi was likely added as a side-effect of [nomerge] add sbt-whitesource scala#8167)
2.12.8 (and probably before) has an optional dependency on 2.14.6
the distribution for 2.12/8/9/10 have jline-2.14.6.jar in the lib folder. no jansi
2.12.8/9/10 all ship with jline and jansi shaded into the scala-compiler.jar
- binaries in META-INF/native/linux64/libjansi.so
- classfiles in scala/tools/fusesource_embedded/jansi and scala/tools/jline_embedded

It looks like the shaded jline was originally added in scala/scala#4563 (ant build) for 2.11.7. The sbt version of this is scala/scala@9debc84dcd. It seems that the repl would only use the shaded jline as a fallback if jline cannot be found on the classpath. I don't know what was the motivation, but it seems to have something to do with spark.

eed3si9n · 2020-03-19T07:01:21Z

I got a $160 laptop for Windows testing last November, and I finally set it up to do some testing today. On it, it has a recent Windows 10.

Scala 2.13.1

sbt:hello> show scalaInstance
[info] Scala instance { version label 2.13.1, actual version 2.13.1,
library jars: C:\Users\eugen\AppData\Local\Coursier\cache\v1\https\repo1.maven.org\maven2\org\scala-lang\scala-library\2.13.1\scala-library-2.13.1.jar, 
compiler jar: C:\Users\eugen\AppData\Local\Coursier\cache\v1\https\repo1.maven.org\maven2\org\scala-lang\scala-compiler\2.13.1\scala-compiler-2.13.1.jar,
other jars:
C:\Users\eugen\AppData\Local\Coursier\cache\v1\https\repo1.maven.org\maven2\jline\jline\2.14.6\jline-2.14.6.jar,
C:\Users\eugen\AppData\Local\Coursier\cache\v1\https\repo1.maven.org\maven2\org\fusesource\jansi\jansi\1.12\jansi-1.12.jar,
C:\Users\eugen\AppData\Local\Coursier\cache\v1\https\repo1.maven.org\maven2\org\scala-lang\scala-reflect\2.13.1\scala-reflect-2.13.1.jar
}

Scala 2.13.2 (JLine 3 branch)

sbt:hello> show scalaInstance
[info] Scala instance { version label 2.13.2-bin-SNAPSHOT, actual version 2.13.2-bin-SNAPSHOT, 
library jars: C:\Users\eugen\.ivy2\local\org.scala-lang\scala-library\2.13.2-bin-SNAPSHOT\jars\scala-library.jar,
compiler jar: C:\Users\eugen\.ivy2\local\org.scala-lang\scala-compiler\2.13.2-bin-SNAPSHOT\jars\scala-compiler.jar,
other jars:
C:\Users\eugen\AppData\Local\Coursier\cache\v1\https\repo1.maven.org\maven2\org\fusesource\jansi\jansi\1.18\jansi-1.18.jar,
C:\Users\eugen\.ivy2\local\org.scala-lang\scala-reflect\2.13.2-bin-SNAPSHOT\jars\scala-reflect.jar,
C:\Users\eugen\AppData\Local\Coursier\cache\v1\https\repo1.maven.org\maven2\org\jline\jline\3.14.0\jline-3.14.0.jar,
C:\Users\eugen\AppData\Local\Coursier\cache\v1\https\repo1.maven.org\maven2\org\scala-lang\modules\scala-asm\7.3.1-scala-1\scala-asm-7.3.1-scala-1.jar
}

Dotty 0.23.0-RC1

[info] Scala instance { version label 0.23.0-RC1, actual version 0.23.0-RC1,
library jars: 
C:\Users\eugen\.ivy2\cache\ch.epfl.lamp\dotty-library_0.23\jars\dotty-library_0.23-0.23.0-RC1.jar,
C:\Users\eugen\.ivy2\cache\org.scala-lang\scala-library\jars\scala-library-2.13.1.jar,
compiler jar: C:\Users\eugen\.ivy2\cache\ch.epfl.lamp\dotty-compiler_0.23\jars\dotty-compiler_0.23-0.23.0-RC1.jar,
other jars: 
C:\Users\eugen\.ivy2\cache\ch.epfl.lamp\dotty-interfaces\jars\dotty-interfaces-0.23.0-RC1.jar, 
C:\Users\eugen\.ivy2\cache\ch.epfl.lamp\tasty-core_0.23\jars\tasty-core_0.23-0.23.0-RC1.jar, 
C:\Users\eugen\.ivy2\cache\org.scala-lang.modules\scala-asm\bundles\scala-asm-7.3.1-scala-1.jar, 
C:\Users\eugen\.ivy2\cache\org.scala-sbt\compiler-interface\jars\compiler-interface-1.2.5.jar, 
C:\Users\eugen\.ivy2\cache\org.scala-sbt\util-interface\jars\util-interface-1.2.2.jar, 
C:\Users\eugen\.ivy2\cache\org.jline\jline-reader\jars\jline-reader-3.9.0.jar, 
C:\Users\eugen\.ivy2\cache\org.jline\jline-terminal\jars\jline-terminal-3.9.0.jar, 
C:\Users\eugen\.ivy2\cache\org.jline\jline-terminal-jna\jars\jline-terminal-jna-3.9.0.jar, 
C:\Users\eugen\.ivy2\cache\net.java.dev.jna\jna\jars\jna-4.2.2.jar
}

[error] (run-main-0) java.lang.IllegalStateException: Unable to create a system terminal
[error] java.lang.IllegalStateException: Unable to create a system terminal
[error]         at org.jline.terminal.TerminalBuilder.doBuild(TerminalBuilder.java:316)
[error]         at org.jline.terminal.TerminalBuilder.build(TerminalBuilder.java:262)
[error]         at dotty.tools.repl.JLineTerminal.<init>(JLineTerminal.scala:22)
[error]         at dotty.tools.repl.ReplDriver.runUntilQuit(ReplDriver.scala:105)
[error]         at xsbt.ConsoleInterface.run(ConsoleInterface.java:52)
[error]         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[error]         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
[error]         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[error]         at java.lang.reflect.Method.invoke(Method.java:498)
[error]         at sbt.internal.inc.AnalyzingCompiler.call(AnalyzingCompiler.scala:248)
[error]         at sbt.internal.inc.AnalyzingCompiler.console(AnalyzingCompiler.scala:210)
[error]         at sbt.Console.console0$1(Console.scala:48)
[error]         at sbt.Console.$anonfun$apply$2(Console.scala:51)
[error]         at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
[error]         at sbt.util.InterfaceUtil$$anon$1.get(InterfaceUtil.scala:10)
[error]         at sbt.TrapExit$App.run(TrapExit.scala:257)
[error]         at java.lang.Thread.run(Thread.java:748)
[error] Nonzero exit code: 1

I was hoping that Dotty can run console task on Windows, but apparently not.

eed3si9n · 2020-03-19T18:11:34Z

I was hoping that key input not working and "WARNING: Unable to create a system terminal, creating a dumb terminal (enable debug logging for more information)" were specifically a Command Prompt (Cmd) problem, but I am getting the same unfortunately on Windows Terminal as well.

eed3si9n · 2020-03-19T20:45:15Z

Got this to work after switching to JNA.

Ref scala/scala-dev#678 https://github.com/jline/jline3#jansi-vs-jna says > On the Windows platform, relying on native calls is mandatory, so you need to have either Jansi or JNA library in your classpath along with the `jline-terminal-jansi` or `jline-terminal-jna` jar. When I tested on Windows 7 or Windows 10 using `sbt console` I got "WARNING: Unable to create a system terminal, creating a dumb terminal (enable debug logging for more information)" and key input also didn't work. (This could be more to do with sbt). I am guessing that Jansi used by sbt itself maybe interfering with Jansi that gets loaded from the compiler. Normally these duplicates are resolved by creating a sandbox ClassLoader, but maybe it doesn't exactly work for native libraries loaded into the same process. When I switched to using JNA the console worked ok at least for sbt 1.3.8. ### notes To avoid the interference of dependencies between repl and sbt, maybe sbt needs to switch to forking to a fresh JVM for console, or at least provide an option to do so.

eed3si9n · 2020-03-20T03:59:39Z

As I wrote on som-snytt/scala#5 I am guessing that Jansi used by sbt itself may be interfering with Jansi that gets loaded from the compiler. Normally these duplicates are resolved by creating a sandbox ClassLoader, but maybe it doesn't exactly work for native libraries loaded into the same process.

I wondering if this would go away if we forked for console instead of reusing the same JVM
sbt/sbt#1918

Ref scala/scala-dev#678 https://github.com/jline/jline3#jansi-vs-jna says > On the Windows platform, relying on native calls is mandatory, so you need to have either Jansi or JNA library in your classpath along with the `jline-terminal-jansi` or `jline-terminal-jna` jar. When I tested on Windows 7 or Windows 10 using `sbt console` I got "WARNING: Unable to create a system terminal, creating a dumb terminal (enable debug logging for more information)" and key input also didn't work. (This could be more to do with sbt). I am guessing that Jansi used by sbt itself maybe interfering with Jansi that gets loaded from the compiler. Normally these duplicates are resolved by creating a sandbox ClassLoader, but maybe it doesn't exactly work for native libraries loaded into the same process. When I switched to using JNA the console worked ok at least for sbt 1.3.8. To avoid the interference of dependencies between repl and sbt, maybe sbt needs to switch to forking to a fresh JVM for console, or at least provide an option to do so.

SethTisue · 2020-03-24T21:01:46Z

@eed3si9n changes on scala/scala#8036 LGTY?

Goal: avoid conflicting with sbt's own Jansi version Ref scala/scala-dev#678 https://github.com/jline/jline3#jansi-vs-jna says > On the Windows platform, relying on native calls is mandatory, so > you need to have either Jansi or JNA library in your classpath along > with the `jline-terminal-jansi` or `jline-terminal-jna` jar. When I tested on Windows 7 or Windows 10 using `sbt console` I got "WARNING: Unable to create a system terminal, creating a dumb terminal (enable debug logging for more information)" and key input also didn't work. (This could be more to do with sbt). I am guessing that Jansi used by sbt itself maybe interfering with Jansi that gets loaded from the compiler. Normally these duplicates are resolved by creating a sandbox ClassLoader, but maybe it doesn't exactly work for native libraries loaded into the same process. additional changes by Seth: * change Jansi -> JNA in legal info, Intellij config * make REPL's JNA & JLine dependencies optional. most users of scala-compiler.jar just want the compiler and aren't embedding the REPL, so there's no need to pull REPL stuff onto their classpath (and doing so might even cause a conflict. if someone is embedding the REPL and does want JNA, they can take care of adding the optional dependency themselves. Co-authored-by: Seth Tisue <seth@tisue.net>

Goal: avoid conflicting with sbt's own Jansi version Ref scala/scala-dev#678 https://github.com/jline/jline3#jansi-vs-jna says > On the Windows platform, relying on native calls is mandatory, so > you need to have either Jansi or JNA library in your classpath along > with the `jline-terminal-jansi` or `jline-terminal-jna` jar. When I tested on Windows 7 or Windows 10 using `sbt console` I got "WARNING: Unable to create a system terminal, creating a dumb terminal (enable debug logging for more information)" and key input also didn't work. (This could be more to do with sbt). I am guessing that Jansi used by sbt itself maybe interfering with Jansi that gets loaded from the compiler. Normally these duplicates are resolved by creating a sandbox ClassLoader, but maybe it doesn't exactly work for native libraries loaded into the same process. additional changes by Seth: * change Jansi -> JNA in legal info, Intellij config Co-authored-by: Seth Tisue <seth@tisue.net>

SethTisue · 2020-03-25T15:06:23Z

@eed3si9n should we close this, or do you want to do some more testing first?

eed3si9n · 2020-03-25T15:07:44Z

I guess we can close it.

SethTisue added the t:repl JLine 3 upgrade (scala/scala#8036) label Mar 5, 2020

SethTisue added the blocker label Mar 6, 2020

SethTisue assigned eed3si9n Mar 10, 2020

eed3si9n mentioned this issue Mar 20, 2020

Prefer JNA som-snytt/scala#5

Closed

eed3si9n closed this as completed Mar 25, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

JLine 3: make sure it works with `sbt console` as well as standalone #678

JLine 3: make sure it works with `sbt console` as well as standalone #678

SethTisue commented Mar 5, 2020 •

edited

SethTisue commented Mar 5, 2020

dwijnand commented Mar 5, 2020

SethTisue commented Mar 5, 2020

eed3si9n commented Mar 5, 2020

lrytz commented Mar 9, 2020

eed3si9n commented Mar 19, 2020

eed3si9n commented Mar 19, 2020

eed3si9n commented Mar 19, 2020

eed3si9n commented Mar 20, 2020 •

edited

SethTisue commented Mar 24, 2020

SethTisue commented Mar 25, 2020

eed3si9n commented Mar 25, 2020

JLine 3: make sure it works with sbt console as well as standalone #678

JLine 3: make sure it works with sbt console as well as standalone #678

Comments

SethTisue commented Mar 5, 2020 • edited

SethTisue commented Mar 5, 2020

dwijnand commented Mar 5, 2020

SethTisue commented Mar 5, 2020

eed3si9n commented Mar 5, 2020

lrytz commented Mar 9, 2020

eed3si9n commented Mar 19, 2020

Scala 2.13.1

Scala 2.13.2 (JLine 3 branch)

Dotty 0.23.0-RC1

eed3si9n commented Mar 19, 2020

eed3si9n commented Mar 19, 2020

eed3si9n commented Mar 20, 2020 • edited

SethTisue commented Mar 24, 2020

SethTisue commented Mar 25, 2020

eed3si9n commented Mar 25, 2020

JLine 3: make sure it works with `sbt console` as well as standalone #678

JLine 3: make sure it works with `sbt console` as well as standalone #678

SethTisue commented Mar 5, 2020 •

edited

eed3si9n commented Mar 20, 2020 •

edited