Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improved Repl #325

Open
adriaanm opened this issue Mar 2, 2017 · 30 comments
Open

Improved Repl #325

adriaanm opened this issue Mar 2, 2017 · 30 comments
Milestone

Comments

@adriaanm
Copy link
Member

@adriaanm adriaanm commented Mar 2, 2017

Here are some ideas for improving the repl:

  • syntax highlighting
  • class-based wrapping (PR)
  • take advantage of Java 9 features
  • improve embeddability of repl in worksheets for fast data platforms

Steal Incorporate the improvements already implemented in the Ammonite REPL:

  • improved multi-line editing/history
  • magic imports (ivy dependencies etc)

More ideas / requirements:

  • reload class path when changing/recompiling project (faster test cycle)
  • ctrl-c stops execution of program instead of repl itself
  • test in IntelliJ shell window
@adriaanm adriaanm added this to the 2.13 milestone Mar 2, 2017
@DavidPerezIngeniero
Copy link

@DavidPerezIngeniero DavidPerezIngeniero commented Mar 10, 2017

+1 for incorporating Ammonite, this is a fantastic tool.

@kjsingh
Copy link

@kjsingh kjsingh commented Mar 12, 2017

As long as we keep the performance smooth!

@yinshurman
Copy link

@yinshurman yinshurman commented May 5, 2017

in the REPL, enter a Tab will display all the available names imported in the scope which makes a lot troubles in some case,for example,when pasting codes with tab as indent from editors like vim even in the :paste mode ,the screen if full of unexpected names.
by the way ,the auto completion by Tab direct print all the names in the screen makes it hard read on the console.Can't you just display it another way ? For example, like IPython' way.

@yinshurman
Copy link

@yinshurman yinshurman commented May 5, 2017

In Python you can investigate any object foo (type, object in a global variable, built-in function, etc.) using foo to see what the thing evaluates to, type(foo) to show its type, dir(foo) to tell you the methods you can call on it, and help(foo) to get the built-in documentation. You can even do things like help("re") to find out the documentation on the package named re (which holds regular-expression objects and methods), even though there isn't an object associated with it.Can scala REPL does things like this ?

@som-snytt
Copy link

@som-snytt som-snytt commented May 5, 2017

@yinshurman about tab, are you using a recent version with scala/bug#9336 ? That enables the kludge in jline; but maybe it could do more in paste mode.

@yinshurman
Copy link

@yinshurman yinshurman commented May 5, 2017

I'm use scala of version 2.11.7

@adriaanm
Copy link
Member Author

@adriaanm adriaanm commented May 5, 2017

@yinshurman
Copy link

@yinshurman yinshurman commented May 8, 2017

It's fixed by add a tip "Display all 435 possibilities? (y or n)" every time you enter a tab.
Instead of direct printing all the available options ,I think it's better to display them another way,for example,when you type foo. and enter a tab, all the available options display in another little window right behind the foo.,and when you selected the option you needed,the window disappeared too. I think it is better comfortable

@som-snytt
Copy link

@som-snytt som-snytt commented May 8, 2017

It's a nice consequence of English pronunciation that a windowing REPL or wrepl sounds the same as repl.

@adriaanm
Copy link
Member Author

@adriaanm adriaanm commented May 9, 2017

The plan is to first split the current repl into:

  • a layer on top of the compiler (similar to the presentation compiler)
  • a shell that implements the UI (and interfaces with jline3).

The api exposed by the compiler should also support other shells, such as ammonite / spark notebooks. Eventually, maybe we can support connecting to this repl compiler instance using the language server protocol (as in lampepfl/dotty#2195)

I've just started hacking on this in my WIP branch: https://github.com/scala/scala/compare/2.13.x...adriaanm:repl-modularize?expand=1

@adriaanm
Copy link
Member Author

@adriaanm adriaanm commented May 19, 2017

scala/scala#5903 takes the first steps towards splitting the REPL into a shell (UI) component and a compiler to support evaluating expressions and definitions + auto-completion (similar to the presentation compiler). The idea is that there could be multiple front-ends (the default, ammonite, spark worksheets,...) that all use the same underling compiler for consistent evaluation and type checking semantics.

Also, the default shell will likely use jline3, which has better support multi-line editing, highlighting, parsing as you type, etc.

@adriaanm
Copy link
Member Author

@adriaanm adriaanm commented Jun 13, 2017

By the way, we're currently not planning to backport the REPL work to 2.12. Please target 2.13.x for any PR that changes the REPL (or, please fix it first in 2.13 and backport to 2.12).

We are open to discussing a backport of the rework, though -- especially if someone's interested to contribute it :-) (A backport would have to be more careful not to break anyone currently embedding the 2.12 repl).

Finally, we'd love to hear feedback on the interface proposed in scala/scala#5903 from anyone embedding the REPL.

@retronym
Copy link
Member

@retronym retronym commented Aug 10, 2017

Another wishlist item that was in part prototyped in REPLs of yore would some features that exploit the runtime types and reflection.

  • Autocompletion of x.asInstanceOf[<TAB>] could look at the x.getClass, figure out the corresponding ClassSymbol, and then intelligently merge that with the static type of x. Example if x.getClass == classOf[Some[_], and x is of type Option[String], offer to downcast to Some[String].
  • Another UI option for that would be to have another level of tab completion used a safe downcast as an adaptation.
  • Provide a utility method or command to do some reflection based stringification of objects
  • Cheat mode: Add the ability to call private methods and fields by generating reflective calls

An unrelated feature that I think would be beneficial would be a UI to show the expected type at the cursor. Maybe use this to filter or prioritize autocomplete suggestions. I prototyped this in my last round of REPL renovations, but didn't follow through.

@retronym
Copy link
Member

@retronym retronym commented Aug 18, 2017

One more wishlist item for power mode: a version of reify that just funnels the actual typed argument tree back into userland.

You can simulate this at the moment with a macro and a backchannel:

⚡ scala -cp /tmp
Welcome to Scala 2.12.3 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_112).
Type in expressions for evaluation. Or try :help.

scala> :power
Power mode enabled. :phase is at typer.
import scala.tools.nsc._, intp.global._, definitions._
Try :help or completions for vals._ and power._

scala> jz.TreeOf { class C { def foo = 42 } }
res0: $r.intp.global.Tree =
{
  class C extends scala.AnyRef {
    def <init>(): C = {
      C.super.<init>();
      ()
    };
    def foo: Int = 42
  };
  ()
}

scala> :quit

⚡ cat sandbox/treeof.scala
package jz
import scala.language.experimental.macros
import scala.reflect.macros.whitebox.Context

object TreeOf {
  def apply[A](a: A): Any = macro impl
  def impl(c: Context)(a: c.Tree): c.Tree = {
    val key = "jz.treeof.backchannel"
    System.getProperties.put(key, a)
    import c.universe._
    q"_root_.java.lang.System.getProperties.remove($key).asInstanceOf[Tree]"
  }
}
@adriaanm
Copy link
Member Author

@adriaanm adriaanm commented Aug 19, 2017

that's beautiful

@zaxebo1
Copy link

@zaxebo1 zaxebo1 commented Sep 29, 2017

is there a "plugin loading mechanism for REPL", with a stable API and proper documentation/example, "through which one can add new inbuilt commands(colon commands) to REPL" , say ":my_new_command" , just like the inbuilt commands ":help" , ":paste" etc.

@som-snytt
Copy link

@som-snytt som-snytt commented Sep 30, 2017

Welcome to Scala 2.12.3 (OpenJDK 64-Bit Server VM, Java 1.8.0_131).
Type in expressions for evaluation. Or try :help.

scala> import scala.tools.nsc._,interpreter._
import scala.tools.nsc._
import interpreter._

scala> val loop = new ILoop { override def commands = super.commands :+ LoopCommand.cmd("greet","helper","go for it", s => { println("hi") ; Result(false, None) }) }
loop: scala.tools.nsc.interpreter.ILoop = $anon$1@58f31629

scala> loop.process(new Settings)
Welcome to Scala 2.12.3 (OpenJDK 64-Bit Server VM, Java 1.8.0_131).
Type in expressions for evaluation. Or try :help.

scala> :greet
hi
res0: Boolean = true

Other than that, no. So, no.

@som-snytt
Copy link

@som-snytt som-snytt commented Feb 22, 2018

REPL should clip my stackoverflows for me. Give me a last command to display it. And elide repeating segments:

  at .$anonfun$f$1$adapted(<console>:11)
  at .$anonfun$f$1(<console>:11)
  at .$anonfun$f$1$adapted(<console>:11)
  at .$anonfun$f$1(<console>:11)
@hvesalai
Copy link

@hvesalai hvesalai commented Mar 6, 2018

See also scala/scala#6382

@dwijnand
Copy link
Member

@dwijnand dwijnand commented Aug 8, 2018

jline3 upgrade split off to https://github.com/scala/scala-dev/issues/535 (and scheduled for 2.13.0-RC1)

@szeiger szeiger removed this from the 2.13.1 milestone Sep 9, 2019
@retronym
Copy link
Member

@retronym retronym commented Nov 28, 2019

Here's a test case that shows the inefficiency of the current wrapper schemes that require wrapper nesting proportional to the length of the REPL session.

https://gist.github.com/ad2505e99fc4084e4f9e79aa5b147aad

We should revive scala/scala@2.13.x...adriaanm:repl-wrappers to address that problem. In that branch, the test above fails suggesting an unwanted semantic change:

scala> import bar.foo
<console>:16: error: reference to bar is ambiguous;
it is imported twice in the same scope by
import bar
and import bar
       import bar.foo
              ^

scala>  foo
<console>:17: error: reference to foo is ambiguous;
it is imported twice in the same scope by
import foo
and import bar.foo
        foo
        ^
@som-snytt
Copy link

@som-snytt som-snytt commented Nov 28, 2019

I wanted the nesting level import to compile a directory as a single compilation unit, where files are not just concatenated, but each file is in a magic nesting block. (That requires both increasing and decreasing, for entering and leaving the block.)

I think REPL lines should compile to functions, taking values instead of importing them. That would be closer to the scripting environment, where contexts maintain bindings.

@SethTisue
Copy link
Member

@SethTisue SethTisue commented Feb 23, 2020

@dwijnand currently this is on the 2.13.2 milestone. is there anything here that 1) is crucial for 2.13.2 but 2) isn't already covered by existing PRs/issues that are already milestoned 2.13.2?

@som-snytt
Copy link

@som-snytt som-snytt commented Feb 23, 2020

@SethTisue definitely #325 (comment) (self-deprecation)

@dwijnand What's going on? Are you the person tasked with getting jline 3 over the goal line, or something else? From the outside, it's hard to know if it's worth touching anything in that area for obvious reasons (merge conflicts rather than personality conflicts).

@SethTisue
Copy link
Member

@SethTisue SethTisue commented Feb 23, 2020

Are you the person tasked with getting jline 3 over the goal line

That's me (and it's literally the top item in my queue at present, other than making sure all our 2.13.2 ducks are in a line generally).

...but perhaps Dale will help also, depending on how much else he has going on. (He's up to speed now on REPL stuff, having just gotten scala/scala#8712 over the 2.12.x finish line, though merging it onto 2.13 is still in process.)

it's hard to know if it's worth touching anything in that area

Mm hmm. It makes sense to hold off a bit on anything that's likely to more-than-trivially merge-conflict with the JLine 3 work and the class-based changes.

@dwijnand
Copy link
Member

@dwijnand dwijnand commented Feb 23, 2020

@dwijnand currently this is on the 2.13.2 milestone. is there anything here that 1) is crucial for 2.13.2 but 2) isn't already covered by existing PRs/issues that are already milestoned 2.13.2?

I think we're using this ticket as a proxy for "let's not forget about the REPL" issue, every release. So with the class-based changes and hopefully the JLine 3 upgrade, both targeting 2.13.2, this one can be pushed to .3.

@dwijnand What's going on? Are you the person tasked with getting jline 3 over the goal line, or something else? From the outside, it's hard to know if it's worth touching anything in that area for obvious reasons (merge conflicts rather than personality conflicts).
it's hard to know if it's worth touching anything in that area

Mm hmm. It makes sense to hold off a bit on anything that's likely to more-than-trivially merge-conflict with the JLine 3 work and the class-based changes.

@som-snytt Would you benefit from having a, let's say, every-two-week video call? Might help on a regular basis touch base on things that slip or that we forget to communicate clearly.

Seth's on JLine 3 duty. But my branch-merging of class-based hit merge conflicts. Not a huge amount, but enough to likely incur another rebase push on the JLine 3 upgrade. I'm happy to do that rebase, btw, to help the cause (and b/c I have a good tooling setup for rebases, and b/c I know what the cause of the rebase). Oh than that, I don't have any work coming up for the REPL, so no potential future conflicts from me (both merge conflicts and personality conflicts).

@SethTisue SethTisue modified the milestones: 2.13.2, 2.13.3 Feb 23, 2020
@SethTisue
Copy link
Member

@SethTisue SethTisue commented Mar 10, 2020

The JLine 3 upgrade is progressing well at scala/scala#8036 and https://github.com/scala/scala-dev/labels/t%3Ajline3 . There are separate tickets and we'd love to have contributors tackle any of them.

In general, further REPL improvements are a likely area of focus for us in 2020. (It stalled somewhat in 2019, but only because we had to focus on getting 2.13.0 and 2.13.1 out the door. That's done now.)

@SethTisue
Copy link
Member

@SethTisue SethTisue commented May 15, 2020

I did a little Twitter poll at https://twitter.com/SethTisue/status/1255633365232840704

unscientific results:

Screen Shot 2020-05-14 at 8 30 52 PM

"add dependencies live" is the clear winner, but also the most ambitious as it involves bringing in coursier as a dependency. it might best be tackle-able in combination with replacing our current start scripts with coursier-based ones (as @lrytz observed)

it was nice to see the other two choices I gave, which are easier, both got a good amount of support

in the replies there was interest in displaying type names in their shortest reachable form, as Ammonite already does:

@ scala.collection.mutable.Buffer.empty 
res0: collection.mutable.Buffer[Nothing] = ArrayBuffer()

@ import collection._ 
import collection._

@ scala.collection.mutable.Buffer.empty 
res2: mutable.Buffer[Nothing] = ArrayBuffer()

so I think that should be on the short list as well.

@retronym
Copy link
Member

@retronym retronym commented May 15, 2020

An intermediate solution would be to require coursier to be installed separately and interact with it via its command line interface.

@bjornregnell
Copy link

@bjornregnell bjornregnell commented May 15, 2020

@SethTisue I think showing scaladoc in REPL is, in the minds of the beginner coders that I'm teaching, the most important feature on your Twitter poll short-list. (The other features in the REPL upgrade already in there are really great for beginners as well, like multi line editing. 👍 🥇 )

@SethTisue SethTisue modified the milestones: 2.13.3, 2.13.4 Jun 23, 2020
@dwijnand dwijnand modified the milestones: 2.13.4, 2.13.14 Oct 16, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
You can’t perform that action at this time.