implement a proper REPL #9898

Closed
thestinger opened this Issue Oct 17, 2013 · 32 comments

Projects

None yet
@thestinger
Contributor

It shouldn't re-compile and re-run the entire input so far with each new addition, and should be able to recover from a compilation error or failure. This should likely be implemented in a separate repository until it's solid and has a working test suite.

A good basis for the design would be the cling REPL for C++ built on top of libclang using LLVM's JIT compiler.

This was referenced Dec 2, 2013
@adnelson

not sure why this (seemingly) isn't in higher demand. I would love to have a REPL.

@rklaehn
rklaehn commented Feb 11, 2014

This would be very nice. When developing scala, I really enjoy trying out language features in the REPL. It does not have to be perfect.

@thestinger
Contributor

Posting comments on an issue saying you want it fixed isn't going to make it happen any faster. It just decreases the signal to noise ratio on the bug tracker.

@Jurily
Contributor
Jurily commented Feb 12, 2014

@adnelson, @rklaehn: Rust's test mechanism makes it really easy to run snippets in the same context you'd have in real code. I actually prefer it to ghci and friends. HTH.

@catharsis
Contributor

Would it be a good idea to implement this using the existing lexer/parser? I've had a look at mod.rs in libsyntax/parse and it seems to me that it wouldn't be too hard to build something that reads tokens from stdin and emitting tokens as they arrive. Actually, looking at driver.rs (librustc/driver/driver.rs) it looks like this is sort of already possible at a higher level through compile_input, though perhaps that doesn't provide quite the flexibility you'd want in a REPL - not sure.

I'd love to play around with this some (provided I can find the time), but I'd really appreciate if some of the more experienced devs would chime in and warn me off if it's a terrible idea or if there are other plans on how this should be done. Any other hints or suggestions would of course also be highly appreciated.

@abonander
Contributor

Could we make use of lli to interpret/JIT LLVM bitcode?

@thestinger
Contributor

There's no point in shelling out to a command-line utility when it can be done with the same library functions lli is using. The LLVM JIT is trivial to use, and the only difference relative to AOT compilation is that it's outputting code in-memory and making it callable via linker hacks rather than making an object file. It's not what makes this hard.

@alexchandel

+1 for ease of adoption. I've learned most languages with the help of an interpreter (python, C#, JS, Haskell all have good ones).

@willprice

I too agree with alexchandel, is there any work being done on this?

@hastebrot

I also like to see the missing bindings for LLVMs ExecutionEngine implemented in the rustc crate. It seems that ExecutionEngine is needed for a REPL.

@jauhien
Contributor
jauhien commented Sep 21, 2014

Yes, ExecutionEngine seems to be needed. Are there any plans to implement full LLVM bindings in rust?

@thestinger
Contributor

Implementing a REPL is a complex project, and exposing / using MCJIT is a trivial piece of that puzzle. There's no point of getting side tracked on that issue here.

@japaric
Member
japaric commented Oct 14, 2014

This issue is a feature request, and needs an approved RFC to be implemented (*). It should be moved to the rust-lang/rfcs repo. (As per the issues policy)

cc @nick29581

(*) I think it makes sense to develop the REPL out of tree (it would still be able to link to librustc, libsyntax or whatever is needed), if it needs more rustc internals/llvm bindings exposed, those bits can be requested via RFCs. (Note that I've no experience writing REPLs/compilers (the closest thing I've worked with are syntax extensions!) so I could be speaking nosense)

@sinistersnare
Contributor

I do not think this warrants an RFC because it is not a change to the language or the standard libraries. I am personally OK with keeping A-an-interesting-project labeled issues in this repo.

@nrc
Contributor
nrc commented Oct 14, 2014

Yeah, RFCs are only for language features or far-reaching changes to the libs. Leaving here.

@jgmize
jgmize commented Oct 14, 2014

One option for implementing a REPL that I find interesting would be to implement an IPython Kernel for Rust, which would allow you to make use of not only the terminal based frontend, but also other existing IPython (or Jupyter) frontends like the notebook, QT console, or emacs notebook.

@schickling

๐Ÿ‘ would be a huge gain for the language!

@murarth
Contributor
murarth commented Nov 12, 2014

I'm not yet familiar with rustc or LLVM, but I'm interested in trying my hand at this project.
I'm looking into MCJIT right now and I've been reading some rustc code. I'm not sure what the problem areas may be, but if anyone has any general advice about this, I would appreciate it.

@thestinger
Contributor

MCJIT just allows machine code to be directly generated and run without a temporary file. It's not the right way to get started on this. The hard part is the logic of the REPL itself, and it could just start off by outputting an object file with MCJIT usage added as an unimportant refinement later.

@murarth
Contributor
murarth commented Nov 12, 2014

Oh. I thought MCJIT might have been a part of managing persistent memory state of the execution environment. I guess I'm not exactly sure what it does and doesn't do yet.
Supposing I compile some object code, how could it be executed with a custom stack and heap (and whatever else may be necessary) controlled by the REPL process?

@thestinger
Contributor

MCJIT allows you to generate the machine code for functions in-memory and then call into them. It would be the basis of a solid REPL but it is really only an optimization.

@thestinger
Contributor

The same thing can be accomplished by progressively generating object files in a temporary directory and linking them together repeatedly. I think it would be easier to build the REPL around the existing backend and then switch over to MCJIT as an optimization, but of course the person implementing it can do it however they want.

@murarth
Contributor
murarth commented Nov 14, 2014

You're probably right about MCJIT. I'm looking exclusively at rustc now.
The way I see it, there are three big problems:

  • Type- and region-checking whole input with each new round of input.
    Naturally, new functions and expressions will depend on the type information of existing items. This should be relatively easy, but it may require changes to rustc code.
  • Incremental code compilation with persistent data.
    Each new round of code will need to know the locations of previously defined stack locals. Also, stack locals must not be dropped at the end of a round of input. I'm not sure how this can be done. I haven't looked into IR and machine code translations yet.
  • Dynamic loading and linking crates when user inputs extern crate ....
    Loading crate metadata is already done in the type-checking phase, so this may also be relatively easy.

(Also, if this isn't the place to discuss implementation, I'll gladly take it elsewhere.)

@murarth
Contributor
murarth commented Nov 28, 2014

Status update (in case anyone is interested):

  • Use of MCJIT through LLVM ExecutionEngine turned out to be rather simple. The old rusti code was making use of that and I used it as a guide. As a result, basic compilation (through librustc) and execution are now functional.
  • Loading crates as dynamic libraries was also rather simple. LLVM ExecutionEngine does the job of linking.
  • I'm currently looking at the type checker code to see how the aforementioned problem can be solved.
  • I hope to release something that will run rounds of input as self-contained functions. It will still be useful as a REPL, though not quite as convenient as one would like.
  • I've decided to delay implementation of persistent access to declared locals because it seems like the biggest problem. After something generally useful has been released, I'll give this another look.
@schickling

Good job @murarth! Really looking forward to see this!

@hastebrot

Sounds promising, @murarth! Can we use ExecutionEngine directly with Rust (e.g. via src/librustc_llvm/lib.rs) or do we need to add an external LLVM dynamic library?

@jauhien
Contributor
jauhien commented Nov 28, 2014

@murarth: where is your repo with code? I would like to look at it and may be join you in working on it, if you need any help.

@murarth
Contributor
murarth commented Nov 28, 2014

@hastebrot: Some C++ wrapper code in src/rustllvm is necessary for it to work properly with Rust. It defines a few functions that do appear in src/librustc_llvm/lib.rs. Once I've gotten everything working and I'm sure there won't be any more changes necessary, I'll be making a PR with those additions.

@jauhien: I don't have any public code right now. I want to get to a point of minimal working functionality before publishing anything, so it doesn't look like it's broken.

@freebroccolo
Contributor

Sounds great! Really looking forward to this.

Eventually, one thing I'd like to have in Rust is some sort of interface that we can build fancier editor modes and tools on top of, where the REPL would be a part of that. I'm thinking something llike what the Idris folks have done with the emacs-mode. They provide an interface (editor agnostic) with some commands which hook into parts of the compiler to provide some very cool functionality.

Would it be within the scope of this project to support such functionality? If not, would it be possible to re-use some of what you're putting together now for something like that?

@murarth
Contributor
murarth commented Dec 16, 2014

The above mentioned problems still exist, but I have made a public release with basic functionality.
I hope it's not confusing that I call it "rusti" just like the old one and the IRC bot.
https://github.com/murarth/rusti

@hastebrot

Very nice, thanks! For reference, the PR that brings LLVM's ExecutionEngine API to Rust is #19750.

@steveklabnik
Contributor

I'm pulling a massive triage effort to get us ready for 1.0. As part of this, I'm moving stuff that's wishlist-like to the RFCs repo, as that's where major new things should get discussed/prioritized.

This issue has been moved to the RFCs repo: rust-lang/rfcs#655

@steveklabnik steveklabnik referenced this issue in rust-lang/rfcs Jan 21, 2015
Open

implement a proper REPL #655

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment