Rust Language Server (IDE support)

[nrc]

This RFC describes how we intend to modify the compiler to support IDEs. The
intention is that support will be as generic as possible. A follow-up internals
post will describe how we intend to focus our energies and deploy Rust support
in actual IDEs.

There are two sets of technical changes proposed in this RFC: changes to how we
compile, and the creation of an 'oracle' tool (name of tool TBC).

Thanks to Phil Dawes, Bruno Medeiros, Vosen, eddyb, Evgeny Kurbatsky, and Dmitry Jemerov for early feedback.

[killercup]

Rendered

[killercup]

Line break → link break

[liigo]

'oracle' is not a perfect name here, since when you google 'rustlang oracle', Google don't know what you intend to search, a compiler tool or a database api?

[daniel-vainsencher]

Seems like "incremental" here is use to describe push-driven in contrast to the lazy pull-driven compilation. Which is more efficient depends strongly on the use case, hence supporting both and combinations is probably useful. If the dependencies, changes (the pushes) and interests (the pulls) are managed explicitly, combining the strategies should be feasible. I would use the word "incremental" instead to describe all of these partial compilations.

[nrc]

The two are orthogonal - you can have lazy incremental, eager incremental, lazy non-incremental, and eager non-incremental.

[daniel-vainsencher]

I'm saying that the reference to incremental in line 101 seems to define it as eager, conflicting the orthogonality set up in lines 110-115. Or maybe that's just the way it reads to me.

[Ericson2314]

@daniel-vainsencher I think the intended distinction is that "incremental" describes what is done (only what hasn't been done before), while "lazy" describes the algorithm of figuring what to do (start at end goal and work through dependency dag). Laziness without caching is non-incremental, and the "push-driven" approach you mention is a different algorithm for incremental compilation than the lazy one. Does that clarify the orthogonality?

[eddyb]

Why exactly can't the oracle drive rustc directly?
Keeping compiler sessions in memory seems like the most efficient and accurate method at our disposal.

[nrc]

It could, I list this as an alternative in the alternatives section (I assume you mean the quick-check version of rustc). I think it may be a better approach. The only real downside is that the oracle then has to know about dependencies between crates.

In terms of when to do a full build of a crate, I think you want to avoid this as much as possible, so the IDE is the best driver to choose when it is necessary.

[eddyb]

I forgot to comment this on the pre-RFC thread (sorry @nrc), but I prefer rider as the name of the oracle tool, and keeping racer (with @phildawes' permission, of course) as the one-shot completion (and perhaps quickcheck) tool.

The way I see it, racer "races" to give an useful result each time, whereas rider "rides" along an IDE, for as long as there are Rust projects to be handled by it, and can have a slow start without impacting UX.

[kud1ing]

I don't have a name suggestion (yet), but i expect calling it "Oracle" will lead to confusion and legal trouble.

[killercup]

(I like how this RFC concentrates on the name first 😉)

How about rustcage? Either pronounced rust cage or rust sage.

[phildawes]

Personally I'm not entirely sold on the oracle/rider concept yet. I think the fundamental requirement is that we need a stable interface to rustc to support IDEs and plugins. I'm not yet sure whether a long running database process is required/desirable.

Some things that might cause problems:

• The oracle concept implies a single view of the sourcecode, and I think this jibes a bit with the tools that will perform refactoring, reformatting and suggestions. I suspect they will want interactive access to the compiler (compile this pre-processed snippet, what errors occur if I do this?)
• If the oracle were to support completion plugins directly from its database, it would need to have very low latency update turnaround (i.e. on every keypress, in <100ms). I'm not sure how well this would work with it being separate from rustc and not driven by the plugins.

It may be that we have an oracle in addition to a tools-oriented interface to rustc.

(aside: I noticed that the go oracle isn't used by the gocode completion tool, but I don't know the reason for this, it could just be historical)

[daniel-vainsencher]

This style of API, where the IDE queries for what it needs right now, is hard for doing push-driven updates (based on a file changing and being saved), because who knows what exactly the IDE is interested in?

An alternative is to allow IDEs to register interest in some queries (for a typical heavy IDE "all definitions in this project", but in Racer, only a fast changing "whatever is under this cursor location"), and then:

• Use the registrations to notify IDE only of interesting changes.
• Use positive registrations to know someone cares about this at all (even before they ask for it).

[nrc]

Could you explain the push-driven update a bit more please? I am assuming the IDE plugin is pretty much state-less with respect to the oracle's data and whenever the user performs an action (right click, hover, hotkey, etc.) then it will query the oracle for the data it needs (thus why the oracle must be quick).

[daniel-vainsencher]

Two things can drive the oracle to do work:

• Code changed (two major usecases: a drip as in editing or massive as in git pull/automatic refactoring/cargo update). So IIUC, what you've called eager evaluation would be to react to the change in code immediately, I called this push driven. This can easily be a waste of time when you are recomputing something that nobody cares about. However, if in the API the IDE said "I care until X,Y,Z and any updates on them until further notice" then you can be correctly selective.
• IDE asked for something (say, a definition) and not everything has been precomputed in advance. Triggers some lazy (I called this pull driven) computation. This is never a waste of time, but may have unreasonable latency (whoops, I should have d/led those new versions of two crates before, huh?) and might also lose opportunities for doing things in parallel.

[nrc]

I think that for the push-based changes, the IDE calls the update functions (for the big changes, that is why we need to invalidate whole files or directories). But I don't think the result of any of that has to be communicated back to the IDE (other than error messages, maybe) - the IDE won't keep any state about the program - that is all kept in the oracle, which will be updated by the compiler (or the two are integrated together).

You should of the oracle as part of the IDE really, the IDE shouldn't manage state of its own about the program.

Does that sound right? Or am I missing something about the work flow?

[daniel-vainsencher]

defintion -> definition

[bungcip]

Found some paper related with IDE integration:
http://wasdett.org/2013/submissions/wasdett2013_submission_10.pdf

[daniel-vainsencher]

Apologize for linking my own paper. [1] is a (proto-) pattern language describing methods to support multiple analyses over a changing source base (some from Smalltalk designs, some also used then in Eclipse). Basically, something like the oracle discussed here. One major decision point is: what objects from the analysis persist when the program text is not completely valid? for example, adding a "{" early in a file can be seen to invalidate every scope after it, do we then not use those function definitions in auto complete? Smalltalk solves this by giving most definitions (classes, methods) a persistent identity over time. The text edited only corresponds to a single definition, and updates the canonical version only when saved (and syntactically valid). Rust currently seems to encourage large files, which makes this more difficult, though not necessarily impossible. For example, if we have both a recent valid version of the source and the changed span, we can use the valid old definitions except where valid current versions are available.

[1] http://hillside.net/plop/2006/Papers/ACMConferenceProceedings/Intimacy_Gradient/a15-vainsencher.pdf

[olivren]

Preliminary notes: I'm commenting on this RFC because I started writing a Rust plugin for the QtCreator IDE. This plugin is in a very preliminary state and it does nothing useful for the moment. I don't know much about compilers in general, and I'm discovering the QtCreator API as I write my plugin. So, be suspicious of anything I could say!

What is the rationale for proposing a daemon process + an IPC API? I fail to see an advantage compared to the oracle just being a library with both a Rust and a C API, that I could call directly from my IDE plugin code. It would remove the pain of writing communication code. it would also allow me to instanciate two services at the same time with isolated content.

The only pros I can see for the daemon approach are:

1. Share information between multiple IDE instances.
   It seems like a rare use case, and I doubt there are much data to share.
2. Be easier to integrate in languages with awful FFI capabilities.
   The IPC API could be designed on top of an existing library, if the need is real.

When I investigated to integrate Racer to my IDE plugin, I came to the conclusion that it would be easier to use Racer as a library rather than invoking it as a process and parsing the output.

[eddyb]

Having an API means all sorts of stabilization concerns, and also passing structured data through a C API.

[phildawes]

Would we not have to apply the same stablization concerns to the IPC interface?
(genuine question, what's the difference between a rust api and an ipc api wrt stabilization?)

[nrc]

The major reason is parallelism - we want the IDE to carry on doing its thing whilst the compiler (coordinated by the oracle) compiles the non-urgent parts of changed source. Then the oracle needs to update its database with the results. Although the IDE could handle the threading to make all this work, it is simpler (and more reusable) just to do it all in a separate process.

With a decent IPC library, having an IPC API is not a lot more complex than having an FFI API. And in turns of implementation having a totally separate process is marginally easier.

In terms of stability, I don't think there is much difference between the two. An IPC API might be a little more robust because using an intermediate data structure and having parsing/serialisation adds a little abstraction.

[olivren]

At least for QtCreator, the extension API is already designed with asynchrony in mind. Having to deal with asynchronous IO would be a lot more painful for me than having a simple synchronous API. In fact, if you give me an IPC API, the first thing I'll do will be to wrap IO and parsing into a simple synchronous API. In the end there will just be an unnecessary
indirection layer.

The oracle library will of course deal with its own update work in a dedicated thread, but you will have to have such a thread anyway, with a process-based design.

[nrc]

Hmm, interesting, that suggests implementing the oracle as a library might be a better solution than as a separate process. I'm not 100% convinced, but it seems like a reasonable alternative to consider. Let me have a think about it...

[nrc]

Does anyone know how this would work out in the Java world? How does a JNI/FFI interface compare to an IPC interface?

[eddyb]

I guess this got lost in the bikeshed: the reason I independently came up with the IPC idea is that compartimentalization and serialization is unavoidable.

I have implemented a system where rustc threads are spawned in the background, and there is a duplex request/response channel to control the thread and get data from it.

The messages are owned ADTs, which means rustc-specific data (e.g. an interned string) has to be converted to a general representation (String).

To avoid tying the in-memory format to Rust or C-based representations, an actual serialization format can be used.
Cap'n Proto, for example, had built-in versioning and extensibility.

And, finally, for added reliability and to get rid of the FFI surface, the threads can be moved to a separate process, which is how we end up with the oracle/rider model.

[olivren]

I understand this is not a detailed API, but I dont see how this works when there are multiple spans to invalidate.Do all spans must be computed based on the initial content of the file? It sounds like it could be a bit painful to compute it from the plugin side. Maybe a simpler API could live alongside this one, that takes the full content of the file and that lets the oracle compute the differences based on its current state.

[nrc]

The spans are relative to the last update passed to the oracle. I don't think that should be hard to compute - the plugin just has to keep track of any text deleted or edited since the last update call.

The trouble with diff'ing before and after snapshots is that it is hard to do well, and so we'd end up making mistakes or overestimating the invalidated region. I imagine it is not super-cheap either.

[olivren]

This makes sense. I think you are right and the proposed design is best.

[olivren]

Why is the input a span here? I would expect an offset. Would I have to find the span of the entire word I want the definition of, or is a partial span (or even an empty span) a valid input? (Note that the same remark applies to all the subsequent API functions)

[nrc]

I'm assuming that the IDE has a tokeniser and therefore already knows the span of the identifier (even simple editors usually have a tokeniser in order to implement syntax highlighting). On the other hand, I suppose that from the oracle's point of view it is as easy to identify the identifier by a single position as it is a span, so it might be as well to take a single position.

[olivren]

The hand written tokenizer on the plugin side will certainly not be as accurate as the oracle's one. I'd rather delegate to the oracle as much as possible.

[olivren]

The output should also tell the "kind" of each reference found. For example, if we want to find the references of a function declared in a trait, the output could be either a "method call" kind, or a "definition in a trait impl" kind, or a "function as a value" kind.

[nrc]

The "reference data" includes the kind of definition, see lines 299, 300

[olivren]

The term used in the the previous section was "definition data". It is not obvious that "reference data" designates the "kind" of references.

[nrc]

I'll try and polish the text here.

[olivren]

I think it only depends whether the oracle is able to give one type of answer more quickly than another one. If the oracle can only be "ready to answer any request" or "not ready", then a unique get definition function is enough.

[daniel-vainsencher]

I don't think that get type (presumably over any expression, not just identifiers) is the same kind of query as get definition: the user wants the type specialized to the current call's context, including values of any type parameters.

[olivren]

The search for identifier should also take as an input the scope of the search. QtCreator's Locator lets the user search for a symbol in the current opened file, and It's one of the Locator function I use the most.

[nrc]

This is a good idea, I'll add it.

Note that the IDE is expected to present an interface over this functionality, and plain text search should be done entirely in the IDE, the only search the oracle helps with is finding uses of a particular identifier (i.e., semantic search). Defining a scope to search over would still be useful though.

[olivren]

Yes, I was indeed talking about symbols, not full text searches. The Locator will give top level items only.

[olivren]

I would add another function to the API: a way to search among the documentation. QtCreator's Locator can search in the available documentation, and display the html doc of the matches.

[nrc]

Is this a text search of the docs, or does it look up documentation for a particular function (or whatever)?

[olivren]

Not sure about the specifics. With a Qt project, this Locator feature matches either a symbol name, or a word that appears inside a title of the generated html doc (they are structured docs). This is clearly not a critical feature, it may not belong to an initial RFC.

[olivren]

In this case I think a span is ok. If a user highlights a part of the text and invokes the autocompletion, I expect the completion to give results that could replace the highlighted selection.

[olivren]

Could you give an example of how an HTML error message could be useful? Or give a link to a discussion on the subject?

[nrc]

The idea is that we could use colouring or shading or whatever to indicate things like the scope of borrows or lifetimes and have links to relevant documentation, etc. This is not useful for IDEs so much as a general way to improve our error messages.

[olivren]

I too was wondering how to deal with multiple versions of Rust. In my IDE plugin, I would like to let the user choose its compiler, and be able to invoke the nightly compiler or the stable one explicitly using different targets. I don't know how I could give this choice to the user, for the code model used for navigation/completion/refactoring.

[eddyb]

@phildawes To remove as much latency as possible and to prevent rustc stabilization concerns, I believe that the oracle should use rustc's internal APIs and be behind the stability wall.
That is, the oracle ships with rustc and uses its libraries directly.

I really see no point in having some output format from rustc itself, that's added design complexity and inefficiency for no gain (in case of the oracle, at least).

[phildawes]

Hi @eddyb!

I agree with that idea, if oracle is to be a thing then it makes sense to put oracle behind the stabilization wall and link it directly to rustc.

However I'm worried about the whole oracle thing. Definitely we need a stable interface to rustc. My concerns with oracle being the stable interface to rustc are:

• 'database of code' is seductive, but imposing this top-down abstraction could easily result in incidental complexity (overengineering)
• it might turn out that building the various refactorings and functionality require an interface that doesn't fit well into the database-of-code paradigm.

It already appears to me that completions maybe don't fit naturally into this design, and we've barely got started.

I'd prefer to see us drive the interface out bottom-up from the ide plugins. Try to build stuff using rustc directly, understand what stable interface is required.

[nrc]

'oracle' is not a perfect name here

@liigo oracle is a terrible name! But it does have some precedent from Go, and I can't think of a better one. Suggestions welcome!

[nikomatsakis]

Personally I'm wary of the database-of-code metaphor. I think it
hides the two hard problems of rust code completion, and instead
encourages thinking about indexing and cache invalidation.

This makes a lot of sense. I certainly agree that adding an oracle
doesn't get us fast code completion for free -- that seems to require
not just incr. comp., but also a bit more work towards lazy
compilation and better error recovery (some of which is ongoing, as
@nrc notes). It does seem like an oracle would help for Find All
References, and I'm not sure what else.

Still, it seems to me that even in code completion, there is perhaps a
role for an oracle. In particular, you talk about wanting to determine
the type of the receiver of a method, but once you know that receiver,
you still want to figure out things like "what are the set of traits
that are implemented for that type?" (and note that those traits may
not be imported, so the compiler might not consider them
normally). This might be an ideal place for an oracle to come in.

Moreover, some completion scenarios don't necessarily require knowing
the full type, e.g., if I write some_fn(, then you can likely
resolve some_fn to a specific item (but in fact you really want to
be resolving some_fn all along to potential matches, even before
I've finished typing it).

Maybe there's another way to say what I'm saying. It seems clear that
"ideal IDE integration" is going to be a long time coming, and we're
going to be evolving how rustc works for some time. But in the short
to medium time frame, I expect you are going to want to do fast,
approximate queries across several crates, and it seems like we can
get that more easily through some spearate oracle (you will also
want to be able to ask rustc for stuff and get back a fast answer, of
course).

And if in the future it happens that rustc gets fast enough that we
don't need a database to answer those queries, we can just recompute
the data, that's fine, right? All the old code that expected
potentially stale answers should be fine when their answers are 100%
up to date?

[bruno-medeiros]

https://crates.io/crates/syntex_syntax/

Speaking of, who maintains this project? Are the Rust developers involved?

I think changes and improvements to the Rust parser would a nice incremental step and starting point to improving the toolchain, with regards to how it's used by IDEs or IDE related tools (like Racer).

I've been working on one such tool, https://github.com/RustDT/Rainicorn (formerly called rust-parse-describe, I mentioned in a previous comment some weeks ago), also using syntex_syntax, and I can see some avenues for improvement already. For starters a panic-less parser would be nice, the way the parser currently works makes my code a bit more complicated than would otherwise be necessary.

Forking a project is a socially aggressive action, and it is a shame to have political fallout and hurt over some small technical point or direction.

I didn't mean forking as in a divergent, or organizational fork (duplication of efforts, vastly incompatible code baselines, no communication, development heading in different directions, etc.).

I meant fork as in the Git way: a clone/branch that you create, you make some modifications or additions, but it doesn't diverge that much from the upstream source, and you try to merge updates from the upstream source regularly. And occasionally you might submit patches to upstream as well.

[nrc]

@bruno-medeiros syntex is maintained by @erickt, who is a core member of the community (community and moderation teams, as well as just being generally involved), but not part of the compiler team. It is pretty much just a straight clone of libsyntax from the compiler, so improvements to the compiler's parser show up in syntex quite quickly.

Part of the plan with procedural macros/syntax extensions is to present a stable interface for them to work on, at which point syntex gets a lot less necessary (only useful for tools). In the long term I'd like to stabilise enough of libsyntax that tools don't need it either.

There is work going on to make the parser panic-less, it no longer panics on error. I've also been doing some work on error recovery, for example rust-lang/rust#31065 adds some error correction for missing identifiers. Ideally it should be panic-free under normal use and recover from most errors within the next few months.

[nrc]

@nikomatsakis I did not intend the DB to be used for the code completion suggestions. It is useful for find all references, also queries like find all impls of a given trait, find all sub-traits, etc.

[bruno-medeiros]

Still, it seems to me that even in code completion, there is perhaps a role for an oracle

Just to be clear, the Oracle, as in, "the resident process that is responsible for serving requests of various sorts to the IDE", it should handle code completion as well. Even if the data structures that are used to determine the results of code completion are entirely different from the data structures used for say, find-references, it makes no sense for this to be two different processes, or two different tools. This is because at the very minimum, these two operations can share cached AST data, not to mention that eventually (and sooner that later) we will want the Oracle to support functionality to manage/supply dirty editor buffers (ie, use a document that is being edited in memory in an IDE, but has not yet been persisted to disk). Even the functionality I'm coding in the Rainicorn tool should ideally also eventually be integrated into an oracle.

Of course, as an early prototype, it's okay for different operations to be handled by different tools, etc. but the end goal should be to integrate everything in the oracle, all-knowing that it is. 😉 (gotta hand it to the Go guys, they choose the perfect name - apart from the trademark thing.. 😝 )

@nrc BTW, I was just looking at the Nim language, and to my surprise found out they have an "oracle" tool already: http://nim-lang.org/docs/idetools.html
And by the looks of it quite more advanced than the Go-oracle, it actually seems to achieve all those key aspects that were mentioned before:

• Handles incorrect/incomplete code well
• Is a resident process
• Supports managing dirty buffers
• Supports all sorts of IDE query operations in a single tool: resolve-definition ("Definitions"), code-completion ("Suggestions"), find references ("Symbol usages"), etc.. (Something like parse-analysis seems to be missing though)

[bruno-medeiros]

Ideally it should be panic-free under normal use and recover from most errors within the next few months.

Sweet 👍

[alexcrichton]

🔔 This RFC is now entering its two-week long final comment period 🔔

[lilianmoraru]

Don't know if it is of any help but I will throw it out there.
QtCreator(mainly C++ IDE) has a plugin that uses Clang to offer C++ code completion.
Here is the code:
http://code.qt.io/cgit/qt-creator/qt-creator.git/tree/src/libs/clangbackendipc
http://code.qt.io/cgit/qt-creator/qt-creator.git/tree/src/plugins/clangcodemodel

QtCreator also has an older custom C++ code model(that is being replaced by the clang one) that was a lot faster and seems like it is accepted in the community that the code model that uses the compiler will be considerably slower, but it offers more information and is more accurate.
I mention the speed because I saw @phildawes mentioning the hard requirements on the compiler to deliver the information in <100ms and I am not sure if we should expect it to be fast.

[DemiMarie]

@lilianmoraru The reason the compiler needs to deliver the information so quickly is that the GUI is waiting on it. The user is expecting the completion to appear as soon as the user presses . and that requires the compiler to respond fast.

[michaelwoerister]

I'm in favor of accepting this RFC. The approach seems worth exploring and it does not preclude improving the compilers amenability for being used as a library. On the contrary, I think the compiler's APIs will benefit from trying to build the RLS on top of it and it can only help if people on the compiler team are actual clients of their own APIs. The RFC leaves many open questions when it comes to specifics and we just need a prototype implementation and the experience that comes from building that in order to decide how to proceed further. Worst case, we'll learn a bunch of stuff on what doesn't work :)

[erkinalp]

1-based line numbers and 0-based column numbers please.

[nrc]

@erkinalp why?

[bruno-medeiros]

@erkinalp why?

I was wondering the same, why is a mixed format being used (1-based in one, and 0-based for the other)

[erkinalp]

Emacs uses 0-based column numbers.

[ticki]

@erkinalp That's a perfect reason for not doing that 😜 .

[erkinalp]

@nrc @bruno-medeiros: And it uses 1-based line-numbers https://gcc.gnu.org/bugzilla/show_bug.cgi?id=19165#c21

[nrc]

afaik, there is no standard for editors to use 1-based line numbers. Having both 0-based seems like the least confusing thing to do, it's easy enough for editors to add one to each line number.

[dgrunwald]

Compilers (including rustc) tend to use 1-based line and column numbers in their error messages. Following that standard seems like the least confusing thing to do. I certainly wouldn't expect 0-based line numbers.

But the concepts of "line" and "column" are ill-defined anyways.
Does a vertical tab (\v) count as a new line? What about form feed? What about all the other exotic Unicode newlines?
Does a tab count as N columns (where N is configurable; usually 4 or 8?), does it advance to the next multiple of N columns, or does it count as only 1 character?
Are full-width characters like 'ｘ' one or two columns wide?
Or does the editor count each Unicode codepoint as 1 column? Maybe a 'column' really is a UTF-8 byte offset within the line? Maybe it's measured in UTF-16 code units? Grapheme clusters?

It's difficult to find two editors that agree in their line/column counting for all possible input files.

[Valloric]

I've integrated 5+ semantic engines in ycmd, and the only thing that makes sense is 1-based line and column numbers. Columns are byte offsets in UTF-8. Done.

it's easy enough for editors to add one to each line number.

But why should they? Line & column numbers coming from your oracle will be shown to the user and they expect 1-based numbering.

there is no standard for editors to use 1-based line numbers.

And yet they ~all do use 1-based numbers in the user interface. When you put your caret on the first line in the file, the editor doesn't say the line number is 0, it says it's 1. Same for columns.

[erkinalp]

Vertical tab means skip one line below and continue from same column offset.
CR, CR+LF, LF, LS and NEL are regular line feeds.
FF and PS count as two lines instead of one.

[bruno-medeiros]

I've integrated 5+ semantic engines in ycmd, and the only thing that makes sense is 1-based line and column numbers. Columns are byte offsets in UTF-8. Done.

Why byte offsets and not Unicode character offsets? It's not like an error or position for a Rust symbol will ever start in the middle of a Unicode character.

there is no standard for editors to use 1-based line numbers.

And yet they ~all do use 1-based numbers in the user interface. When you put your caret on the first line in the file, the editor doesn't say the line number is 0, it says it's 1. Same for columns.

Because the internal API for lines and columns can be 0-based, despite the UI being 1-based. This is certainly the case for Eclipse, for IntelliJ, and probably for most IDEs/editors out there. It would not surprise me if Vim is the odd one out... 😆

[adelarsq]

Keep it simple. Just use 0-based for lines and columns.

[alexcrichton]

This RFC was discussed during the tools team triage today and the decision was to merge. This RFC is still at a somewhat high level and some minor details can continue to be ironed out in the implementation over time, but there seems to be widespread agreement about the body of the RFC here.

Thanks again for the discussion everybody!