Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Rust Language Server (IDE support) #1317

Merged
merged 6 commits into from Feb 11, 2016
Merged

Rust Language Server (IDE support) #1317

merged 6 commits into from Feb 11, 2016

Conversation

@nrc
Copy link
Member

nrc commented Oct 13, 2015

This RFC describes how we intend to modify the compiler to support IDEs. The
intention is that support will be as generic as possible. A follow-up internals
post will describe how we intend to focus our energies and deploy Rust support
in actual IDEs.

There are two sets of technical changes proposed in this RFC: changes to how we
compile, and the creation of an 'oracle' tool (name of tool TBC).

Thanks to Phil Dawes, Bruno Medeiros, Vosen, eddyb, Evgeny Kurbatsky, and Dmitry Jemerov for early feedback.

@nrc nrc self-assigned this Oct 13, 2015
@killercup
Copy link
Member

killercup commented Oct 13, 2015


A solution to the first problem is replacing invalid names with some magic
identifier, and ignoring errors involving that identifier. @sanxiyn implemented
something like the second feature in a [PR](https://github.com/rust-

This comment has been minimized.

Copy link
@killercup

killercup Oct 13, 2015

Member

Line break → link break

@liigo
Copy link
Contributor

liigo commented Oct 14, 2015

'oracle' is not a perfect name here, since when you google 'rustlang oracle', Google don't know what you intend to search, a compiler tool or a database api?

proposal](https://github.com/rust-lang/rfcs/pull/1298) for supporting
incremental compilation involves some lazy compilation as an implementation
detail.

This comment has been minimized.

Copy link
@daniel-vainsencher

daniel-vainsencher Oct 14, 2015

Seems like "incremental" here is use to describe push-driven in contrast to the lazy pull-driven compilation. Which is more efficient depends strongly on the use case, hence supporting both and combinations is probably useful. If the dependencies, changes (the pushes) and interests (the pulls) are managed explicitly, combining the strategies should be feasible. I would use the word "incremental" instead to describe all of these partial compilations.

This comment has been minimized.

Copy link
@nrc

nrc Oct 15, 2015

Author Member

The two are orthogonal - you can have lazy incremental, eager incremental, lazy non-incremental, and eager non-incremental.

This comment has been minimized.

Copy link
@daniel-vainsencher

daniel-vainsencher Oct 15, 2015

I'm saying that the reference to incremental in line 101 seems to define it as eager, conflicting the orthogonality set up in lines 110-115. Or maybe that's just the way it reads to me.

This comment has been minimized.

Copy link
@Ericson2314

Ericson2314 Oct 15, 2015

Contributor

@daniel-vainsencher I think the intended distinction is that "incremental" describes what is done (only what hasn't been done before), while "lazy" describes the algorithm of figuring what to do (start at end goal and work through dependency dag). Laziness without caching is non-incremental, and the "push-driven" approach you mention is a different algorithm for incremental compilation than the lazy one. Does that clarify the orthogonality?

incrementally update its knowledge of the source code. How exactly to do this
when neither names nor ids are stable is an interesting question, but too much
detail for this RFC (especially as the implementation of ids in the compiler is
evolving).

This comment has been minimized.

Copy link
@eddyb

eddyb Oct 14, 2015

Member

Why exactly can't the oracle drive rustc directly?
Keeping compiler sessions in memory seems like the most efficient and accurate method at our disposal.

This comment has been minimized.

Copy link
@nrc

nrc Oct 15, 2015

Author Member

It could, I list this as an alternative in the alternatives section (I assume you mean the quick-check version of rustc). I think it may be a better approach. The only real downside is that the oracle then has to know about dependencies between crates.

In terms of when to do a full build of a crate, I think you want to avoid this as much as possible, so the IDE is the best driver to choose when it is necessary.

@eddyb
Copy link
Member

eddyb commented Oct 14, 2015

I forgot to comment this on the pre-RFC thread (sorry @nrc), but I prefer rider as the name of the oracle tool, and keeping racer (with @phildawes' permission, of course) as the one-shot completion (and perhaps quickcheck) tool.

The way I see it, racer "races" to give an useful result each time, whereas rider "rides" along an IDE, for as long as there are Rust projects to be handled by it, and can have a slow start without impacting UX.

@kud1ing
Copy link
Contributor

kud1ing commented Oct 14, 2015

I don't have a name suggestion (yet), but i expect calling it "Oracle" will lead to confusion and legal trouble.

@killercup
Copy link
Member

killercup commented Oct 14, 2015

(I like how this RFC concentrates on the name first 😉)

How about rustcage? Either pronounced rust cage or rust sage.

@phildawes
Copy link

phildawes commented Oct 14, 2015

Personally I'm not entirely sold on the oracle/rider concept yet. I think the fundamental requirement is that we need a stable interface to rustc to support IDEs and plugins. I'm not yet sure whether a long running database process is required/desirable.

Some things that might cause problems:

  • The oracle concept implies a single view of the sourcecode, and I think this jibes a bit with the tools that will perform refactoring, reformatting and suggestions. I suspect they will want interactive access to the compiler (compile this pre-processed snippet, what errors occur if I do this?)
  • If the oracle were to support completion plugins directly from its database, it would need to have very low latency update turnaround (i.e. on every keypress, in <100ms). I'm not sure how well this would work with it being separate from rustc and not driven by the plugins.

It may be that we have an oracle in addition to a tools-oriented interface to rustc.

(aside: I noticed that the go oracle isn't used by the gocode completion tool, but I don't know the reason for this, it could just be historical)

The returned data is a list of 'defintion' data. That data includes the span for
the item, any documentation for the item, a code snippet for the item,
optionally a type for the item, and one or more kinds of definition (e.g.,
'variable definition', 'field definition', 'function declaration').

This comment has been minimized.

Copy link
@daniel-vainsencher

daniel-vainsencher Oct 14, 2015

This style of API, where the IDE queries for what it needs right now, is hard for doing push-driven updates (based on a file changing and being saved), because who knows what exactly the IDE is interested in?

An alternative is to allow IDEs to register interest in some queries (for a typical heavy IDE "all definitions in this project", but in Racer, only a fast changing "whatever is under this cursor location"), and then:

  • Use the registrations to notify IDE only of interesting changes.
  • Use positive registrations to know someone cares about this at all (even before they ask for it).

This comment has been minimized.

Copy link
@nrc

nrc Oct 15, 2015

Author Member

Could you explain the push-driven update a bit more please? I am assuming the IDE plugin is pretty much state-less with respect to the oracle's data and whenever the user performs an action (right click, hover, hotkey, etc.) then it will query the oracle for the data it needs (thus why the oracle must be quick).

This comment has been minimized.

Copy link
@daniel-vainsencher

daniel-vainsencher Oct 15, 2015

Two things can drive the oracle to do work:

  • Code changed (two major usecases: a drip as in editing or massive as in git pull/automatic refactoring/cargo update). So IIUC, what you've called eager evaluation would be to react to the change in code immediately, I called this push driven. This can easily be a waste of time when you are recomputing something that nobody cares about. However, if in the API the IDE said "I care until X,Y,Z and any updates on them until further notice" then you can be correctly selective.
  • IDE asked for something (say, a definition) and not everything has been precomputed in advance. Triggers some lazy (I called this pull driven) computation. This is never a waste of time, but may have unreasonable latency (whoops, I should have d/led those new versions of two crates before, huh?) and might also lose opportunities for doing things in parallel.

This comment has been minimized.

Copy link
@nrc

nrc Oct 16, 2015

Author Member

I think that for the push-based changes, the IDE calls the update functions (for the big changes, that is why we need to invalidate whole files or directories). But I don't think the result of any of that has to be communicated back to the IDE (other than error messages, maybe) - the IDE won't keep any state about the program - that is all kept in the oracle, which will be updated by the compiler (or the two are integrated together).

You should of the oracle as part of the IDE really, the IDE shouldn't manage state of its own about the program.

Does that sound right? Or am I missing something about the work flow?

covered by the span. Can return an error if the span does not cover exactly one
identifier or the oracle has no data for an identifier.

The returned data is a list of 'defintion' data. That data includes the span for

This comment has been minimized.

Copy link
@daniel-vainsencher

daniel-vainsencher Oct 14, 2015

defintion -> definition

@bungcip
Copy link

bungcip commented Oct 14, 2015

Found some paper related with IDE integration:
http://wasdett.org/2013/submissions/wasdett2013_submission_10.pdf

@daniel-vainsencher
Copy link

daniel-vainsencher commented Oct 14, 2015

Apologize for linking my own paper. [1] is a (proto-) pattern language describing methods to support multiple analyses over a changing source base (some from Smalltalk designs, some also used then in Eclipse). Basically, something like the oracle discussed here. One major decision point is: what objects from the analysis persist when the program text is not completely valid? for example, adding a "{" early in a file can be seen to invalidate every scope after it, do we then not use those function definitions in auto complete? Smalltalk solves this by giving most definitions (classes, methods) a persistent identity over time. The text edited only corresponds to a single definition, and updates the canonical version only when saved (and syntactically valid). Rust currently seems to encourage large files, which makes this more difficult, though not necessarily impossible. For example, if we have both a recent valid version of the source and the changed span, we can use the valid old definitions except where valid current versions are available.

[1] http://hillside.net/plop/2006/Papers/ACMConferenceProceedings/Intimacy_Gradient/a15-vainsencher.pdf

The oracle is a long running daemon process. It will keep a database
representation of an entire project's source code and semantic information (as
opposed to the compiler which operates on a crate at a time). It is
incrementally updated by the compiler and provides an IPC API for providing

This comment has been minimized.

Copy link
@olivren

olivren Oct 14, 2015

Preliminary notes: I'm commenting on this RFC because I started writing a Rust plugin for the QtCreator IDE. This plugin is in a very preliminary state and it does nothing useful for the moment. I don't know much about compilers in general, and I'm discovering the QtCreator API as I write my plugin. So, be suspicious of anything I could say!

What is the rationale for proposing a daemon process + an IPC API? I fail to see an advantage compared to the oracle just being a library with both a Rust and a C API, that I could call directly from my IDE plugin code. It would remove the pain of writing communication code. it would also allow me to instanciate two services at the same time with isolated content.

The only pros I can see for the daemon approach are:

  1. Share information between multiple IDE instances.
    It seems like a rare use case, and I doubt there are much data to share.
  2. Be easier to integrate in languages with awful FFI capabilities.
    The IPC API could be designed on top of an existing library, if the need is real.

When I investigated to integrate Racer to my IDE plugin, I came to the conclusion that it would be easier to use Racer as a library rather than invoking it as a process and parsing the output.

This comment has been minimized.

Copy link
@eddyb

eddyb Oct 15, 2015

Member

Having an API means all sorts of stabilization concerns, and also passing structured data through a C API.

This comment has been minimized.

Copy link
@phildawes

phildawes Oct 15, 2015

Would we not have to apply the same stablization concerns to the IPC interface?
(genuine question, what's the difference between a rust api and an ipc api wrt stabilization?)

This comment has been minimized.

Copy link
@nrc

nrc Oct 15, 2015

Author Member

The major reason is parallelism - we want the IDE to carry on doing its thing whilst the compiler (coordinated by the oracle) compiles the non-urgent parts of changed source. Then the oracle needs to update its database with the results. Although the IDE could handle the threading to make all this work, it is simpler (and more reusable) just to do it all in a separate process.

With a decent IPC library, having an IPC API is not a lot more complex than having an FFI API. And in turns of implementation having a totally separate process is marginally easier.

In terms of stability, I don't think there is much difference between the two. An IPC API might be a little more robust because using an intermediate data structure and having parsing/serialisation adds a little abstraction.

This comment has been minimized.

Copy link
@olivren

olivren Oct 15, 2015

At least for QtCreator, the extension API is already designed with asynchrony in mind. Having to deal with asynchronous IO would be a lot more painful for me than having a simple synchronous API. In fact, if you give me an IPC API, the first thing I'll do will be to wrap IO and parsing into a simple synchronous API. In the end there will just be an unnecessary
indirection layer.

The oracle library will of course deal with its own update work in a dedicated thread, but you will have to have such a thread anyway, with a process-based design.

This comment has been minimized.

Copy link
@nrc

nrc Oct 16, 2015

Author Member

Hmm, interesting, that suggests implementing the oracle as a library might be a better solution than as a separate process. I'm not 100% convinced, but it seems like a reasonable alternative to consider. Let me have a think about it...

This comment has been minimized.

Copy link
@nrc

nrc Oct 16, 2015

Author Member

Does anyone know how this would work out in the Java world? How does a JNI/FFI interface compare to an IPC interface?

This comment has been minimized.

Copy link
@eddyb

eddyb Oct 16, 2015

Member

I guess this got lost in the bikeshed: the reason I independently came up with the IPC idea is that compartimentalization and serialization is unavoidable.

I have implemented a system where rustc threads are spawned in the background, and there is a duplex request/response channel to control the thread and get data from it.

The messages are owned ADTs, which means rustc-specific data (e.g. an interned string) has to be converted to a general representation (String).

To avoid tying the in-memory format to Rust or C-based representations, an actual serialization format can be used.
Cap'n Proto, for example, had built-in versioning and extensibility.

And, finally, for added reliability and to get rid of the FFI surface, the threads can be moved to a separate process, which is how we end up with the oracle/rider model.

the user has saved the file) and a list of spans to invalidate. Where there are
no invalidated spans, the update call adds data (which will cause an error if
there are conflicts). Where there is no input data, update just invalidates.

This comment has been minimized.

Copy link
@olivren

olivren Oct 14, 2015

I understand this is not a detailed API, but I dont see how this works when there are multiple spans to invalidate.Do all spans must be computed based on the initial content of the file? It sounds like it could be a bit painful to compute it from the plugin side. Maybe a simpler API could live alongside this one, that takes the full content of the file and that lets the oracle compute the differences based on its current state.

This comment has been minimized.

Copy link
@nrc

nrc Oct 15, 2015

Author Member

The spans are relative to the last update passed to the oracle. I don't think that should be hard to compute - the plugin just has to keep track of any text deleted or edited since the last update call.

The trouble with diff'ing before and after snapshots is that it is hard to do well, and so we'd end up making mistakes or overestimating the invalidated region. I imagine it is not super-cheap either.

This comment has been minimized.

Copy link
@olivren

olivren Oct 15, 2015

This makes sense. I think you are right and the proposed design is best.

Takes a span, returns all 'definitions and declarations' for the identifier
covered by the span. Can return an error if the span does not cover exactly one
identifier or the oracle has no data for an identifier.

This comment has been minimized.

Copy link
@olivren

olivren Oct 14, 2015

Why is the input a span here? I would expect an offset. Would I have to find the span of the entire word I want the definition of, or is a partial span (or even an empty span) a valid input? (Note that the same remark applies to all the subsequent API functions)

This comment has been minimized.

Copy link
@nrc

nrc Oct 15, 2015

Author Member

I'm assuming that the IDE has a tokeniser and therefore already knows the span of the identifier (even simple editors usually have a tokeniser in order to implement syntax highlighting). On the other hand, I suppose that from the oracle's point of view it is as easy to identify the identifier by a single position as it is a span, so it might be as well to take a single position.

This comment has been minimized.

Copy link
@olivren

olivren Oct 15, 2015

The hand written tokenizer on the plugin side will certainly not be as accurate as the oracle's one. I'd rather delegate to the oracle as much as possible.


Takes a span, returns a list of reference data (or an error). Each datum
consists of the span of the reference and a code snippet.

This comment has been minimized.

Copy link
@olivren

olivren Oct 14, 2015

The output should also tell the "kind" of each reference found. For example, if we want to find the references of a function declared in a trait, the output could be either a "method call" kind, or a "definition in a trait impl" kind, or a "function as a value" kind.

This comment has been minimized.

Copy link
@nrc

nrc Oct 15, 2015

Author Member

The "reference data" includes the kind of definition, see lines 299, 300

This comment has been minimized.

Copy link
@olivren

olivren Oct 15, 2015

The term used in the the previous section was "definition data". It is not obvious that "reference data" designates the "kind" of references.

This comment has been minimized.

Copy link
@nrc

nrc Oct 16, 2015

Author Member

I'll try and polish the text here.


Takes a span, returns the same data as *get definition* but limited to type information.

Question: are these useful/necessary? Or should users just call *get definition*?

This comment has been minimized.

Copy link
@olivren

olivren Oct 14, 2015

I think it only depends whether the oracle is able to give one type of answer more quickly than another one. If the oracle can only be "ready to answer any request" or "not ready", then a unique get definition function is enough.

This comment has been minimized.

Copy link
@daniel-vainsencher

daniel-vainsencher Oct 15, 2015

I don't think that get type (presumably over any expression, not just identifiers) is the same kind of query as get definition: the user wants the type specialized to the current call's context, including values of any type parameters.

Takes a search string or an id, and a struct of search parameters including case
sensitivity, and the kind of items to search (e.g., functions, traits, all
items). Returns a list of spans and code snippets.

This comment has been minimized.

Copy link
@olivren

olivren Oct 14, 2015

The search for identifier should also take as an input the scope of the search. QtCreator's Locator lets the user search for a symbol in the current opened file, and It's one of the Locator function I use the most.

This comment has been minimized.

Copy link
@nrc

nrc Oct 15, 2015

Author Member

This is a good idea, I'll add it.

Note that the IDE is expected to present an interface over this functionality, and plain text search should be done entirely in the IDE, the only search the oracle helps with is finding uses of a particular identifier (i.e., semantic search). Defining a scope to search over would still be useful though.

This comment has been minimized.

Copy link
@olivren

olivren Oct 15, 2015

Yes, I was indeed talking about symbols, not full text searches. The Locator will give top level items only.

from just using the caret position?). Each suggestion consists of the text for
completion plus the same information as returned for the *get definition* call.


This comment has been minimized.

Copy link
@olivren

olivren Oct 14, 2015

I would add another function to the API: a way to search among the documentation. QtCreator's Locator can search in the available documentation, and display the html doc of the matches.

This comment has been minimized.

Copy link
@nrc

nrc Oct 15, 2015

Author Member

Is this a text search of the docs, or does it look up documentation for a particular function (or whatever)?

This comment has been minimized.

Copy link
@olivren

olivren Oct 15, 2015

Not sure about the specifics. With a Qt project, this Locator feature matches either a symbol name, or a word that appears inside a title of the generated html doc (they are structured docs). This is clearly not a critical feature, it may not belong to an initial RFC.

Takes a span (note that this span could be empty, e.g, for `foo.` we would use
the empty span which starts after the `.`; for `foo.b` we would use the span for
`b`), and returns a list of suggestions (is this useful? Is there any difference
from just using the caret position?). Each suggestion consists of the text for

This comment has been minimized.

Copy link
@olivren

olivren Oct 14, 2015

In this case I think a span is ok. If a user highlights a part of the text and invokes the autocompletion, I expect the completion to give results that could replace the highlighted selection.

We should support the current text format, JSON (or some other structured
format) for tools to use, and HTML for rich error messages (this is somewhat
orthogonal to this RFC, but has been discussed in the past as a desirable
feature).

This comment has been minimized.

Copy link
@olivren

olivren Oct 14, 2015

Could you give an example of how an HTML error message could be useful? Or give a link to a discussion on the subject?

This comment has been minimized.

Copy link
@nrc

nrc Oct 15, 2015

Author Member

The idea is that we could use colouring or shading or whatever to indicate things like the scope of borrows or lifetimes and have links to relevant documentation, etc. This is not useful for IDEs so much as a general way to improve our error messages.

Alternatives are 'Rider', 'Racer Server', or anything you can think of.

How do we handle different versions of Rust and interact with multi-rust?
Upgrades to the next stable version of Rust?

This comment has been minimized.

Copy link
@olivren

olivren Oct 14, 2015

I too was wondering how to deal with multiple versions of Rust. In my IDE plugin, I would like to let the user choose its compiler, and be able to invoke the nightly compiler or the stable one explicitly using different targets. I don't know how I could give this choice to the user, for the code model used for navigation/completion/refactoring.

@eddyb
Copy link
Member

eddyb commented Oct 15, 2015

@phildawes To remove as much latency as possible and to prevent rustc stabilization concerns, I believe that the oracle should use rustc's internal APIs and be behind the stability wall.
That is, the oracle ships with rustc and uses its libraries directly.

I really see no point in having some output format from rustc itself, that's added design complexity and inefficiency for no gain (in case of the oracle, at least).

@phildawes
Copy link

phildawes commented Oct 15, 2015

Hi @eddyb!

I agree with that idea, if oracle is to be a thing then it makes sense to put oracle behind the stabilization wall and link it directly to rustc.

However I'm worried about the whole oracle thing. Definitely we need a stable interface to rustc. My concerns with oracle being the stable interface to rustc are:

  • 'database of code' is seductive, but imposing this top-down abstraction could easily result in incidental complexity (overengineering)
  • it might turn out that building the various refactorings and functionality require an interface that doesn't fit well into the database-of-code paradigm.

It already appears to me that completions maybe don't fit naturally into this design, and we've barely got started.

I'd prefer to see us drive the interface out bottom-up from the ide plugins. Try to build stuff using rustc directly, understand what stable interface is required.

@nrc
Copy link
Member Author

nrc commented Oct 15, 2015

'oracle' is not a perfect name here

@liigo oracle is a terrible name! But it does have some precedent from Go, and I can't think of a better one. Suggestions welcome!

@nikomatsakis
Copy link
Contributor

nikomatsakis commented Jan 22, 2016

Personally I'm wary of the database-of-code metaphor. I think it
hides the two hard problems of rust code completion, and instead
encourages thinking about indexing and cache invalidation.

This makes a lot of sense. I certainly agree that adding an oracle
doesn't get us fast code completion for free -- that seems to require
not just incr. comp., but also a bit more work towards lazy
compilation and better error recovery (some of which is ongoing, as
@nrc notes). It does seem like an oracle would help for Find All
References, and I'm not sure what else.

Still, it seems to me that even in code completion, there is perhaps a
role for an oracle. In particular, you talk about wanting to determine
the type of the receiver of a method, but once you know that receiver,
you still want to figure out things like "what are the set of traits
that are implemented for that type?" (and note that those traits may
not be imported, so the compiler might not consider them
normally). This might be an ideal place for an oracle to come in.

Moreover, some completion scenarios don't necessarily require knowing
the full type, e.g., if I write some_fn(, then you can likely
resolve some_fn to a specific item (but in fact you really want to
be resolving some_fn all along to potential matches, even before
I've finished typing it).

Maybe there's another way to say what I'm saying. It seems clear that
"ideal IDE integration" is going to be a long time coming, and we're
going to be evolving how rustc works for some time. But in the short
to medium time frame, I expect you are going to want to do fast,
approximate queries across several crates, and it seems like we can
get that more easily through some spearate oracle (you will also
want to be able to ask rustc for stuff and get back a fast answer, of
course).

And if in the future it happens that rustc gets fast enough that we
don't need a database to answer those queries, we can just recompute
the data, that's fine, right? All the old code that expected
potentially stale answers should be fine when their answers are 100%
up to date?

@bruno-medeiros
Copy link

bruno-medeiros commented Jan 22, 2016

https://crates.io/crates/syntex_syntax/

Speaking of, who maintains this project? Are the Rust developers involved?

I think changes and improvements to the Rust parser would a nice incremental step and starting point to improving the toolchain, with regards to how it's used by IDEs or IDE related tools (like Racer).

I've been working on one such tool, https://github.com/RustDT/Rainicorn (formerly called rust-parse-describe, I mentioned in a previous comment some weeks ago), also using syntex_syntax, and I can see some avenues for improvement already. For starters a panic-less parser would be nice, the way the parser currently works makes my code a bit more complicated than would otherwise be necessary.

Forking a project is a socially aggressive action, and it is a shame to have political fallout and hurt over some small technical point or direction.

I didn't mean forking as in a divergent, or organizational fork (duplication of efforts, vastly incompatible code baselines, no communication, development heading in different directions, etc.).

I meant fork as in the Git way: a clone/branch that you create, you make some modifications or additions, but it doesn't diverge that much from the upstream source, and you try to merge updates from the upstream source regularly. And occasionally you might submit patches to upstream as well.

@nrc
Copy link
Member Author

nrc commented Jan 22, 2016

@bruno-medeiros syntex is maintained by @erickt, who is a core member of the community (community and moderation teams, as well as just being generally involved), but not part of the compiler team. It is pretty much just a straight clone of libsyntax from the compiler, so improvements to the compiler's parser show up in syntex quite quickly.

Part of the plan with procedural macros/syntax extensions is to present a stable interface for them to work on, at which point syntex gets a lot less necessary (only useful for tools). In the long term I'd like to stabilise enough of libsyntax that tools don't need it either.

There is work going on to make the parser panic-less, it no longer panics on error. I've also been doing some work on error recovery, for example rust-lang/rust#31065 adds some error correction for missing identifiers. Ideally it should be panic-free under normal use and recover from most errors within the next few months.

@nrc
Copy link
Member Author

nrc commented Jan 22, 2016

@nikomatsakis I did not intend the DB to be used for the code completion suggestions. It is useful for find all references, also queries like find all impls of a given trait, find all sub-traits, etc.

@bruno-medeiros
Copy link

bruno-medeiros commented Jan 22, 2016

Still, it seems to me that even in code completion, there is perhaps a role for an oracle

Just to be clear, the Oracle, as in, "the resident process that is responsible for serving requests of various sorts to the IDE", it should handle code completion as well. Even if the data structures that are used to determine the results of code completion are entirely different from the data structures used for say, find-references, it makes no sense for this to be two different processes, or two different tools. This is because at the very minimum, these two operations can share cached AST data, not to mention that eventually (and sooner that later) we will want the Oracle to support functionality to manage/supply dirty editor buffers (ie, use a document that is being edited in memory in an IDE, but has not yet been persisted to disk). Even the functionality I'm coding in the Rainicorn tool should ideally also eventually be integrated into an oracle.

Of course, as an early prototype, it's okay for different operations to be handled by different tools, etc. but the end goal should be to integrate everything in the oracle, all-knowing that it is. 😉 (gotta hand it to the Go guys, they choose the perfect name - apart from the trademark thing.. 😝 )

@nrc BTW, I was just looking at the Nim language, and to my surprise found out they have an "oracle" tool already: http://nim-lang.org/docs/idetools.html
And by the looks of it quite more advanced than the Go-oracle, it actually seems to achieve all those key aspects that were mentioned before:

  • Handles incorrect/incomplete code well
  • Is a resident process
  • Supports managing dirty buffers
  • Supports all sorts of IDE query operations in a single tool: resolve-definition ("Definitions"), code-completion ("Suggestions"), find references ("Symbol usages"), etc.. (Something like parse-analysis seems to be missing though)
@bruno-medeiros
Copy link

bruno-medeiros commented Jan 22, 2016

Ideally it should be panic-free under normal use and recover from most errors within the next few months.

Sweet 👍

@alexcrichton
Copy link
Member

alexcrichton commented Jan 29, 2016

🔔 This RFC is now entering its two-week long final comment period 🔔

@lilianmoraru
Copy link

lilianmoraru commented Jan 30, 2016

Don't know if it is of any help but I will throw it out there.
QtCreator(mainly C++ IDE) has a plugin that uses Clang to offer C++ code completion.
Here is the code:
http://code.qt.io/cgit/qt-creator/qt-creator.git/tree/src/libs/clangbackendipc
http://code.qt.io/cgit/qt-creator/qt-creator.git/tree/src/plugins/clangcodemodel

QtCreator also has an older custom C++ code model(that is being replaced by the clang one) that was a lot faster and seems like it is accepted in the community that the code model that uses the compiler will be considerably slower, but it offers more information and is more accurate.
I mention the speed because I saw @phildawes mentioning the hard requirements on the compiler to deliver the information in <100ms and I am not sure if we should expect it to be fast.

@DemiMarie
Copy link

DemiMarie commented Feb 1, 2016

@lilianmoraru The reason the compiler needs to deliver the information so quickly is that the GUI is waiting on it. The user is expecting the completion to appear as soon as the user presses . and that requires the compiler to respond fast.

@michaelwoerister
Copy link
Contributor

michaelwoerister commented Feb 4, 2016

I'm in favor of accepting this RFC. The approach seems worth exploring and it does not preclude improving the compilers amenability for being used as a library. On the contrary, I think the compiler's APIs will benefit from trying to build the RLS on top of it and it can only help if people on the compiler team are actual clients of their own APIs. The RFC leaves many open questions when it comes to specifics and we just need a prototype implementation and the experience that comes from building that in order to decide how to proceed further. Worst case, we'll learn a bunch of stuff on what doesn't work :)

@erkinalp
Copy link

erkinalp commented Feb 8, 2016

1-based line numbers and 0-based column numbers please.

@nrc
Copy link
Member Author

nrc commented Feb 9, 2016

@erkinalp why?

@bruno-medeiros
Copy link

bruno-medeiros commented Feb 9, 2016

@erkinalp why?

I was wondering the same, why is a mixed format being used (1-based in one, and 0-based for the other)

@erkinalp
Copy link

erkinalp commented Feb 9, 2016

Emacs uses 0-based column numbers.

@ticki
Copy link
Contributor

ticki commented Feb 9, 2016

@erkinalp That's a perfect reason for not doing that 😜 .

@erkinalp
Copy link

erkinalp commented Feb 9, 2016

@nrc
Copy link
Member Author

nrc commented Feb 9, 2016

afaik, there is no standard for editors to use 1-based line numbers. Having both 0-based seems like the least confusing thing to do, it's easy enough for editors to add one to each line number.

@dgrunwald
Copy link
Contributor

dgrunwald commented Feb 9, 2016

Compilers (including rustc) tend to use 1-based line and column numbers in their error messages. Following that standard seems like the least confusing thing to do. I certainly wouldn't expect 0-based line numbers.

But the concepts of "line" and "column" are ill-defined anyways.
Does a vertical tab (\v) count as a new line? What about form feed? What about all the other exotic Unicode newlines?
Does a tab count as N columns (where N is configurable; usually 4 or 8?), does it advance to the next multiple of N columns, or does it count as only 1 character?
Are full-width characters like 'x' one or two columns wide?
Or does the editor count each Unicode codepoint as 1 column? Maybe a 'column' really is a UTF-8 byte offset within the line? Maybe it's measured in UTF-16 code units? Grapheme clusters?

It's difficult to find two editors that agree in their line/column counting for all possible input files.

@Valloric
Copy link

Valloric commented Feb 9, 2016

I've integrated 5+ semantic engines in ycmd, and the only thing that makes sense is 1-based line and column numbers. Columns are byte offsets in UTF-8. Done.

it's easy enough for editors to add one to each line number.

But why should they? Line & column numbers coming from your oracle will be shown to the user and they expect 1-based numbering.

there is no standard for editors to use 1-based line numbers.

And yet they ~all do use 1-based numbers in the user interface. When you put your caret on the first line in the file, the editor doesn't say the line number is 0, it says it's 1. Same for columns.

@erkinalp
Copy link

erkinalp commented Feb 10, 2016

Vertical tab means skip one line below and continue from same column offset.
CR, CR+LF, LF, LS and NEL are regular line feeds.
FF and PS count as two lines instead of one.

@bruno-medeiros
Copy link

bruno-medeiros commented Feb 10, 2016

I've integrated 5+ semantic engines in ycmd, and the only thing that makes sense is 1-based line and column numbers. Columns are byte offsets in UTF-8. Done.

Why byte offsets and not Unicode character offsets? It's not like an error or position for a Rust symbol will ever start in the middle of a Unicode character.

there is no standard for editors to use 1-based line numbers.

And yet they ~all do use 1-based numbers in the user interface. When you put your caret on the first line in the file, the editor doesn't say the line number is 0, it says it's 1. Same for columns.

Because the internal API for lines and columns can be 0-based, despite the UI being 1-based. This is certainly the case for Eclipse, for IntelliJ, and probably for most IDEs/editors out there. It would not surprise me if Vim is the odd one out... 😆

@adelarsq
Copy link

adelarsq commented Feb 10, 2016

Keep it simple. Just use 0-based for lines and columns.

@alexcrichton
Copy link
Member

alexcrichton commented Feb 11, 2016

This RFC was discussed during the tools team triage today and the decision was to merge. This RFC is still at a somewhat high level and some minor details can continue to be ironed out in the implementation over time, but there seems to be widespread agreement about the body of the RFC here.

Thanks again for the discussion everybody!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

You can’t perform that action at this time.