Context-sensitive IDE completions #3960

odersky · 2018-02-04T15:51:38Z

This provides support for full context-sensitive completions. Local scopes and imports are considered. Member completions take possible implicit conversions into account.

Based on #3949.

dottybot

Hello, and thank you for opening this PR! 🎉

All contributors have signed the CLA, thank you! ❤️

Commit Messages

We want to keep history, but for that to actually be useful we have
some rules on how to format our commit messages (relevant xkcd).

Please stick to these guidelines for commit messages:

Separate subject from body with a blank line

When fixing an issue, start your commit message with Fix #<ISSUE-NBR>:

Limit the subject line to 72 characters

Capitalize the subject line

Do not end the subject line with a period

Use the imperative mood in the subject line ("Added" instead of "Add")

Wrap the body at 80 characters

Use the body to explain what and why vs. how

adapted from https://chris.beams.io/posts/git-commit

Have an awesome day! ☀️

odersky · 2018-02-04T15:53:15Z

I have one issue that's still remaining: When doing selection completions, let's say following

sym.in

we often get a completion on an error tree sym.<error>. This is a pity since it means we cannot pre-filter on the prefix in (yes, VSCode does the filtering as well, but it usually includes way too many symbols, and also cannot distinguish between type names and term names).

Is there a way to get round this? I.e. force the client to give us the latest source?

odersky · 2018-02-04T16:33:58Z

Separate TODO: The REPL should use the new completer in Interactive instead of the old one.

smarter · 2018-02-04T18:00:21Z

Is there a way to get round this? I.e. force the client to give us the latest source?

An error tree created by errorTree is just an untyped tree with an ErrorType set, so you should still be able to see the name.

smarter · 2018-02-04T18:03:16Z

compiler/src/dotty/tools/dotc/interactive/Interactive.scala

+      ctx
+  }
+
+  def contextOfPath(path: List[Tree])(implicit ctx: Context): Context = path match {


This looks very fragile. Since Typer already has to do a similar job for typedIdent to work, I was wondering if instead we could make a subclass of Typer that:

Skips trees outside of the current position

As soon as it reaches a leaf, find all completions and then exit.

That's pretty much how scalac's PC works. It's a possibility, but would be a much larger rewrite compared to what we have. Other downsides: (1) it could be a lot slower, depending how much needs to be typechecked. (2) we create new symbols, which might confuse things. (3) Generally, more entanglement of normal compiler and presentation code.

(2) we create new symbols, which might confuse things.

But a ReTyper doesn't create symbols, does it? It also should be possible to not retypecheck anything and rely on the existing typechecked tree.

That's true. Retypers don't create symbols. But I think messing with them will ultimately be more fragile. It comes down to whether we can coax a 3000 line module to do something quite different than what it was intended for, or whether we write a 40 line method that extracts the information directly. I was actually quite pleased how straightforward that was. True, there are some small differences, i.e. we ignore the "in supercall" distinctions. But I don't think these matter. And it's not clear that a retyper approach would not have similar differences.

odersky · 2018-02-04T22:55:04Z

An error tree created by errorTree is just an untyped tree with an ErrorType set, so you should still be able to see the name.

That's not what I am seeing. I a see a lot of trees where the selector is literally "<error>". These are generated by the parser when the "." is not followed by an identifier. My theory is that the client is giving the server these trees to complete even though the user has since typed some more characters. But I don't know why that happens or how to work around it.

It does not happen if I come from a full identifier like x.abc, then backspace into the abc and do completions. But it does happen we I type forward x.abc from left to right and then do the completion after the abc. Somehow the client passes the tree x.<error> instead of x.abc.

retronym · 2018-02-04T23:59:02Z

Sort of related, let's think about how the parser enable completions like global.def<TAB>. I want definitions there, not the keyword def.

I worked around this in the completion work I did for the 2.12 REPL by inserting some dummy text at the cursor, but you then need to account for the length of that text when looking at positions.

So maybe the parser should be told the cursor location directly in interactive mode, and promote unexpected keywords to idents or selections.

smarter · 2018-02-05T00:01:00Z

So maybe the parser should be told the cursor location directly in interactive mode, and promote unexpected keywords to idents or selections.

Couldn't we just do that all the time?

retronym · 2018-02-05T00:04:11Z

Also, during my REPL work, I added a layer of abstraction over the scope- and member-completion APIs, completionsAt. That handled some corner cases that are tested in CompletionsTest. While that test is at the REPL completion level, it covers some interesting test cases for general completion.

retronym · 2018-02-05T00:08:19Z

Couldn't we just do that all the time?

I guess that's an implementation choice. So long as "foo".def remains a parse error, and we don't end up emitting a no such member "def" type error.

odersky · 2018-02-05T08:13:24Z

I reverted 349d79b in 858d52c. Without the wider matchSymbol definition, we would not count apply methods of case classes as references, for instance.

odersky · 2018-02-05T08:24:51Z

I did some more investigation. If I type e.g. sym.in CTRL-TAB, even very quickly, the source for the completion contains only sym.. If I wait a couple of seconds and then press CTRL-TAB again, the source is sym.in. I am not sure why that happens and how we can mitigate it.

It's not so bad, since we will simply return all completions from the ., and let the client choose which one to present. When presenting, VSCode does know about the prefix, but it throws a lot of other completions out which contain the prefix letters somewhere in the identifiers, not start with the prefix.

For the same reason typing global.def, or even def is not so much of an issue. It will simply return all completions from global. or all completions in the current scope.

This allows the compiler to succeed even if the project is not fully built. It also gives navigation capability into directly dependent files.

@smarter

Strangely, it seems that at least for simple dependencies, the IDE already knows about classes even if class name != source file name. @smarter Do you have an idea why?

Use less state: just a single field that can be either a tree or a tree provider This should make it easy to integrate sourfe files in IDE. The refactoring required a major refactoring in ReadTastyTreesFromClasses, which was a mess before. Hopefully it's not right. There are still "spurious" warnings coming from classes loaded twice. These will require a refactoring of FromTastyTest to avoid. They do not happen if the FromTasty frontend is invoked stand-alone.

1. Since Symbol.tree no longer automatically reads positions (it should not, really, that's what we have contexts for!), ReadPositions has to be turned on in therelevant tests. 2. Disabled simpleClass which fails again because wrong class file ends up being decompiled.

... so that it can be shared with SourcefileLoader in the future.

Various tweaks necessary so that symbols can be loaded from source.

After having implemented the interface between IDE and SymbolLoaders it turns out lateUnits is not needed, and lateFiles is used only internally.

- Don't fail if a class file is not found - Consult name tables of Tasty info for a possible match before loading tree. - Do the same for rename

Found to be usused by invoking findReferences, yay!

Following "name all the things" mantra.

We need that to be able to establish precise contexts.

Collect them all in NamerContextOps. Previously some were also in Typer and in Context itself.

This can cause MergeErrors which should be caught and ignored.

When invoking "goto-definition" on a definition node itself, we now return all overriding or overridden definitions, in all sources. Previously, we returned only overriding definitions in the same source. But it's arguably more useful to know about overriding definitions in other sources because these are harder to find manually. Performance, is not an issue because it's only invoked if one is already on a definition, which should be relatively rare.

Revert to the original definition. For a while this was changed so that a tree would match if its source symbol matched the required symbol but this yields to many trees (e.g. primary constructor of a class in addition to the class itself).

Revert to the original definition. For a while this was changed so that a tree would match if its source symbol matched the required symbol but this yields to many trees (e.g. primary constructor of a class in addition to the class itself). (reverted from commit 349d79b)

smarter

Great improvements!

smarter · 2018-02-09T18:21:32Z

language-server/src/dotty/tools/languageserver/DottyLanguageServer.scala

@@ -65,7 +66,8 @@ class DottyLanguageServer extends LanguageServer
      myDrivers = new mutable.HashMap
      for (config <- configs) {
        val classpathFlags = List("-classpath", (config.classDirectory +: config.dependencyClasspath).mkString(File.pathSeparator))
-        val settings = defaultFlags ++ config.compilerArguments.toList ++ classpathFlags
+        val sourcepathFlags = List("-sourcepath", config.sourceDirectories.mkString(File.pathSeparator), "-scansource")
+        val settings = defaultFlags ++ config.compilerArguments.toList ++ classpathFlags ++ sourcepathFlags


When running launchIDE on Dotty we get:

[error] Flag -sourcepath set repeatedly

Because -sourcepath is part of the settings we set for dotty-library in our build. This could be solved by not adding -sourcepath when it's already set, or by appending source directories to an existing -sourcepath found in config.compilerArguments

Not sure what is the right thing to do here. I was assuming config.sourceDirectories gives us all the info. Does it make sense to include the dotty-library sourcepath?

sourceDirectories does give us all the info, it's just that when compiling dotty-library itself we already use -sourcepath in our build to avoid accidentally linking against the dotty-library on the classpath. In general we cannot assume that -sourcepath is not part of compilerArguments.

Ah, OK, so we can just filter out the existing sourcepath, right?

smarter · 2018-02-09T18:21:45Z

compiler/src/dotty/tools/dotc/config/Printers.scala

  val gadts: Printer = noPrinter
  val hk: Printer = noPrinter
  val implicits: Printer = noPrinter
  val implicitsDetailed: Printer = noPrinter
  val inlining: Printer = noPrinter
+  val interactiv: Printer = new Printer


Should be noPrinter

smarter · 2018-02-09T18:22:05Z

compiler/src/dotty/tools/dotc/core/Symbols.scala

+        case None =>
+          val idSet = mutable.SortedSet[String]()
+          tree.foreachSubTree {
+            case tree: tpd.NameTree => idSet += tree.name.toString


Should be restricted to SimpleName, like in DottyUnpickler#mightContain I think.

smarter · 2018-02-09T18:22:21Z

compiler/src/dotty/tools/dotc/interactive/Interactive.scala

+
+    (  sym == tree.symbol
+    || sym.exists && sym == sourceSymbol(tree.symbol)
+    || include != 0 && sym.name == tree.symbol.name && sym.owner != tree.symbol.owner


sym.owner will crash if sym is NoSymbol

smarter · 2018-02-09T18:22:25Z

compiler/src/dotty/tools/dotc/interactive/Interactive.scala

  private def safely[T](op: => List[T]): List[T] =
    try op catch { case ex: TypeError => Nil }

  /** Get possible completions from tree at `pos`
   *
   *  @return offset and list of symbols for possible completions
   */
+  // deprecated


Add a TODO for removing this

smarter · 2018-02-09T18:22:38Z

compiler/src/dotty/tools/dotc/interactive/Interactive.scala

+     */
+    def include(sym: Symbol) =
+      sym.name.startsWith(prefix) &&
+      !sym.name.toString.drop(prefix.length).contains('$') &&


Can't we just check if the name kind is SimpleName after a stripModuleClassSuffix ?

I want to exclude all compiler-generated names, not just module names.

But all these names will have a kind different from SimpleName, no?

smarter · 2018-02-09T18:22:47Z

compiler/src/dotty/tools/dotc/interactive/Interactive.scala

+          addMember(imp.site, name)
+          addMember(imp.site, name.toTypeName)
+        }
+        for (renamed <- imp.reverseMapping.keys) addImport(renamed)


I don't think this can work since the name renamed doesn't exist in imp.site

Good point. I removed the code and added a FIXME. Will open a separate issue.

smarter · 2018-02-09T18:22:58Z

compiler/src/dotty/tools/dotc/interactive/Interactive.scala

+
+      if (ctx.owner.isClass) {
+        for (mbr <- accessibleMembers(ctx.owner.thisType))
+          addMember(ctx.owner.thisType, mbr.name)


Equivalent to addAccessibleMembers(ctx.owner.thisType)

smarter · 2018-02-09T18:23:03Z

compiler/src/dotty/tools/dotc/interactive/Interactive.scala

+  def contextOfStat(stats: List[Tree], stat: Tree, exprOwner: Symbol, ctx: Context): Context = stats match {
+    case Nil =>
+      ctx
+    case first :: _ if first eq stat =>


Why is the first statement handled specially?

It's not. There's a tail recursion in the third clause.

smarter · 2018-02-09T18:23:08Z

compiler/src/dotty/tools/dotc/interactive/Interactive.scala

+          }
+          localCtx
+        case tree @ Template(constr, parents, self, _) =>
+          if ((constr :: self :: parents).exists(nested `eq` _)) ctx


Any reason to not use .contains(nested) here?

None. I overlooked that tree comparison is by eq anyway.

smarter · 2018-02-09T18:25:47Z

In InteractiveDriver.scala, doing a Find All References query on the definition of allTrees does not bring up the usages in DottyLanguageServer.

odersky · 2018-02-11T17:07:13Z

In InteractiveDriver.scala, doing a Find All References query on the definition of allTrees does not bring up the usages in DottyLanguageServer.

Yes, I think that's because these are different projects. I am not sure what needs to be done to also search external projects. Opening a fresh issue.

smarter · 2018-02-11T18:15:38Z

Right, but to be conservative I would keep it and append our sourceDirectories to it if it already exists

odersky · 2018-02-12T08:42:45Z

Right, but to be conservative I would keep it and append our sourceDirectories to it if it already exists

The problem is that uncontrolled source directories can be surprising and expensive. So I'd rather keep a tight lid on it and not include some random stuff on the classpath.

dottybot reviewed Feb 4, 2018

View reviewed changes

smarter reviewed Feb 4, 2018

View reviewed changes

odersky mentioned this pull request Feb 5, 2018

Restart presentation compilers if memory is low #3967

Merged

odersky added 17 commits February 6, 2018 10:50

Add sourcepath to IDE setup

5a94139

This allows the compiler to succeed even if the project is not fully built. It also gives navigation capability into directly dependent files.

Turn -scansource on for IDE

b27cf86

Strangely, it seems that at least for simple dependencies, the IDE already knows about classes even if class name != source file name. @smarter Do you have an idea why?

Refactoring: Move stuff from DottyUnpickler to TreeProvider

dd8ad5c

... so that it can be shared with SourcefileLoader in the future.

Disable more from tasty tests

b615dc1

Allow symbols to be loaded from source in IDE

dedaeac

Various tweaks necessary so that symbols can be loaded from source.

Fix computation of moduleClass

c863922

Re-enable FromTasty test

ef0473c

Disable more tasty tests

83ccf31

Disable one more from tasty test.

a72b577

Drop lateUnits

de4f83f

After having implemented the interface between IDE and SymbolLoaders it turns out lateUnits is not needed, and lateFiles is used only internally.

Disable yet another fromTasty cmd test

cb86aa4

Make findReferences more robust and faster

b6b09cc

- Don't fail if a class file is not found - Consult name tables of Tasty info for a possible match before loading tree. - Do the same for rename

Disable even more cmd tests

e407c29

Cache identifier search in trees

82d25a0

Delete unused method

5bc01c2

Found to be usused by invoking findReferences, yay!

odersky added 12 commits February 6, 2018 10:52

Refactor completions

4cfb00f

Following "name all the things" mantra.

Only collect simple names when reading unpickler name tables

cdb202d

Collect references as well as definitions for computing ids of a tree

9f4dac2

Maintain compilationUnits mapping

7d86966

We need that to be able to establish precise contexts.

Centralize methods for local context creation

9747f6b

Collect them all in NamerContextOps. Previously some were also in Typer and in Context itself.

Redo completions

5d3b040

Improve completion

518e9fd

Consider implicit conversions for possible completions

d93a8eb

Harden accessibleMembers

c56f126

This can cause MergeErrors which should be caught and ignored.

Tighten matchSymbol again

6342078

Revert to the original definition. For a while this was changed so that a tree would match if its source symbol matched the required symbol but this yields to many trees (e.g. primary constructor of a class in addition to the class itself).

odersky force-pushed the ide-completions branch from 858d52c to 0519aca Compare February 6, 2018 09:55

smarter requested changes Feb 9, 2018

View reviewed changes

Address review comments

3e96c0f

Fix syntax error and refine which names are cached.

a2eb753

odersky assigned smarter Feb 12, 2018

Fix contextOfStats

4d0b9ed

smarter approved these changes Feb 12, 2018

View reviewed changes

smarter merged commit 5c4d7d2 into scala:master Feb 12, 2018

allanrenucci mentioned this pull request Mar 1, 2018

Blog Post: Announcing Dotty 0.6.0 and 0.7.0-RC1 #4053

Merged

karolchmist mentioned this pull request Jun 7, 2018

dotty: 0.6.0-RC1 -> 0.8.0-RC1 NixOS/nixpkgs#41626

Merged

8 tasks

Duhemm mentioned this pull request Aug 20, 2018

Add test for #2864 #4967

Merged

allanrenucci deleted the ide-completions branch August 21, 2018 14:14

Duhemm mentioned this pull request Nov 18, 2018

IDE: Implicits should show up in completion results #2864

Closed

Duhemm mentioned this pull request Nov 30, 2018

Fix #3979: Completion for renamed imports #4977

Merged

Context-sensitive IDE completions #3960

Context-sensitive IDE completions #3960

Conversation

odersky commented Feb 4, 2018

dottybot left a comment

Choose a reason for hiding this comment

Commit Messages

odersky commented Feb 4, 2018

odersky commented Feb 4, 2018

smarter commented Feb 4, 2018

Choose a reason for hiding this comment

odersky Feb 4, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

odersky commented Feb 4, 2018 • edited Loading

retronym commented Feb 4, 2018

smarter commented Feb 5, 2018

retronym commented Feb 5, 2018

retronym commented Feb 5, 2018

odersky commented Feb 5, 2018

odersky commented Feb 5, 2018 • edited Loading

smarter left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

odersky Feb 11, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

odersky Feb 11, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

smarter commented Feb 9, 2018

odersky commented Feb 11, 2018 • edited Loading

smarter commented Feb 11, 2018 via email

odersky commented Feb 12, 2018

odersky Feb 4, 2018 •

edited

Loading

odersky commented Feb 4, 2018 •

edited

Loading

odersky commented Feb 5, 2018 •

edited

Loading

odersky Feb 11, 2018 •

edited

Loading

odersky Feb 11, 2018 •

edited

Loading

odersky commented Feb 11, 2018 •

edited

Loading