Switch between filenames #19 #33

olafurpg · 2017-07-01T22:29:21Z

No description provided.

olafurpg · 2017-07-01T22:31:19Z

Just opening to show progress, this is still not working. Nothing happens when calling "Show definition" to the Chunk identifier which appears in a different uri, but "show references" does trigger the MetadocTextModelService even when the uri is Doc.scala. It currently fails with the error "scala.scalajs.js.JavaScriptException: Error: Cannot instantiate a second Model with the same URI)", which sounds reasonable.

jonas · 2017-07-02T09:50:23Z

metadoc-js/src/main/scala/metadoc/ScalaTextContentProvider.scala

+@ScalaJSDefined
+class ScalaTextContentProvider extends ITextModelContentProvider {
+  override def provideTextContent(resource: Uri): Promise[IModel] = {
+    val model = Editor.createModel(


One possible fix for the "cannot instantiate second model with the same URI" error would be to first lookup if a model already exists using https://microsoft.github.io/monaco-editor/api/modules/monaco.editor.html#getmodel.

That did the trick, thanks!

jonas · 2017-07-02T10:14:50Z

metadoc-js/src/main/scala/monaco/Monaco.scala

@@ -2624,3 +2671,27 @@ package worker {
 object Monaco extends js.Object {
  type MarkedString = String | js.Any
 }
+
+package services {
+


I suggest we put the service definitions in a separateMonacoServices.scala file and document rigorously anything that would help a future upgrade or Monaco.

Good point, I've added a note about the ITextModelResolverService rename.

jonas · 2017-07-02T10:21:40Z

metadoc-js/src/main/scala/monaco/Monaco.scala

@@ -11,12 +11,12 @@ import monaco.languages.Languages.Definition
 import monaco.editor.Editor.BuiltinTheme
 import monaco.editor.Editor.LineNumbersOption

-@js.native
+@ScalaJSDefined


Is this needed? If we can get away with only using monaco.Promise I think that is preferable.

It's not necessary. I managed to avoid it by using Future[T], JSConverters.toJSPromise and a case to monaco.Thenable[T].

jonas · 2017-07-02T10:22:59Z

metadoc-js/src/main/scala/monaco/Lifecycle.scala

+package common {
+
+  @ScalaJSDefined
+  trait IDisposable extends js.Object {


This looks similar to the one in monaco.IDisposable

Good catch, removed.

olafurpg

Thank you for the review! I managed to get something working 🎉

It's not very robust, but it's a start. Jump to definition still doesn't switch files, only "find references"

olafurpg · 2017-07-02T12:43:11Z

metadoc-js/src/main/scala/metadoc/ScalaTextContentProvider.scala

+@ScalaJSDefined
+class ScalaTextContentProvider extends ITextModelContentProvider {
+  override def provideTextContent(resource: Uri): Promise[IModel] = {
+    val model = Editor.createModel(


That did the trick, thanks!

olafurpg · 2017-07-02T12:43:19Z

metadoc-js/src/main/scala/monaco/Lifecycle.scala

+package common {
+
+  @ScalaJSDefined
+  trait IDisposable extends js.Object {


Good catch, removed.

olafurpg · 2017-07-02T12:44:17Z

metadoc-js/src/main/scala/monaco/Monaco.scala

@@ -11,12 +11,12 @@ import monaco.languages.Languages.Definition
 import monaco.editor.Editor.BuiltinTheme
 import monaco.editor.Editor.LineNumbersOption

-@js.native
+@ScalaJSDefined


It's not necessary. I managed to avoid it by using Future[T], JSConverters.toJSPromise and a case to monaco.Thenable[T].

olafurpg · 2017-07-02T12:44:41Z

metadoc-js/src/main/scala/monaco/Monaco.scala

@@ -2624,3 +2671,27 @@ package worker {
 object Monaco extends js.Object {
  type MarkedString = String | js.Any
 }
+
+package services {
+


Good point, I've added a note about the ITextModelResolverService rename.

It seems the ITextContentProvider is more relevant for us than the editor service, but I can't figure out how to override the ITextContentProvider. This is just a demo for how to add random content when switching URI.

- Missing a robust converstion from Future[T] to Thenable[T]. - This currently does something when calling "Show references" but nothing when calling "show definition"

- Search "Chunk" - "See references" - double click on refernce in Chunk.scala - File contents change and active model too. This is not a very robust solution. There seems to be an assumption that the editor is a SimpleEditor, and implements methods like getControl(). This commit just adds a stub method for this method. Eventually, we may need to implement the full editor ourselves.

olafurpg · 2017-07-02T13:03:26Z

This PR is ready for another review. It seems there's been a regression in the definition provider, but the reference provider works for multiple files now. I'm investigating what's gone wrong with the definition provider.

The model was undefined when jumping to Chunk.scala, causing a NPE. This commit changes the definition provider to create the model if it doesn't exist before returning the definition location.

This makes it possible to use the browser's back/forward buttons to switch files.

Previously, we assumed Attributes for a single file was always available, which was used to power services such as definition provider. This works no longer since the active file can change. This commit reorganizes the codebase so that attributes are lazily fetched by uri. We currently don't cache attributes, but that might be a good idea eventually.

olafurpg · 2017-07-02T14:39:00Z

I'm think I'm getting the hang of this 😸 It's not possible to jump to definitions across files, and use the browser's back/forward button. Reference provider returns weird positions for cross-file references, I'm investigating what's causing that.

Now that we can switch between files and attributes are lazily fetched, we can load symbols lazily from `symbols/`. This shrinks metadoc.index, which should help with scaling metadoc for larger codebases.

Document symbol provider only needs the definition.

The uri encoding did not handle weird characters like `#`. Persisting the references in metadoc.index is wasteful, this change makes it possible to lookup the references only when needed.

Previously, the reference provider used the same model for all found references, including references from other files. This caused the ranges to be OK for the active file but wrong for other files. This commit changes reference provider to create a new model for each reference before creating the range selection.

- Remove unused imports - Reorganize code.

olafurpg · 2017-07-02T16:47:50Z

All services are fully functioning now for multiple files so I think this PR is ready to go. The diff became quite big, a lot of things had to be refactored to handle the new multi-file scheme.

I added the paiges test sources to the corpus and it's pretty clear that the current reference provider will not become too slow for larger repos since the reference provider eagerly loads ALL references and creates the model for file that contains a reference. I think this is a very promising start, however.

olafurpg · 2017-07-02T16:48:34Z

We can worry about caching/optimizations later 😄

Previously, searching for a non-existent symbol resulted in an uncaught exception. Now we handle 404s and missing symbols, causing the UI to render a friendly "No results" page instead of "Loading....".

olafurpg · 2017-07-02T18:43:11Z

I took the liberty to push the site from the latest commit in the PR http://scalameta.org/metadoc I noticed that the results are weird when using reference provider on Chunk in Doc.scala. I think we can leave that for another PR

olafurpg · 2017-07-02T22:46:22Z

metadoc-cli/src/main/scala/metadoc/cli/MetadocCli.scala

+      // do nothing for symbols that are not globally relevant.
+      case Symbol.Local(_) =>
+      case Symbol.Global(_, _: Signature.TypeParameter) =>
+      case Symbol.Global(_, _: Signature.TermParameter) =>


I just noticed these changes break the definition provider for local symbols. We should be able to work around that by parsing the active model value instead storing them in the global index.

jonas

Wow 👏 good stuff. If you prefer we can merge ASAP and do follow ups in separate PRs.

jonas · 2017-07-02T22:07:59Z

metadoc-js/src/main/scala/metadoc/MetadocApp.scala

-    )
-    editor.setModel(model)
-
+    dom.window.onhashchange = { e: Event =>


Nice. Didn't know it was this simple to get basic routing working.

It seems doable to also get "jump to line" working if we push the line into the URL and have onDidChangeModel call revealPosition.

Yup! Line numbers would definitely be nice, I opened #38

jonas · 2017-07-02T22:11:00Z

metadoc-js/src/main/scala/metadoc/MetadocAttributeService.scala

+import scala.concurrent.ExecutionContext.Implicits.global
+import scala.concurrent.Future
+import scala.meta._
+import scala.meta.internal.semantic.{schema => s}


What does using this API offer compared to Database.load?

This one is "lower-level", it's the plain protobuf messages. For metadoc I've found the protobuf messages to be very useful, they match closely with the monaco api.

jonas · 2017-07-02T22:14:19Z

metadoc-js/src/main/scala/metadoc/MetadocEditorService.scala

+    val model = Editor.getModel(input.resource)
+    editor.setModel(model)
+    selection.foreach { range =>
+      val pos = new Position(range.startLineNumber, range.startColumn)


Use range.getStartPosition?

Nice! Didn't know about that one, although it requires a cast to Range from IRange.

jonas · 2017-07-02T22:21:39Z

metadoc-js/src/main/scala/metadoc/MetadocEditorService.scala

+
+  def findModel(model: IModel, data: IResourceInput): IEditorModel = {
+    if (model.uri.toString() != data.resource.toString()) {
+      null


I'd be interested in having this log something to make sure we don't miss any cases. I am also wondering if findModel ever gets called. Looking at the Simple services code it is private.

Good catch, this is indeed dead code. Removed.

And I agree we should log or fail fast on error cases!

jonas · 2017-07-02T22:25:03Z

metadoc-js/src/main/scala/metadoc/MetadocTextModelService.scala

+  // It's possible to hit those errors for example in the reference provider, where
+  // we call `Future.sequence(Seq.map(modelReference))`. Since js is single threaded,
+  // we are safe from race conditions.
+  private val cache = mutable.Map.empty[String, IModel]


So editor.getModel(s) didn't work as a cache? We need to figure out if Monaco ever calls model.dispose() and we subsequently need to invalidate those entries.

editor.getModel(s) worked fine until I called Future.sequence(Seq[Future[IModel]]), the guard on line 39 is not sufficient since it's possible another callback creates a new model after the guard but before we finish fetching attributes on line 44. Maybe there's a cleaner way to accomplish this, this was a quick hack.

jonas · 2017-07-02T22:55:49Z

metadoc-js/src/main/scala/monaco/Monaco.scala

@@ -513,6 +513,57 @@ class Token protected () extends js.Object {
 }

 package editor {
+  @ScalaJSDefined
+  trait IEditorService extends js.Object {


Mind if we also move this (eventually) to monaco.services?

Done. Good idea.

jonas · 2017-07-02T22:57:16Z

metadoc-js/src/main/scala/monaco/Monaco.scala

@@ -1962,6 +2017,7 @@ package editor {
        diffEditor: IStandaloneDiffEditor,
        opts: IDiffNavigatorOptions = ???
    ): IDiffNavigator = js.native
+    // NOTE: Do not call this method, use MetadocTextModelService.createModel instead.


For the cache and duplicate URI issue?

Amongst other things, it will also be useful to have a single place to create models in #34

jonas · 2017-07-02T22:57:50Z

metadoc-js/src/main/scala/monaco/Monaco.scala

@@ -2286,7 +2342,7 @@ package languages {
        position: Position,
        context: ReferenceContext,
        token: CancellationToken
-    ): js.Array[Location] | Thenable[js.Array[Location]]
+    ): Thenable[js.Array[Location]]


To simplify the signature in our implementation?

Yes, and to (sadly) please IntelliJ.

jonas · 2017-07-02T23:01:59Z

metadoc-js/src/main/scala/monaco/Monaco.scala

+    def `object`: T
+  }
+
+  object IReference {


IReference signifies "interface" so I suggest we do something like this instead to also signal the lifecycle guarantees:

@ScalaJSDefined class ImmortalReference[T](override val `object`: T) extends IReference[T] { override def dispose(): Unit = () }

Good idea! Done.

jonas · 2017-07-02T23:04:01Z

metadoc-cli/src/main/scala/metadoc/cli/MetadocCli.scala

-case class MetadocSite(semanticdb: Seq[AbsolutePath], index: d.Index)
+case class MetadocSite(
+    semanticdb: Seq[AbsolutePath],
+    symbols: Seq[d.Symbol],


Not sure I understand the reason for adding back symbols here.

I added it to support the reference provider without storing references in Index. The index gets really fat quickly and it's unnecessary to download all references to all symbols on startup time. This allows us to lookup references lazily when reference provider is invoked.

Note that Index.symbols is still there, but I strip out the references.

olafurpg

Thank you for the review! I also just noticed #39

Let's merge this and follow up with fixes.

olafurpg · 2017-07-03T06:22:16Z

metadoc-cli/src/main/scala/metadoc/cli/MetadocCli.scala

-case class MetadocSite(semanticdb: Seq[AbsolutePath], index: d.Index)
+case class MetadocSite(
+    semanticdb: Seq[AbsolutePath],
+    symbols: Seq[d.Symbol],


I added it to support the reference provider without storing references in Index. The index gets really fat quickly and it's unnecessary to download all references to all symbols on startup time. This allows us to lookup references lazily when reference provider is invoked.

Note that Index.symbols is still there, but I strip out the references.

olafurpg · 2017-07-03T06:26:08Z

metadoc-js/src/main/scala/metadoc/MetadocApp.scala

-    )
-    editor.setModel(model)
-
+    dom.window.onhashchange = { e: Event =>


Yup! Line numbers would definitely be nice, I opened #38

olafurpg · 2017-07-03T06:26:48Z

metadoc-js/src/main/scala/metadoc/MetadocAttributeService.scala

+import scala.concurrent.ExecutionContext.Implicits.global
+import scala.concurrent.Future
+import scala.meta._
+import scala.meta.internal.semantic.{schema => s}


This one is "lower-level", it's the plain protobuf messages. For metadoc I've found the protobuf messages to be very useful, they match closely with the monaco api.

olafurpg · 2017-07-03T06:27:08Z

metadoc-js/src/main/scala/metadoc/MetadocAttributeService.scala

+    case _: NoSuchElementException => None // 404, not found
+  }
+
+  def fetchsAttributes(filename: String): Future[s.Attributes] = {


This is a terrible name, it should be something like fetchProtoAttributes instead

olafurpg · 2017-07-03T06:29:56Z

metadoc-js/src/main/scala/metadoc/MetadocEditorService.scala

+    val model = Editor.getModel(input.resource)
+    editor.setModel(model)
+    selection.foreach { range =>
+      val pos = new Position(range.startLineNumber, range.startColumn)


Nice! Didn't know about that one, although it requires a cast to Range from IRange.

olafurpg · 2017-07-03T06:35:30Z

metadoc-js/src/main/scala/metadoc/package.scala

+    def toMonacoPromise: Promise[T] =
+      Promise.wrap(toMonacoThenable)
+    def toMonacoThenable: Thenable[T] =
+      future.toJSPromise.asInstanceOf[Thenable[T]]


Yess 🙏 You can imagine the relief when I saw this worked 😅

olafurpg · 2017-07-03T06:37:00Z

metadoc-js/src/main/scala/monaco/Monaco.scala

@@ -513,6 +513,57 @@ class Token protected () extends js.Object {
 }

 package editor {
+  @ScalaJSDefined
+  trait IEditorService extends js.Object {


Done. Good idea.

olafurpg · 2017-07-03T06:38:46Z

metadoc-js/src/main/scala/monaco/Monaco.scala

@@ -1962,6 +2017,7 @@ package editor {
        diffEditor: IStandaloneDiffEditor,
        opts: IDiffNavigatorOptions = ???
    ): IDiffNavigator = js.native
+    // NOTE: Do not call this method, use MetadocTextModelService.createModel instead.


Amongst other things, it will also be useful to have a single place to create models in #34

olafurpg · 2017-07-03T06:39:34Z

metadoc-js/src/main/scala/monaco/Monaco.scala

@@ -2286,7 +2342,7 @@ package languages {
        position: Position,
        context: ReferenceContext,
        token: CancellationToken
-    ): js.Array[Location] | Thenable[js.Array[Location]]
+    ): Thenable[js.Array[Location]]


Yes, and to (sadly) please IntelliJ.

olafurpg · 2017-07-03T06:40:37Z

metadoc-js/src/main/scala/monaco/Monaco.scala

+    def `object`: T
+  }
+
+  object IReference {


Good idea! Done.

jonas reviewed Jul 2, 2017

View reviewed changes

olafurpg commented Jul 2, 2017

View reviewed changes

olafurpg force-pushed the filenames branch from 80a2486 to 6c7bc22 Compare July 2, 2017 12:46

olafurpg added 9 commits July 2, 2017 14:59

Implement Text Model Content Provider.

c92b226

It seems the ITextContentProvider is more relevant for us than the editor service, but I can't figure out how to override the ITextContentProvider. This is just a demo for how to add random content when switching URI.

Experiment with ITextModelService.

84e2f7f

- Missing a robust converstion from Future[T] to Thenable[T]. - This currently does something when calling "Show references" but nothing when calling "show definition"

Address review comments.

c1a2b75

Address review feedback.

4b057b4

Almost there, it works except there's no content.

df8bf17

Return ITextEditorModel, this fixes the missing content problem!

92d5b73

Add note on ITextModelResolverService getting renamed.

c111bc4

Add helper to convert Future to monaco.Promise.

c4998ed

olafurpg force-pushed the filenames branch from b13a7cb to eff9590 Compare July 2, 2017 12:59

Clean up for review.

4026bf7

olafurpg force-pushed the filenames branch from eff9590 to 4026bf7 Compare July 2, 2017 13:01

olafurpg mentioned this pull request Jul 2, 2017

Switch between files #19

Closed

olafurpg changed the title ~~[WIP] Filenames #19~~ Switch between filenames #19 Jul 2, 2017

olafurpg added 4 commits July 2, 2017 15:41

Fix regression in definition provider.

862e021

Make symbol provider work for jumping to other file.

5f5c583

The model was undefined when jumping to Chunk.scala, causing a NPE. This commit changes the definition provider to create the model if it doesn't exist before returning the definition location.

Sync window.location.hash with editor model.

aceabdf

This makes it possible to use the browser's back/forward buttons to switch files.

olafurpg added 3 commits July 2, 2017 17:20

Add symbol/ index back.

7755223

Now that we can switch between files and attributes are lazily fetched, we can load symbols lazily from `symbols/`. This shrinks metadoc.index, which should help with scaling metadoc for larger codebases.

Include definition in index, but exclude references.

469bf2c

Document symbol provider only needs the definition.

Base 64 encode symbol files instead instead uri encode.

ef850ec

The uri encoding did not handle weird characters like `#`. Persisting the references in metadoc.index is wasteful, this change makes it possible to lookup the references only when needed.

olafurpg added 2 commits July 2, 2017 18:24

Clean up for review.

b4c4ac9

- Remove unused imports - Reorganize code.

olafurpg force-pushed the filenames branch from a185448 to b4c4ac9 Compare July 2, 2017 16:43

olafurpg mentioned this pull request Jul 2, 2017

sbt plugin #37

Closed

Gracefully handle error cases.

be26b4a

Previously, searching for a non-existent symbol resulted in an uncaught exception. Now we handle 404s and missing symbols, causing the UI to render a friendly "No results" page instead of "Loading....".

This was referenced Jul 2, 2017

Figure out uri scheme #35

Closed

Run demo on a larger corpus #22

Closed

Expose the semanticdb messages as model markers #34

Closed

olafurpg commented Jul 2, 2017

View reviewed changes

jonas approved these changes Jul 2, 2017

View reviewed changes

Address review comments

2e215af

olafurpg commented Jul 3, 2017

View reviewed changes

olafurpg merged commit bb59414 into master Jul 3, 2017

olafurpg deleted the filenames branch July 3, 2017 08:48

Switch between filenames #19 #33

Switch between filenames #19 #33

Conversation

olafurpg commented Jul 1, 2017

olafurpg commented Jul 1, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

olafurpg left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

olafurpg commented Jul 2, 2017

olafurpg commented Jul 2, 2017

olafurpg commented Jul 2, 2017

olafurpg commented Jul 2, 2017

olafurpg commented Jul 2, 2017

Choose a reason for hiding this comment

jonas left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jonas Jul 2, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

olafurpg left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jonas Jul 2, 2017 •

edited

Loading