Prototype of SqliteSymbolIndex #94

xeno-by · 2017-12-04T03:37:21Z

This is a prototype implementation of SymbolIndex that works on top of SQLite databases created by https://github.com/xeno-by/scalameta/tree/topic/language-server. The code will need significant work before becoming merge-worthy, but nonetheless I wanted to share it with you guys asap to discuss associated design decisions.

The first couple commits tune the existing infrastructure to work well on huge codebases that don't necessarily conform to the SBT project layout. Nothing much to say here - the diffs should be self-explanatory.

The remaining commits factor out the SymbolIndex contract, move the existing SymbolIndex into InMemorySymbolIndex and establish SqliteSymbolIndex. SqliteSymbolIndex hasn't yet reached feature-parity with InMemorySymbolIndex, but even in its current form it is confirmed to provide instantaneous responses to textDocument/definition and textDocument/references requests with negligible memory requirements even on huge codebases.

ShaneDelmore · 2017-12-04T04:20:52Z

metaserver/src/main/scala/scala/meta/languageserver/search/InMemorySymbolIndex.scala

+    for {
+      document <- documentIndex.getDocument(path.toNIO.toUri)
+      _ = logger.info(s"Found document for $path")
+      _ <- isFreshSemanticdb(path, document)


This is odd. The naming convension says it should return a Boolean, the doc comment says it returns a bool, yet it returns Option[Unit] and performs side effects.

Agreed, this should have a different name but it's unrelated to this PR, this is copy-paste from my code ;)

ShaneDelmore · 2017-12-04T04:32:15Z

metaserver/src/main/scala/scala/meta/languageserver/search/InMemorySymbolIndex.scala

+  }
+
+  /**
+   * Returns false this this document is stale.


This sounds like the correct signature, but it actually returns Option[Unit]

Unrelated to this PR.

ShaneDelmore · 2017-12-04T04:32:16Z

metaserver/src/main/scala/scala/meta/languageserver/search/InMemorySymbolIndex.scala

+  ): Option[Unit] = {
+    val ok = Option(())
+    val s = buffers.read(path)
+    if (s == document.contents) ok


Is this an expensive check?

Unrelated to this PR. This is not on a critical path so it's not a problem, we should be doing edit distance here

ShaneDelmore · 2017-12-04T04:39:08Z

metaserver/src/main/scala/scala/meta/languageserver/search/SqliteSymbolIndex.scala

  def findSymbol(path: AbsolutePath, line: Int, column: Int): Option[Symbol] = {
-    ???
+    conn match {


olafurpg

Woot ⭐️ 💯 Just gave this a swing in slick/slick and references+definition works perfectly 👍 👍 👍

Memory usage is tiny too, big win.

Since we're still experimenting I think it's great to have keep both in-memory and sql indices. It makes it easy to add a third implementation if we get a new idea. Once we find something more stable that we like a lot then we should consider settling on that approach and remove competing implementations.

olafurpg · 2017-12-04T06:27:06Z

metaserver/src/main/scala/scala/meta/languageserver/Main.scala

@@ -11,7 +12,9 @@ object Main extends LazyLogging {
  def main(args: Array[String]): Unit = {
    // FIXME(gabro): this is vscode specific (at least the name)
    val workspace = System.getProperty("vscode.workspace")
-    val logPath = s"$workspace/target/metaserver.log"
+    val logDir = new File(s"$workspace/.metaserver")


s/java.io/java.nio.file/

olafurpg · 2017-12-04T06:29:03Z

metaserver/src/main/scala/scala/meta/languageserver/search/InMemorySymbolIndex.scala

+    for {
+      document <- documentIndex.getDocument(path.toNIO.toUri)
+      _ = logger.info(s"Found document for $path")
+      _ <- isFreshSemanticdb(path, document)


Agreed, this should have a different name but it's unrelated to this PR, this is copy-paste from my code ;)

olafurpg · 2017-12-04T06:31:07Z

metaserver/src/main/scala/scala/meta/languageserver/search/InMemorySymbolIndex.scala

+  ): Option[Unit] = {
+    val ok = Option(())
+    val s = buffers.read(path)
+    if (s == document.contents) ok


Unrelated to this PR. This is not on a critical path so it's not a problem, we should be doing edit distance here

olafurpg · 2017-12-04T06:31:21Z

metaserver/src/main/scala/scala/meta/languageserver/search/InMemorySymbolIndex.scala

+  }
+
+  /**
+   * Returns false this this document is stale.


Unrelated to this PR.

olafurpg · 2017-12-04T06:32:44Z

metaserver/src/test/scala/tests/search/SymbolIndexTest.scala

-      )
-      assertNoDiff(reconstructedDatabase.syntax, originalDatabase.syntax)
+      ???
+      // TODO: No idea how important this test is.


This can be turned into a test against in-memory db.

olafurpg · 2017-12-04T06:38:23Z

metaserver/src/main/scala/scala/meta/languageserver/search/SqliteSymbolIndex.scala

+      case (Some(symbolIdStmt), Some(definitionDataStmt)) =>
+        // TODO: Take into account symbol.definitionAlternative.
+        symbolIdStmt.setString(1, symbol.toString)
+        val symbolIdRs = symbolIdStmt.executeQuery()


For longer-term maintainability, I think we should consider using a higher-level API like Quill http://getquill.io/

I can vouch for quill: extremely pleasant to work with! That said, I would make sure it works reliably with sqlite, which is often a bit "strange" compared to other dbs. I've had problems with slick and sqlite too in the past.

One more vote for quill, it has been my favorite db lib to use so far.

@fwbrasil Can you comment on Quill's compatibility with SQLite?

Quill has the dialect implementation for SQLite but I've seen a complaint on Twitter that it has bugs. I guess not many people use Quill with SQLite since there are no open bugs for it. I can review the dialect later this week if you're planning to use it.

Another alternative is H2. It has better performance, can be used in a client-server setup if necessary, and at least supports standard SQL.

I have no personal attachment nor preference for sqlite over h2. It sounds like we should consider using h2, could you check how h2 impacts performance/space @xeno-by ?

@fwbrasil Perhaps unrelated, but maybe you have some insights on a problem we're facing: how do we compose indices from different sources? I imagine this could impact the choice of a db, but I'm quite a noob in this space 😅. For example, we want to be able to query for all references of a symbol across library1.jar:library2.jar:test-classes/

We could store everything in a single database per user, but that would mean concurrent writes since indexing independent libraries can be done in parallel.

We could store everything in a single database per workspace, but that could mean a lot of wasted disk space for machines with many projects. I personally have 587 projects in my ~/dev directory, so this is a no-go for me.

We could separate each index by its source (library1.jar.db), and for each configuration we open connections to each database per entry in the classpath. I'm not sure how practical this approach is for large classpaths or evolving projects with moving modules.

I'm personally leaning towards option 1 if we can take advantage of concurrent writes. I had previously outruled this option because sqlite doesn't seem to support concurrent writes.

Both H2 and HSQLDB embedded dbs support multiple writers using mvcc.

I didn't know about sqlite's WAL, nice find!

Reading up a bit, it seems we should also consider performance of deletions. We will need to regularly clean up semanticdb indices from stale snapshots of source files. It would be good to investigate difference between h2/sqlite with regards to deletions @xeno-by

xeno-by · 2017-12-04T16:12:27Z

metaserver/src/main/scala/scala/meta/languageserver/search/SymbolIndex.scala

+  def definitionData(symbol: Symbol): Option[SymbolData]
+  def referencesData(symbol: Symbol): List[SymbolData]
+  def indexDependencyClasspath(sourceJars: List[AbsolutePath]): Effects.IndexSourcesClasspath
+  def indexDatabase(document: s.Database): Effects.IndexSemanticdb
 }


@olafurpg What do you think about turning SymbolIndex into a small trait with a curated list of methods? If you're on board with that, I can submit a pull request with just this change right away.

I like it, that's a great first step 👍

xeno-by · 2017-12-15T19:36:16Z

Thank you, everyone, for your insightful comments! We'll be reworking this pull request according to the comments, and in the meanwhile, I've spun off a non-controversial commit into a separate PR: #132.

xeno-by added 8 commits December 3, 2017 19:16

Add explicit log entry for initialization time

c3c1d9e

Move the log to $workspace/.metaserver/log

7fafb8e

Introduce configurable initialization scope

9b4be49

Factor out the SymbolIndex contract

b39f36c

Introduce SqliteSymbolIndex

1605298

Add support for opening database connection in SqliteSymbolIndex

8ed2443

Add support for textDocument/definition in SqliteSymbolIndex

f6e1092

Add support for textDocument/references in SqliteSymbolIndex

a7ced02

xeno-by requested review from gabro, laughedelic and olafurpg December 4, 2017 03:37

xeno-by assigned olafurpg Dec 4, 2017

xeno-by mentioned this pull request Dec 4, 2017

Relational schema for SemanticDB and a converter to SQLite scalameta/scalameta#1174

Closed

xeno-by added a commit to xeno-by/scalameta that referenced this pull request Dec 4, 2017

Add indices required for scalameta/metals#94

f59c520

ShaneDelmore reviewed Dec 4, 2017

View reviewed changes

olafurpg reviewed Dec 4, 2017

View reviewed changes

xeno-by commented Dec 4, 2017

View reviewed changes

xeno-by mentioned this pull request Dec 5, 2017

add sqlite-contrib tpolecat/doobie#223

Closed

xeno-by added a commit to xeno-by/scalameta that referenced this pull request Dec 11, 2017

Add indices required for scalameta/metals#94

d36cace

xeno-by closed this Dec 15, 2017

xeno-by deleted the topic/sqlite branch December 15, 2017 19:39

xeno-by mentioned this pull request Jan 19, 2018

SemanticDB Specification, Version 3.0.0 scalameta/scalameta#1220

Closed

Prototype of SqliteSymbolIndex #94

Prototype of SqliteSymbolIndex #94

xeno-by commented Dec 4, 2017 •

edited

ShaneDelmore Dec 4, 2017

olafurpg Dec 4, 2017

ShaneDelmore Dec 4, 2017

olafurpg Dec 4, 2017

ShaneDelmore Dec 4, 2017

olafurpg Dec 4, 2017

ShaneDelmore Dec 4, 2017

olafurpg left a comment

olafurpg Dec 4, 2017

olafurpg Dec 4, 2017

olafurpg Dec 4, 2017

olafurpg Dec 4, 2017

olafurpg Dec 4, 2017

olafurpg Dec 4, 2017

gabro Dec 4, 2017

ShaneDelmore Dec 4, 2017

xeno-by Dec 4, 2017

fwbrasil Dec 4, 2017

olafurpg Dec 4, 2017

ShaneDelmore Dec 5, 2017

olafurpg Dec 5, 2017

xeno-by Dec 4, 2017

olafurpg Dec 4, 2017

xeno-by commented Dec 15, 2017

Prototype of SqliteSymbolIndex #94

Prototype of SqliteSymbolIndex #94

Conversation

xeno-by commented Dec 4, 2017 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

olafurpg left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

xeno-by commented Dec 15, 2017

xeno-by commented Dec 4, 2017 •

edited