Get server back into working state

[olafurpg]

The combination of #104 #103 and #100 made the server almost unusable. This PR tries to repair a bit of the damage by

• pushing notifications and commands onto the Scheduler instead of main thread. Commands were accidentally running on the main thread because I used Task.now 😅 . Notifications were also blocking the main thread because we compile on every change after Produce a stream of semanticdbs as you type #100.
• disable interactive semanticdb Produce a stream of semanticdbs as you type #100, scalac and scalafix were racing to publish diagnostics making red squigglies jitter in the editor. I propose we dedicate a PR to fix that issue, this PR is only about returning the server to a functioning state for people who checkout master.
• reduce the logging noise to the clients from Move all requests to Task[T] #103 by skipping Message{Reader/Writer} entries. Maybe there's a better way to implement that filter in logback.groovy, but my logback/groovy foo is not great

[olafurpg]

The PC is still getting stuck in an infinite loop

   java.lang.Thread.State: RUNNABLE
	at scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:788)
	at scala.tools.nsc.interactive.Global$Members.addNonShadowed(Global.scala:1006)
	at scala.tools.nsc.interactive.Global.localsToEnclosing$1(Global.scala:1030)
	at scala.tools.nsc.interactive.Global.scopeMembers(Global.scala:1043)
	at scala.tools.nsc.interactive.Global.completionsAt(Global.scala:1242)
	at scala.meta.languageserver.compiler.CompilerUtils$.safeCompletionsAt(CompilerUtils.scala:13)
	at scala.meta.languageserver.providers.CompletionProvider$.completions(CompletionProvider.scala:77)
	at scala.meta.languageserver.ScalametaLanguageServer.$anonfun$completion$1(ScalametaLanguageServer.scala:183)
	at scala.meta.languageserver.ScalametaLanguageServer$$Lambda$1726/167989916.apply(Unknown Source)
	at monix.eval.internal.TaskRunLoop$.monix$eval$internal$TaskRunLoop$$loop$1(TaskRunLoop.scala:187)
	at monix.eval.internal.TaskRunLoop$RestartCallback$1.onSuccess(TaskRunLoop.scala:119)
	at monix.eval.Task$.$anonfun$forkedUnit$2(Task.scala:1463)
	at monix.eval.Task$$$Lambda$903/353321401.run(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)

I have two helpful scripts to debug it

bash-3.2$ cat lsp_kill
jps | grep coursier | awk '{ print $1 }' | xargs kill
bash-3.2$ cat lsp_stack
jps | grep coursier | awk '{ print $1 }' | xargs jstack | grep RUNNABLE -A 20

lsp_kill is especially helpful to kill a zombie process when the shutdown procedure fails to run.

[gabro]

👍 thanks for taking this on! I've left two non blocking comments and I'll add the logging configuration change directly on top of this branch.

[gabro]

this can be configured moreflexibly in logback.groovy. I'll push this change directly on the PR.

[gabro]

Did you remove this entirely because you think there's a better approach? Otherwise just commenting/removing the code that invokes this should be enough.

[olafurpg]

I think There may be a better way to model this as a stream from the notifications, want to experiment with it today

[gabro]

@olafurpg this should be easier to tweak. I've opted for raising the level to INFO, since the noisy RPC messages have a DEBUG level.