Skip to content
Roberto Aloi edited this page Aug 15, 2019 · 2 revisions

erlang_ls

An Erlang server implementing Microsoft's Language Server Protocol 3.0.

Rationale

The Erlang community is, at the time of this writing, quite small when compared to other communities. Within the same Erlang community, different developers have been using different editors to write their code, ranging from Emacs to Vim, from Eclipse to VS Code. The limited number of community members, combined with the fragmentation of editors used, historically caused the lack of a serious development environment for Erlang.

The Language Server Protocol enables developers to build a single language server that multiple development tools and editors can leverage.

The goal of this project is to create that missing language server for Erlang. For the time being, the scope of the project is limited to Erlang only, but with the mindset of supporting other BEAM languages in the long term.

Principles

  • Follow OTP design patterns as much as possible
  • Re-use existing Erlang standard libraries as much as possible. Do not re-invent the wheel
  • Use property-based testing to test the erlang_ls software
  • Develop for collaboration
  • Feature driven
  • User documentation

History of the project

I am an Erlang developer and an affectionate Emacs user. For most of my professional career I have been using Emacs's erlang-mode and, once available, the EDTS suite, whose author is a former colleague. These two tools, combined with a few extra customizations and additions, provided me with a decent development environment for Erlang. I felt quick and productive when writing and navigating Erlang code and I never really missed a more powerful IDE.

I occasionally experienced a few bugs with these tools and identified a few potential improvements, but nothing major that would trigger the quest for a better alternative. At some point I started consulting for a company which used comma-first indentation and Emacs' automatic indentation started to break here and there. I started switching heavily between projects using different versions of the Erlang Runtime system and EDTS, even if in theory supported multiple OTP runtimes, occasionally started to crash and spam my EDTS buffer with error messages related to xref, the Erlang cross-reference tool. Reverting Emacs buffers, killing and restarting the EDTS process, became normal part of my developer workflow. After a recent upgrade, EDTS started to perform poorly. It would get occasionally stuck and I had to press C-g a few times to recover control of my Emacs. When working on huge projects (in terms of lines of code), EDTS was freezing my entire Emacs every time I was navigating into a newly-opened module.

It was 2018 when I contacted EDTS' author to ask if I was the only one experiencing these issues. Some of these problems were apparently known, but not easily reproducible. Also, Thomas (EDTS's author) was not using Erlang anymore, so he had very little incentive in improving or even maintaining EDTS. I shivered for a second, since I was relying so much on EDTS. It was at that point that Thomas asked me if I was interested in contributing to EDTS or even becoming a maintener for it.

I started diving into the EDTS code-base and learnt about its architecture. It was surprisingly interesting. At start-up, EDTS would spin up a distributed Erlang node (named edts) and then a separate Erlang node for each project. The two Erlang nodes would communicate between each other using the standard Erlang Distribution Protocol. The edts node exposed an HTTP API that the Emacs client interacted with. This setup allowed (with some limitations) different versions of the Erlang runtime for different projects. The approach had a lot of potential. One could implement a lot of interesting features in an easy way by using the standard Erlang libraries. In principle, if one could make the HTTP API generic enough and not Emacs-specific, other editors could have leveraged these libraries. This would have been particularly beneficial in a small community like the Erlang one. Just think about it: a bug fix to the way return values for a spec are indented would automatically be available to users of all major text editors and IDEs. Unfortunately, the client part in EDTS was still quite heavy and a lot of functionalities (e.g. auto-completion) were too coupled with Emacs and implemented directly in Emacs Lisp. My interest, though, was growing.

A couple of weeks later I attended the Code Beam conference in Stockholm. I was sharing my ideas with a few attendees when Csaba (co-author of vim-erlang) told me that Vlad (co-author of the erlide Eclipse plugin) was working on a similar project. It was time for me to attend Vlad's talk.

At Vlad's talk I learnt that the idea of having a well defined API between text editors (or IDEs) on one side and language-specific servers on the other one was not so new. It was not surprising to see that the driver for such an approach was Microsoft, which obviously faced the problem of supporting multiple programming languages (e.g.. C, C++, C#, F#, TypeScript) in a common IDE (i.e. Visual Studio). What was surprising (at least for me) was that Microsoft actually defined and open-sourced a fully-bledged Language Server Protocol (aka LSP) to solve the problem, and that all major text editors and IDEs (including Emacs \o/) already implemented a client for it.

I started looking for implementations of Language Servers for Erlang and found a few ones (including one for Elixir), all at different development stages and using different approaches (see the other LSP implementations section for details).

It was time to roll my own.

Other LSP Implementations

erlang_ls is not by any means the first, the only or the best Erlang implementation of the Language Server Protocol (aka LSP). This section aims at covering the other existing implementations, explaining differences and similarities with erlang_ls. Since things around erlang_ls will inevitably evolve, this section will tend to become outdated very quickly. Therefore, feel free to contribute by pull-requesting a change whenever deemed necessary. Also consider that erlang_ls is an extremely young project and that some of the functionalities discussed here are not fully implemented, yet. The intent of this section is not to criticize other implementations, but to explain why a new implementation was created. As always in software design, there are no best solutions, but choices and trade-offs.

sourcer

To use the author's own words, sourcer is the reincarnation of a 7 years old, battle-tested, Erlang plugin for Eclipse, named ErlIDE, to use Microsoft's Language Server Protocol.

The project is split into two separate Erlang applications: lsp_server and sourcer. lsp_server is the actual implementation of the Language Server Protocol (aka LSP) and it supports both TCP and STDIO transports. The application consists of a single one_for_one supervisor with three child processes. One of the three children, named jsonrpc, is a TCP server responsible for encoding/decoding LSP messages and for dispatching these messages from/to the other two children: lsp_server and lsp_client, respectively the LSP server itself and a middleman towards the actual LSP client (i.e. the text editor). The sourcer application contains the port of the erlide libraries to the new format. Here the origin of the project is clearly visible in the structure of the code-base, in the references to erlide and the presence of a src2 folder, containing all the erlide code not yet ported to the new project. In reality, the project contains a third Erlang application, named erlang_ls (yes, what a fortunate name collision), but that is just a wrapper to create an Erlang escript to bootstrap the whole server, so we can safely ignore it here.

In sourcer, a separate worker process is spawn to handle each individual LSP method (e.g. textDocument/completion or textDocument/hover). sourcer uses a mix of OTP patterns and basic, low-level, Erlang spawning and message passing, which this project tries to avoid, preferring to adopt OTP guidelines and patterns across the entire code-base.

sourcer defines the concept of a cancellable worker. Essentially, since every LSP method is executed within a separate process, this process can be killed at any time. Each worker processes can send partial results to a central server, which can return these partial results to the client upon cancellation. This is not the cases for the erlang_ls project, where processes can be killed, but no partial results are sent back to the client. The rationale behind this choice is that partial result could often be misleading to the end user. Please notice that request cancellation and partial results are optional features according to the LSP protocol, so both approaches are valid.

sourcer supports full-text synchronization, meaning that upon each change the entire content of the modified buffer is sent to the server. All contents for all buffers are stored into the state of a single process. On the contrary, erlang_ls has a dedicated supervisor for text synchronization purposes, where each opened buffer is modelled as a separate Erlang process with its own state, favouring decoupling.

An interesting idea in sourcer is the presence of a database which is populated by worker processes and queried by the LSP server. The database acts as a memoization tool and avoids re-calculating already calculated information.

sourcer implements a custom scanner and parser, which try to handle situations where, for example, an un-closed string would provoke the rest of an Erlang module to be un-parsable. It does so by artificially adding quotes and other potentially missing tokens. Having a temporarily un-parsable module is not considered an actual problem by the erlang_ls project, which prefers simplicity in this scenario and uses the default Erlang scanner and parser.

sourcer implements some very basic project support, mainly focusing on rebar3 conventions. erlang_ls tries to be agnostic of the building tool whilst providing workspace functionalities.

sourcer provides auto-formatting of Erlang modules using custom tools (i.e. the sourcer_indent module). erlang_ls prefers standard Erlang tools, such as the erl_tidy module. In the same way, sourcer prefers to run its own custom cross-reference analysis of Erlang modules, whilst erlang_ls prefers the standard xref tool.

vscode_erlang

TODO

EDTS

TODO

Clone this wiki locally