Skip to content

chore: update README.md and docs #46

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 4 commits into from
Aug 21, 2025
Merged

chore: update README.md and docs #46

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
cfcc8b2
chore: update README.md and docs
doorgan Aug 18, 2025
815f2af
chore: cleanup CHANGELOG.md
doorgan Aug 18, 2025
820128c
chore: add section for the development server
doorgan Aug 21, 2025
e8a463d
fix: change wording about what the project is
doorgan Aug 21, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
479 changes: 0 additions & 479 deletions CHANGELOG.md
View file
Open in desktop

Large diffs are not rendered by default.

210 changes: 20 additions & 190 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,212 +1,42 @@
[![Discord](https://img.shields.io/badge/Discord-5865F3?style=flat&logo=discord&logoColor=white&link=https://discord.gg/FvdkuVyted)](https://discord.gg/FvdkuVyted)
![GitHub Workflow Status (with event)](https://img.shields.io/github/actions/workflow/status/lexical-lsp/lexical/elixir.yml)

<img alt="Lexical logo: Lexi the lynx" src="assets/lexi-logo.svg" width="150" height="200" align="right"/>

# Expert

Expert is a next-generation language server for the Elixir programming language.

<br><br><br>

### Note: Development on Lexical will slow while we transition to the new combined effort, Expert.

## Features

* Context aware code completion
* As-you-type compilation
* Advanced error highlighting
* Code actions
* Code Formatting
* Go To Definition
* Completely isolated build environment

## What makes Lexical different?
There are a couple things that lexical does differently than other language servers. Let's look at what separates it from
the pack.

#### Architecture

When lexical starts, it boots an erlang virtual machine that runs the language server and its code. It then boots a
separate virtual machine that runs your project code and connects the two via distribution. This provides the following benefits:

* None of lexical's dependencies will conflict with your project. This means that lexical can make use of dependencies to make developing in it easier without having to "vendor" them. It also means that you can use lexical to work on your project, **even if lexical depends on your project**.
* Your project can depend on a different version of elixir and erlang than lexical itself. This means that lexical can make use of the latest versions of elixir and erlang while still supporting projects that run on older versions.
* The build environment for your project is only aware of your project, which enables as-you-type compilation and error reporting.
* In the future, there is a possibility of having the lexical vm instance control multiple projects

#### Ease of contribution

Lexical has been designed to be easy to contribute to and work on. It features:

* A consistent data model that represents the Language Server Protocol and `mix` tasks to generate new Language Server models.
* A clearly defined separation between the language server and project code
* A set of utilities that deal with manipulating code
* A set of unit tests and test cases that make testing new features easy.

#### Focus on developer productivity

Lexical is also built with an eye on increasing developer productivity, and approaches some common features a little bit
differently. For example, Lexical's code completion is _context aware_, which means that if you type `alias MyModule.|`
you will only receive completions for modules and not the names of functions in `MyModule`. This awareness will extend
to other areas, which means:

* You won't see completions for random functions and types in strings. In fact, when extended to string literals, Lexical will only show you completions if you're inside of an interpolation (`"hello there #{na|}'`).
* If you're inside of a struct reference (`%StructModule.|`), you will only see modules listed that define structs, or are the parents of modules that define structs.

Because of this focus, Lexical aims to deliver depth of features rather than breadth of them. We'll likely spend
more time making sure each thing we add works and feels _just right_ rather than adding a whole slew of features
that people mostly won't use.

#### As you type compilation
Because your project is run in a separate virtual machine, we can compile the code that you're working on as you
type. This means you see errors _immediately_ rather than having to wait for a save. The result is you see and
fix typos, warnings, unused variables and a whole host of errors when they occur, which makes your code better,
faster.
Expert is the official language server implementation for the Elixir programming language.

## Installation

Follow the [Detailed Installation Instructions](pages/installation.md)

```
mix package
```

Lexical will now be available in `_build/dev/package/lexical`

If you would like to change the output directory, you can do so with the `--path` option

```
mix package --path /path/to/lexical
```

Lexical will be available in `/path/to/lexical`.

## Development

Lexical is intended to run on any version of Erlang 25+ and Elixir
1.15+. Before beginning development, you should install Erlang
`25.0` and Elixir `1.15.8` and use those versions when you're
building code.

You should also look at the [complete compatibility
matrix](pages/installation.md#caveats) do see which versions are
supported.
You can download Expert from the [releases page](https://github.com/elixir-lang/expert/releases) for your
operating system and architecture.

You're going to need a local instance in order to develop lexical, so follow the [Detailed Installation Instructions](pages/installation.md) first.
For editor specific installation instructions, please refer to the [Installation Instructions](pages/installation.md)

Then, install the git hooks with

```
mix hooks
```

These are pre-commit hooks that will check for correct formatting and run credo for you.

After this, you're ready to put together a pull request for Lexical!

#### Benchmarks

The `remote_control` project has a set of benchmarks that measure the speed of various internal functions and data structures. In order to use them, you first need to install [git large file storage](https://docs.github.com/en/repositories/working-with-files/managing-large-files/installing-git-large-file-storage), and then run `git pull`. Benchmarks are stored in the `benchmarks` subdirectory, and can be run via

```
mix benchmark /benchmarks/<benchmark_file>.exs
```
### Nightly Builds

### Logging
If you want to try out the latest features, you can download a nightly build.

When lexical starts up, it creates a `.lexical` directory in the root
directory of a project. Inside that directory are two log files,
`lexical.log` and `project.log`. The `.lexical.log` log file contains
logging and OTP messages from the language server, while the
`project.log` file contains logging and OTP messages from the
project's node. While developing lexical, it's helpful to open up a
terminal and tail both of these log files so you can see any errors
and messages that lexical emits. To do that, run the following in a
terminal while in the project's root directory:
Using the GH CLI, you can run the following command to download the latest nightly build:

```shell
tail -f .lexical/*.log
gh release download nightly --pattern 'expert_linux_amd64'
```

Note: These log files roll over when they reach 1 megabyte, so after a
time, it will be necessary to re-run the above command.

### Debugging
Then point your editor to the downloaded binary.

Lexical supports a debug shell, which will connect a remote shell to a
currently-running language server process. To use it, `cd` into your
lexical installation directory and run
### Building from source

```
./bin/debug_shell.sh <name of project>
```
To build Expert from source, you need Zig `1.14.1` installed on your system.

For example, if I would like to run the debug server for a server
running in your `lexical` project, run:

```
./bin/debug_shell.sh lexical
```
Then you can run the following command:

...and you will be connected to a remote IEx session _inside_ my
language server. This allows you to investigate processes, make
changes to the running code, or run `:observer`.

While in the debugging shell, all the functions in
`Expert.IEx.Helpers` are imported for you, and some common
modules, like `Lexical.Project` and `Lexical.Document` are
aliased.

You can also start the lexical server in interactive mode via
`./bin/start_lexical.sh iex`. Combining this with the helpers that are
imported will allow you to run projects and do completions entirely in
the shell.

*Note*: The helpers assume that all of your projects are in folders that are siblings with your lexical project.

Consider the example shell session:

```
./bin/start_lexical.sh iex
iex(1)> start_project :other
# the project in the ../other directory is started
compile_project(:other)
# the other project is compiled
iex(2)> complete :other, "defmo|"
[
#GenLSP.Structures.CompletionItem<[
detail: "",
insert_text: "defmacro ${1:name}($2) do\n $0\nend\n",
insert_text_format: :snippet,
kind: :class,
label: "defmacro (Define a macro)",
sort_text: "093_defmacro (Define a macro)"
]>,
#GenLSP.Structures.CompletionItem<[
detail: "",
insert_text: "defmacrop ${1:name}($2) do\n $0\nend\n",
insert_text_format: :snippet,
kind: :class,
label: "defmacrop (Define a private macro)",
sort_text: "094_defmacrop (Define a private macro)"
]>,
#GenLSP.Structures.CompletionItem<[
detail: "",
insert_text: "defmodule ${1:module name} do\n $0\nend\n",
insert_text_format: :snippet,
kind: :class,
label: "defmodule (Define a module)",
sort_text: "092_defmodule (Define a module)"
]>
]
```sh
just release-local
```

The same kind of support is available when you run `iex -S mix` in the
lexical directory, and is helpful for narrowing down issues without
disturbing your editor flow.
This will build the Expert binary and place it in the `apps/expert/burrito_out` directory. You can then point your
editor to this binary.

### Other resources

* [Architecture](pages/architecture.md)
* [Glossary](pages/glossary.md)
- [Architecture](pages/architecture.md)
- [Development Guide](pages/development.md)
- [Glossary](pages/glossary.md)
- [Installation Instructions](pages/installation.md)
27 changes: 11 additions & 16 deletions pages/architecture.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,32 +2,27 @@

## Project Structure

Lexical is designed to keep your application isolated from lexical's code. Because of this, lexical is structured as an umbrella app, with the following sub-apps:
Expert is designed to keep your application isolated from Expert's code. Because of this, Expert is structured as an umbrella app, with the following sub-apps:

* `common`: Contains all code common to the other applications.
* `proto`: Used by `protocol` to generate the Elixir representation of LSP data structures.
* `protocol`: Code related to speaking the language server protocol.
* `remote_control`: The application that's injected into a project's code, which
gives lexical an API to do things in the context of your app.
* `server` The language server itself.
* `forge`: Contains all code common to the other applications.
* `engine`: The application that's injected into a project's code, which
gives expert an API to do things in the context of your app.
* `expert` The language server itself.

Lexical is an umbrella app so we can control how many dependencies the remote control app has. By separating lexical into sub-applications, each is built as a separate archive, and we can pick and choose which of these applications (and their dependencies) are injected into the project's VM, thus reducing how much contamination the project sees. If lexical was a standard application, adding dependencies to lexical would cause those dependencies to appear in the project's VM, which might cause build issues, version conflicts in mix or other inconsistencies.

Since the `remote_control` app only depends on `common`, `path_glob` and `elixir_sense`, only those applications pollute the project's vm. Keeping `remote_control`'s dependencies to a minimum is a design goal of this architecture.
By separating expert into sub-applications, each is built as a separate archive, and we can pick and choose which of these applications (and their dependencies) are injected into the project's VM, thus reducing how much contamination the project sees. If Expert was a standard application, adding dependencies to Expert would cause those dependencies to appear in the project's VM, which might cause build issues, version conflicts in mix or other inconsistencies.

Since the `engine` app only depends on `forge`, `path_glob` and `elixir_sense`, only those applications pollute the project's vm. Keeping `engine`'s dependencies to a minimum is a design goal of this architecture.

## Language Server

The language server (the `server` app) is the entry point to Lexical. When started by the `start_lexical.sh` command, it sets up a [transport](https://github.com/lexical-lsp/lexical/blob/main/apps/expert/lib/lexical/server/transport.ex) that [reads JsonRPC from standard input and writes responses to standard output](https://github.com/lexical-lsp/lexical/blob/main/apps/expert/lib/lexical/server/transport/std_io.ex).
The language server (the `expert` app) is the entry point to Expert. When started, it sets up a transport via GenLSP that reads JsonRPC and responds to it. The default transport is Standard IO, but it can be configured to use TCP.

When a message is received, it is parsed into either a [LSP Request](https://github.com/lexical-lsp/lexical/blob/main/apps/protocol/lib/lexical/protocol/requests.ex) or a [LSP Notification](https://github.com/lexical-lsp/lexical/blob/main/apps/protocol/lib/lexical/protocol/notifications.ex) and then it's handed to the [language server](https://github.com/lexical-lsp/lexical/blob/main/apps/expert/lib/lexical/server.ex) to process.
When a message is received, it is parsed into either a LSP Request or a LSP Notification and then it's handed to the [language server](https://github.com/elixir-lang/expert/blob/main/apps/expert/lib/expert.ex) to process.

The only messages the [lexical server process](https://github.com/lexical-lsp/lexical/blob/main/apps/expert/lib/lexical/server.ex) handles directly are those related to the lifecycle of the language server itself:
The only messages the Expert server process handles directly are those related to the lifecycle of the language server itself:

- Synchronizing document states.
- Processing LSP configuration changes.
- Performing initialization and shutdown.

All other messages are delegated to a _Provider Handler_. This delegation is accomplished by the server process adding the request to the [provider queue](https://github.com/lexical-lsp/lexical/blob/main/apps/expert/lib/lexical/server/provider/queue.ex). The provider queue asks the `Expert.Provider.Handlers.for_request/1` function which handler is configured to handle the request, creates a task for the handler and starts it.

A _Provider Handler_ is just a module that defines a function of arity 2 that takes the request to handle and a `%Expert.Configuration{}`. These functions can reply to the request, ignore it, or do some other action.
All other messages are delegated to a _Provider Handler_. A _Provider Handler_ is a module that defines a function of arity 2 that takes the request to handle and a `%Expert.Configuration{}`. These functions can reply to the request, ignore it, or do some other action.
67 changes: 67 additions & 0 deletions pages/development.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,67 @@
# Development

## Logging

When expert starts up, it creates a `.expert` directory in the root
directory of a project. Inside that directory are two log files,
`expert.log` and `project.log`. The `.expert.log` log file contains
logging and OTP messages from the language server, while the
`project.log` file contains logging and OTP messages from the
project's node. While developing expert, it's helpful to open up a
terminal and tail both of these log files so you can see any errors
and messages that expert emits. To do that, run the following in a
terminal while in the project's root directory:

```sh
tail -f .expert/*.log
```

Note: These log files roll over when they reach 1 megabyte, so after a
time, it will be necessary to re-run the above command.

## Development server

To start a development server with an interactive shell, you can run the
following command:

```sh
just start
```

This will launch an IEx session, and it will start Expert listening
in the TCP port `9000`.

You will need to configure your editor to connect to the Expert LSP
via TCP at that port. After that, opening you project in your editor
will connect it to the running dev server, and it will terminate it
when you close the editor.

In this dev server you can run `:observer.start()`, and call any
function from Expert to inspect the state of the server, or run
arbitrary code.

Since Expert needs namespacing to work, modules from the `forge`
application will be namespaced as `XPForge`; the same applies for
any module that is shared between the `expert` and `engine`
applications.

## Debugging

Expert supports a debug shell, which will connect a remote shell to a
currently-running language server process. To use it, `cd` into your expert
installation directory and run

```sh
./apps/expert/bin/debug_server.sh <name of project>
```

For example, if I would like to run the debug server for a server running in
your `my_project` project, run:

```sh
./apps/expert/bin/debug_server.sh my_project
```

...and you will be connected to a remote IEx session _inside_ the language
server for `my_project`. This allows you to investigate processes, make changes
to the running code, or run `:observer`.
Loading
Loading