Erroneous behaviour while formatting with expert

[Terbium-135]

This is with 583ee15 (and Zed editor - I can't test with other editors right now)
Might be the reason for / or connected to: #184

Given this snippet:

  @test %{
    a: :a,
    b: :b
  }

After adding a few leading spaces in front of b: :b and doing a reformat with expert - (might have to repeat this several times) there is suddenly an error message unexpected expression after keyword list [...]

expert.log:

04:07:43.093 [debug] handled request client -> server textDocument/hover in 20ms
04:07:43.180 [debug] handled request client -> server textDocument/codeAction in 11ms
04:07:44.562 [error] Formatter failed %SyntaxError{file: "/home/elixir-dev/projects/ganymed/lib/backend/tasks/CreateInvoiceZugferdWithVerification.ex", line: 868, column: 8, snippet: "      ],", description: "unexpected expression after keyword list. Keyword lists must always come last in lists and maps. Therefore, this is not allowed:\n\n    [some: :value, :another]\n    %{some: :value, another => value}\n\nInstead, reorder it to be the last entry:\n\n    [:another, some: :value]\n    %{another => value, some: :value}\n\nSyntax error after: ','"}
04:07:44.562 [debug] handled request client -> server textDocument/formatting in 26ms
04:07:44.605 [debug] Node port message: 
04:07:44.604 [info] Plugins found 0 results in 0.013 ms

Another example:

    ~H"""
    <span class={["mr-3 mb-2 rounded-lg bg-yellow-700 py-1 pr-3 pl-2", @class]}>
        {render_slot(@inner_block)}
    </span>
    """

Inserting leading spaces in front of {render_slot(@inner_block)}, formatting gives

[mhanberg]

I also saw some weird formatting results yesterday. I'll keep an eye on it.

[Terbium-135]

I have the gut feeling this might be a one-off error or counting graphemes wrong.
Really miss Lexical's format while typing...

For now I switched to external formatting;

      "use_on_type_format": false,
      // "use_on_type_format": true,
      "enable_language_server": true,
      "formatter": {
        "external": {
          "command": "mix",
          "arguments": ["format", "--stdin-filename", "{buffer_path}", "-"]
        }
      },

[mhanberg]

Really miss Lexical's format while typing...

Comments like this aren't productive.

[doorgan]

I'm not able to reproduce on Neovim nor Zed

Screen.Recording.2025-11-27.at.2.46.38.AM.mov

Screen.Recording.2025-11-27.at.2.53.48.AM.mov

Although I think this may be related to as-you-type compilation and not formatting per se, sometimes diagnostics get stuck until you save the file

[Terbium-135]

Thanks for the screen recordings. I tried that snippet in different source files for myself and got no error.
Still getting the error in the exact same source file very reliable. I am lost a bit now.
Removed _build and index folder as well as the log files in .expert and started fresh.

This is what I find when the error happens (now a different one but with the same snippet):

10:26:51.850 [info] Local completions are: [callback: (backoff)   function: (backoff, billing_specific_periof, build, build_invoice, build_invoice, binary_part, binary_slice, binary_slice, bit_size, byte_size)   macro: (binding, binding)   ]
10:26:51.853 [info] Emitting Completions: [8:backoff(job) 3:backoff(job) 3:billing_specific_periof(start_date, end_date) 3:build(:en16931, invoice) 3:build_invoice(invoice) 3:build_invoice(invoice, options) 3:binary_part(binary, start, size) 3:binary_slice(binary, range) 3:binary_slice(binary, start, size) 3:bit_size(bitstring) 3:byte_size(bitstring) 3:binding() 3:binding(context) ]
10:26:51.854 [error] ** (FunctionClauseError) no function clause matching in String.slice/3
    (elixir 1.17.3) lib/string.ex:2310: String.slice("        b : :b", 0, -1)
    (xp_forge 0.1.0-0ff78f1) lib/forge/code_unit.ex:23: XPForge.CodeUnit.utf8_position_to_utf16_offset/2
    (xp_expert 0.1.0-0ff78f1) lib/expert/protocol/conversions.ex:146: XPExpert.Protocol.Conversions.extract_lsp_character/1
    (xp_expert 0.1.0-0ff78f1) lib/expert/protocol/conversions.ex:120: XPExpert.Protocol.Conversions.to_lsp/1
    (xp_expert 0.1.0-0ff78f1) lib/expert/protocol/conversions.ex:98: XPExpert.Protocol.Conversions.to_lsp/1
    (xp_expert 0.1.0-0ff78f1) lib/convertibles/forge.document.edit.ex:11: XPForge.Protocol.Convertible.Forge.Document.Edit.to_lsp/1
    (xp_forge 0.1.0-0ff78f1) lib/forge/protocol/convertible.ex:9: anonymous fn/3 in XPForge.Protocol.Convertible.Helpers.apply/2
    (elixir 1.17.3) lib/enum.ex:4858: Enumerable.List.reduce/3

10:26:51.854 [debug] sent notification server -> client window/logMessage
10:26:52.046 [debug] handled request client -> server textDocument/codeAction in 27ms
10:26:52.053 [debug] handled request client -> server textDocument/codeLens in 1ms
10:26:52.135 [debug] Node port message: 
10:26:52.135 [info] Plugins found 0 results in 0.013 ms

10:26:52.135 [debug] sent notification server -> client textDocument/publishDiagnostics
10:26:52.136 [debug] sent notification server -> client textDocument/publishDiagnostics
10:26:52.136 [debug] sent notification server -> client textDocument/publishDiagnostics
10:26:52.137 [debug] sent notification server -> client textDocument/publishDiagnostics
10:26:52.188 [debug] sent notification server -> client textDocument/publishDiagnostics
10:26:52.188 [debug] sent notification server -> client textDocument/publishDiagnostics
10:26:52.188 [debug] Node port message: 
10:26:52.188 [info] Plugins found 0 results in 0.01 ms

10:26:52.189 [debug] sent notification server -> client textDocument/publishDiagnostics
10:26:52.190 [debug] sent notification server -> client textDocument/publishDiagnostics
10:27:08.018 [error] Formatter failed %SyntaxError{file: "/home/elixir-dev/projects/ganymed/lib/backend/tasks/test.ex", line: 45, column: 15, snippet: "        b     : :b", description: "unexpected token: \":\" (column 15, code point U+003A)"}
10:27:08.018 [debug] handled request client -> server textDocument/formatting in 28ms
10:27:08.052 [debug] handled notification client -> server workspace/didChangeWatchedFiles in 3ms
10:27:08.053 [debug] handled notification client -> server textDocument/didSave in 2ms
10:27:08.059 [error] Formatter failed %SyntaxError{file: "/home/elixir-dev/projects/ganymed/lib/backend/tasks/test.ex", line: 45, column: 15, snippet: "        b     : :b", description: "unexpected token: \":\" (column 15, code point U+003A)"}
10:27:08.059 [debug] handled request client -> server textDocument/formatting in 16ms
10:27:08.081 [debug] handled notification client -> server workspace/didChangeWatchedFiles in 3ms
10:27:08.081 [debug] handled notification client -> server textDocument/didSave in 2ms
10:27:08.133 [debug] sent request server -> client window/workDoneProgress/create
10:27:08.135 [debug] sent notification server -> client $/progress
10:27:08.619 [debug] Node port message: Compiling 1 file (.ex)

[Terbium-135]

This is with Zed editor on windows using a WSL connection.
The application developed using:

Erlang/OTP 28 [erts-16.1.2] [source] [64-bit] [smp:12:12] [ds:12:12:10] [async-threads:1] [jit:ns]
Elixir 1.19.3 (compiled with Erlang/OTP 28)

While expert looks like being compiled with (elixir 1.17.3). But this shouldn't matter, should it?

Zed is:

Zed: v0.215.2-82+3b733feb12db25fc4347a5f04b04c55a0f738621 (Zed Preview) 
OS: Windows 10.0.26200
Memory: 63.7 GiB
Architecture: x86_64
GPU: Intel(R) Iris(R) Xe Graphics || Intel Corporation || 32.0.101.7040

[doorgan]

I don't think the version expert is compiled with should be relevant, our goal is that version mismatches don't matter.
It does seem though that there's something wrong in our conversion logic as your trace shows, I'll investigate

[doorgan]

Can you confirm the file you're reliably encountering this issue at contains non-ascii characters like emoji?
The error trace seems very similar to this issue: lexical-lsp/lexical#865

[doorgan]

Does the issue still happen if you build expert from #239?

[Terbium-135]

This specific file contains no emoji but german umlauts (äöüÄÖÜß) withing strings and comments (taking non-ascii as non US-ascii)
I will have a look at your draft very soon.
Thanks for your investigation!

[doorgan]

This specific file contains no emoji but german umlauts (äöüÄÖÜß) withing strings and comments (taking non-ascii as non US-ascii)

In Expert, we count codepoints >127 as non-ascii. Expert does a lot of byte conversions to deal with the LSP utf-16 shenanigans, and from the trace I suspect we're messing up somewhere which leads us to produce an impossible negative offset when making completion suggestions or apparently when formatting too.

[Terbium-135]

That also reminds me of another (formatting) error I had with lexical: lexical-lsp/lexical#843

There was also a utf-16 character in that line (in this case arrow right: ➜).

The line
<%= gettext("Transaction") %> ➜ <%= gettext("Show details") %>
becomes:
{gettext("Transaction")} ➜ <%{ttext("Show details") %}

[Terbium-135]

Somehow I can't get a local expert running. So I was unable to test your fix.
There must be a mistake on my side which I couldn't figure out so far
But that's a problem at my end.

Zed configuration:

  "languages": {
    "Elixir": {
      "show_completions_on_input": true,
      // "show_whitespaces": "all",
      "enable_language_server": true,
      "use_on_type_format": true,
      "formatter": "language_server",
      "language_servers": [
        "expert",
        "!tailwindcss-language-server",
        "!elixir-ls",
        "!next-ls",
        "!lexical",
        "..."
      ]
    }
  },
  "lsp": {
    "expert": {
      "binary": {
        "path": "/home/elixir-dev/projects/expert/apps/expert/burrito_out/expert_linux_amd64",
        "arguments": ["--stdio"]
      }
    }
  },

Looks like expert never initializes
I get:

// Send:
{"jsonrpc":"2.0","id":6,"method":"textDocument/hover","params":{"textDocument":{"uri":"file:///home/elixir-dev/projects/ganymed/lib/backend/tasks/CreateInvoiceZugferdWithVerification.ex"},"position":{"line":44,"character":16}}}

// Receive:
{"jsonrpc":"2.0","method":"window/logMessage","params":{"message":"Received request textDocument/hover before engine was initialized. Ignoring.","type":2}}