LSP: docs/hover handler doesn't handle HTML escape sequences

[Kasama]

Problem

Related to this issue

When the Language server responds a markdown payload for a textDocument/hover request containing html escape sequences (in this example, many   sequences), the stylize_markdown function doesn't handle it, causing it to be written as-is to the floating window's buffer.

Example response:

{
  "contents": {
    "kind": "markdown",
    "value": " # # # #   g i t l a b \\ - c i \n \n T h e   n a m e   o f   o n e   o r   m o r e   j o b s   t o   i n h e r i t   c o n f i g u r a t i o n   f r o m \\ . \n \n S o u r c e :   [ c i . j s o n ] ( h t t p s : / / g i t l a b . c o m / g i t l a b - o r g / g i t l a b / - / r a w / m a s t e r / a p p / a s s e t s / j a v a s c r i p t s / e d i t o r / s c h e m a / c i . j s o n ) "
  },
  "range": {
    "end": {
      "character": 9,
      "line": 2377
    },
    "start": {
      "character": 2,
      "line": 2377
    }
  }
}

Example hover on neovim

It would be great to have neovim decode those html sequences, similar to what this luarocks package does.

---

A workaround for me is to use a wrapper function around vim.lsp.handlers.hover to remove those sequences manually

local function hover_wrapper(err, request, ctx, config)
  local bufnr, winnr = vim.lsp.handlers.hover(err, request, ctx, config)
  if (bufnr == nil or winnr == nil) then
    return
  end
  local contents = vim.api.nvim_buf_get_lines(bufnr, 0, -1, false)
  contents = vim.tbl_map(
    function(line)
      return string.gsub(line, " ", "")
    end,
    contents
  )
  vim.api.nvim_buf_set_option(bufnr, 'modifiable', true)
  vim.api.nvim_buf_set_lines(bufnr, 0, -1, false, contents)
  vim.api.nvim_buf_set_option(bufnr, 'modifiable', false)
  vim.api.nvim_win_set_height(winnr, #contents)

  return bufnr, winnr
end

vim.lsp.handlers["textDocument/hover"] = hover_wrapper

Adding this to init.lua fixes the appearance of the hover for this language server, but there are others that probably have similar issues.

Steps to reproduce using "nvim -u minimal_init.lua"

1. create a new temporary directory cd $(mktemp -d) and create an init.lua file with the following contents

local pattern = 'yaml'
local cmd = { 'yaml-language-server', '--stdio' }
-- Add files/folders here that indicate the root of a project
local root_markers = { '.gitlab-ci.yaml' }
-- Change to table with settings if required
local settings = {
  yaml = {
    schemas = {
      ["https://gitlab.com/gitlab-org/gitlab/-/raw/master/app/assets/javascripts/editor/schema/ci.json"] = ".gitlab-ci.y*l",
    },
    hover = true,
  }
}

vim.api.nvim_create_autocmd('FileType', {
  pattern = pattern,
  callback = function(args)
    local match = vim.fs.find(root_markers, { path = args.file, upward = true })[1]
    local root_dir = match and vim.fn.fnamemodify(match, ':p:h') or nil
    vim.lsp.start({
      name = 'yamlls',
      cmd = cmd,
      root_dir = root_dir,
      settings = settings
    })
  end
})

vim.keymap.set('n', '<space>', vim.lsp.buf.hover, {})

2. create a .gitlab-ci.yaml file with the following contents

stages:
  - hello

3. open it with neovim nvim --noplugin -u init.lua .gitlab-ci.yaml
4. wait a couple of seconds for yaml-language-server to be launched and press <space> with the cursor over the word stages

Expected behavior

I expected neovim to be able to parse html entities in a markdown hover response to properly display them in the floating buffer.

Neovim version (nvim -v)

NVIM v0.8.3 Build type: Release

Language server name/version

yaml-language-server 1.12.0 https://github.com/redhat-developer/yaml-language-server/releases/tag/1.12.0

Operating system/version

Linux Kasama 6.2.7-arch1-1 #1 SMP PREEMPT_DYNAMIC Sat, 18 Mar 2023 01:06:36 +0000 x86_64 GNU/Linux

Log file

No response

[Kasama]

I also volunteer to write a PR for this, but there are a couple of questions I'd like to have answered first.

• Is this something we want to do?
• Do we want to include the luarocks package that already handles these entities?
  • if not, should we include all entities that it handles, or just a subset of common ones? i.e. >, <, "

[clason]

Is that even valid Markdown? Neovim's policy is to strictly implement the specification and not cater to every off-spec quirk that language servers may have "because VS Code handles it fine".

(And the near-term goal is to replace this function with one that uses the tree-sitter Markdown parser.)

[justinmk]

the near-term goal is to replace this function with one that uses the tree-sitter Markdown parser

Yes 👍

Do we want to include the luarocks package that already handles these entities?

No (see above), but a small code change to workaround the issue is fine as a short-term solution, imo.

[Kasama]

Is that even valid Markdown? Neovim's policy is to strictly implement the specification and not cater to every off-spec quirk that language servers may have "because VS Code handles it fine".

Indeed, I don't think this is valid in "normal" markdown even though it works on github-flavored markdown and some others.

I'll try to continue the discussion on the language server side to see why the returned value is like this to begin with.

(And the near-term goal is to replace this function with one that uses the tree-sitter Markdown parser.)

nice.

No (see above), but a small code change to workaround the issue is fine as a short-term solution, imo.

Got it. Would something like this be a good, small, workaround for now?

-- in function M.stylized_markdown in runtime/lua/vim/lsp/util.lua
stripped = vim.tbl_map(
  function(line)
    line = string.gsub(line, '>', '>')
    line = string.gsub(line, '&lt;', '<')
    line = string.gsub(line, '&quot;', '"')
    line = string.gsub(line, '&apos;', "'")
    line = string.gsub(line, '&ensp;', ' ')
    line = string.gsub(line, '&emsp;', ' ')
    line = string.gsub(line, '&amp;', '&')
    return line
  end,
  stripped
)

[clason]

Yes, that looks fine.