Add UTF-8 string manipulation functions to the lua stdlib #14281
Comments
|
we should expose select functions from mbyte.c . In particalar, i would be good for scripts to have direct access to neovim's specific idea of grapheme cluster (which is a bit limited but might get upgraded later). |
|
Nice! In the mean time, we can always clone and use for example https://github.com/Stepets/utf8.lua |
|
Btw, it will be very nice to have access to the functions in |
|
On a related note (assuming this library includes offset calculations etc.), it would be useful to be able to directly compute those for files that are not loaded (without the current |
Also make the implementation via neovim API rather than normal mode. I had hell before I realized the LSP server was speaking characters but nvim_buf_set_text speaks bytes. See neovim/neovim#14281 for the upstream issue that'd probably help. Refs: #57
|
@bfredl What do you think remains here with vim.str_utf_{pos,start,end} now in core? |
|
We have added functions for codepoints. for some usecases, functions for composing chars (i e vims idea of grapheme clusters, the smallest editable unit) might be needed. |
|
Related to the original request (not so much the current plan): https://github.com/uga-rosa/utf8.nvim |
resources: vim.str_utf_pos // utf 8 vim.fn.getline vim.fn.setcursorcharpos vim.fn.cursor vim.str_utfindex vim.str_utfindex // [32, 16] vim.str_byteindex clangd/clangd#3 neovim/neovim#14542 https://github.com/neovim/neovim/pull/16252/files neovim/neovim#14281
resources: vim.str_utf_pos // utf 8 vim.fn.getline vim.fn.setcursorcharpos vim.fn.cursor vim.str_utfindex vim.str_utfindex // [32, 16] vim.str_byteindex clangd/clangd#3 neovim/neovim#14542 https://github.com/neovim/neovim/pull/16252/files neovim/neovim#14281 offsetEncoding is off spec and using it might break in future, see neovim/neovim#17049 (comment) and hrsh7th/nvim-cmp#726 (comment) microsoft/language-server-protocol@f9c85d5
|
Hi! Are there any plans to expose those functions anytime soon? |
|
Are you volunteering to work on it? |
|
Unfortunately, I cannot. I am just asking if there is any update since I do not follow Neovim development updates... Is there anything wrong about it? The lack of UTF-8 support was the reason why this bug was opened: Vonr/align.nvim#15 That's why I decided to ping this issue. For all my use cases, I bundled https://github.com/uga-rosa/utf8.nvim |
|
On second thoughts, is there anything extremely wrong in bundling the code in https://github.com/uga-rosa/utf8.nvim inside Neovim? This can be used as the API is very similar to what we have in Lua 5.3. At a latter stage, we can replace those functions by low-level ones written in C. If this is the case, I think I might help. |
|
No, we will not simply bundle an external plugin. As bfredl explained, we need an API that exposes Neovim's own string handling capabilities. This is the only way of making sure that these functions play nicely with other Neovim APIs. And that requires thought and effort -- as much as other things that are higher priority currently. |
|
Ok, so the answer to my question "Are there any plans to expose those functions anytime soon?" would be just "No, unfortunately, there are not". |
|
I mean, this does not simply happen by itself, despite the suggestive impersonal phrasing. It will happen when someone who personally cares about this topic invests the time and effort to do it -- same as any open source development. |
|
What is your point? I pretty much know how open-source development works. I did not demand anything from anyone. I just asked if anyone has any plans to close this issue. You really do not need to be harsh over one simple question. |
Since we are using Lua 5.1, we do not have any library to manage UTF-8 string. This can be a problem when creating lua scripts that manipulates the buffer with UTF-8 strings.
The text was updated successfully, but these errors were encountered: