[WIP] Lua helpers for plugin development

[KillTheMule]

If we want people to write lua plugins, we should give them some tools to ease development.

A somewhat loose to-do list, just some things I need to remember:

• Add v:lua to eval.txt, reference to if_lua.txt for the individial functions

• Need help:

  • Actually make v:lua work
  • Decision about proper namespacing (My suggestion: vim.helpers for plain lua help function like split, and vim.lua_api.osname for the function osname that gets exported to viml).

to actually make v:lua work

[ci skip]

[justinmk]

Definitely in favor of this, thanks for getting it started.

I would suggest not to think of it as "plugin help", but rather "augmenting the Nvim stdlib". Up to now the stdlib has been defined by :help eval.txt, AKA all of the f_* functions in src/nvim/*.c.

We want to enhance the Nvim stdlib with Lua functions. These functions are implemented in Lua, but available also from VimL.

• These new Nvim stdlib (Lua-implemented) functions might be listed in :help if_lua.txt, but that's mostly a detail: they are conceptually sibling-to :help eval.txt.
• Using Lua functions from VimL should be as "seamless" as possible (TODO/out-of-scope for this PR).
  • One idea is to populate v:lua with wrappers so that VimL arguments/retval can be marshalled to/from Lua easily without the need for luaeval({escaped-string}). Similar to how helpers.funcs works in the tests.
• uname probably doesn't make sense as implemented from test/helpers.lua because it deals with the case where nvim is not available. Nvim already has has('win32|unix|mac').
• Some functions will just wrap existing VimL functions, such as system(), but with "Lua idioms".
  • E.g. v:lua.system() could return multiple return-values so that the caller doesn't need to save v:shell_error in a separate step.
• In other cases like v:lua.glob(), it might make sense to have a completely separate implementation which doesn't suffer from the quirks of VimL's glob(), and uses Lua patterns instead of VimL wildcards.
• These functions need to be designed with the same mindset as a new VimL function: forward-compatibility, documentation, tests. So we can't just dump a bunch of functions or penlight into userspace, because then we have to support it forever.

One weakness of Lua is the lack of canonical "best practices" to draw from. I think https://github.com/leafo/moonscript and https://github.com/leafo/lapis might be good sources for that.

[KillTheMule]

Ok, so I'll try to do this with 1-2 functions I found I needed, and we'll see how this works out. About v:lua: This basically means just sticking call luaeval(require(helpers).function... as a functtion into this table, right? Should this be autogenerated? I'd handcraft it and try to write a test that each such lua function has a viml entry in v:lua...

[KillTheMule]

After a good night's sleep I was able to solve this, but I'm not sure that's the way it should be done. Seems like I can't use eval('v:lua.osname()') but have to resort to eval('lua.osname()'), and I'm not sure why. Is this the way it should be?

[KillTheMule]

With the patient help of bfredl I was able to make it work! Now there's a test to check if all functions in vim.helpers (suggested by bfredl) have their counterpart in lua.

There's snag though: The code isn't loaded automatically. Bfredl suggested putting a file in runtime/plugin to achieve that, which would surely be easy. Otoh, some people care for startup times, and I'm not sure if that's what we want then.

Suggestions?

Next step would be to write documentation (I'd say osname can stay the way it is), and think about where to ask people which functions they'd want in this spot. If anybody wants to suggest functions to add, please do so :)

(e) My confusion above stemmed from the fact that I confused g: and v:. I'll try to find out how to create v:lua, but for now I guess a global lua works.

[bfredl]

Otoh, some people care for startup times, and I'm not sure if that's what we want then.

Startup time difference could be measured :) Otherwise vim.helpers could be imported on first reference using a metatable. But maybe it is not so important: any script could use local helpers = require('helpers') anyway, it matters mostly for quick :lua print(vim.helpers.somefunc) testing

[KillTheMule]

I tried to make lua a VVAR instead of a global one. I can confirm it exists by manually trying, but seems it's not modifiable, although I've not set any flags. The type of it is a dict, but every try to add an entry results in Vim(function):E689: Can only index a List or Dictionary. What am I missing here? Docs are kinda scarce about vvars...

Otoh, all the other vvars seem different in usage. They seem more like local variables, that make only sense in a certain context (like v:foldend only makes sense when you're talking about a particular fold). Are we sure that's what we want? Why aren't we using a global?

[KillTheMule]

I've started bugging people about what they'd like to see in a "lua stdlib". @tjdevries, @nhooyr, @keidax I think you've dabbled in lua code for nvim, care to chime in? Thanks!

[hkupty]

Thanks for calling me in.
There are a few patterns I see myself repeating over and over when using lua.

They mostly drill down to those:

For table management:

• clone
• keys
• values
• get (as a safe alternative to direct referencing)
• get-in (get in nested tables)

For string management:

• split

For functional-ish style:

• curry(this is an applied example)

For debbuging (now this is somewhat biased and optional though):

• inspect(it is a third-party library).

Even though there may be more, specially on string management, I feel those are the bits I need to stop thinking about the plugin and start thinking about the language. It'd be great if some of those could be provided.

Also, I'd like to add that it feels great to use lua to develop plugins for neovim except for when I need to expose lua to VimL.

Somethings like mapping a lua function or exposing some commands can be tricky/messy and sometimes require some indirection to work properly.

I would love if the API could expose functions to define mappings and/or create commands in a way that those could natively access lua functions/tables. Obviously I understand the complexity of those, but asking wouldn't hurt, right?

Sorry if this got too dense. I'm glad to help.

Cheers,
Henry

[bfredl]

I second inspect.lua, it is invaluable when debugging functionaltest code. As it has license inline it could just be vendored as runtime/lua/inspect.lua.

For mappings and commands, we already have API functions to inspect them, it seems logical to also have API functions for adding mappings and commands (this will also benefit init.lua usage, and remote plugins).

To natively support lua references in API would require a bit work, but a quicker workaround could be to have a lua-side wrapper function with the same signature as the API functions, but also accepts lua closures and converts them to the vimL lua commands accessing them from a reference table (as in the linked examples of indirection).

[hkupty]

Taking the indirection out of plugins scope can greatly reduce the cost of writing lua plugins. Even if such mapping is still required, it is acceptable and could be further optimized without requiring to migrate the plugins.

[bfredl]

We definitely want jobstart natively in lua, it will allow proper binary output as lua string instead of list-of-lines representation of binary output (which everyone confuses with line-based output)

[tjdevries]

Seconded what @hkupty has said.

I made some helper functions while working on the LSP branch:

https://github.com/tjdevries/neovim/blob/0d2dd6446c6cf105806b8b4ab11fe8147fbf4c25/runtime/lua/neovim/meta.lua

https://github.com/tjdevries/neovim/blob/0d2dd6446c6cf105806b8b4ab11fe8147fbf4c25/runtime/lua/neovim/util.lua

Not exactly "complete" but should give an idea of a few of the things it would be helpful to have when writing lua plugins.

[tjdevries]

Alternatively, we could also add some native lua without having to do things to the vim api.

function string:split(...) will modify all strings to now have split. As well as function table:copy or things like that. Not sure if people prefer that over having vim.std.string.split(str, ...)

[bfredl]

function string:split(...) will modify all strings to now have split.

Not sure I like monkeypatching builtin metatables: it might then look a piece of code only uses builtin methods, but then one runs it outside of nvim and it unexpectedly fails.

Not sure if people prefer that over having vim.std.string.split(str, ...)

For lua util functions we should also consider the standard module pattern of local split = require('util').split (we can use a more elaborate name if we are afraid of collisions)

[hkupty]

The name is less important than the structure I'd say...

I think the intent is to make sure all those functions were provided by neovim, thus are accessible from the vim.* "namespace".

We could have them as vim.string, vim.table if we agree that adding more levels is irrelevant and actually worse.

I just want to explicitly separate those 4 cases. A flat structure would be:

vim.api - current api
vim.native_api - api with native wrapper
vim.shared - exposed to viml
vim.* - all the helpers

[KillTheMule]

As far as I can see, the current api is useable directly from lua, what do you mean by "native wrapper"? Or am I missing something?

[KillTheMule]

I've now added inspect. License is included in the file, which should be enough. Is it enough to refer to the github site for documentation? Personally, I wouldn't want to blow up the doc file with all the possible options and details, since most of the time it simply "does what you want".

[KillTheMule]

I also added trim and deepcopy. I'd say for a first shot, vim.helpers is done. I'd do some things for vim.lua_api next.

[KillTheMule]

@hkupty has started working on a possible api for map functions, and we'll use his repository to iterate on that a bit. If anyone wants to participate, head over.

[KillTheMule]

After some iteration over at tycho, we settled for a simple design for map, which I've implemented. I've also made it much more complicated by accounting for modes, recursiveness and cmd-ness. Needs tests :)

I've removed the dependency on the extmarks PR by providing namespaces in the lua code, but in a maximally trivial way.

[hkupty]

Just pinging to check if this is still alive. I've been on a rush and couldn't be closer to this change.

[KillTheMule]

It's somewhat on hiatus, since I don't have much time these days. I'll pick it up later, or if somebody else wants to take over, be my guest :)

[KillTheMule]

Superseeded by #9463 and #9517.