rplugin: cache registrations in 'shada' file

[justinmk]

followup to #4438 (comment)

Also mentioned there: extend :helptags to eliminate :UpdateRemotePlugins.

[bfredl]

Also, I wonder if the logic of finding rplugin files/packages, and later, finding changes to rplugin files to remove the need for :UpdateRemotePlugins, should be delegated to the rplugin hosts. Because for multi-file rplugins it is not obvious which file/s/ (pkg/__init__.py in the case of python) whose changes can change the rplugin registrations. Just give it all the relevant rplugin/LANG/ folders. (this still allows "private" hosts like for the legacy python provider, though we will handle that one differently soon)

[tarruda]

Also, I wonder if the logic of finding rplugin files/packages, and later, finding changes to rplugin files to remove the need for :UpdateRemotePlugins, should be delegated to the rplugin hosts

The whole point of having :UpdateRemotePlugins is to avoid the overhead of loading all hosts at startup, deferring to only when required. The logic for discovering and detecting changes would be better if delegated to plugin hosts, but since they would need to be loaded first, this becomes a bit of chicken/egg problem.

A way to remove the need for :UpdateRemotePlugins without losing the benefits of lazy loading is to have nvim spawn a daemon that keeps running until the user logs out. Such daemon would only be spawned once, and subsequent nvim instances will connect to the existing daemon.

The main job of this daemon would be to keep track of all remote plugins in all rtp directories across all nvim instances, so nvim startup process becomes something like this:

• spawn the rplugin daemon if not done already
• connect to the rplugin daemon socket and asks for a list of remote plugin specifications for all its runtime paths.

Since the daemon(which could be a libuv program using the fs watcher API) is always keeping track of remote plugins files/directories, new nvim instances will always get up to date rplugin information without needing :UpdateRemotePlugins. And it would be as fast as sourcing the manually updated rplugin spec file, since the daemon is only spawned once.

Anothe nice thing about this daemon is that it could notify running nvim instances when a rplugin changes. Imagine this:

• The user is editing a python rplugin being watched by the daemon
• The daemon detects changes, and sends a notification to all connected nvim instances.
• The user is prompted about plugin changes and has the option to reload the affected plugin host(if running).

[justinmk]

A way to remove the need for :UpdateRemotePlugins without losing the benefits of lazy loading is to have nvim spawn a daemon that keeps running until the user logs out.

What about the idea of extending :helptags?

Also, in general, could we avoid a daemon and instead use a p2p strategy using leader election ? This would be more robust than a daemon. I've never heard of any application doing this for local communication, but I've also never heard anyone say why it would be a bad idea: from a high-level, it seems to me that it would avoid the various problems associated with detecting a zombie daemon, restarting the daemon, etc. (solutions for that are platform-specific; implementing "raft" protocol internal to nvim would be cross-platform).

[bfredl]

The question of who decides what is a rplugin file is independent of when the rplugin host needs to start. We could start the host when any file in the rplugin tree is newer than the host's shada timestamp. Proper shada synchronization will ensure that only happens at most once per file update.

[justinmk]

The whole point of having :UpdateRemotePlugins is to avoid the overhead of loading all hosts at startup, deferring to only when required.

We can achieve that by having "remote" plugins define their setup just like any other plugin, in a lua/foo.lua or plugin/foo.lua file. The missing piece is a standard vim.rplugin() interface that makes it easy to (1) start an API client and (2) point the client to the plugin main/library.

• simplify remote plugins, massively #27949

The logic for discovering and detecting changes would be better if delegated to plugin hosts, but since they would need to be loaded first, this becomes a bit of chicken/egg problem.

Why does the plugin host (aka API client) need to detect changes at runtime?

A way to remove the need for :UpdateRemotePlugins without losing the benefits of lazy loading is to have nvim spawn a daemon that keeps running until the user logs out.

#27949 proposes something like that, but we can ignore the "daemon" part initially. Instead we just need an easy way for plugins to start an API client and point it to a module.