feat(pack): add built-in plugin manager vim.pack

[echasnovski]

This PR adds a minimal built-in plugin manager vim.pack. This is a first step towards resolving #20893. A general design is a minimized (496 LOC for core functionality when PR is opened) and reworked 'mini.deps' plugin. There is a first-pass documentation mostly aimed for reviewers and no tests yet (as it is an open question about how to do that reasonably).

Here is a demo of common workflows at the latest PR state:

vim-pack-demo-4.mp4

Demo of initial PR state (for history)

vim-pack-demo.mp4

Top right window comes from 'mini.notify' plugin; that is not part of vim.pack:

Timestamps:

• 00:00 - initial "clean" run (i.e. just like after installing Neovim and adding 'init.lua' that is shown in demo).
• 00:10 - install new plugin (add line and restart).
• 00:28 - updating existing plugins (simulate new upstream changes, run vim.pack.update(), review, confirm).
• 01:10 - change version of existing plugin (update 'init.lua', restart, run vim.pack.update() with { offline = true } to skip downloading new changes, review, confirm).

---

General notes about design decisions:

Design decisions notes

• Plugin spec is designed to be minimal yet versatile with an eye for future automated packspec support, i.e. plugins themselves containing a special 'pkg.json' file which should contain at least the following information:

  • Dependencies: repo urls and versions. If plugin doesn't support packspec, user can register dependencies manually by explicitly adding them to vim.pack.add().
  • Hooks: paths to scripts to be executed before/after install/update. Without them user can register hooks by creating autocommand for dedicated events. Note: custom events are generally better than hooks because they can be used by plugins to tweak install/update behavior of other plugins. Plus smaller vim.pack.Spec.

• Showing "interactive" confirmation buffer might be considered too much, but being able to review updates before approving them is a huge QoL improvement. Right now it also includes in-process LSP server to show structure via vim.lsp.buf.document_symbol() (or with default gO). The whole LSP approach might be an overkill, but:

  • It works nicely with plugin ecosystem around LSP lifting the burden of having to create dedicated conventions for vim.pack. Most useful mappings are already built-in.
  • Can serve as a reference implementation of an in-process LSP server (which is a decided way forward for several built-in things, like custom vim.lsp.completion sources and code actions).
  • It reinforces the "LSP out of the box" approach.

  There are more LSP-based interactive features planned (see next section).

---

Left out from this PR but planned after discussions and executive decisions:

Planned future work

• "Scripting way" to remove installed plugins from disk. I do have plans to support "delete plugin" code action from inside confirmation buffer, but it needs green light. The "scripting way" can be designed in several ways:

  • A vim.pack.remove() - simple wrapper around vim.fn.delete(). Might also trigger dedicated events.
  • A vim.pack.clean() - delete plugins that are not added to current session. Without interactive confirmation, this can become tricky for configs which will lazy load plugins on events/ModeChanged/etc.
  • Do nothing and suggest using vim.fn.delete() directly with (possibly exported) path to a special plugin's directory. This is meant as minimal plugin manager, after all.

  After relevant discussions (this and this), there is a vim.pack.del() present in this PR.

• User commands. Like :Pack add or :PackAdd. The latter is more natural when it comes to ! and easier to implement completion.

• Allowing "local plugins" (as "unmanaged plugins" in 'vim-plug'). This can be skipped entirely in favor of suggesting putting those plugins in custom 'pack/mine/opt' package and use :packadd. But initial Matrix discussions showed interest in having this supported.

  As far as I can tell, they don't quite fit into the current design: they usually don't require automated installation or update. They require manual handling during add (explicit source of 'plugin/' and 'after/plugin/') and book keeping to ignore it during update(). Doable, but requires executive decision and extra effort/code.

• Consider adding vim.pack.config(). It will configure how vim.pack operates. Possible options:

  • job_threads - how many threads can parallel steps (like "install" and "download updates") use. Current and default value - 80% of all available threads.
  • job_timeout - amount of time to wait until force stopping any job. Current and default value - 30 seconds.
  • default_version - default version to use in vim.pack.Spec. It can be used to have "install latest available version" by default with vim.version.range('*') value. Current and default value - nil to infer the default repo branch.

• Consider moving documentation to a dedicated pack.txt. Maybe also move the :help packages section in there.

• Consider re-introducing dedicated highlight groups for 'nvim-pack' confirmation report. Needs discussing the scope of highlighting groups and the best place to define them.

• Consider making vim.pack.add() more flexible. Either via flags/enums on vim.pack.add() level or as part of per-plugin spec. Some common requested use cases:

  • Allow skipping automated installation of not presently installed plugins in favor of doing that manually later.
  • Allow skipping :packadd (even :packadd!) to only ensure that all plugins are installed in favor of manually loading them later. One way to do it is to allow load as function that accepts a single resolved plugin spec and is responsible for loading into current session.
  • Allow skipping installation confirmation.

• More interactive update features:

  • Code lenses/actions "update this plugin", "skip updating this plugin", maybe "delete" this plugin.

• Make progress update user-customizable. Either after vim.ui.progress (progress messages #32537) is a thing or there is a built-in $/progress handler that actually shows progress ([GSoC] Implementation of AI-Oriented LSP feature & LSP Data Caching module #34254, statusline: reimplement default, include LSP progress #28809)..

• Discuss and enforce order of event triggering. Right now due to totally async install/update/checkout there is no guaranteed order of Pack{Install,Update}{Pre,} events across different plugins. This might lead to conflicts when callback of some plugin event relies on features from dependency plugin. As there is no guaranteed order, callback for "main" plugin can trigger before installing/updating "dependency" plugin. The installation order can be enforced by separate vim.pack.add() calls, but update/checkout can not.

  One approach can be to guarantee processing necessary steps in order that plugins are :packadded. This is currently done for vim.pack.del() and its events.

  See more discussions here.

• Lockfile support. Basically, store state/commit per source and prefer it only during initial install over resolving version. Will help with reproducible setups.

  It (or its "private" alternative) can also contain an information about version to be used during update() if the plugin was not (yet) added to current session. In theory, this can also be used to detect if the user has changed plugin's version in an attempt to switch version/branch and automatically do so during startup (without having to do an extra vim.pack.update() call). One thing why this might not be good is that it breaks the separation of "add() is for installing and loading plugins" and "update() is for updating already installed plugins".

• More straightforward plugin freeze/pin/lock. Originally this PR proposed a special value of version ('HEAD' string or false boolean) to be treated as "freeze the installed plugin from updates". It was decided to postpone the decision about the better "freeze" approach until there is a lockfile support. See this comment for details.

• More straightforward version change of already installed plugin. Currently it requires extra step of vim.pack.update() (which needs internet connection) to "sync" plugin on disk with to have proper version. After there is lockfile support, an alternative approach would be to have "add() ensures both presence and correct on disk spec (like source and version)" and "update() is only for downloading and applying changes from source". This simplifies the "change version" step at the cost of not being able to see beforehand and confirm/deny which changes will be applied. They would still be present in the log file for inspection, though. Relevant discussion is here.

• :checkhealth support. It can include suggestions like "the latest tag of this plugin is very far behind default branch; consider setting version = 'master'".

• Packspec support. It is rather big and needs discussions about the degree of support vs complexity.

---

Not planned as vim.pack functionality:

Not planned

• Manage plugins from 'start/' directory. As vim.pack.add() only installs in 'opt/' directory (as it is all that is needed), it seems unnecessary to also manage 'start/' from the same package path. Plus it is known to cause confusion about how to not load its plugins.
• Lazy loading out of the box. This can be done like in 'mini.deps': via now() and later() functions. They safely execute its input function immediately and "soon" while reporting errors only after all those functions are finished. These are general enough to live outside of vim.pack and should be useful to have in vim.func.

[echasnovski]

Left some comments in places that I know worth discussing.

[echasnovski]

What about url schemas, like Nix has with its fetchTree (and, by extension, flakes): "gitlab:some-fella/plug.nvim" for "https://gitlab.com/some-fella/plug.nvim.git"? Or will that be too much for this?

The actual flexibility that is welcomed here will be a topic of discussion. Right now the most established pattern among plugin managers is to only allow "user/plugin" as a stand-in for "https://github.com/user/plugin". If even this kind of "magic" is welcome needs discussion with core team. There is this comment that brings up this and similar issues.

---

I'd also like to take an opportunity and kindly ask for people outside of Neovim team to deeply consider if adding a comment will help the discussion. There will probably be a lot of comments as is and navigating a PR with lots of comments becomes problematic very quickly. Thanks for understanding!

[telemachus]

Can I ask about the choice never to use the start/ directory? More generally, I'm wondering why vim.pack.add always calls :packadd behind the scenes. Unless I'm confused, that makes it impossible to use vim.pack to install something that a user prefers to load only sometimes. (I have several plugins that fall into that category for me.) It seems better to me to use start/ and opt/ as designed, so that items in start/ are always sourced and items in opt/ are left to be manually sourced by the user. But you may have reasons I haven't thought of. Anyhow, I'd be curious to hear your rationale for this part of the design.

[brianhuster]

Can I ask about the choice never to use the start/ directory?

Then if you want to disable plugins, you have to manually move them to opt. That is bad UX.

I remember @justinmk once said the Nvim team generally think pack/*/start is unnecessary

[telemachus]

Can I ask about the choice never to use the start/ directory?

Then if you want to disable plugins, you have to manually move them to opt.

No, that doesn't follow. The plugin manager can move plugins between start/ and opt/ as necessary so that the user does not have to do anything manually. (It shouldn't take much extra code: plugin configurations can have a boolean field for opt that defaults to false.)

That is bad UX.

I remember justinmk once said the Nvim team generally think pack/*/start is unnecessary

I think it's poor UX for neovim's built-in manager to work directly against the underlying vim/neovim package system. It's true that pack/*/start is not absolutely necessary, but the system of start/ versus opt/ makes good sense and works well. Why design a plugin manager that undermines that system?

If the core team is against it, I doubt I will convince them, but here's my final argument. It's better to support both start/ and opt/ because that allows more people to use the built-in plugin manager. People who prefer opt/ for everything can put all their plugins in opt/ and :packadd them all at startup. Other people can put most things in start/ and a few things in opt/ (and :packadd those when they want). But if the plugin manager ignores start/, then only users who prefer everything in opt/ are supported. I think that a built-in tool should support as many users as possible rather than be overly opinionated.

[brianhuster]

The plugin manager can move plugins between start/ and opt/ as necessary so that the user does not have to do anything manually. (It shouldn't take much extra code: plugin configurations can have a boolean field for opt that defaults to false.)

It does make the code more complicated than just

-- pseudo code
if not pack.opt then
    vim.cmd('packadd! ' .. pack.name)
end

Not to say checking if a plugin is in start/ directory and then moving it could be slow.

I think it's poor UX for neovim's built-in manager to work directly against the underlying vim/neovim package system.

It seems you misunderstood what "UX" means.

It's true that pack/*/start is not absolutely necessary, but the system of start/ versus opt/ makes good sense and works well. Why design a plugin manager that undermines that system?

I already answered that, and Chasnovski has also answered that right in the description of this PR.

People who prefer opt/ for everything can put all their plugins in opt/ and :packadd them all at startup. Other people can put most things in start/ and a few things in opt/ (and :packadd those when they want). But if the plugin manager ignores start/, then only users who prefer everything in opt/ are supported.

Why would people care if the plugin manager use start/ or not?

[echasnovski]

Unless I'm confused, that makes it impossible to use vim.pack to install something that a user prefers to load only sometimes.

Yes, it is possible. Call vim.pack.add() with plugin spec only when you want to use the plugin. If plugin is absent it will be automatically installed.
Without plugin manager it is two step: 1) manually put plugin in pack/*/opt; 2) call :packadd when you want to use it.

I think it's poor UX for neovim's built-in manager to work directly against the underlying vim/neovim package system. It's true that pack/*/start is not absolutely necessary, but the system of start/ versus opt/ makes good sense and works well. Why design a plugin manager that undermines that system?

The fact that there is choice to make doesn't mean every choice fits every situation. Nothing here "undermines the system".

People who prefer opt/ for everything can put all their plugins in opt/ and :packadd them all at startup. Other people can put most things in start/ and a few things in opt/ (and :packadd those when they want). ... I think that a built-in tool should support as many users as possible rather than be overly opinionated.

Both "put all in 'opt/' and :packadd all of them" and "put most in 'start/' and some in 'opt/'" is supported with vim.pack.

Putting a plugin in 'start/' is (in aspects that usually matter, not like 'rtp' order, etc.) the same as putting a vim.pack.add() line in an 'init.lua' that is executed on every startup.

Putting a plugin in 'opt/' and load when needed is the same as executing vim.pack.add() only when needed. With future user commands that will also be easier to do from command line.

[echasnovski]

Maybe this is where I am confused. Will vim.pack uninstall packages that are present in opt/ but no longer in a user's start-up files? If not, then I think I see what you're saying. I was worried that plugins not currently in use would be uninstalled—and then need to be reinstalled later.

No, vim.pack doesn't remove those plugins from disk. They are there just waiting to be loaded. They can even be updated when not loaded (just not with default :lua vim.pack.update() at the moment), but that is problematic currently because vim.pack doesn't know about which version to use. That would require some version of lockfile (which is planned).

In fact, there is currently no functionality of removing plugins from disk at all. I do want to have an "interactive way" to do that inside confirmation buffer, but not sure if the whole design is still acceptable. And the "scripting way" is also planned (see "Planned future work" in first comment).

---

This has two side effects that I don't think are desirable: if the plugin is not loaded when vim.pack.update() is called the plugin will not be updated and it will be removed when vim.pack.clean() is run.

The plugin not being updated during vim.pack.update() is addressed couple of paragraphs above (i.e. "it requires some sort of lockfile"). The "will be removed after clean()" is intentional: "clean" means remove all plugins that are not used. And 'mini.deps' allows you to interactively adjust which plugins to not remove (by deleting lines), which can be done in vim.pack also.

[telemachus]

The "will be removed after clean()" is intentional: "clean" means remove all plugins that are not used.

But "are not used" is now ambiguous. Some plugins "are not used (at startup)"—meaning plugins that are wanted but which users will only sometimes call on with a manual vim.pack.add. But it also can mean "are not used (at all)"—meaning plugins that the user wants removed from disk.

I agree with micampe that "a way to separately indicate 'this is a plugin I want installed' and 'this is a plugin I want loaded"' is useful." As he says, one way to do this is with start/ and opt/, but you can also do it other ways.

[echasnovski]

But "are not used" is now ambiguous. Some plugins "are not used (at startup)"—meaning plugins that are wanted but which users will only sometimes call on with a manual vim.pack.add. But it also can mean "are not used (at all)"—meaning plugins that the user wants removed from disk.

If anything, this is a point towards not having automated "cleaning" in favor of manually selected one (which I'd suggest having in vim.pack).

I agree with micampe that "a way to separately indicate 'this is a plugin I want installed' and 'this is a plugin I want loaded"' is useful." As he says, one way to do this is with start/ and opt/, but you can also do it other ways.

Having 'start/' and 'opt/' causes more troubles than brings good. Yes, having an extra load flag in vim.pack.Spec is possible. At the moment I personally lean towards implementing lockfile (to be able to properly update "plugins that are not yet loaded") plus only manual plugin removal (inside confirmation buffer or vim.pack.remove()). Mostly because this both solves actual problems without introducing new fields in spec (which is usually better for usability and maintainability).

---

You (collective) were heard on this topic. Please, let's not continue this discussion here and wait for reviews and (hopefully) merge. After that, these separate subjects can be discussed in a more structured fashion inside separate issues. It is clearly stated as yet work in progress.

[echasnovski]

I realized that original demo didn't showcase the "in-process LSP" part of the PR. So I decided to add 'textDocument/hover' support (as it is pretty straightforward) and here is the result:

vim-pack-demo-lsp.mp4

This is how a clean install looks, but in combination with "pretty LSP" plugins it can be customized without any burden on vim.pack itself.

[echasnovski]

Just to be clear, I'd still like to add at least basic end-to-end tests in initial PR. This will make future changes/discussions much easier. But I'll wait for @justinmk review first.

[justinmk]

Beautiful. Thank you @echasnovski for the careful design choices.

After resolving comments, I'm in favor of merging this without tests, since we plan to iterate on it anyway and tests can be part of that.

[MironPascalCaseFan]

Is there already an option to disable confirmation? I would like to replace lazy.nvim with it while preserving the modular config structure it encourages. This would mean having multiple files with around 40 vim.pack.add calls instead of shoving all plugin installations into a single one. I’m afraid the first init would require me to press confirm 40 times, so there should definitely be an option to disable this invasive behavior (globally)

[echasnovski]

I’m afraid the first init would require me to press confirm 40 times, so there should definitely be an option to disable this invasive behavior (globally)

On one hand, it is a problem only for the first install, which is not a huge deal. On the other hand, I've added this to the follow up work as part of "Consider making vim.pack.add() more flexible".

[clason]

. I just described one of the cases where a confirmation prompt prevents me from using vim.pack as I want.

And I just said "thanks, duly noted, but the final decision is the maintainers'." Input is welcome; arguing is not. (This PR already has over 200 comments, and every single one adds friction.)

[echasnovski]

Pushed the changes based on the recent round of review:

• Several renames:
  • nvimpack filetype and URI prefix is nvim-pack. I do like the idea of using just pack, but it feels a bit too much. Plus there is something in C# that uses pack://application/... URI scheme (not sure how much this matters).
  • source in plugin spec is now src.
  • bang in vim.pack.add() is now a more descriptive load. It is boolean for now, but might be extended to be function to take care of plugin loading in the future.
  • added in vim.pack.get() output is now active. Relevant discussion starts here.
• Instead of two dedicated events for each action, there is now only PackChangedPre and PackChanged with populated data.kind ("install", "update", "delete"). Relevant discussion starts here.
• There is no dedicated highlight groups for confirmation report. It uses directly already present built-in groups, mostly DiagnosticXxx plus Added / Removed. Titles also use diagnostic groups for consistency and based on default highlight groups for headings (h1/h2/h3), success, attention #32646. Relevant discussion starts here.

Here is the demo of the current state:

vim-pack-demo-4.mp4

[justinmk]

It is bold to state it is not a huge deal to press confirm 40 times!!!

This is Phase 1. We will think about next steps after this. Please, leave the off-topic remarks out of this PR.

[echasnovski]

Added basic mention of external plugins and vim.pack.add() to the 'example_init.lua'. After vim.pack API matures, maybe it can also have basic description of common workflows, but it is a bit early right now.

[justinmk]

Discuss followup work and feedback in the sub-issues of #34763