Pivot to "federated package spec" (ecosystem-agnostic dependency spec)

[justinmk]

(Note: all feedback is captured in this description + the first comment.)

---

To add some momentum to this effort, I suggest:

1. Supporting all Lua-supporting versions of Vim/Nvim all "assets" or "artifacts" of any kind in the spec should be a requirement.
   • No need for Vim/Nvim to have a different package spec, plus this adds legitimacy and weight to the idea.
   • No need for this spec to be specific to vim or text editors. What's needed is simply a way to define artifacts that depend on other artifacts.
2. Shrink the scope by at least 2x.
   • Only support one format, Lua (because it allows comments). Vim must be compiled with +lua or must have lua available on $PATH.
   • Only support one format, JSON. Because:
     • ubiquitous
     • "machine readable" (sandboxed by design): can recursively download an entire dependency tree before executing any code, including hooks. Aggregators such as https://neovimcraft.com/ can consume it.
     • Turing-complete formats invite endless special-case features (nvim-lspconfig is a living example).
   • Remove any fields that are not immediately, obviously needed.
     • Remove any fields that are provided by git. Git is a client requirement.
     • But keep anything that is needed to ensure forward-compatible enhancements.
3. (optional) use identical field names from npm's package.json, where possible (assuming this reduces confusion rather than increasing it)
4. Strict "no side-effects" requirement: pkg.json should be totally sandboxed (evaluating it must have no side-effects, only input and output). (Does #7 address this?)
5. Transfer this repo to https://github.com/neovim/ ?

Client requirements

• git (packages can live at any git URL)
• json parser

Package server requirements

• the package URL must be a git repo

Why NPM package.json?

NPM is special because it's ubiquitous--and lots of discussion hasn't yielded a strong case for a novel format. Using things that are ubiquitous means you gain their tooling, docs, validators (and possibly even... infrastructure).

node and NPM aren't perfect, but that doesn't matter. Choosing to be a subset of that ecosystem provides optionality: it's almost entirely upside with limited downside. We could also choose PHP's package format.
The point is to surf on something massive and immortal.

What about LuaRocks?

I've advocated for LuaRocks as the Nvim plugin manager, but defining a "plugin spec" "federated package spec" also makes sense because:

• There is no "federated" plugin spec (corrections welcome!). LuaRocks is a "centralized" approach.
• LuaRocks + Nvim is starting to see real progress in the form of https://github.com/nvim-neorocks , but thus far has not gained momentum. A decentralized, lowest-common-denominator, "infectious" approach is high-leverage, while work continues on the centralized LuaRocks approach at its own pace.
• There's no central asset registry, just a bunch of URLs.
  • Could have a central list of plugins, but not assets.
• We can do both, at low cost. pkg.json is a fairly "cheap" approach. LuaRocks
• Luarocks itself is a somewhat complex dependency. Nvim removed it from its own build:
  • "luarocks is very slow, its servers unreliable, and its caching mechanism is straight up broken." ref ref

References

• https://json-schema.org/
• lazy.nvim pkg.json impl

[justinmk]

Revised, minimal package spec: pkg.json

{
  "name" : "lspconfig", // OPTIONAL cosmetic name, not used for resolution nor filesystem locations.
  "description" : "Quickstart configurations for the Nvim-lsp client", // OPTIONAL
  "engines": {
      "nvim": "^0.10.0",
      "vim": "^9.1.0"
  },
  "repository": { // REQUIRED
      "type": "git", // reserved for future use
      "url": "https://github.com/neovim/nvim-lspconfig"
  },
  "dependencies" : { // OPTIONAL
    "https://github.com/neovim/neovim" : "0.6.1",
    "https://github.com/lewis6991/gitsigns.nvim" : "0.3"
  },
}

• Dependencies ("leaf nodes") aren't required to have a pkg.json file. Only required for "downstream".
  • pkg.json can declare a dependency on any random artifact fetchable by URL. The upstream dependency doesn't need a pkg.json.
• Version specifiers in dependencies follow the NPM version range spec cargo spec
  • Supported by Nvim vim.version.range().
  • Extensions to npm version spec:
    • "HEAD" means git HEAD. (npm version spec defines "" and "*" as latest stable version.)
  • Do NOT support "Combined ranges".
  • Treat any string of length >=7 and lacking "." as a commit-id.
  • Only support commit-id, tags, and HEAD.
  • Tags must contain a non-alphanumeric char.
• Out of scope:
  • "pack" (creating a package)
  • "publish" is out of scope, because pkg.json is decentralized. Publishing a package means pushing it to a git repo with a top-level pkg.json.
  • "uninstall" https://docs.npmjs.com/cli/v9/using-npm/scripts#a-note-on-a-lack-of-npm-uninstall-scripts

Changes

• renamed packspec.json to pkg.json deps.json (to hint that it's basically a subset of NPM's package.json)
• removed "version" : "0.1.2", because package version is provided by the .git repo info
• removed external_dependencies
• removed specification_version. The lack of a "spec version" field means the spec version is 1.0.0. If breaking changes are ever needed then we could introduce a "spec version" field.
• renamed "source" : "git:…", to repository.url
• renamed package to name (to align with NPM)
• changed the shape of description from object to string (to align with NPM)
• changed dependencies shape to align with NPM. Except the keys are URLs.
  • Leaves the door open for non-URL keys in the future.

Closed questions

• Hooks and "build-time" tasks are defined in scripts (lifecycle)
  • Each scripts item is a path to a Lua file on the Nvim 'runtimepath'.
    • Example "scripts.prepare": "lua/my_statusline/prepare.lua"
  • Scripts are run from the root of the package folder, regardless of what the current working directory is.
  • Predefined script names and lifecycle order:
    • These all run after fetching and writing the package contents to the engine-defined package path, in order.
    • preinstall
    • install
    • postinstall
• Top-level application-defined "metadata" field (client, user, metadata, ...?) for use by clients (package managers)?
  • pkg.json allows arbitrary application-defined fields, as package.json does.
• "Ecosystem-agnostic" means that https://luarocks.org packages can't be consumed?
  • If Nvim plugins can successfully use luarocks then pkg.json is redundant. pkg.json is only useful for ecosystems that don't have centralized package management.
• Are git submodules/subtrees a viable solution for git-only dependency trees?
  • https://stackoverflow.com/a/61961021/152142
  • pro: avoids another package/deps format
  • con:
    • not easy for package authors to implement (run git commands instead of editing a json file)
    • no engines field: how will aggregators build a package list?
    • no support for non-git blobs
• Does the lack of a version field mean that a manifest file always tracks HEAD of the git repo?
  • The dependents declare what version they need, which must be available as a git tag in the dependency. Thus no need for pkg.json to repeat that information. The reason that package.json and other package formats need a version field is because they don't require a .git repo to be present.
• Should consumers of dependencies need to control how a dependency is resolved?
  • repository.type is available for future use if we want to deal with that.
• It'd be nice if the spec enforces globally unique names... Then dependencies could look like { "dependencies": { "plenary.nvim": "1.0.0" } }
  • Requiring URIs achieves that, without a central registry.
• Should name be removed? Because repository.url already defines the "name" (which can be prettified in UIs).
  • Defined name as OPTIONAL and strictly cosmetic (not used for programmatic decisions or filesystem paths).
• package.json has an engines field that declares what software can run the package. Example:

  "engines": {
      "vscode": "^1.71.0"
  },
  

• How to deal with dependencies moving to a new host? Should pkg.json support "fallback" URLs?
  • The downstream must update its URLs.

Open questions

• via @folke: most important to ideally be in the spec:
  • ✅ dependencies
  • ❓ metadata probably makes sense, NPM itself allows arbitrary fields in package.json
  • ❓ build
  • ❓whether the plugin needs/supports setup()
  • ❓main module for the plugin (lazy currently figures that out automatically, but would be better to have this part of the spec)
• Can pkg.json be a strict subset of NPM package.json ? The ability to validate it with https://www.npmjs.com/package/read-package-json is attractive...
• Non-git dependencies ("blobs"): require version specifier to be object (instead of string):
  • "https://www.leonerd.org.uk/code/libvterm/libvterm-0.3.2.tar.gz": { "type": "tar+gzip", "version": "…" }
  • How can a package manager know the blob has been updated if there's no git info? (Answer: undefined.)
• Naming conflict: what happens if https://github.com/.../foo and https://sr.ht/.../foo are in the dependency tree?

   .local/share/nvim/site/pack/github.com/start/
   .local/share/nvim/site/pack/sr.ht/start/
  

Strategy

• specify packspec (above)
• specify ecosystem-agnostic client behavior (report conflicts, fetch things into pack/ dir, update existing dir, ...)
• specify what is undefined (i.e. owned by the per-ecosystem "engine", for example vim/nvim packages are fetched into 'packpath')

[Sanix-Darker]

removed "version" : "0.1.2", because package version is provided by the .git repo info

based on tag/release ?

should name be removed? because repository.url already defines the "name" (which can be prettified in UIs)

I think the name attribute can stay IMO

[famiu]

Should we perhaps have a way of describing why a dependency exists? What about optional dependencies? Should that be within scope for this package spec, and if so, how would it work?

[justinmk]

Should we perhaps have a way of describing why a dependency exists?

You mean like pseudo "comments"? No, that doesn't sound like something needed in a minimal "P0" approach.

What about optional dependencies?

Why is that needed in the minimal, initial spec?

[famiu]

Should we perhaps have a way of describing why a dependency exists?

You mean like pseudo "comments"? No, that doesn't sound like something needed in a minimal "P0" approach.

Makes sense.

What about optional dependencies?

Why is that needed in the minimal, initial spec?

I don't think it is "needed", I was just asking if it should be included or not, just in case the thought of optional deps may have slipped your mind.

[williamboman]

Cool! I'll add some of my thoughts, hope you don't mind;

• removed "version" : "0.1.2", because package version is provided by the .git repo info

Does this mean that a manifest file always tracks HEAD of the git repo? Given a manifest file as input, how would one know which version of the package it describes? I think the version number in the manifest file is very much needed, and should be the canonical version identifier. I also feel like the absence of the version number would violate the 4th principle?

• renamed specification_version to specversion

I think excluding this field would help with overall ergonomics. If the spec starts out small with only fundamental requirements and additions are carefully considered I don't see the spec schema changing in such a way it'd require a spec version bump. Bumping the spec version should imo be a last resort. I think a strict non-breaking change policy would make sense. Consumers would simply check the presence of a field to enable that "feature".

• changed dependencies shape to align with NPM. Except the keys are URLs.

  • Leaves the door open for non-URL keys in the future.

Should consumers of dependencies need to control how a dependency is resolved? I think it'd be nice if the spec enforces globally unique names instead which central registries would have to enforce. The dependency schema could then simply be:

{
  "dependencies": {
    "plenary.nvim": "1.0.0"
  }
}

As for describing version ranges, it looks like the suggested syntax employs npm's syntax, which I think is entirely proprietary to npm. I think cargo's approach is a bit simpler, where:

1.2.3  :=  >=1.2.3, <2.0.0
1.2    :=  >=1.2.0, <2.0.0
1      :=  >=1.0.0, <2.0.0
0.2.3  :=  >=0.2.3, <0.3.0
0.2    :=  >=0.2.0, <0.3.0
0.0.3  :=  >=0.0.3, <0.0.4
0.0    :=  >=0.0.0, <0.1.0
0      :=  >=0.0.0, <1.0.0

I feel like this alone is enough? In the future it could be extended with the additional modifiers ~, ^ and * (which have different meaning than in npm).

Dependencies aren't required to have a packspec.json file. That's only required for the "leaf nodes".

I think the entire dependency tree should be "packspec-enabled". Having plugin dependencies is in my experience pretty rare (maybe only a contemporary consequence of the lack of a manifest file), and those who do tend to depend on plugins that are intended to be dependent on (e.g. plenary) - it'd be in everyone's interest for them to provide a manifest. Having this requirement could also act as a motivator for authors to include one. It's also a hygiene and stability factor as it'd hinder "nonserious" plugins from making their way into the package ecosystem - it's a good proxy for signaling stability.

Finally I think it'd also be good to explicitly define which fields are required. I don't see it being mentioned here but I feel like there's an implication that some of them currently are?

[justinmk]

Great feedback!

removed "version" : "0.1.2",

Does this mean that a manifest file always tracks HEAD of the git repo?

(Added to "Closed questions") The dependents declare what version they need, which must be available as a git tag in the dependency. Thus there is no need for packspec.json to repeat that information. The reason that package.json and other package formats need a version field is because they don't require a .git repo to be present.

excluding specversion field would help with overall ergonomics. ... Bumping the spec version should be a last resort.

Agreed, updated spec. The absence of specversion means "spec version 1.0".

Should consumers of dependencies need to control how a dependency is resolved?

(Added to "Closed questions") repository.type is available for future use if we want to deal with that.

it'd be nice if the spec enforces globally unique names

(Added to "Closed questions") Requiring URIs achieves that.

globally unique names ... which central registries would have to enforce. The dependency schema could then simply be: { "dependencies": { "plenary.nvim": "1.0.0"

(Added to "Closed questions") That looks nice but it makes the protocol more complicated and less "distributed".

[justinmk]

As for describing version ranges, it looks like the suggested syntax employs npm's syntax, which I think is entirely proprietary to npm. I think cargo's approach is a bit simpler

Good idea, updated spec.

I think the entire dependency tree should be "packspec-enabled". Having plugin dependencies is in my experience pretty rare

• Why shouldn't I be able to depend on random artifacts available at a git URL? E.g. I have a key bindings plugin that depends on fugitive, I don't care if fugitive doesn't have a packspec.json.
• This spec extends to anything available from a URL, not just vim plugins.

It's also a hygiene and stability factor as it'd hinder "nonserious" plugins from making their way into the package ecosystem - it's a good proxy for signaling stability.

That just isn't a goal (and it's the "RMS" approach to software 😏: cathedral instead of crazy, infectious bazaar) . The original scope was to be able to declare and resolve dependencies. Not anything else.

[titaniumtraveler]

optional dependencies

Why do people keep mentioning this? What in the world is an "optional" dependency? Either it's a dependency or it isn't.

I mean lots of package managers support optional dependencies.
In my experience it's often a way to make integrations with other
packages more discoverable.

Like it works without the dependency, but has additional features if the
dependency is added.

[justinmk]

Via https://twitter.com/oilsforunix/status/1680957458431213569 :

I have been lightly working on almost exactly this, many notes on https://oilshell.zulipchat.com (please join). Working name is "Silo" for dumb artifacts; "medo"/meadow for git-versioned trees. It's a "meta" package manager because it invokes containerized apt, pip, etc.

[titaniumtraveler]

❓ metadata probably makes sense, NPM itself allows arbitrary fields in package.json

I think having a metadata field reserved for client defined data is a pretty good idea. Especially when considering that this spec is likely to change in the future and breaking old packages because they used fields that collide with fields that were added in new versions would be bad.

Also validation and deserialization is much easier if you know what to expect.

[llllvvuu]

whether the plugin needs/supports setup()

VSCode handles this in the package.json. actually we could rip off quite a few of their contribution points although the rest of them have less obvious value.

configuration and configurationDefaults maps most cleanly to vim.g instead of setup(). I guess setup() could be a bool on top of that.

Contribution points can be used for anything lua_ls annotations are currently used for (completions, hover, diagnostics, vimdoc) plus:

• configuration UI (more generally, UI-driven plugin experience like VSCode's extensions tab)
• enhancement of web view on neovimcraft/dotfyle
• document vim.g
• detecting multiple plugins use the same vim.g option name
• behavioral hack to encourage people to document their options
• probably more things I missed

Technically lua_ls annotations could also be contorted to do some of the above (not sure about vim.g or vimscript support) although the annotated setup() "standard" is too tacit I think. But if contribution points were added to the spec then I imagine someone would make a generator. Similar to how you can go schema-first or code-first in GraphQL and OpenAPI. (But schema-first is the way 😤)