Standardization of plugin file-structure layout

[andreyorst]

Hello. I'm developing a plugin manager for Kakoune (for personal use, but if anyone interested, you're welcome). Since there's no currently de facto standard of Kakoune plugin project layout, I'd like to propose Vim-like one (with some exceptions, because of structure), because it works really well.

Vim's plugin structure

Owerall structure of most plugins can be described like this:

.
├── after
│   └── plugin
├── autoload
├── doc
├── ftplugin
├── indent
├── plugin
└── syntax

Not all folders are necessary, it is highly dependent on the plugin's task. However, the possibilities that come with abilities off putting different modules of the plugin to the supported methods of loading via different directories are huge.

In Vim plugin we have essential plugin folder, which stores all the plugin scripts (I'm dropping autoload part, because there's no equivalent in Kakoune for it, which is sort of a shame, but whatever), doc folder, which contains documentation, and sometimes ftplugin folder, which acts the same way as plugin except it allows loading plugin only for a certain filetype (which again isn't implemented in Kakoune, AFAIK).

This sturcture allows easy unified mechanics for all plugins, to be loaded, loaded with specific filetype, parts of plugin can be autoloaded only when really needed, which boosts overall load time, adding syntax as a part of plugins functions (like Rust plugin adds command interface to cargo through Vim, plus adds advanced syntax highlighting on top of Vim's one), indentation settings for this filetipe, and more. All of this leads to ability of developing really great and advanced plugins, that can turn Vim into full featured IDE.

Documentation

One of the most powerful Vim feature is help system, and one of the most powerful feature of vim's help system is that it can be extended easily. As a plugin creator responsible of writing a comprehensive documentation, editor is responsible of making it easy to refer to that documentation.

Plugins tend grow, and it is natural for them to have a documentation. For example, look at the documentation of LanguageClient-Neovim or UltiSnips. It's kinda huge. And it is meaningless to store all that in the README.md, since this file should contain only essential part about a plugin (although some plugin maintainers do that).

The ability to extend Kakoune's doc base with plugin, or user docs will make overall usage and configuration process lot easier already. This can go alongside with already existing ticket #1991

Proposed plugin layout

Currently, most popular plugins have this layout:

.
├── rc
│   └── plugin.kak
...
├── LICENSE
└── README.asciidoc

There's no separation to documentation, syntax, indentation rules right now.

I'm proposing layout similar to Vim's plugins, but simplified a bit:

.
├── rc
│   ├── doc
│   │   └── plugin.asciidoc
│   ├── indent
│   │   └── indent.kak
│   ├── plugin
│   │   └── plugin.kak
│   └── syntax
│       └── syntax.kak
...
├── LICENSE
└── README.asciidoc

With rc folder being a root of meaningful to kakoune part of a plugin, plugin managers will have easy mechanisms of detecting various parts of plugin, and using dedicated mechanism of loading those files. Kakoune's goal is to support this kind of separation, which is not an issue as I can see right now.
The different method of working currently is needed only for documentation and colorschemes, as former must be located at /installation/path/kak/doc, and latter at the in the $XDG_CONFIG_HOME/kak/colors. The simple solution of loading user or plugin docs will be extension of current doc command making it search docs at user's config folder in a doc folder

[andreyorst]

I was experimenting with command plug-doc which acts essentially the same as Kakoune's built in :doc command, just because there's no way of doing this with native features. And it turns out that this requires almost identical functionality in plugin manager, which is unnecessary, because it is duplicated code, and only path will differ in the end.

I also would like a mechanism of loading colorschemes as arbitrary files, but not enabling them (or a way to distinguish colorscheme from script file in interactive way to store it inside needed folder. This will make possible installation of several colorschemes with single plugin, similar to how base16-vim does it.

[Screwtapello]

I understand that Vim benefits from keeping indent and syntax config separate from general plugins, since it will selectively execute those scripts depending on the active configuration (for example, I believe indent scripts aren't read if you don't have autoindent enabled).

Kakoune doesn't really need to keep those things separate, because it always reads everything anyway, and for the few plugins I use, I'm perfectly happy with adding them as git submodules with no extra ceremony. For colorschemes it's the same deal: Kakoune recursively scans kak/colors for *.kak files, so git submodules are adequate there, too. It would be difficult to install a plugin that provided a colorscheme and a script, but I don't think there's going to be many of those.

On the other hand, I'd love it if the :doc command could pull up documentation from plugins (and colorschemes?). Especially if it could convert raw path-names into friendly human names (my-plugin/doc/tutorial.asciidoc → my-plugin-tutorial, my-plugin/README.asciidoc → my-plugin), but even just paths relative to the autoload directory would work (:doc my-plugin/doc/tutorial.asciidoc).

[andreyorst]

The main point of separation different kind of scripts is to make it easier to automate the process. Right now my plugin manager downloads git repo to certain path, and sources every .kak script from it. Without explicitly putting script to color folder in plugin repo, I can't deduce if the script is actually a colorscheme and, instead of sourcing it, place a symbolic link of it to colors folder in config path automatically. Adding separate colors, rc, doc folders will make it easy to write fast and consistent plugin manager that will do everything automatically without explicitly deducing what script is and how to treat it.

[alexherbo2]

How about:

document (name) (content)

Example 1

document Awesome %{
  Awesome
  ‾‾‾‾‾‾‾
  This is the first line of Awesome documentation.
}

Example 2

.
├── doc
│  └── awesome.asciidoc
├── rc
│  └── awesome.kak
└── README.asciidoc

document awesome.asciidoc %sh(cat $(dirname $0)/../doc/awesome.asciidoc)

[andreyorst]

Well this can work. But I don't like the part with cat, it is better to use symbolic links, so documentation could be updated without messing with cat again

[Screwtapello]

I think the idea of @alexherbo2's example is that the documentation could be included inline in the plugin, or read from a file when Kakoune starts up (by shelling out to cat) and stored in memory, presumably in some str-to-str-map option. Once cat is given the correct relative path from the plugin to the docs, it should work every time.

[alexherbo2]

@Screwtapello I would like the same for color-schemes.

[andreyorst]

Actually, naming documentation with the same name as plugin may be enough to deduce that this is documentation, and not a readme (Comparing every file's name to all sort's of ReAdmE is generally a bad idea for me). doc folder is just a more explicit way:

awesome.kak (repo)
├── doc
│  └── awesome.asciidoc
├── rc
│  └── awesome.kak
└── README.asciidoc

Or for lazy people small scripts:

awesome.kak (repo)
├── awesome.asciidoc
├── awesome.kak
└── README.asciidoc

[mawww]

Yeah, I think the plugins managers should basically link *.kak rc/*.kak into $kak_config/autoload, and *.asciidoc doc/*.asciidoc into $kak_config/doc, (possibly ignoring README.*). This would be simple, easily predictible, and hopefully good enough for 99% of plugins.

[andreyorst]

@mawww no. The point of plugin manager is not only installation of plugins, which involves downloading and linking, but also loading them, and maintaining them as well. Why should plugin manager use it's own method of loading a plugin instead of relying on symbolic link to plugin's script file? That's easy question that can be answered by a set of simple examples. With my plugin manager, you can download, update, and use any plugin if such entry exists in your configuration:

plug "git_user/repo_name"

This interactive command essentially does two main things: It ensures that plugin exists, at installation path, and loads it if it is installed. Given that we could assume that we can use a symbolic link, but actually we can't. Here's why:

Imagine, that we added these two lines to our configuration file:

plug "delapouite/kakoune-text-objects"
plug "occivink/kakoune-vertical-selection"

After resourcing of configuration file, or restarting Kakoune, plug.kak knows about these plugins, and knows that they are not installed, because it tried to load them and failed. We can fix this with plug-install command. It will tell plug.kak to look for all plugins that were mentioned, but are not installed, and download each. After downloading a plugin it is not sourced automatically, but this can be forced either by resourcing config/restarting Kakoune, or by invoking plug command manually.

So, user had installed two plugins, and was able to load them without restarting whole session. You can even install plugin's without adding plug "..." entry to your configuration file, but by invoking it from commandline instead. So if installation process was using a symbolic link it will become less flexable, and the following situation would be impossible:

So user didn't liked that some plugin is loaded for every filetype, or want to disable plugin temporarely. If symbolic link was used, user need to go to autoload folder, find that link, and delete it. And this isn't possible to do for choosing to load or not to load plugin for different filetypes. Because plugin may be a general purpose tool, but meaningless for user in certain workflows.

With plug.kak user can do these:

plug "delapouite/kakoune-text-objects"
# plug "occivink/kakoune-vertical-selection"

By simply commenting plugin in kakrc, user prevents plugin from loading. It is much more flexible, because user may load it manually later if needed, and, may load it only at certain event, by using hooks:

plug "delapouite/kakoune-text-objects"
hook global WinSetOption filetype=markdown %{
    plug "occivink/kakoune-vertical-selection"
}

Plugin will be loaded only when a markdown file was opened. It will not be loaded for markdown file only though, but this isn't possible to do at all with symbolic link at autoload directory. Because again, plugin may be a general purpose, and not used as a filetype extension, but may be meaningless with certain conditions of user's workflow.

Again, removing plugins is simplified too, because you just need to comment out desired plugin and remove it with plug-clean plugin_name command. It will not be unloaded for current session, because it is impossible to do AFAIK (I would like to ask about ability to unload loaded scripts by the way), but will not be loaded on the next startup.

There's one more opportunity to drop usage of symbolic links, and it is interactive plugin loading:

With plug.kak you can specify which branch or tag you want to use in particular plugin:

plug "andreyorst/fzf.kak" "branch: preview"

That is, before loading a plugin, during the startup of the Kakoune, plug.kak will select needed branch, and load a script file from that branch. By using a symbolic link you need to manually go and use git checkout command, and if the plugin layout changed during checking out, which might be a case for complex plugins, or plugins that used one layout at one version, and upgraded to standardized layout with another version, your symbolic link will be broken. By using plugin manager that knows how to load any type of a plugin, which currently the case, you ensure that every plugin will be loaded properly.

Plugin manager also should be used to create safe configuration file, in first place:

1. plug.kak will ensure that each and every plugin will be loaded only once, so you could install and load plugins without reloading whole session, and if you wish to reload, it will ensure that no plugin will cause any trouble to you.
2. You can use this trick to make sure that custom configurations, that are meaningful only if plugin is installed/loaded, won't become an issue if you decide to not to load a plugin:

plug "TeddyDD/kakoune-edit-or-dir"
evaluate-commands %sh{
    [ -z "${kak_opt_plug_loaded_plugins##*kakoune-edit-or-dir*}" ] || exit
    echo "unalias global e"
    echo "alias global e edit-or-dir"
}

This makes possible to have alias only if plugin is installed, so if you run fresh installation of kakoune, with your configuration file from your dotfiles repo, Kkoune will still be loaded normally even if plugins wasn't installed.

---

Using symbolic links is easy, but a naive way to maintain plugins, and it was already a known bad experience in vim's world. No need to step on the same rake.

[andswitch]

I like the simple solution as well. On the other hand, I've used Vim's pathogen plugin manager and I found it delightful from an end-user perspective.

Setting up pathogen is a matter of putting pathogen.vim in autoload and adding a command to .vimrc.

After setting up pathogen, installing a plugin is a matter of running cd ~/.vim/bundle && git clone <plugin> and you're done in many cases. Pathogen finds and autoloads all the plugins in bundle/. As a consequence, updating a plugin is a matter of cd ~/.vim/bundle/<plugin> && git pull and restarting vim.

https://github.com/tpope/vim-pathogen/blob/master/autoload/pathogen.vim

[andreyorst]

Pathogen is good old tool useful for loading plugins, but it is not plugin manager. It's a runtimepath manager, which is a kind of different thing. With Kakoune you don't even need pathogen, because you can clone plugins to autoload directory and It will work just in the same way, because it loads every .kak file from autoload and from directories inside it recursively.

My approach is more like vundle.vim, or vim-plug, both are step forward from what pathogen afforded.

[TeddyDD]

I think standard plugin structure should resemble structure of ~/.config/kak directory

foo #repo
    | README.md
    | autoloads/
        | foo.kak
        | foo-extra.kak # additional features, user can remove this at will
        | foo/
            | foo-something.kak # loaded in runtime by foo.kak
    | colors
        | special-foo-colorscheme.kak
    | docs # same as autoload, but for documentation, issue #2466
         | Foo.asciidoc
    | bin/ # not loaded or interpreted by kak itself. might be conventional way to ship scripts
         | foo.sh

User can basically copy and paste it into .config/kak/ (excluding README/License etc) or symlink everything or they can use plugin managers like one created by @andreyorst or mine. Plugin manager can do all smart tricks with loading in runtime, reloading etc. Autolodad name doesn't mean plugin manager has to use this mechanism to load plugins. Names of files are convention, not enforced in any way.

I think it's simple, and consistent with way Kakoune works right now.

[alexherbo2]

@negativedensity I like @tpope’s approach as well. I’ve a similar, at the exception it’s declarative – in the kakrc – and it doesn’t make use of the autoload/.

https://github.com/alexherbo2/configuration/blob/master/config/kak/kakrc
https://github.com/alexherbo2/pathogen.kak
https://github.com/alexherbo2/git-hub

[andreyorst]

I like @tpope’s approach as well. I've..

How to accidentally mention a top notch viml expert in another editor's tracker.
(twice)

[alexherbo2]

It’s not accidental, nor for a purpose, but I like to mention the persons I’m talking about.

[mawww]

@andreyorst Symlinking in autoload or just sourcing them is fine by me, regarding doc/ the plugin manager could create a doc/plugin-manager/ dir and create symlinks in there dynamically (clear the content of the directory on startup, add symlinks to active plugin docs on plugin load).

What I was mostly pointing to is that plugins manager could just source *.kak rc/*.kak. Which might not be enough if we want "modules" support with dependencies (the "require"/"provides" set of commands that has been discussed a few times).

That said, if it is really preferred, I am not that strongly opposed to an option storing the paths in which to look for docs.

[andreyorst]

So I've developed a plugin, called powerline.kak that utilizes the style where sub-modules of the plugin are stored in separate folder. I've did it more or less just to try this paradigm where modules are located separately and turns out that, this really simplifies the code, and makes scripts small and consistent. The very same issue with big and bloated script exists for my another plugin andreyorst/fzf.kak#22, but there were no way how to do it properly back then.

This way of writing plugins also allows other developers to write their modules for my plugin, which are not located directly inside mine plugin, but can use the same interface which plugin provides with variables.

Here's how it is structured:
Main script, located at plugin/rc/, defines all variables that are used by sub-modules, and sub-modules, located in plugin/rc/modules/, are expecting those variables to exist. Plugin itself highly depend on those sub-modules and can't work without them, but can work with only one of them if user wants to.
There are also another set of modules, located at plugin/rc/themes/, which act differently and doesn't needed by plugin to work. So user may skip sourcing those files, and nothing would break inside plugin, but those files are still rely on variables defined by main script.

Full layout looks like so:

powerline.kak
├── LICENSE
├── rc
│   ├── modules
│   │   ├── bufname.kak
│   │   ├── client.kak
│   │   ├── filetype.kak
│   │   ├── git.kak
│   │   ├── line_column.kak
│   │   ├── mode_info.kak
│   │   ├── position.kak
│   │   └── session.kak
│   ├── powerline.kak
│   └── themes
│       ├── base16-gruvbox.kak
│       ├── base16.kak
│       ├── default.kak
│       ├── desertex.kak
│       ├── github.kak
│       ├── gruvbox.kak
│       ├── red-phoenix.kak
│       ├── reeder.kak
│       ├── solarized-dark.kak
│       ├── solarized-light.kak
│       ├── tomorrow-night.kak
│       └── zenburn.kak
└── README.md

I've encountered this caveat (which I've partly solved with plugin manager):

Plugin must be loaded with proper order of script files, since if modules are loaded before the plugin, they complain about variables are not being defined. I can use this thing in each module to workaround the issue:

try %{
    set-option global module module_name
} catch %{
    declare option str-list modules ''
    set-option global modules module_name
}

But this looks ugly (duplicated code) and I'm not sure what will happen if main script will try to declare the same variable.

However, if Kakoune claims that it will never redefine variable, and will always support the syntax where you declare a variable without value, then this is fine to use:

# we declare option every time, but since it has no value
# it doesn't affect variable with the same name
declare option str-list modules
set-option global modules module_name

Otherwise it is error prone code.

Therefore I have a question how Kakoune loads files, from autoload folder. I've noticed that when I try to execute colorscheme command, it actually tries to use find to find colorscheme files under very specific path, which defaults to %val{config}/colors. GNU Find can display items in whatever order it wants, which was a problem for my plugin manager too, since it uses find, but I've solved it by sorting find output by depth with some awk | sort | cut command after find.

If Kakoune recursively loads autoload folder by sourcing less depth files first, then everything is perfectly fine. However if it isn't so, user just can't place such plugin to autoload folder, and either...

Either plugin file-structure layout should be standardized so such problems can be grouped into easily avoidable situations described by the document of how plugin should be structured to be loaded properly by plugin managers or Kakoune itself, or

Or tweak Kakoune loading mechanics to support loading plugins which store sub-modules in nested folders, or

Or leave this whole task to plugin managers, and not support loading such complex plugins automatically, leaving this task on the user. (and fight those plugins like Vim did a long time, but adopt it after 20 years because it actually good thing to have)