doc.kak: Also search through plugins (stdlib and per-user) for docs.

[Screwtapello]

This makes the somewhat-dubious assumption that every plugin will have uniquely-named documentation files, instead of automatically putting every plugin's docs into a namespace. However, plugins already have to deal with flat namespaces for commands, options, filetypes, etc. so one more shouldn't hurt.

Also includes some documentation for a few standard plugins (doc.kak itself, autoreload.kak, lint.kak) so you can see the change in action, and as a proposal for how plugin documentation might be organised.

[caksoylar]

This is a great initiative! I think it would help a lot with new user experience too, since many people look for tools like linting or window management which are already covered by stdlib but they are not very easy to figure out. Documentation for them sometimes exists in the Github wiki but it is not very discoverable. It'd be great if we had docs for the tools that ship with Kakoune like you started doing.

[lenormf]

I'm not sure this is usable because the hardest and most important questions about built-in scripts documentation still stand.

Where is the documentation supposed to be located? I'm not convinced storing it in the same directory as the .kak files is the right way. Should we assume documentation files can be overridden through autoload? This PR makes doc call find three times.

Are we going to manually write all the documentation, making it a massive maintenance burden?

It'd be more productive to have a PR that auto-generates it (using a separate script? a CLI flag that uses the internals?). Ideally there shouldn't be any manual modification made to the resulting flag, we can fit all the information needed into docstrings.

[Screwtapello]

Where is the documentation supposed to be located? I'm not convinced storing it in the same directory as the .kak files is the right way.

When it comes to third-party plugins, either we support documentation beside plugin code, or users need a plugin manager to install all the different parts of the plugin into all the different places they need to go. The fact that Kakoune doesn't need a plugin manager today is something I value, and I'd like to see that continue to be the case.

Storing the documentation for standard-library plugins beside their .kak files is less important, but it does provide an example for third-party plugin authors to follow.

Should we assume documentation files can be overridden through autoload?

Ideally documentation should work the same way as plugins do - if $kak_config/autoload exists, we use it, otherwise we use $kak_runtime/autoload. However, the way doc.kak works right now, that logic would need to be repeated two or three times (for the completion logic, and for the actual command), and the only practical difference is that right now you can't remove stdlib documentation. Since unwanted documentation is not as problematic as unwanted code, I'm happy to leave this for a future PR.

This PR makes doc call find three times.

@krobelus pointed out that wasn't necessary, and I've fixed it.

Are we going to manually write all the documentation, making it a massive maintenance burden?

The nice thing is we don't have to write it all at once - most of the modules have gone this long without any kind of online documentation, they can go a little longer.

The other nice thing is that writing and updating documentation is an awful lot easier than learning C++. I bet there's a lot of people who use Kakoune for writing Python or Rust or JavaScript or Markdown who would like to contribute, who don't have time to learn C++ but could pretty easily take 10 minutes to document a plugin — especially once there's a bunch of examples to follow.

Honestly, if this PR (or something like it) lands, I expect third-party plugins to adopt it really quickly, and standard-library plugins very slowly or not-at-all.

Ideally there shouldn't be any manual modification made to the resulting flag, we can fit all the information needed into docstrings.

Today, people put a bunch of information into docstrings because there's no other way to attach documentation to things. And so, when I added some example documentation to this PR, I copy/pasted docstrings because they were easy to find. However, because they're easy to find, we probably don't actually need to write them down separately. Instead, plugin documentation should talk about how to use all the commands and options together, and leave the command/option docstrings where they are.

I've updated my PR to remove the "commands" and "options" sections from the example documentation, and to focus on documenting how to use the plugin as a whole.

[caksoylar]

I don't think duplicating the docstring in the doc file is a bad idea. That is what happens for built-in commands in :doc commands after all (actually wording is inconsistent between the two also). It is good to have a list of everything on one page so that you can get an overview or can search through it.

[krobelus]

One of the things I miss from Emacs is the ability to quickly go to the implementation of any function.
Since the Kakoune convention is to prefix commands with the name of the rc file, it's probably enough provide a nice way to go to that script.

If we take this PR I'd like to see the rc-file docs have a "go to source" link that does an edit "%val{runtime}/rc/tools/lint.kak" and so on.

the hardest and most important questions about built-in scripts documentation still stand.

Which issues? I assume some of the points raised here
so yeah those are mostly independent.
I think :doc is great when you already roughly know what you are searching for.
When learning Kakoune, the best option right now is to read through all files
in doc/ because it's not very convenient to repeatedly call :doc with the
topics in random order.

I guess that this adding more pages further emphasizes the need for some kind of
overview similar to Vim's :h usr_doc.txt where the most important topics are
listed first. We could add a :doc toc or simply :doc.

Where is the documentation supposed to be located? I'm not convinced storing it in the same directory as the .kak files is the right way.

Existing plugins are usually documented in a README.md. Supporting that might
turn out too complex, unless we encourage them to use AsciiDoc instead.

An idea is to have documentation at the beginning of an rc file, though that
would make it harder to get AsciiDoc syntax highlighting and complicate :doc.

Should we assume documentation files can be overridden through autoload?

Accidentally overriding docs could be a problem in future but I think that's minor.

Are we going to manually write all the documentation, making it a massive maintenance burden?

With the updated version that does not duplicate docstrings this seems better.
We don't need the documentation to be complete. For example, doc.asciidoc seems quite useful already.

It'd be more productive to have a PR that auto-generates it (using a separate script? a CLI flag that uses the internals?). Ideally there shouldn't be any manual modification made to the resulting flag, we can fit all the information needed into docstrings.
I've updated my PR to remove the "commands" and "options" sections from the example documentation, and to focus on documenting how to use the plugin as a whole.

Yeah that makes sense.
My gut feeling is that we don't want to duplicate the docstrings. If we want to have the "commands" and "options" sections we should auto-generate them.

[Screwtapello]

The most important thing I wanted to do with this PR was allow third-party plugins the same access to Kakoune's online documentation that "native" code has, and I think the code changes in this PR achieve that goal — especially now that I've included krobelus' suggestions.

The second thing I wanted to do was to provide a model for third-party plugins to follow, so they can feel consistent with Kakoune's native documentation and with each other. Also, this helps first-time documentation writers make sure they've covered all the basics, leading to better quality documentation overall.

The exact content of the model is not as important as the existence of a model — docs aren't written in stone, after all. Nevertheless, I think it's useful to spend some time thinking about what should be in plugin docs. Using the grand unified theory of documentation as a base, I think the primary plugin docs should mostly be "explanation" content, since "reference" content is better suited to docstrings, and tutorials and how-to guides tend to be whole documents in themselves.

I'm open to the idea of plugin docs including reference content, since it can be handy when your particular instance of Kakoune doesn't have the docstrings loaded (for example, the tmux module isn't loaded when Kakoune runs outside of tmux). On the other hand, if Kakoune's maintainers are worried about maintaining consistency between docstrings and prose documentation, I'm quite happy to give them up.

I think just adding explanation content for Kakoune's standard plugins would be a big improvement. Today, the only documentation for the format the lint plugin expects is in the lintcmd docstring, and that's not really where anyone would expect to find it. Other useful and important things, like how Kakoune's windowing plugins interact or how you're supposed to use :tmux-terminal-vertical aren't documented at all, so adding plugin docs would really help.

There's certainly a lot more that could be done with Kakoune's documentation, including automatically generating AsciiDoc from docstrings, cross-linking between defined items and their documentation, and all the things a fancy documentation system like Sphinx can do. However, I have no idea what that would look like, or how to get there from here, so I don't think it's worthwhile trying to think that far ahead.

We could add a :doc toc or simply :doc.

I wrote the current doc.asciidoc with the idea that bare :doc could be made to automatically run :doc doc and display the documentation for how to use the documentation.

Existing plugins are usually documented in a README.md. Supporting that might
turn out too complex, unless we encourage them to use AsciiDoc instead.

I don't think it's practical to support existing third-party plugin docs as-is, since there's no standards or conventions. It's much more practical to set up a convention and plugins can conform to it if they want the nice integration.

I'd hope the convention would be somethnig like:

my-kakoune-plugin.git
 |
 +- README.md (cover page for people browsing the repo)
 |
 +- my-kakoune-plugin.kak (actual code)
 |
 +- my-kakoune-plugin.asciidoc (online explanation docs)
 |
 +- my-kakoune-plugin-tutorial.asciidoc (online tutorial, linked from main docs)
 |
 +- my-kakoune-plugin-how-to-foo.asciidoc (online how-to docs, linked from main docs)
 |
 +- my-kakoune-plugin-how-to-bar.asciidoc (online how-to docs, linked from main docs)

[Screwtapello]

As a better illustration of what I mean, I've amended lint.kak to move the higher-level "what is required of a lint command" information from the docstring to the prose docs, and made sure each docstring mentions the prose documentation, so people who stumble across an interesting command or option can easily find out more.

[krobelus]

The exact content of the model is not as important as the existence of a model — docs aren't written in stone, after all.

I agree, having some convention for integrated documentation is much better than having none.

The new See ':doc doc' for details. is something that could be added automatically at some point.

[mawww]

Did not look at the implementation yet, but one quick question comes to mind, should we require a doc/ subfolder and only find doc/*.asciidoc in order to avoid matching asciidoc files that might not be intended as Kakoune documentation or should we rely on plugin author to know about doc.kak and avoid asciidoc files that are not intended to be used through :doc altogether.

[Screwtapello]

I looked at the top 12 plugins in https://kakoune.org/plugins.html to check their current documentation conventions. Of those:

• eraserhd's plugins (parinfer-rust and rep) have AsciiDoc documentation, but they use the .adoc extension instead of .asciidoc.
• lenormf's kakoune-extra repo already follows the conventions I happened to pick: some .kak files don't have documentation, but the ones that do have a matching .asciidoc right beside them.
• ul's plugins (kak-lsp and kak-tree) both have README.asciidoc and no other documentation. If a user had both plugins installed, they'd only be able to read one plugin's docs with :doc README.
• A few plugins have documentation in non-AsciiDoc formats (Markdown, plain text).
• Most plugins just have README.md.
  • although my plugins aren't in the top 12, they're certainly in this category

I expect most plugins will be completely unaffected by this change. Even for the plugins that are affected, the :doc completion menu won't be flooded with noise because it'll mostly be a few generic names like README.asciidoc. People using kakoune-extra will get a much larger completion menu, but it'll be specific and relevant rather than noise.

I haven't surveyed plugin authors or anything, but I expect that if this change lands, I expect they'll be happy to rename/split/reformat their docs to match the new conventions (especially since Pandoc can automatically convert Markdown to AsciiDoc).

Perhaps I should start a thread on the forum/subreddit canvassing people's opinions?