Increase possibilities for yazi<->embedding editor integration

[mikavilpas]

Please describe the problem you're trying to solve

In https://github.com/mikavilpas/yazi.nvim, I have ideas for some features that I think might require allowing a deeper level of interaction with editors that want to embed yazi:

• open the currently open neovim splits in yazi tabs
• open yazi in a floating window inside neovim, then select files in yazi, and finally do a batch rename for them using the neovim that is already open
• properly forward events about the neovim terminal being resized.
  • Neovim currently has some bugs in its implementation and a similar plugin for neovim was able to work around them by using remote procedure call (rpc) calls to notify the Ranger file manager manually
• make neovim support showing images that are selected in yazi
  • right now the terminal implementation in neovim does not support images, but neovim itself does not have this limitation. If I have live access to yazi events, this can be done
• explore options related to displaying yazi in a permanent split in neovim, not a floating window that constantly needs to be closed and reopened
  • (not sure what could be done with this approach yet)
• refactor and make the currently available yazi.nvim "open in split/tab/quickfix list" commands more robust
  • right now they are done by shadowing the keymappings that are used in the yazi floating window.
  • the problem with this is that they make an assumption that the user has not changed the default yazi "open with enter" keymapping. While most users probably have not, this approach cannot be used for many things

Would you be willing to contribute this feature?

• Yes, I'll give it a shot

Describe the solution you'd like

Suggestions

Suggestion 1: programmatic access to yazi actions

The api could be disabled by default, and enabled with a command line flag such as yazi --enable-rpc-api, maybe even enabled in the user's configuration.

Currently, yazi can send events via the data distribution service (dds), and the editor integrations can listen to these events. Events can also be sent to yazi via ya pub and ya pub-static.

Even though the communication is now bidirectional, there are some difficulties with this approach:

• it's very specific to yazi. Although the communication protocol is not too complex, it's still a custom protocol
  • it's not a huge thing, but currently due to neovim limitations, I cannot read live events from yazi. The apis seem to only allow reading stdout + stderr as a combined stream while the application (yazi) is running, so right now I can only read the events after yazi has been closed.
  • contrast this to RPC support which is built in in neovim because it's a common protocol
• no yazi actions (such as "open this file") are exposed out of the box, although with plugins it seems to be possible to expose custom actions
  • yazi's api is large with lots of functionality
  • this is the same problem that the language server protocol (lsp) solves - namely, each plugin interested in this must implement their own actions. So there are m actions * n plugins while there could be m actions + n plugins if they were implemented once in yazi

Suggestion 2: allow loading additional plugins from the command line

With deeper integration between yazi and neovim, some features only make sense when both yazi and neovim are running. I love the fact that yazi is a very composable tool and I can run it from the terminal as well as in my editor.

I want myself and other users to have a good experience in both environments. I think the best way to do this would be adding a new flag such as yazi --load-plugin='~/.local/share/nvim/lazy/yazi.nvim/bundled-yazi-plugin/' which would load this additional plugin that provides yazi.nvim specific functionality. When running yazi in the terminal environment, the plugin would not get loaded.

Additional context

Benefits and ideas

Finally, I want to list some ideas and benefits that I think would come from these changes:

• Suggestion 1 (programmatic access to yazi actions)
  • would enable writing integration tests for yazi
    • I think it would make the most sense to test at this level
  • would enable plugin writers to also write integration tests for their plugins
    • it would also be really fun to explore writing helpers for development workflows
      • e.g. when I save my file, automatically restart yazi and do these 5 plugin specific actions
  • would be a great way to "dogfood" yazi's lua api
    • if full lua typing can be added (see the discussion here), the types would also be dogfooded
• Suggestion 2 (allow loading additional plugins from the command line)
  • plugin experiences can be optional and not disturb users who do not want to use them
  • performance is also not affected when running yazi in the terminal

[sxyazi]

but currently due to neovim limitations, I cannot read live events from yazi. The apis seem to only allow reading stdout + stderr as a combined stream while the application (yazi) is running, so right now I can only read the events after yazi has been closed

If Neovim doesn't provide an API for reading stdout from a running process, you can start another Yazi process in the background as a server using yazi --remote-event, and the process with which the user interacts will automatically become the client.

which would load this additional plugin that provides yazi.nvim specific functionality. When running yazi in the terminal environment, the plugin would not get loaded.

I'm not quite sure what you mean by "loading" plugins. Normally, Yazi plugins don't need to be manually loaded; they are lazily loaded automatically after user actions (such as pressing a key). Are you referring to the initialization process of plugins? If so, that would involve loading an init.lua rather than specific plugins.

[mikavilpas]

If Neovim doesn't provide an API for reading stdout from a running process, you can start another Yazi process in the background as a server using yazi --remote-event, and the process with which the user interacts will automatically become the client.

I love this idea. I think I will try it out soon.

I'm not quite sure what you mean by "loading" plugins.

I'm not sure if "plugin" is the correct term to use here. I'm looking for a way to

• get editor related features available when I use yazi inside neovim
• (ideally) have the code be loaded by yazi from a location that I can easily control. The motivation is to not force the user to handle custom installations and keeping them in sync with their neovim, yazi.nvim and yazi versions

[mikavilpas]

If Neovim doesn't provide an API for reading stdout from a running process, you can start another Yazi process in the background as a server using yazi --remote-event, and the process with which the user interacts will automatically become the client.

Tried this approach, but I quickly found out yazi could not be started when it's "hidden", possibly because it cannot draw to a screen.

Opened #1004 which could be used to work around this limitation - the idea is to start ya sub-static in addition to the already running yazi instance, and listen to incoming events without displaying a TUI.

[mikavilpas]

In comparison to the LSP (language server protocol) which uses json RPC (https://microsoft.github.io/language-server-protocol/overviews/lsp/overview/), several types of messages are supported:

1. request (editor -> server -> editor): editor asks the server, and the server responds with data
2. notice (editor -> server): the editor notifies the server but doesn't need a response
3. notice (server -> editor): the server notifies the editor but doesn't need a response

Right now, the ya sub implementation that's ongoing in #1004 should allow for type 3 (notice from server -> editor). I'm curious how you might feel about types 1 and 2.

Is this a direction you could see yazi going?

[sxyazi]

It sounds like the three modes you mentioned are already covered by the current DDS, but it's based on an asynchronous publish-subscribe model rather than a synchronous request-response model.

For example, for the first mode, editor -> server -> editor, you can use the Lua API to call ya.pub_to to send a message to the server, which then forwards the message to the corresponding Yazi instance. The instance then executes the event, and you receive the result through ya.sub_remote.

[mikavilpas]

Message passing with the DDS itself should not be an issue. My issue is that there is no known real time communication channel that I could use from neovim.

It's becoming clearer and clearer to me that at some point yazi.nvim might need its own plugin for various features. I will think about this idea more.

Maybe there's something that could be done on the plugin level to make the communication possible.

[sxyazi]

My issue is that there is no known real time communication channel that I could use from neovim.

I think adding a new ya sub to the CLI (as in the content of this PR #1004) to workaround the current limitations of nvim is the right direction. But since it's closed, I'm wondering if there's something specific that it couldn't cover in this way?

It's becoming clearer and clearer to me that at some point yazi.nvim might need its own plugin for various features.

Could you describe your needs? What are the things that DDS currently can't provide directly and must be done through plugins?

[mikavilpas]

Actually, I'm coming around to #1004 being a good approach. Let's reopen it and see if we can finish it.

Previously I was confused about a few things, but I think this approach can work for me:

• ya sub can be used to receive events from yazi in neovim in real time
• ya pub can be used to send events to yazi from neovim
• I can write a yazi plugin to act as the orchestrator for custom functionality can be used in neovim

I will think about what the best way to include the plugin would be.. For example, when running yazi inside neovim, some keybindings should probably run plugin specific functionality, but when running outside of neovim they should use the default functionality instead.

Have you thought about this kind of an approach?

[39555]

I've also played with this issue because I'm using Nushell, but yazi.nvim depends on the posix shell when running --local-events with redirection to a temp file to read events. Without the redirection, the stdout prints all stuff to the screen and breaks the UI

Starting the second process with yazi --remote-events is not possible. All of termopen jobstart, vim.uv.spawn exit with Device is not configured because yazi tries to draw the UI. --remote-events becomes useless without a --headless flag or something similar.

[sxyazi]

I've also played with this issue because I'm using Nushell, but yazi.nvim depends on the posix shell when running --local-events with redirection to a temp file to read events. Without the redirection, the stdout prints all stuff to the screen and breaks the UI

Starting the second process with yazi --remote-events is not possible. All of termopen jobstart, vim.uv.spawn exit with Device is not configured because yazi tries to draw the UI. --remote-events becomes useless without a --headless flag or something similar.

Please try #1004 to see if it meets your needs.

[sxyazi]

some keybindings should probably run plugin specific functionality

Could you give me an example - which keys need to trigger a specific plugin inside Neovim and when?

[mikavilpas]

Sure, I was thinking good use cases might be to

• add more robust support for bulk renaming files with yazi.nvim. Outside of neovim, bulk renaming works fine, but when using yazi.nvim, the experience is not consistent and a few assumptions have to be made. Some users might see another neovim being opened where yazi was shown (the floating terminal), others might have plugins such as https://github.com/willothy/flatten.nvim which prevent this nesting, etc.
  • a more robust way to do this would to send a custom event to yazi.nvim, and then always execute the same actions. The actions could be, as an example, somehow hide the floating terminal (but don't close it), display the temporary file used for bulk renaming, and when that is saved and closed, resume the yazi session in the floating terminal.
• the same thing could be done for the yazi.nvim default keymappings (open in vsplit, etc) to make them more robust - currently they depend on some yazi keybindings being their defaults 🙂

Other use cases for this type of integration are also possible (when yazi is open in neovim):

• pressing a key would jump to the next file that's open in a visible neovim split
• pressing a key could copy the relative path to the currently selected file in neovim

There's lots of cool ideas that I think this could help with!

[sxyazi]

I need to think more about an elegant way for the plugin to take over these keybindings. Do you have any ideas on this?

[mikavilpas]

Yes, I have a couple of ideas 🙂

1. Idea: what about an approach like To be able to pass a yazi config as an argument like yazi --config "path_to_config" #1148 where I could make neovim open yazi with an additional configuration file? We could merge the two files together in yazi when it's starting, for example.
2. Idea: Plugins could be allowed to create their own keybindings too. That way I could have the plugin only activate when $NVIM is set, indicating the session is happening inside of neovim's embedded terminal.

Neovim is a special case though, because I control both the (future) plugin as well as the invocation of yazi itself. Maybe idea 2 could be more useful to the yazi community. I think it might allow some plugins to provide a more "batteries included" type of experience (reducing the amount of manual configuration).

[daUnknownCoder]

Idea: Plugins could be allowed to create their own keybindings too. That way I could have the plugin only activate when $NVIM is set, indicating the session is happening inside of neovim's embedded terminal.

this feels like an on_attach function i wonder if yazi's lua format supports it?

[sxyazi]

Outside of neovim, bulk renaming works fine, but when using yazi.nvim, the experience is not consistent and a few assumptions have to be made

I tried this yesterday and found that the issue isn't due to nested Neovim instances (as I mentioned here). Instead, it's because yazi.nvim starts Yazi with yazi --local-events > /tmp/xxx, which causes Neovim invoked by Yazi during bulk renaming to be unable to render the TUI to stdout, making it invisible to the user.

Do you think this could be solved using the existing ya sub? This might also address the issue you mentioned earlier about not being able to listen to Yazi events in real-time.

the same thing could be done for the yazi.nvim default keymappings (open in vsplit, etc) to make them more robust - currently they depend on some yazi keybindings being their defaults

Can you elaborate on this? I don't quite understand why splits in Neovim would be related to Yazi's default key bindings. From what I understand, you want to add a new --config or Lua API to override Yazi's default keybindings, but I don't get why it would be related to solving the split issue. Are you mean configuring two different key mappings for yazi.nvim and Yazi at the same time, like this?

require("yazi").setup {
  yazi_ctrl_s = "none"  -- <C-s> in Yazi, do nothing
  yazi_nvim_ctrl_s = "vsplit"  -- <C-s> in Nvim, vsplit
}

add more robust support for bulk renaming files with yazi.nvim

the same thing could be done for the yazi.nvim default keymappings

I'm not too satisfied with these two ideas (I'll explain why later), so I want to see if we can solve these issues using existing solutions first. If they don't work, we can look for other ways.

[daUnknownCoder]

Do you think this could be solved using the existing ya sub:

error: unrecognized subcommand 'sub'

  tip: a similar subcommand exists: 'pub'

Usage: ya <COMMAND>

For more information, try '--help'.

version:

Yazi 0.2.5 (Arch Linux 2024-04-27)

[sxyazi]

add more robust support for bulk renaming files with yazi.nvim

Even though I think it's doable, there are still some concerns:

• Unclear definition: What exactly does it refer to? yazi.toml or keymap.toml? If I got it right, you want to override some of the user's key bindings, so it's probably keymap.toml. Then how do yazi.toml and theme.toml get specified?
• Doesn't match semantics: Usually, starting a program with --config means using a completely different config file, but it seems like you want to modify the existing user configuration.
• Incomplete: If it's just for specifying a toml file path, how do plugins and flavors get specified? If a user adds a new key and binds it to a plugin, how does that plugin exist?
• Makes configuration less predictable:
  • If --config yazi.toml is used, it will cause three rounds of config file merging: --config - user config - preset config.
  • If --config theme.toml and the user has enabled a flavor, it will cause four rounds of merging: --config - flavor - user config - preset config.
• Needs new merge instructions: Yazi's configuration system does not support merging and has not considered it. Instead, it uses the prepend_keymap and append_keymap directives to insert new content into the existing configuration. To implement this, we would have to consider merging the prepend_keymap from the file specified by --config with the user's prepend_keymap, or we would need to add a new prepend_prepend_keymap to insert content into the user's prepend_keymap. Both approaches would complicate the configuration system.

the same thing could be done for the yazi.nvim default keymappings

It's simply not feasible: To achieve the highest performance, Yazi's configurations are designed to be immutable. They are parsed and initialized only once at startup. So dynamically registering new content into the configuration via the Lua API isn't feasible. To achieve this, we would need to change the immutable configuration and event dispatch design, introduce more locks and memory clones, which would increase code complexity and decrease performance.

[sxyazi]

Do you think this could be solved using the existing ya sub:

error: unrecognized subcommand 'sub'

  tip: a similar subcommand exists: 'pub'

Usage: ya <COMMAND>

For more information, try '--help'.

version:

Yazi 0.2.5 (Arch Linux 2024-04-27)

It was added two weeks ago in #1004 and will soon be released as part of Yazi v0.3.

[mikavilpas]

I tried this yesterday and found that the issue isn't due to nested Neovim instances (as I mentioned mikavilpas/yazi.nvim#137 (comment)). Instead, it's because yazi.nvim starts Yazi with yazi --local-events > /tmp/xxx, which causes Neovim invoked by Yazi during bulk renaming to be unable to render the TUI to stdout, making it invisible to the user.

Do you think this could be solved using the existing ya sub? This might also address the issue you mentioned earlier about not being able to listen to Yazi events in real-time.

I think this sounds like a really good idea. I will have to do some experiments but I think this can probably work as long as plugins such as https://github.com/willothy/flatten.nvim are disabled e.g. when $YAZI_ID is set, indicating a nested session.

[mikavilpas]

Unclear definition: What exactly does it refer to? yazi.toml or keymap.toml? If I got it right, you want to override some of the user's key bindings, so it's probably keymap.toml. Then how do yazi.toml and theme.toml get specified?

Yeah I think I meant a merging of additional yazi.nvim settings with the ones that are used in the terminal yazi. I'm not really sure what exactly would be required, but since there are 3 configuration files it would perhaps add quite a bit of complexity.

I think you laid out the challenges in yazi related to this level of configuration quite well; perhaps it's better to abandon this idea. Maybe it would be possible to do overriding or configuration merging in yazi.nvim though.

[mikavilpas]

I'll summarize a bit, although you are already aware of some of these points:

• The DDS approach to listening for yazi events with ya sub has been very successful 👍🏻. Three features are already using it in yazi.nvim:
  • highlighting the currently hovered window/buffer
  • reading the last directory yazi was in when it exited
  • bulk renaming works now!
  • ya sub even works in CI - I'm able to run integration tests with a real neovim and a real yazi
• Modifying the configuration when starting a new yazi from inside neovim is still open/unsolved, but it's not blocking anything right now. I will let the idea develop further in time. 🤔
• An interesting question is sending commands to yazi.
  • I want to add a feature where opening yazi inside neovim would also open new tabs for related files. They can be e.g. files that are currently visible, if there are many - or e.g. 3 of the most recently modified files.
  • I can probably do this by developing a yazi.nvim.yazi plugin (with a better name 😄) and making it listen into custom DDS events that neovim can send using ya pub.
  • For this to work, I need to know some kind of ID for the one specific yazi instance that's open in neovim. Would you have ideas on how this could be done?

[sxyazi]

Awesome! Thanks for your excellent work, it's great to see most issues being resolved through ya sub!

also open new tabs for related files

Do you mean opening three tabs in Yazi or Neovim?

with a better name

nvim.yazi would be a better name, its like a twin to yazi.nvim haha 😄

Would you have ideas on how this could be done?

Yazi generates a unique ID on startup and sets it as the env variable YAZI_ID. Can Neovim read this env variable? If not, maybe we can add a new command line parameter like yazi --id=123456 to specify it.

The plugin would need to ensure it always generates a globally unique ID; using a nanosecond timestamp is a good method.

---

I've been considering introducing new builtin events, like a dds-emit similar to the existing dds-cd. It would essentially be equivalent to ya.manager_emit(), and we could use it like:

ya pub dds-emit --str "plugin nvim --args=foobar"

which would automatically load and run the nvim.yazi plugin.

[mikavilpas]

Do you mean opening three tabs in Yazi or Neovim?

I mean opening tabs in yazi, based on the files that are open in neovim.

nvim.yazi would be a better name, its like a twin to yazi.nvim haha 😄

I love it, it's somehow beautiful - a bit like ☯︎

Yazi generates a unique ID on startup and sets it as the env variable YAZI_ID. Can Neovim read this env variable? If not, maybe we can add a new command line parameter like yazi --id=123456 to specify it.

I think reading the environment from another process is not possible, but I really like passing it as a parameter. I can maybe PR this at some point.

I've been considering introducing new builtin events, like a dds-emit similar to the existing dds-cd. It would essentially be equivalent to ya.manager_emit(), [...]

Ooh this is an amazing idea. If I understand this correctly, it would enable sending any of the built-in commands to yazi through ya pub. Or do you have some controlled subset of commands in mind?

This could potentially solve both of the remaining open questions at the same time.