[RFC] functions to notify a co-process when the current buffer is modified #5269

phodge · 2016-08-29T16:02:58Z

Hello, I have been working on an efficient API for having neovim notify a co-process (channel?) any time a buffer is modified (because doing this with TextChanged autocmds is inefficient and difficult). This feature will make it easier to write plugins which scans the buffers contents and updates the display in real-time (e.g. syntax highlighters, linters).

The first commit in this PR contains the help text for 3 new functions and a mini-tutorial, please refer to that commit if you would like to learn how the new feature would work. The last commit contains some TODO items that I have yet to resolve.

I'm interested in collecting feedback with regards to:

Do others think the 3 new functions mentioned in the first commit are a sensible interface?
Are there any reasons why the solution I'm working on won't work?
Do you have any suggestions on how I should resolve the TODO items from the last commit?

Thank you for your thoughtful consideration.
Peter

phodge · 2016-08-29T16:16:24Z

Related to:
#719
#1114

oni-link · 2016-08-29T16:29:22Z

runtime/doc/msgpack_rpc.txt

+	was replaced and will always be between `1` and `line('$')`. {int2} tells
+	how many lines were replaced and will always be at least `1`. For most
+	small changes {int2} will be `1`, meaning only the line mentioned in {int1}
+	was changed. {int3} tells how many new lines are replacing the lines that


{int3} would also be at least 1?

bfredl · 2016-08-29T16:31:58Z

Interesting work! There is a simpler way than adding vimL functions tough, use api methods. Compare nvim_ui_attach in api/ui.c, if the first argument is uint64_t channel_id, then the channel_id is implicitly sent when the api function is called over the channel. If the function is placed in api/buffer.c and called buffer_add_watcher (soon nvim_buf_add_watcher, #4934 ) then for instance it can be called on a python buffer object as vim.current.buffer.api.add_watcher(...)

ghost · 2016-08-29T16:34:54Z

When a single channel is watching multiple buffers, is there a way to determine which buffer was modified?

bfredl · 2016-08-29T16:45:23Z

runtime/doc/msgpack_rpc.txt

+
+Setting Up~
+
+Use |rpcstart()| to start up your co-process. This will give you a valid


no need to repeat this here. It is likely also confusing: I suppose many consumers would be plugins living in a rplugin host, and don't be separate co-processes. (As the notifications are high-throughput we should also add multiplexing logic to e.g. the python-client, so that multiple plugins active in the same buffer can share the same notification stream)

phodge · 2016-08-30T07:19:27Z

There is a simpler way than adding vimL functions tough, use api methods.

Thank you, I will look into this ... I don't have any experience with writing or using api methods but it sounds like a reasonable approach.

When a single channel is watching multiple buffers, is there a way to determine which buffer was modified?

Er, no. I'm embarrassed that I didn't notice this glaring omission myself. I will make sure this is catered for in my next push.

I think it should use zero-based indexing

This seems reasonable.

bfredl · 2016-08-30T09:41:29Z

Also please try to write tests in lua. api/server_notifications_spec.lua might have some useful starting points: channel = nvim('get_api_info')[1] to get the channel to the lua test driver (which is just another rpc client to nvim) and eq(..., next_message()) to assert the next notification from nvim. If you make api functions on buffer objects, use curbufmeths.watch_changes() (or what they will be called) to call them.

justinmk · 2016-10-27T07:06:24Z

@phodge why close this?

phodge · 2016-10-29T02:15:02Z

@justinmk Sorry I think when I deleted my rpcbufwatch branch it closed the PR, and just now I've accidentally restored the old branch because my trackpad is way too sensitive.

phodge · 2016-10-29T02:52:12Z

OK I've won my battle with git rebase and managed to push a redesigned version of this feature. It's quite stable and performance seems good. Unfortunately with a new baby having arrived a few weeks ago and a bunch of other projects piling up, I haven't had time to learn lua and rewrite the tests (although they all pass with a python3 py.test). All feedback is appreciated!

phodge · 2016-10-29T20:01:03Z

I've written a simple word highlighting plugin to help with evaluating performance: https://github.com/phodge/vim-hiword

tjdevries · 2016-10-30T12:21:40Z

This would be great for www.github.com/tjdevries/nvim-langserver-shim. There are a couple messages that need to be sent on text changed, text editted, etc. Really cool idea!

bfredl · 2016-10-30T14:25:16Z

src/nvim/eval.c

@@ -10583,6 +10583,7 @@ static void f_has(typval_T *argvars, typval_T *rettv, FunPtr fptr)
    "linebreak",
    "lispindent",
    "listcmds",
+    "liveupdate",


Normally we don't add named features for new nvim-specific features, as nvim will always be compiled with all features (with a few exceptions). Instead a plugin should check for the thing the feature adds, an api function in this case. This is possible in principle by inspecting the api metadata, but maybe we should make this simpler by overloading exist() (exist('+') does not work as not all api functions are vimL functions, like this won't be).

I've made a new commit to revert this addition.

bfredl · 2016-10-30T15:07:51Z

src/nvim/liveupdate.c

+  }
+
+  // add the buffer to the active array and send the full buffer contents now
+  uint64_t *newlist = xmalloc((active_size + 2) * sizeof channel_id);


we should use kvec_t for this, it implements a resizable array for us. See lib/kvec.h for implementation, and you can find example i e bufhl_vec_T lineinfo in buffer.c (type defined in bufhl_defs.h)

I've made a new commit that changes buf->liveupdate_channels to a kvec_t(uint64_t).

bfredl · 2016-10-30T15:36:44Z

src/nvim/api/dispatch_deprecated.lua

@@ -7,6 +7,7 @@ local deprecated_aliases = {
  nvim_buf_get_number="buffer_get_number",
  nvim_buf_get_option="buffer_get_option",
  nvim_buf_get_var="buffer_get_var",
+  nvim_buf_live_updates="buffer_live_updates",


if you use master version of neovim python-client (release will be made soon) this will not be needed.

I'd prefer to leave this backwards-compatibility in since I'd like to try and have live updates merged soon, and I want it to work with current python clients.

No, this is for older functions that was part of 0.1.5 as 0.1.6 changed the naming convention and this has nothing to be backwards-compatible to. python-client 0.1.11 will be released soon, well before this is merged. But we will need to add helper functions in python-client to make it feasible to use this in rplugins (if different python plugins subscribe to the same buffer).

ghost

The method names ("LiveUpdate" etc) should be rationalized with other method names used by nvim. See discussion starting at https://gitter.im/neovim/neovim?at=58160c8a482c168b22d0369b

ghost · 2016-11-01T15:50:48Z

src/nvim/liveupdate.c

+      args.items = xcalloc(sizeof(Object), args.size);
+
+      // the first argument is always the buffer number
+      args.items[0] = INTEGER_OBJ(buf->handle);


Buffer should be sent using EXT encoding. See discussion starting at https://gitter.im/neovim/neovim?at=581609f85a1cfa016e61e784

Which means BUFFER_OBJ should be used instead of INTEGER_OBJ.

bfredl · 2016-11-01T19:21:19Z

src/nvim/liveupdate.c

-    while (buf->liveupdate_channels[active_size] != LIVEUPDATE_NONE) {
-      if (buf->liveupdate_channels[active_size] == channel_id) {
+  size_t size = kv_size(buf->liveupdate_channels);
+  if (size) {


this if can just be removed

@phodge

Originally written by @phodge in neovim#5269.

@phodge

Originally written by @phodge in neovim#5269.

@phodge

Originally written by @phodge in neovim#5269.

@phodge

Originally written by @phodge in neovim#5269.

@phodge

Originally written by @phodge in neovim#5269.

@phodge

Originally written by @phodge in neovim#5269.

@phodge

Originally written by @phodge in neovim#5269.

@phodge

Originally written by @phodge in neovim#5269.

jcjolley · 2018-04-20T03:24:42Z

Thank you for all the work that has been done towards this! This issue is important for https://github.com/Chillee/VSCodeNeovim, and I'm excited to see all the progress that has been made.

@phodge

Originally written by @phodge in neovim#5269.

@phodge

Originally written by @phodge in neovim#5269.

@phodge

Originally written by @phodge in neovim#5269.

@phodge

Originally written by @phodge in neovim#5269.

@phodge

Originally written by @phodge in neovim#5269.

@phodge

Originally written by @phodge in neovim#5269.

@phodge

Originally written by @phodge in neovim#5269.

@phodge

Originally written by @phodge in neovim#5269.

@phodge

Originally written by @phodge in neovim#5269.

@phodge

Originally written by @phodge in neovim#5269.

@phodge

Originally written by @phodge in neovim#5269.

@phodge

Originally written by @phodge in neovim#5269.

@phodge

Originally written by @phodge in neovim#5269.

@phodge

Originally written by @phodge in neovim#5269.

@phodge

Originally written by @phodge in neovim#5269.

justinmk · 2018-06-06T00:35:08Z

This will be addressed by #7917

marvim added the RFC label Aug 29, 2016

oni-link reviewed Aug 29, 2016
View reviewed changes

bfredl reviewed Aug 29, 2016
View reviewed changes

phodge closed this Oct 26, 2016

phodge deleted the rpcbufwatch branch October 26, 2016 20:28

jszakmeister removed the RFC label Oct 26, 2016

phodge restored the rpcbufwatch branch October 29, 2016 02:10

phodge reopened this Oct 29, 2016

phodge closed this Oct 29, 2016

phodge force-pushed the rpcbufwatch branch from e57747f to 242c3d5 Compare October 29, 2016 02:22

phodge reopened this Oct 29, 2016

marvim added the RFC label Oct 29, 2016

bfredl reviewed Oct 30, 2016

View reviewed changes

ghost reviewed Nov 1, 2016

View reviewed changes

bfredl reviewed Nov 1, 2016

View reviewed changes

justinmk mentioned this pull request Nov 4, 2016

Add a notification provider #5562

Closed

KillTheMule pushed a commit to KillTheMule/neovim that referenced this pull request Feb 9, 2018

API: Document buffer updates

d05b278

Originally written by @phodge in neovim#5269.

KillTheMule pushed a commit to KillTheMule/neovim that referenced this pull request Feb 9, 2018

Tests for buffer updates

0957da0

Originally written by @phodge in neovim#5269.

KillTheMule pushed a commit to KillTheMule/neovim that referenced this pull request Feb 12, 2018

API: Implement buffer updates

ff16d3e

Originally written by @phodge in neovim#5269.

KillTheMule pushed a commit to KillTheMule/neovim that referenced this pull request Feb 12, 2018

API: Document buffer updates

36a23f8

Originally written by @phodge in neovim#5269.

KillTheMule pushed a commit to KillTheMule/neovim that referenced this pull request Feb 12, 2018

Tests for buffer updates

c9f6083

Originally written by @phodge in neovim#5269.

KillTheMule pushed a commit to KillTheMule/neovim that referenced this pull request Feb 17, 2018

API: Implement buffer updates

f46c6a5

Originally written by @phodge in neovim#5269.

KillTheMule pushed a commit to KillTheMule/neovim that referenced this pull request Feb 17, 2018

API: Document buffer updates

967721a

Originally written by @phodge in neovim#5269.

KillTheMule pushed a commit to KillTheMule/neovim that referenced this pull request Feb 17, 2018

Tests for buffer updates

3873ebf

Originally written by @phodge in neovim#5269.

KillTheMule pushed a commit to KillTheMule/neovim that referenced this pull request Apr 24, 2018

API: Implement buffer updates

2b3a3ec

Originally written by @phodge in neovim#5269.

KillTheMule pushed a commit to KillTheMule/neovim that referenced this pull request Apr 24, 2018

API: Document buffer updates

d7ca698

Originally written by @phodge in neovim#5269.

KillTheMule pushed a commit to KillTheMule/neovim that referenced this pull request Apr 24, 2018

Tests for buffer updates

04d2717

Originally written by @phodge in neovim#5269.

KillTheMule pushed a commit to KillTheMule/neovim that referenced this pull request Apr 27, 2018

API: Implement buffer updates

cf6e24e

Originally written by @phodge in neovim#5269.

KillTheMule pushed a commit to KillTheMule/neovim that referenced this pull request Apr 27, 2018

API: Document buffer updates

35cbae3

Originally written by @phodge in neovim#5269.

KillTheMule pushed a commit to KillTheMule/neovim that referenced this pull request Apr 27, 2018

Tests for buffer updates

aa4fb22

Originally written by @phodge in neovim#5269.

KillTheMule pushed a commit to KillTheMule/neovim that referenced this pull request May 10, 2018

API: Implement buffer updates

5642ab1

Originally written by @phodge in neovim#5269.

KillTheMule pushed a commit to KillTheMule/neovim that referenced this pull request May 10, 2018

API: Document buffer updates

2ed7026

Originally written by @phodge in neovim#5269.

KillTheMule pushed a commit to KillTheMule/neovim that referenced this pull request May 10, 2018

Tests for buffer updates

fc5fb07

Originally written by @phodge in neovim#5269.

KillTheMule pushed a commit to KillTheMule/neovim that referenced this pull request May 18, 2018

API: Implement buffer updates

7c4f24e

Originally written by @phodge in neovim#5269.

KillTheMule pushed a commit to KillTheMule/neovim that referenced this pull request May 18, 2018

API: Document buffer updates

b6b45dc

Originally written by @phodge in neovim#5269.

KillTheMule pushed a commit to KillTheMule/neovim that referenced this pull request May 18, 2018

Tests for buffer updates

de5cd57

Originally written by @phodge in neovim#5269.

KillTheMule pushed a commit to KillTheMule/neovim that referenced this pull request May 23, 2018

API: Implement buffer updates

edcc73e

Originally written by @phodge in neovim#5269.

KillTheMule pushed a commit to KillTheMule/neovim that referenced this pull request May 23, 2018

API: Document buffer updates

db15a3b

Originally written by @phodge in neovim#5269.

KillTheMule pushed a commit to KillTheMule/neovim that referenced this pull request May 23, 2018

Tests for buffer updates

7918480

Originally written by @phodge in neovim#5269.

justinmk closed this Jun 6, 2018

justinmk removed the RFC label Jun 6, 2018

justinmk mentioned this pull request Jun 8, 2018

Nvim API now supports buffer update events Chillee/VSCodeNeovim#13

Closed

justinmk mentioned this pull request Jun 21, 2018

Buffer changes, "shadow buffers" and the MSGPACK API #1114

Closed

phodge deleted the rpcbufwatch branch June 24, 2018 20:54

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[RFC] functions to notify a co-process when the current buffer is modified #5269

[RFC] functions to notify a co-process when the current buffer is modified #5269

phodge commented Aug 29, 2016

phodge commented Aug 29, 2016

oni-link Aug 29, 2016

bfredl commented Aug 29, 2016 •

edited

Loading

ghost commented Aug 29, 2016 •

edited by ghost

Loading

bfredl Aug 29, 2016

phodge commented Aug 30, 2016

bfredl commented Aug 30, 2016

justinmk commented Oct 27, 2016

phodge commented Oct 29, 2016

phodge commented Oct 29, 2016 •

edited

Loading

phodge commented Oct 29, 2016

tjdevries commented Oct 30, 2016 •

edited

Loading

bfredl Oct 30, 2016

phodge Nov 1, 2016

bfredl Oct 30, 2016

phodge Nov 1, 2016

bfredl Oct 30, 2016

phodge Nov 1, 2016

bfredl Nov 1, 2016

ghost left a comment •

edited by ghost

Loading

ghost Nov 1, 2016 •

edited by ghost

Loading

bfredl Nov 1, 2016

bfredl Nov 1, 2016

jcjolley commented Apr 20, 2018 •

edited

Loading

justinmk commented Jun 6, 2018


		Setting Up~

		Use \|rpcstart()\| to start up your co-process. This will give you a valid

[RFC] functions to notify a co-process when the current buffer is modified #5269

[RFC] functions to notify a co-process when the current buffer is modified #5269

Conversation

phodge commented Aug 29, 2016

phodge commented Aug 29, 2016

Choose a reason for hiding this comment

bfredl commented Aug 29, 2016 • edited Loading

ghost commented Aug 29, 2016 • edited by ghost Loading

Choose a reason for hiding this comment

phodge commented Aug 30, 2016

bfredl commented Aug 30, 2016

justinmk commented Oct 27, 2016

phodge commented Oct 29, 2016

phodge commented Oct 29, 2016 • edited Loading

phodge commented Oct 29, 2016

tjdevries commented Oct 30, 2016 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ghost left a comment • edited by ghost Loading

Choose a reason for hiding this comment

ghost Nov 1, 2016 • edited by ghost Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jcjolley commented Apr 20, 2018 • edited Loading

justinmk commented Jun 6, 2018

bfredl commented Aug 29, 2016 •

edited

Loading

ghost commented Aug 29, 2016 •

edited by ghost

Loading

phodge commented Oct 29, 2016 •

edited

Loading

tjdevries commented Oct 30, 2016 •

edited

Loading

ghost left a comment •

edited by ghost

Loading

ghost Nov 1, 2016 •

edited by ghost

Loading

jcjolley commented Apr 20, 2018 •

edited

Loading