milestone 2.07beta1

[rkitover]

@lucc

So 2.07 will be a huge release, it's been more than half a year since 2.06 and 2.07 will be in Debian. We need to determine what must be done for it.

The most important issue for me is fixing file streaming, I have some ideas about how to do this, using CursorHold among other things.

[lucc]

I am looking forward to a 2.07. By the way you could use the milestones feature of github to mark some issues and PRs. Then we could start individual issues for these things and discuss them.

Can you post some code and ideas for the streaming stuff, then we can discuss it. I made some remarks here.

[lucc]

Depending on your idea for the CursorHold autocmd it might also be interesting to do something similar for these events: VimResized,FocusGained,FocusLost,CursorMoved.

Also you might consider conforming to semantic versioning for future releases.

[rkitover]

@lucc This is my idea for implementing input streaming:

1. vimpager forks off a background process that receives data from stdin, or from a large file if it needs to be processed
2. any necessary processing is done on the data with callbacks, the processing is done block-wise in 4k blocks or so, the processing callbacks need to be aware that they may receive incomplete data and wait for more data if necessary
3. an initial set of a few blocks is written to an encoded file in the tmp dir as is now done
4. the viml is made aware of the fact that that argidx is an active stream and makes sure it is in an open and possibly hidden buffer
5. the reader job writes blocks of data to ${tmp}/file/ as sequentially numbered files
6. a global CursorHold event running every second or so goes through the list of streams and their associated buffers, checks for any newly created numbered data files and sequentially appends them to the target buffer, and deletes them.
7. for a normal large file or STDIN reader job that finishes streaming, on receiving a SIGPIPE it will write a marker file to the file's tmp dir to let the viml know that the streaming has completed.
8. the CursorHold event picks up the end marker and removes that buffer from the list of streamed buffers.
9. for an infinite stream, the viml will pick up data blocks for as long as the user keeps the vim session open
10. it would be nice to use the same code for input streaming in vimcat, as output streaming is already mostly implemented.

re: semantic versioning, I will research this.

re: github milestone, I will set that up later today.

[lucc]

If you want to force your users to have a new version of vim (or neovim) we could think about implementing that with jobs (job_start() for vim and/or jobstart() for neovim). The plan roughly would look like this:

1. vimpager doesn't touch the file to be streamed. If stdin has to be used vimpager connects it to a pipe.
2. vimpager starts vim and calls a special function with -c 'vimpager#handle_streams($PIPE1, $PIPE2, ...)' other files.
3. vimpager#handle_files() will use job{,_}start() to read from these pipes and use a callback function to read any output from the pipes to viml lists.
4. some autocmd is used to pop from these lists when necessary.

I think the basic algorithm is the same (reading data in chunks) as what you said only instead of using /tmp we could use the job control feature in vim. This obviously depends on the general question that you would have to decide beforehand: Do you want to force users to have a new version of vim?

[rkitover]

@lucc sorry I confiused you with this other guy.

I added you as a collaborator, please continue to send PRs, except if it's a minor change, in which case feel free to commit to master.

This jobs thing looks interesting, but I'm not sure it helps here. And I'm wary of using fifos, their behavior is often unpredictable except in the simplest of cases.

I'm still shooting for at least vim 7.3 compatibility, even when it requires workarounds.

[lucc]

Thank you for the honor! But I will continue to ask about most of the changes I propose (send normal PRs instead of pushing myself). During the last conversations we had I saw that you often target systems I know nothing about.

If you want to be compatible to 7.3 jobs are out anyways.

[lucc]

I took the liberty to create the milestone and add some issues to it.

Do you think we can close these issues (because they are resolved and all further details are covered by the new issues in parentheses)?

• doesn't play nicely with git #11 (AnsiEsc: escape sequence '\x1b[1;39m' (bold) not properly recognized #105)
• Can't copy-paste #45

[rkitover]

@lucc I've implemented streaming for vimcat see ^^, this is the prototype for vimpager as well, so I'm well on my way to finishing this up.