Vim for pipeline filtering

[MilesCranmer]

I made a 40-line script that lets you call Vim commands for Unix pipeline filtering, like sed or awk: https://github.com/MilesCranmer/vim-stream.

As an example,

cat file.txt | vims -e 'label' 'f|d$'

will filter the output of "cat file.txt" by running the Vim command ‘f|d$’ on each line matching 'label' to delete everything after "|". The repo gives the other options.

I’m eager to hear: are you potentially interested in integrating this command line mode into Vim itself? The code looks tiny but pulls a lot of weight; I use "vim-stream" for batch text processing in nearly every script I write. I no longer spend time hunting for awk/sed commands, since Vim is ingrained in my fingers.

Is your feature request about something that is currently impossible or hard to do? Please describe the problem.
Yes. To do this with vanilla vim, you would have type:

cat file.txt | vim - -nes -c ':%g/label/exe "norm f|d$"' -c ':%p' -c ':q!' | tail -n +2

Describe the solution you'd like

Perhaps a flag could activate this pipeline filter mode, and this functionality could be integrated into vim.

I'm interested to hear your thoughts.
Cheers,
Miles

[oblitum]

This article may be useful on implementation (or moving to):

https://vimways.org/2018/vims-social-life/#under-the-tree-neovim

[MilesCranmer]

Thanks @oblitum, this looks useful! I should emphasize, though, that the main additions of the script aren't just to make command-line vim less verbose, but to also make it easier to call commands like %g/{...}/exe "norm {...}" on piped text. Such a command is very very useful for batch text processing in a script, but is tedious to write out.

This is why I think it would be useful to integrate such short-hand in vim's execution arguments (and neovim potentially!).

[MilesCranmer]

The command could look like this: cat file.txt | vim -P ...., where the remaining commands would be the shorthand. Here, the "-P" flag stands for "pipe mode" and would essentially run the remaining arguments to vim according to the short hand defined in https://github.com/MilesCranmer/vim-stream/.

For example,

cat file.txt | vim -P '%s/foo/bar/g'

^Replaces foo with bar using vim syntax.

cat file.txt | vim -P -l 'f|d$' -e 'label' 'AHello World'

Goes through every line (-l), and deletes everything after |. Then, every line with "label", it appends "Hello World".

-l usually turns on Lisp mode, but since we have called -P as the first argument, every other argument to vim is interpreted as a vim-stream function.

I'm happy to provide code for this to integrate it to vim, by the way.

[brammool]

It's an interesting idea. Your first example shows an Ex command, while the second one is a Normal mode command. So -l implies that?
The match & append could be done with 'g/label/s/$/Hello World' Is it needed to add another way to select lines?

[MilesCranmer]

Exactly! The heart of the script (in bash) looks like this:

none)        vim_cmds+=(-c "$1");;
-s)          vim_cmds+=(-c ":exe \"norm ""$1""\"");;
-l)          vim_cmds+=(-c ":%g/.*/exe \"norm ""$1""\"");;
-e)          vim_cmds+=(-c "%g/$1/exe \"norm $2\""); shift;;
-r)          vim_cmds+=(-c "%v/$1/exe \"norm $2\""); shift;;

So by default, you run Ex commands. The "-s" flag lets you type in vim command mode (it just runs :exe "norm {...}"), which gets applied to the piped text. "-l" runs a command mode command on every line. "-e" runs a command on every line matching a regex, and "-r" not matching the regex.

It's simple but it's easy to fall in love with. You are right; there is usually an equivalent Ex command, but for more complicated jobs (e.g., requiring copy-pasting around a command output), command-mode vim seems to require less thinking. For example, say I run "ls -lath" and want to isolate the time of modification for each file. I could write out an awk/sed/Ex command, but that would require more thinking (and counting columns) than just running a vim command on every line:

ls -lath | vims -l 'f:hhd0f d$'

This runs "f:" to get to the colon (in the printed time), "hh" to move left twice, "d0" to delete to the start of the line, then "f " to move in front of the time, then "d$" to delete to the end.

It feels much more natural than using sed/etc, so I use this script all the time when text processing. I don't think I have used awk or sed ever since; I only need vim (and occasionally python) for scripted text parsing.