feat(lua): add vim.system()

[lewis6991]

Problem

Handling system commands in Lua is tedious and error-prone:

• vim.fn.jobstart() is vimscript and comes with all limitations attached to typval.
• vim.loop.spawn is too low level

Also see: #17620 (comment)

Solution

Add vim.system().

• Partly inspired by Python's subprocess module
• Does not expose any libuv objects.
• Supersedes feat(lua): add subprocess module #17620

local on_exit = function(code, signal, stdout, stderr)
  print(code)
  print(signal)
  print(stdout)
  print(stderr)
end

-- Asynchronous
vim.system({'echo', 'hello'}, nil, on_exit)

-- Synchronous
local stdout = vim.system({'echo', 'hello'}):wait().stdout

system({cmd}, {opts}, {on_exit})                                *vim.system()*
    Run a system command

    See |uv.spawn()| for more details.

    Parameters: ~
      • {cmd}      (string[]) Command to execute
      • {opts}     (SystemOpts|nil) Options:
                   • cwd: (string) Set the current working directory for the
                     sub-process.
                   • env: table<string,string> Set environment variables for
                     the new process. Inherits the current environment with
                     `NVIM` set to |v:servername|.
                   • clear_env: (boolean) `env` defines the job environment
                     exactly, instead of merging current environment.
                   • stdin: (string|string[]|boolean) If `true`, then a pipe
                     to stdin is opened and can be written to via the
                     `write()` method to SystemObj. If string or string[] then
                     will be written to stdin and closed. Defaults to `false`.
                   • stdout: (boolean|function) Handle output from stdout.
                     When passed as a function must have the signature
                     `fun(err: string, data: string)`. Defaults to `true`
                   • stderr: (boolean|function) Handle output from stdout.
                     When passed as a function must have the signature
                     `fun(err: string, data: string)`. Defaults to `true`.
                   • text: (boolean) Handle stdout and stderr as text.
                     Replaces `\r\n` with `\n`.
                   • timeout: (integer)
                   • detach: (boolean) If true, spawn the child process in a
                     detached state - this will make it a process group
                     leader, and will effectively enable the child to keep
                     running after the parent exits. Note that the child
                     process will still keep the parent's event loop alive
                     unless the parent process calls |uv.unref()| on the
                     child's process handle.
      • {on_exit}  (function|nil) Called when subprocess exits. When provided,
                   the command runs asynchronously. Receives SystemCompleted
                   object, see return of SystemObj:wait().

    Return: ~
        SystemObj Object with the fields:
        • pid (integer) Process ID
        • wait (fun(timeout: integer|nil): SystemCompleted)
          • SystemCompleted is an object with the fields:
            • code: (integer)
            • signal: (integer)
            • stdout: (string), nil if stdout argument is passed
            • stderr: (string), nil if stderr argument is passed

        • kill (fun(signal: integer))
        • write (fun(data: string|nil)) Requires `stdin=true`. Pass `nil` to
          close the stream.
        • is_closing (fun(): boolean)

Todo

• Add tests

Future work

• pty support
• /dev/null support. (Can pass an empty handler for the same effect)
• Execution via a shell

[clason]

Is there a way to pipe stuff to stdin (like Vimscript's system(cmd, string))?

[lewis6991]

Yes, stdin can be a string or string[].

[lewis6991]

Windows test failures are germane (stupid Windows line endings...)

Probably needs something like c65e220/runtime/lua/vim/lsp/util.lua#L458C29-L459

(which seems like something that could be refactored -- e.g., as part of a vim.string library).

I don't think vim.system should be doing anything special with line endings. One example is in Gitsigns \r needs to be preserved since the system output is used to diff against the buffer text so we need to take the output verbatim.

For now, I think we just need to make the tests handle outputs with \n or \r\n. There might be an argument that the default handler strips \r and users must use custom handlers to preserve them.

[clason]

I don't think vim.system should be doing anything special with line endings. One example is in Gitsigns \r needs to be preserved since the system output is used to diff against the buffer text so we need to take the output verbatim.

That is a strong argument (also since it's easier to strip than to restore). Is this something that should/could be mentioned somewhere, or is this too niche or obvious in the context?

[clason]

Is this low-hanging fruit now, or better left for a follow-up PR?

neovim/runtime/lua/provider/health.lua

Lines 78 to 141 in 0370e4d

	-- Run a system command and timeout after 30 seconds.
	local function system(cmd, ...)
	local args = { ... }
	local args_count = vim.tbl_count(args)
	local stdin = (args_count > 0 and args[1] or '')
	local stderr = (args_count > 1 and args[2] or false)
	local ignore_error = (args_count > 2 and args[3] or false)
	local opts = {
	add_stderr_to_output = stderr,
	output = '',
	stderr = '',
	on_stdout = system_handler,
	on_stderr = system_handler,
	on_exit = system_handler,
	}
	local jobid = vim.fn.jobstart(cmd, opts)
	if jobid < 1 then
	local message = 'Command error (job='
	.. jobid
	.. '): `'
	.. shellify(cmd)
	.. '` (in '
	.. vim.fn.string(vim.fn.getcwd())
	.. ')'
	error(message)
	shell_error_code = 1
	return opts.output
	end
	if not is_blank(stdin) then
	vim.api.nvim_chan_send(jobid, stdin)
	end
	local res = vim.fn.jobwait({ jobid }, 30000)
	if res[1] == -1 then
	error('Command timed out: ' .. shellify(cmd))
	vim.fn.jobstop(jobid)
	elseif shell_error() and not ignore_error then
	local emsg = 'Command error (job='
	.. jobid
	.. ', exit code '
	.. shell_error_code
	.. '): `'
	.. shellify(cmd)
	.. '` (in '
	.. vim.fn.string(vim.fn.getcwd())
	.. ')'
	if not is_blank(opts.output) then
	emsg = emsg .. '\noutput: ' .. opts.output
	end
	if not is_blank(opts.stderr) then
	emsg = emsg .. '\nstderr: ' .. opts.stderr
	end
	error(emsg)
	end
	-- return opts.output
	local _ = ...
	return vim.trim(vim.fn.system(cmd))
	end

[lewis6991]

I would say it is just hanging fruit.

I did have a go at refactoring it but the code has some niche options that I'm not sure are meaningful to keep. And tbh, the whole file could do with some cleanup so probably better to do it in one go (in a follow up PR).

[clason]

external_deps test is straight-up broken (@dundargoc) across PRs, so after squashing this should be good to go.

[lewis6991]

For the \r\n problem, python has a text=true argument which I've copied here (default false). Note: before text they had universal_newlines, so they've done the churn for us.

Note vim.fn.system() also has this:

	Result is a String, filtered to avoid platform-specific quirks:
	- <CR><NL> is replaced with <NL>
	- NUL characters are replaced with SOH (0x01)

Lua can happily handle embedded NUL chars so we don't need to do that.

So now the following are roughly equivalent.

local out = vim.fn.system({'echo', 'hello'})
local out = vim.system({'echo', 'hello'}, {text=true}):wait().stdout

vim.system is a bit more verbose than I planned (with :wait(), text=true and .stdout), but it's probably a good middle-ground given its versatility.

[A-Lamia]

vim.system does not work on windows 10 using:
:=vim.system({'echo', 'hello'}, {text=true}):wait().stdout
is this just me or can anyone replicate ?

NVIM: v0.10.0-dev-751+gd086bc1e8

[lewis6991]

The message seems like it's indicating echo doesn't exist.

[A-Lamia]

@lewis6991 it does though, tested on CMD and PS, isn't echo a native command lol ?

EDIT: you are right though fd seems to return something but not echo, I'm very confused :)

[lewis6991]

The tests added in this PR cover echo on windows so the issue must be something specific to your environment.

[A-Lamia]

@lewis6991 didn't work on my second desktop but worked on my laptop, so yeah it's probably something on my side, do these commands all run on default CMD ?

[lewis6991]

No idea, vim.system is just a wrapper around vim.uv.spawn which is provided by libuv.

[mfussenegger]

uv.spawn doesn't run the process in a shell. You'll have to run something like vim.system({"cmd.exe", "/C", "echo"}):wait() if you need one.

(And please stop using this PR as support forum. You're notifying all reviewers. Use matrix if you've general usage questions. )