outputStream(p: Process) in osproc in standard library cannot call peekChar

[demotomohiro]

I tried to send a command to neovim and receive a response using msgpack-rpc.
I added * to proc stringify[ByteStream](s: ByteStream, zz: ByteStream) in msgpack4nim.nim and tried to output stringified response from neovim to stdout.
But I got runtime error as Stream returned from proc outputStream(p: Process): Stream doesn't have peekDataImpl on windows.

Should peekDataImpl be implemented for proc outputStream(p: Process): Stream?
Or msgpack4nim should not use peek?

import osproc, streams
import msgpack4nim, macros

macro writeRPCReq[ByteStream](
                              s: ByteStream;
                              methodName: string;
                              id: int;
                              args: varargs[typed]) =
  let
    argsLen = args.len.newLit
  result = quote do:
    `s`.pack_array 4
    `s`.pack_type 0
    `s`.pack_type `id`
    `s`.pack_type `methodName`
    `s`.pack_array `argsLen`
 
  for a in args:
    result.add newCall(bindSym"pack", s, a)

proc main =
  var
    p = startProcess("nvim", args = ["--embed"], options = {poUsePath})
    inStrm = p.inputStream
    outStrm = p.outputStream
    stdoutStrm = stdout.newFileStream
  inStrm.writeRPCReq("nvim_command", 1, "echo $MYVIMRC")
  inStrm.flush()
  outStrm.stringify stdoutStrm.Stream
  p.close()

main()

I don't know this code sends request message to neovim correctly.

[jangko]

this is not msgpack4nim problem, it's stdlib problem.
why not just read them all into a string?

[demotomohiro]

why not just read them all into a string?

I want to make a program that repeats sending a request message to neovim and receiving a response using msgpack-rpc through pipe.
As neovim process is running after it sent msgpack-rpc response message, there is no EOF after the message from neovim's stdout.
Also there is no '\n' or specific byte sequence that tells me end of response.
That means I cannot use proc readLine(s: Stream): TaintedString or proc readAll(s: Stream): string in Stream module.
They block even if they read whole response message.

According to spec of msgpack and msgpack-rpc, it seems there is no way to get size of a whole message by reading only first fixed size bytes.
Even if I knew that I got the array with 4 elements, it can contain different type of elements.
That means I cannot use proc readData(s: Stream; buffer: pointer; bufLen: int): int or proc readDataStr(s: Stream; buffer: var string; slice: Slice[int]): int in Stream module.
That means I cannot read 1 message from Stream and put it to string without parsing response message.

That is why I wanted to pass output stream of neovim process to proc stringify[ByteStream](s: ByteStream, zz: ByteStream) in msgpack4nim.nim, but peek procs are unavailable on output stream.

I have created new issue about it on Nim repository.
nim-lang/Nim#14906

[demotomohiro]

I added peekableOutputStream proc with this PR so that I can peek output pipe stream of a process.
nim-lang/Nim#14949

I still don't think I can take message as string from pipe while the process is running.
So I need to use proc stringify[ByteStream](s: ByteStream, zz: ByteStream) instead of proc stringify*(data: string): string.
Could you make it being exported?
I will send PR that add * to that symbol if you want.

Here is working sample code to run neovim and send neovim command from Nim.
This code requires devel Nim and msgpack4nim exports proc stringify[ByteStream](s: ByteStream, zz: ByteStream).

import osproc, streams
import msgpack4nim, macros

macro writeRPCReq[ByteStream](
                              s: ByteStream;
                              methodName: string;
                              id: int;
                              args: varargs[typed]) =
  let
    argsLen = args.len.newLit
  result = quote do:
    `s`.pack_array 4
    `s`.pack_type 0
    `s`.pack_type `id`
    `s`.pack_type `methodName`
    `s`.pack_array `argsLen`
 
  for a in args:
    result.add newCall(bindSym"pack", s, a)

proc main =
  var
    p = startProcess("nvim", args = ["--embed"], options = {poUsePath})
    inStrm = p.inputStream
    outStrm = p.peekableOutputStream
    stdoutStrm = stdout.newFileStream

  var msgId = 0
  for l in stdin.lines:
    if not p.running:
      break
    inStrm.writeRPCReq("nvim_command_output", msgId, l)
    inStrm.flush()
    outStrm.stringify stdoutStrm.Stream
    echo ""
    inc msgId
  p.close()

main()

[demotomohiro]

Thank you!