Feature suggestion: real time command output using streams

[yvesnrb]

It would be really useful if one could get the output of commands in real time as they are written to stdout with a readable stream instead of awaiting for the command to exit. For example:

const pingStream = $.stream`ping 8.8.8.8`

pingStream.on('data', () => {
  console.log('ping wrote to stdout');
});

pingStream.on('end', () => {
  console.log('ping terminated with exit code 0');
});

pingStream.on('error', () => {
  console.log('ping terminated with an error');
});

[antonmedv]

Why not just use nodejs API directly for this?

[yvesnrb]

Sorry if I'm misunderstanding the way this works. I thought the promise of $some-command only resolved after the command exited? In this way it would not be possible to get the output of a command as it is happening before it terminated.

[antonmedv]

Yes, only then the command finished.

[catpea]

I am worried about this.

zx is a great idea and $`` is wonderfully lightweight - you are wonderful!

But my feline instincts, are saying, get https://pegjs.org/ in here, and instead of shelling everything out to /bin/sh get node to take over.

This will not be lightweight, but it will unfurl what is going on between $`a | b` and grant access to all the output streams of all the commands being executed.

Now you will be able to get the stream of a's stderr for example, as it stdout into b's stdin. This is what programming is about, beautiful great things that keep you awake at night... that rustle your jimmies... that use node to make running commands even more powerful...

Instead of being clever and lightweight, which is very noble, you become a leader, you become legendary, which is even better.

I think we forget not to $`` all to often, here's Brian to remind us, what it means to hold a legendary idea:
https://www.youtube.com/watch?v=tc4ROCJYbm0&t=249s

Make flow control a first class citizen via https://www.sweetjs.org/ and instead of $`a | b` which is "what it is", allow for something even more brilliant like $ cat purr.txt | wc -l

(I leave the DSL sweetness up to you as you are excellent at it)

Make $`commands` into real commands, not something that is just thrown at /bin/sh, but something that comes to life in node.

@yvesnrb is on the right path, we want to keep getting the data, we want to $.stream`tail -f logs/flarp` or read sockets.

pingStream.on('data', () => {
console.log('ping wrote to stdout');
});

This is an opportunity to change the world,
you made something special, and @yvesnrb has the right idea to make it better.

[biels]

@catpea If I understand correctly are you talking about executing executables using child_process directly instead of executing sh commands?

[catpea]

@biels Yes. It would be a tree of streams that efficiently pipe data from stdout to stdin. That is what Node Streams is patterned after, pipes, pehaps even UNIX pipes. There is this wonderful opportunity here to parse the command line and make it come to life within node.

The main argument against the brilliant $`command1 | command2` is the same argument that is made for Node Streams, read chunk at a time, instead of cramming the whole thing into memory.

I think antonmedv did an amazing job, this is not a critique, I must admit, I think this is outside of the scope of this program.

I am simply saying parse the commands with peg.js build a child_process stream tree and allow for incredible complexity. Complexity and functionality currently obscured by node just sending a string to /bin/sh and then sitting there and waiting until a potentially massive chunk of text is returned.

So it is not just a single child_proces that wraps the entire command, it is a thing that interprets the command line, and creates a child_proces tree. It will work for simple commands like $`tar -cvf bork.tar bork` but also for freakishly complex things that reroute output streams, run as a daemon, respond to something like UNIX's nice(1) command, and enhance the operating system with node.

await $`cat package.json | grep name`

Will actually start two
child_processes one is cat and the other grep, cat will send things to grep chunk at a time, by relying on Node Streams.

The next step to this would be /bin/zxsh a zx based Bash replacement. Which will not be possible to make if we just send strings at /bin/sh

And the step after that would be using Electron as a replacement for X Windows, perhaps with https://nextapps-de.github.io/winbox/

and then antonmedv could release a https://www.linuxfromscratch.org/ called ANTONUX

And we would all end up using it.

My instinct says that

let name = await $`cat package.json | grep name`;

obscures powerful things, that if we can get access too we can create an alternative to https://www.npmjs.com/package/pm2 alternative to Bash, and even mess around with electron as a GUI something that could potentially be as tiny as http://www.damnsmalllinux.org/

The sending a string to /bin/sh and waiting for one back obscures powerful functionality.

antonmedv your work is perfect, zx is awesome, this is not a critique, but a discussion of a potential future of this program.

You started something, that could develop legs.

Way to go!

[yvesnrb]

@catpea What you are suggesting would be quite involved to implement but I think you are on to something.

[Minigugus]

@catpea What about:

await $`cat package.json`.pipe($`grep name`)

let name = await $`cat package.json`.pipe($`grep name`); // works since `.pipe` returns its argument

The main idea is to transform Promise<ProcessOutput> into DuplexStream & PromiseLike<ProcessOutput> and rely on .pipe. This is significantly easier to implement than using pegjs, while increasing the awesomeness of pipes:

const r = fs.createReadStream('file.txt');
const w = fs.createWriteStream('file.txt.gz');
await r.pipe($`gzip`).pipe(w);

Pros:

• Easy to implement (redirect duplex input to stdin and stdout to duplex output, and add a .then function for await to work)
• User-friendly
• Might not break the traditional | (await $`cat package.json | grep name`;)
• Seamless integration with any node library using streams.
• Improve support for Windows

Cons:

• May affect performance (but the user can always fall back on | in this case)
• Node specific feature (but fs and os are already node only)

[antonmedv]

I think this is out of the scope of the zx project. Closing.

UPDATE 2022-05-31

Now zx supports real time output as well.

Pipes can be used to show real-time output of programs:

$.verbose = false

await $`echo 1; sleep 1; echo 2; sleep 1; echo 3;`
  .pipe(process.stdout)

Or like this:

let { stdin, stdout } = $`npm init`

for await (let chunk of stdout) {
  if (chunk.includes('package name:')) stdin.write('test\n')
}

[Minigugus]

@catpea For those looking for the pipeline feature, I made bazx

For instance (#35 (comment))

console.log(await collect($`find ${dir} -type f -print0`.pipe($`xargs -0 grep foo`).pipe($`wc -l`)))

// $ find /tmp/bazx/test -type f -print0 | xargs -0 grep foo | wc -l
// 3
// { success: true, code: 0, stdout: "3", stderr: "" }

https://github.com/Minigugus/bazx/blob/023496cae0a53c0ed9b9c4f72f799a3ca36a6703/test/debug.js#L5-L13

[antonmedv]

Now zx support real time output!

[yvesnrb]

This is excellent! A lot of other great things have been added since I opened this issue as well. Thank you for bringing this library to such a high standard.