[Process] Allow running multiple commands at once

[m-vo]

Description
When running a lot of small shell commands after each other, the overhead of creating the underlying processes becomes noticeable. It would be handy if the process component would support chaining commands.

Working directly on a Linux/Windows shell you would typically do this:

# Linux
$ foo; bar; baz
$ foo && bar && baz

# Windows
$ foo & bar & baz
$ foo && bar && baz

(Yes there are other chaining operators as well like ||.)

This is currently not possible with the process component.

API Example
I like that Symfony\Component\Process\Process became somewhat immutable now (the command line is baked in after the constructor ran), so the API could maybe look like this:

$process = new Process(['foo']);

// foo; bar
$process = $process->withChainedCommand(['bar']); // returns a new process instance

// foo && bar
$process = $process->withAndChainedCommand(['bar']); // returns a new process instance

What do you think? Is this a worthwhile addition?

[m-vo]

In non-windows environments, we're currently replacing the current shell (that PHP starts automatically) with the command to run by prepending it with exec . This makes sure signaling to the process is possible.

For chained commands this cannot work because there is no shell to start the subsequent commands (exec foo && bar). We could however prepend a pipe in that case so that subprocesses are started (exec | foo && bar), though I'm not 100% sure this will work in all cases.

Ideas and insights are welcome. ☺️ /cc @nicolas-grekas

[derrabus]

Sounds like a good idea to me.

[nicolas-grekas]

I'm not sure I would recommend adding withChainedCommand on the Process class.
But a shell command line builder would allow doing that I suppose.

[m-vo]

But a shell command line builder would allow doing that I suppose.

You mean a builder that creates shell system specific shell commands that are then treated as "raw input" like with Process::fromShellCommandline()? I think we would loose things like signaling then as well.

We could also introduce a new Operator class that you could use like this:

$process = new Process([
    'foo', Operator::chain(), 'bar', Operator::and(), 'baz'
]);

So the array of commands would be array<string|Process> and the operators are substituted with unescaped literals when starting the process (here: foo; bar && baz or foo & bar && baz).

[nicolas-grekas]

Why would we loose anything? In the end, the only way to achieve what you describe is to build a shell command line, there is no other way. So yes, I'm thinking about a builder that would generate a string that is then provided to fromShellCommandline().

[nicolas-grekas]

Another idea: since PHP 7.4, proc_open() accepts an array directly. Maybe this removes the overhead you're mentioning, since that's the main motivation for this request. Would you like to have a look?

[m-vo]

Why would we loose anything? In the end, the only way to achieve what you describe is to build a shell command line, there is no other way

I get your point. My thinking was: We're only adding exec to the array version, now, but we want it to be an implementation detail. And then it would be coupled to the builder class as well (if it's needed for chained commands).

Another idea: since PHP 7.4, proc_open() accepts an array directly. Maybe this removes the overhead you're mentioning, since that's the main motivation for this request. Would you like to have a look?

👍 I'll have a look.

[nicolas-grekas]

We're only adding exec to the array version, now, but we want it to be an implementation detail

We could achieve that by wrapping the call to fromShellCommandline inside a getProcess() method on the builder maybe.

since PHP 7.4, proc_open() accepts an array directly

BTW, I'm sure we can cleanup the code on branch 6.0 by leveraging this! PR welcome :)

[carsonbot]

Thank you for this suggestion.
There has not been a lot of activity here for a while. Would you still like to see this feature?

[carsonbot]

Could I get a reply or should I close this?

[carsonbot]

Hey,

I didn't hear anything so I'm going to close it. Feel free to comment if this is still relevant, I can always reopen!

[ausi]

Another idea: since PHP 7.4, proc_open() accepts an array directly. Maybe this removes the overhead you're mentioning, since that's the main motivation for this request. Would you like to have a look?

I just tested this:
using proc_open(['/usr/bin/true'], …) is about 2x as fast as proc_open('/usr/bin/true', …)
(1.34ms vs 3.03ms)

test code used:

$start = microtime(true);
for ($i = 0; $i < 1000; ++$i) {
	proc_close(proc_open(['/usr/bin/true'], [], $pipes));
}
var_dump(microtime(true) - $start); // 1.34 ms

$start = microtime(true);
for ($i = 0; $i < 1000; ++$i) {
	proc_close(proc_open('/usr/bin/true', [], $pipes));
}
var_dump(microtime(true) - $start); // 3.03 ms

$start = microtime(true);
proc_close(proc_open('/usr/bin/true'.str_repeat(' && /usr/bin/true', 999), [], $pipes));
var_dump(microtime(true) - $start); // 0.96 ms

As the performance gain of using && chaining compared to using proc_open() with an array is only 28%, moving to the array usage should be enough (and would also improve the speed of single executions). (UPDATE: with PHP 8.3 proc_open() with array is faster than && chaining)

@nicolas-grekas should I open a new issue (or PR) for that? #52409

2023-11-02 UPDATE: PHP 8.3 brings another 2x speedup to the array version, down to 0.7ms on my system.