Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Piping data to a child process with iterator is very slow #20114

Open
codedokode opened this issue Sep 30, 2016 · 1 comment

Comments

Projects
None yet
3 participants
@codedokode
Copy link

commented Sep 30, 2016

If we start a new process using Process component and feed input to it with an iterator, there is a pause of 200ms for every iteration. Code to reproduce the bug (tested on linux):

<?php 

use Symfony\Component\Process\InputStream;
use Symfony\Component\Process\Process;

require_once __DIR__ . '/vendor/autoload.php';

// yields 4 bytes per iteration
$generator = function () {
    $t0 = microtime(true);
    $size = 0;

    for ($i=0; $i < 10000; $i++) { 
        yield "test";
        $size += 4;
        $time = microtime(true) - $t0;
        printf(
            "Written total %d bytes, %.3f bytes/sec\n", 
            $size, 
            $time ? $size / $time : 0
        );
    }
};

$process = new Process("cat > /dev/null");
$process->setTimeout(10);
$process->setInput($generator());

$process->start();
$process->wait();

Output:

Written total 4 bytes, 636.537 bytes/sec
Written total 8 bytes, 422.834 bytes/sec
Written total 12 bytes, 415.141 bytes/sec
Written total 16 bytes, 69.342 bytes/sec
Written total 20 bytes, 46.270 bytes/sec
Written total 24 bytes, 37.902 bytes/sec
Written total 28 bytes, 33.416 bytes/sec
Written total 32 bytes, 30.632 bytes/sec
Written total 36 bytes, 28.769 bytes/sec
Written total 40 bytes, 27.420 bytes/sec
Written total 44 bytes, 26.399 bytes/sec
Written total 48 bytes, 25.615 bytes/sec
Written total 52 bytes, 24.973 bytes/sec
Written total 56 bytes, 24.469 bytes/sec
Written total 60 bytes, 24.086 bytes/sec

As we output 4 bytes for each iteration and there is a delay of 200ms the average speed is 20 bytes/sec (it is higher in the beginning because there is one non-blocking call).

Process component version: v3.1.4, git rev. e64e93041c80e77197ace5ab9385dedb5a143697
Uname -a: Linux debian 3.16.0-4-586 #1 Debian 3.16.7-ckt9-3~deb8u1 (2015-04-24) i686 GNU/Linux

If we look into UnixPipes and AbstractPipes code, we'll see the algorithm in UnixPipes#readAndWrite():

  • if stdin is ready, feed single item from input to it ($w = $this->write();)
  • wait until stdout or stderr becomes ready for 200ms (Process::TIMEOUT_PRECISION * 1E6)
  • repeat

So this algorithm doesn't work well with processes that don't output anything. It would be better to select() all pipes simultaneously and handle whichever one is ready.

As a workaround one can replace blocking call to wait() with non-blocking output iterator:

$iterator = $process->getIterator(Process::ITER_NON_BLOCKING);
foreach ($iterator as $type => $output) {
    // nothing
}

But this leads to high CPU usage when child process doesn't output anything.

@codedokode

This comment has been minimized.

Copy link
Author

commented Sep 30, 2016

Proposed unit test to check read/write speed for pipes (works only on Linux):

<?php 

use Symfony\Component\Process\Exception\ProcessTimedOutException;
use Symfony\Component\Process\Process;

/**
 * Tests that there is no delays when transferring data 
 * between child and parent processes.
 *
 * Test works only on linux.
 */
class ProcessPipeSpeedTest extends \PHPUnit_Framework_TestCase
{

    public function testStdinWriteSpeed()
    {
        $process = new Process('cat > /dev/null');
        $process->setTimeout(5);

        // Check that we can make more than 1000 writes in 5 seconds
        // If we fail, we'll get ProcessTimedOutException
        $process->setInput($this->createInputIterator(1000));
        $process->start();
        $process->wait();
    }

    public function testStdoutReadSpeed()
    {
        $process = new Process('cat /dev/zero');
        $process->setTimeout(5);
        $process->start();

        // test we can make more than 1000 reads in 5 seconds
        $reads = 0;

        foreach ($process->getIterator() as $type => $output) {
            $reads++;
            if ($reads >= 1000) {
                $process->stop();
                $this->assertTrue(true);
                return;
            }
        }

        // We did not manage to make 1000 reads in time
        $this->assertTrue(false);
    }

    /**
     * @return  \Iterator
     */
    private function createInputIterator($writeCount)
    {
        $generator = function () use ($writeCount) {
            for ($i=0; $i < $writeCount; $i++) { 
                yield "x";
            }
        };        

        return $generator();
    }    

    public function testAllPipesTransferSpeed()
    {
        $process = new Process('tee /dev/stderr');
        $process->setTimeout(5);

        // test we can make more than 1000 writes and reads in 5 seconds
        // if we fail, we'll get ProcessTimedOutException
        $process->setInput($this->createInputIterator(1000));
        $process->start();
        foreach ($process->getIterator() as $type => $output) {
            // nothing
        }
    }
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.