New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

100% of CPU usage on curl_multi_exec #756

Closed
Maykonn opened this Issue Aug 1, 2014 · 8 comments

Comments

Projects
None yet
4 participants
@Maykonn

Maykonn commented Aug 1, 2014

Hi,

I'm getting 100% of CPU usage when executing parallel request with sendAll(). This lock the CPU and the whole system is failing. See in attachment:

upload 1

@mtdowling

This comment has been minimized.

Show comment
Hide comment
@mtdowling

mtdowling Aug 1, 2014

Member

When sending requests in parallel to a low latency server, you will get higher CPU utilization because the select calls made by Guzzle to curl will return almost immediately. This is because you're basically in a tight loop, and the only way to use less CPU would be to force a sleep.

Are you on a fairly low latency connection to the server? How many requests are you running in parallel? Are you uploading data, and if so, how much on average per request? What's the average size of the response bodies you are downloading?

Member

mtdowling commented Aug 1, 2014

When sending requests in parallel to a low latency server, you will get higher CPU utilization because the select calls made by Guzzle to curl will return almost immediately. This is because you're basically in a tight loop, and the only way to use less CPU would be to force a sleep.

Are you on a fairly low latency connection to the server? How many requests are you running in parallel? Are you uploading data, and if so, how much on average per request? What's the average size of the response bodies you are downloading?

@Maykonn

This comment has been minimized.

Show comment
Hide comment
@Maykonn

Maykonn Aug 1, 2014

The servers are into EC2 on same region, so low latency (one server send a group of POST requests to 3 other servers). e.g.:

To server 1:
http://myapp/api/command1.php
http://myapp/api/command1.php
http://myapp/api/command1.php

To server 2:
http://myapp/api/command1.php

To server 3:
http://myapp/api/command1.php
...x50
http://myapp/api/command1.php

What changes are the parameters of requests.

  1. The exactly quantity is unknow, but no more that 50 for each server. Each group of request can have a different quantity of requests.

  2. No upload.

Maykonn commented Aug 1, 2014

The servers are into EC2 on same region, so low latency (one server send a group of POST requests to 3 other servers). e.g.:

To server 1:
http://myapp/api/command1.php
http://myapp/api/command1.php
http://myapp/api/command1.php

To server 2:
http://myapp/api/command1.php

To server 3:
http://myapp/api/command1.php
...x50
http://myapp/api/command1.php

What changes are the parameters of requests.

  1. The exactly quantity is unknow, but no more that 50 for each server. Each group of request can have a different quantity of requests.

  2. No upload.

@mtdowling

This comment has been minimized.

Show comment
Hide comment
@mtdowling

mtdowling Aug 1, 2014

Member

Thanks for the information. Yeah, that's a low latency connection. I've been playing around with the curl handling, but I don't think there's anything wrong with what's implemented in Guzzle.

To prove this, I implemented what Guzzle does in a very basic boiled down script that uses curl directly with no abstractions. This boiled down script also results in nearly 100% CPU utilization when running 50 requests in parallel. When I reduced this to 2 in parallel, it resulted in around 60-70% CPU.

// Yield a bunch of requests
$handleGen = function() {
    for ($i = 0; $i < 100000; $i++) {
        $h = curl_init();
        curl_setopt_array($h, [
            CURLOPT_URL            => 'http://localhost:8125/guzzle-server/perf',
            CURLOPT_POST           => true,
            CURLOPT_POSTFIELDS     => 'foo baz bar',
            CURLOPT_CONNECTTIMEOUT => 150,
            CURLOPT_RETURNTRANSFER => true,
            CURLOPT_HEADER         => true,
        ]);
        yield $h;
    }
};

$mh = curl_multi_init();
$gener = $handleGen();

// Add initial handles to keep a pool size of 50 requests
foreach (new LimitIterator($gener, 0, 50) as $h) {
    curl_multi_add_handle($mh, $h);
}

// Process messages and add more if available in the generator
function process_messages($mh, \Iterator $gener)
{
    $addedRequest = false;

    while ($done = curl_multi_info_read($mh)) {
        curl_multi_remove_handle($mh, $done['handle']);
        curl_close($done['handle']);
        $gener->next();

        if ($gener->current()) {
            echo '.';
            $addedRequest = true;
            curl_multi_add_handle($mh, $gener->current());
        }
    }

    return $addedRequest;
}

do {
    do {
        $mrc = curl_multi_exec($mh, $active);
    } while ($mrc === CURLM_CALL_MULTI_PERFORM);

    if (process_messages($mh, $gener)) {
        $active = true;
    }

    if (curl_multi_select($mh, 1) === -1) {
        usleep(250);
    }

} while ($active);

curl_multi_close($mh);
Member

mtdowling commented Aug 1, 2014

Thanks for the information. Yeah, that's a low latency connection. I've been playing around with the curl handling, but I don't think there's anything wrong with what's implemented in Guzzle.

To prove this, I implemented what Guzzle does in a very basic boiled down script that uses curl directly with no abstractions. This boiled down script also results in nearly 100% CPU utilization when running 50 requests in parallel. When I reduced this to 2 in parallel, it resulted in around 60-70% CPU.

// Yield a bunch of requests
$handleGen = function() {
    for ($i = 0; $i < 100000; $i++) {
        $h = curl_init();
        curl_setopt_array($h, [
            CURLOPT_URL            => 'http://localhost:8125/guzzle-server/perf',
            CURLOPT_POST           => true,
            CURLOPT_POSTFIELDS     => 'foo baz bar',
            CURLOPT_CONNECTTIMEOUT => 150,
            CURLOPT_RETURNTRANSFER => true,
            CURLOPT_HEADER         => true,
        ]);
        yield $h;
    }
};

$mh = curl_multi_init();
$gener = $handleGen();

// Add initial handles to keep a pool size of 50 requests
foreach (new LimitIterator($gener, 0, 50) as $h) {
    curl_multi_add_handle($mh, $h);
}

// Process messages and add more if available in the generator
function process_messages($mh, \Iterator $gener)
{
    $addedRequest = false;

    while ($done = curl_multi_info_read($mh)) {
        curl_multi_remove_handle($mh, $done['handle']);
        curl_close($done['handle']);
        $gener->next();

        if ($gener->current()) {
            echo '.';
            $addedRequest = true;
            curl_multi_add_handle($mh, $gener->current());
        }
    }

    return $addedRequest;
}

do {
    do {
        $mrc = curl_multi_exec($mh, $active);
    } while ($mrc === CURLM_CALL_MULTI_PERFORM);

    if (process_messages($mh, $gener)) {
        $active = true;
    }

    if (curl_multi_select($mh, 1) === -1) {
        usleep(250);
    }

} while ($active);

curl_multi_close($mh);
@Maykonn

This comment has been minimized.

Show comment
Hide comment
@Maykonn

Maykonn Aug 1, 2014

Thanks for answer and tests!

I implement a similar test here and the get same behavior. So, I think that I need to implement it as a qeue, of parallel processes.

Maykonn commented Aug 1, 2014

Thanks for answer and tests!

I implement a similar test here and the get same behavior. So, I think that I need to implement it as a qeue, of parallel processes.

@mtdowling

This comment has been minimized.

Show comment
Hide comment
@mtdowling

mtdowling Aug 1, 2014

Member

No problem. I also had a conversation on Twitter with cURL's creator, and we came to the same conclusion: https://twitter.com/bagder/status/495320333549711360.

I think you could possibly work around this by adding a usleep() to the iterator that yields the parallel requests. Something like:

$gen = function(ClientInterface $client, $total, $url, $body, $skip) {
    for ($i = 0; $i < $total; $i++) {
        if ($i > $skip) {
            usleep(200);
        }
        yield $client->createRequest('PUT', $url, ['body' => $body]);
    }
};

$client->sendAll(
    $gen($client, $total, $url, $body, 50),
    ['parallel' => $parallel]
);

You could even get more fancy and poll the system load average and sleep based on that: http://php.net/manual/en/function.sys-getloadavg.php.

I was able to reduce CPU by adding CURLOPT_MAX_SEND_SPEED_LARGE and CURLOPT_MAX_RECV_SPEED_LARGE to the curl handles. This slows down the transfers, but can be used as a super easy way to self-throttle. See: http://docs.guzzlephp.org/en/latest/faq.html#how-can-i-add-custom-curl-options

All that said, there's nothing that can or should be done in Guzzle by default to work around this.

Member

mtdowling commented Aug 1, 2014

No problem. I also had a conversation on Twitter with cURL's creator, and we came to the same conclusion: https://twitter.com/bagder/status/495320333549711360.

I think you could possibly work around this by adding a usleep() to the iterator that yields the parallel requests. Something like:

$gen = function(ClientInterface $client, $total, $url, $body, $skip) {
    for ($i = 0; $i < $total; $i++) {
        if ($i > $skip) {
            usleep(200);
        }
        yield $client->createRequest('PUT', $url, ['body' => $body]);
    }
};

$client->sendAll(
    $gen($client, $total, $url, $body, 50),
    ['parallel' => $parallel]
);

You could even get more fancy and poll the system load average and sleep based on that: http://php.net/manual/en/function.sys-getloadavg.php.

I was able to reduce CPU by adding CURLOPT_MAX_SEND_SPEED_LARGE and CURLOPT_MAX_RECV_SPEED_LARGE to the curl handles. This slows down the transfers, but can be used as a super easy way to self-throttle. See: http://docs.guzzlephp.org/en/latest/faq.html#how-can-i-add-custom-curl-options

All that said, there's nothing that can or should be done in Guzzle by default to work around this.

@mtdowling mtdowling closed this Aug 1, 2014

@Maykonn

This comment has been minimized.

Show comment
Hide comment
@Maykonn

Maykonn Aug 2, 2014

I'll try to apply the work around and analyze results, some delay must resolve very well. When I have a solution I send you an e-mail to keep you informed about the situation. Thank you for your help!

Maykonn commented Aug 2, 2014

I'll try to apply the work around and analyze results, some delay must resolve very well. When I have a solution I send you an e-mail to keep you informed about the situation. Thank you for your help!

@moon0326

This comment has been minimized.

Show comment
Hide comment
@moon0326

moon0326 Jul 25, 2018

@mtdowling we are having the similar issue. What was your value for CURLOPT_MAX_SEND_SPEED_LARGE and CURLOPT_MAX_RECV_SPEED_LARGE? I'm having hard time understanding how CURLOPT_MAX_RECV_SPEED_LARGE works

moon0326 commented Jul 25, 2018

@mtdowling we are having the similar issue. What was your value for CURLOPT_MAX_SEND_SPEED_LARGE and CURLOPT_MAX_RECV_SPEED_LARGE? I'm having hard time understanding how CURLOPT_MAX_RECV_SPEED_LARGE works

@Chienle

This comment has been minimized.

Show comment
Hide comment
@Chienle

Chienle Jul 26, 2018

tried set MAX_RECV_SPEED_LARGE value to 1024 ?

Chienle commented Jul 26, 2018

tried set MAX_RECV_SPEED_LARGE value to 1024 ?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment