Async file downloads (sink option) are triggering complete handler too soon

[sebastiaanluca]

Is it normal that an async request is considered completed almost instantly when using a pool in combination with the sink option to write to a file?

I've tried every bit of code and expect each transfer to finish completely before moving on to the next, especially since I define the number of parallel transfer, but instead it starts downloading the entire queue at once.

use GuzzleHttp\Pool;
use GuzzleHttp\Client;
use GuzzleHttp\Psr7\Request;

$client = new Client();

$requests = function ($total) use($client) {
    $uri = 'url/to/video.mp4';
    for ($i = 0; $i < $total; $i++) {
        // Write remote file to local source
        yield $client->getAsync($link, ['sink' => 'targetFile']);
    }
};

$pool = new Pool($client, $requests(100), [
    'concurrency' => 2,
    'fulfilled' => function ($response, $index) {
        // this is delivered each successful response
    },
    'rejected' => function ($reason, $index) {
        // this is delivered each failed request
    },
]);

// Initiate the transfers and create a promise
$promise = $pool->promise();

// Force the pool of requests to complete.
$promise->wait();

[mtdowling]

Is it normal that an async request is considered completed almost instantly when using a pool in combination with the sink option to write to a file?

No? Can you provide more information? What do you see when you look at debug output? Are you using cURL or the stream wrapper?

[sebastiaanluca]

How can I enable debug output and where can I find it? Not sure what I'm using but the defaults. Simply instantiating a new Guzzle client and requesting a video, then having it write to a file.

To check if it wasn't related to the pool or any other app-specific code, I placed this in the run method of a PHP thread:

$client = new Client();

$client->get($this->source, [
    // Write output to target file
    'sink' => $this->target,
]);

var_dump('request complete!');

Result is that it executes the request and triggers request complete after about half a second. The thread itself does not exit (when complete) until the target file has been completely downloaded. So in effect the GET request returns completed before it should and while the request is still being processed.

[mtdowling]

Are you using threads (like with pthreads)? Guzzle is not thread safe and will not work in a multithreaded application.

On Jul 3, 2016, at 12:53 PM, Sebastiaan Luca notifications@github.com wrote:

How can I enable debug output and where can I find it? Not sure what I'm using but the defaults. Simply instantiating a new Guzzle client and requesting a video, then having it write to a file.

To check if it wasn't related to the pool or any other app-specific code, I placed this in the run method of a PHP thread:

$client = new Client();

$client->get($this->source, [
// Write output to target file
'sink' => $this->target,
]);

var_dump('request complete!');
Result is that it executes the request and triggers request complete after about half a second. The thread itself does not exit (when complete) until the target file has been completely downloaded. So in effect the GET request returns completed before it should and while the request is still being processed.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or mute the thread.

[sebastiaanluca]

Yes, pthreads. And it works, but it has the same issue as when I'm not using threads (which was just a test), so doesn't really matter. Request is made, then I think it completes when it receives the headers, and then the file downloads.

[rbruhn]

I'm experiencing this same issue. The code to dump the ids is executed, then the script doesn't end until all the downloads finish. I normally fire off a Beanstalk job using those ids, but if the files are still downloading it screws things up.

Is there a way to make it wait for the downloads to complete?

$requests = function () use ($api)
{
    foreach ($this->sources as $index => $source)
    {
        $filePath = config('config.classifications_download') . '/' . $index . '.csv';

        yield $index => function($poolOpts) use ($api, $source, $filePath) {
            $reqOpts = [
                'sink' => $filePath
            ];
            if (is_array($poolOpts) && count($poolOpts) > 0) {
                $reqOpts = array_merge($poolOpts, $reqOpts); // req > pool
            }

            return $api->getHttpClient()->getAsync($source, $reqOpts);
        };
    }
};

$responses = Pool::batch($api->getHttpClient(), $requests(), [
    'concurrency' => 10,
    'fulfilled'   => function ($response, $index)
    {
        return $index;
    },
    'rejected'    => function ($reason, $index)
    {
        return $index;
    }
]);

// Handle responses here.... collect ids into array

dd($ids);

// Fire new Beanstalk job using ids.

[Sadikk]

Bumping this, got the same issue today

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed after 2 weeks if no further activity occurs. Thank you for your contributions.