Generator passed to yield from was aborted without proper return and is unable to continue

[keulinho]

When waiting on multiple Results over Result::wait() we randomly got following error:

Fatal error: Uncaught Error: Generator passed to yield from was aborted without proper return and is unable to continue in /var/www/html/vendor/async-aws/core/src/Stream/IterableStream.php:51
Stack trace:
#0 /var/www/html/vendor/symfony/http-client/HttpClientTrait.php(310): AsyncAws\Core\Stream\IterableStream->getIterator()
#1 [internal function]: Symfony\Component\HttpClient\CurlHttpClient::Symfony\Component\HttpClient\{closure}(Object(AsyncAws\Core\Stream\IterableStream))
#2 /var/www/html/vendor/symfony/http-client/HttpClientTrait.php(292): Generator->valid()
#3 /var/www/html/vendor/symfony/http-client/CurlHttpClient.php(462): Symfony\Component\HttpClient\CurlHttpClient::Symfony\Component\HttpClient\{closure}(65536)
#4 /var/www/html/vendor/symfony/http-client/CurlHttpClient.php(247): Symfony\Component\HttpClient\CurlHttpClient::readRequestBody(65536, Object(Closure), '', false)
#5 [internal function]: Symfony\Component\HttpClient\CurlHttpClient::Symfony\Component\HttpClient\{closure}(Resource id #8604, NULL, 65536)
#6 / in /var/www/html/vendor/async-aws/core/src/Stream/IterableStream.php on line 51

The results that are being waited on are a list of PutObjectOutput.

We aren't able to reproduce the error reliably and don't know what may trigger that error.

[jderusse]

I see 2 cases were the IterableStream is involbed. In signature and web application upload a stream.

Can you show how you call the method? `$client->putObject(['Body' => ????]);'

Maybe you can also help me troubleshooting the issue by adding debug trace in IterableStream class:

public function getIterator(): \Traversable
    {
        try {
            yield from $this->content;
        } catch (\Throwable $e) {
            dump($this->content);
            die;
        }
    }

@nicolas-grekas any idea?

[keulinho]

Our code basically looks like this:

$resource = fopen($$file, 'rb');
$putRequest = (new PutObjectRequest())
->setBucket($adapter->getBucket())
->setBody($resource)
....

So we do upload streams.

[jderusse]

Sorry, I'm not able to reproduce the issue nor see where and how it breaks :(

do you have a parallel script that delete or change the content of the file while you are uploading it?
Running out of memory, or disk space?

I would really appreciate to get a reproducer...

[jderusse]

FYI, this is my test

or ($i=0; $i<50; $i++) {
    file_put_contents('/tmp/s3-'.$i, random_bytes(random_int(65530, 65540)));
}

$client = new S3Client();
$responses = [];
for ($i=0; $i<50; $i++) {
    $responses[] = $client->putObject([
        'Bucket' => 'my-bucket',
        'Key' => 'file-'.$i,
        'Body' => fopen('/tmp/s3-'.$i, 'rb'),
    ]);
}
dump('ready');
/** @var PutObjectOutput $response */
foreach (Result::wait($responses) as $response) {
    dump($response->getEtag());
}

[Nyholm]

It must be that this anonymous function is throwing an exception.

How large are the files your are uploading? (ie, does it take a second or 2 minutes?)
How do you authenticate with AWS?

[keulinho]

The biggest files in that workload are images, meaning the biggest files are a few MB, so the upload should be in seconds, if not ms.

The authentication is done over env vars, as described here.

[Nyholm]

Interesting.

And I assume these environment variables are long lived. I know if you are using AWS' docker (ecs?) then the variables change a few times per hour.

Did you try @jderusse suggestion? #950 (comment)

[keulinho]

We are in fact using ECS on fargate, so the secrets should rotate automatically.

I was not able to add the debug statement, as the problem does only pop up in our production environment, there hot patching a composer dependency is not that easy.

[Nyholm]

I only have one wild guess. That I've just been hinting about... I believe that you credentials are renewed during you are executing the code.

If the script takes milliseconds and this happens frequently, then Im wrong (or you are very unlucky). I'll see if I can test this with by manually reset my credentials during the run of @jderusse test script.

[Nyholm]

Nope. Im super wrong. :/

I read an old stackoverflow chat where they mention the Error message you get. The generator is closed, for some reason..

[keulinho]

seems like today we were pretty unlucky.

I think i will create a fork and put some more logging in there.

I'll keep you posted.

[Nyholm]

Thank you. I appreciate that.

[keulinho]

So the dump revealed that $this->content is an instance of \Generator, which is quite obvious 🤦

I'm now trying to dump $this->content->current(), but that should probably return null as the generator is already closed right?

[Sarke]

I got this as well. We are using Backblaze B2 (S3 compatible) and Flysystem, if that makes a difference. Uploading images, about 1 to 4 mb in size.

I just switched from league/flysystem-aws-s3-v3 to league/flysystem-async-aws-s3 to check it out, and got this after about 100 requests or so. I'm not doing any async or waiting, just one at a time to test.

PHP version: 8.0.7
OpenSSL version: OpenSSL 1.1.1d  10 Sep 2019
cURL version: 7.64.0 libz 1.2.11 ssl OpenSSL/1.1.1d

async-aws/core                     1.10.0             Core package to integrate with AWS. This is a lightweight AWS SDK provider by AsyncAws.
async-aws/s3                       1.9.0              S3 client, part of the AWS SDK provided by AsyncAws.
...
league/flysystem                   2.1.1              File storage abstraction for PHP
league/flysystem-async-aws-s3      2.0.6              AsyncAws S3 filesystem adapter for Flysystem.
...
symfony/http-client                v5.3.2             Provides powerful methods to fetch HTTP resources synchronously or asynchronously
symfony/http-client-contracts      v2.4.0             Generic abstractions related to HTTP clients

Exception 'Error' with message 'Generator passed to yield from was aborted without proper return and is unable to continue'

in /app/vendor/async-aws/core/src/Stream/IterableStream.php:51

Stack trace:
#0 /app/vendor/symfony/http-client/HttpClientTrait.php(321): AsyncAws\Core\Stream\IterableStream->getIterator()
#1 [internal function]: Symfony\Component\HttpClient\CurlHttpClient::Symfony\Component\HttpClient\{closure}(Object(AsyncAws\Core\Stream\IterableStream))
#2 /app/vendor/symfony/http-client/HttpClientTrait.php(303): Generator->valid()
#3 /app/vendor/symfony/http-client/CurlHttpClient.php(440): Symfony\Component\HttpClient\CurlHttpClient::Symfony\Component\HttpClient\{closure}(65536)
#4 /app/vendor/symfony/http-client/CurlHttpClient.php(247): Symfony\Component\HttpClient\CurlHttpClient::readRequestBody(65536, Object(Closure), '', false)
#5 [internal function]: Symfony\Component\HttpClient\CurlHttpClient::Symfony\Component\HttpClient\{closure}(Object(CurlHandle), NULL, 65536)
#6 /app/vendor/symfony/http-client/Response/CurlResponse.php(298): curl_multi_exec(Object(CurlMultiHandle), 0)
#7 /app/vendor/symfony/http-client/Response/TransportResponseTrait.php(170): Symfony\Component\HttpClient\Response\CurlResponse::perform(Object(Symfony\Component\HttpClient\Internal\CurlClientState), Array)
#8 [internal function]: Symfony\Component\HttpClient\Response\CurlResponse::stream(Array, NULL)
#9 /app/vendor/symfony/http-client/Response/ResponseStream.php(47): Generator->rewind()
#10 /app/vendor/symfony/http-client/Response/AsyncResponse.php(253): Symfony\Component\HttpClient\Response\ResponseStream->rewind()
#11 /app/vendor/symfony/http-client/Response/AsyncResponse.php(66): Symfony\Component\HttpClient\Response\AsyncResponse::stream(Array)
#12 /app/vendor/symfony/http-client/Response/CommonResponseTrait.php(152): Symfony\Component\HttpClient\Response\AsyncResponse::Symfony\Component\HttpClient\Response\{closure}(Object(Symfony\Component\HttpClient\Response\AsyncResponse))
#13 /app/vendor/symfony/http-client/Response/AsyncResponse.php(93): Symfony\Component\HttpClient\Response\AsyncResponse::initialize(Object(Symfony\Component\HttpClient\Response\AsyncResponse))
#14 /app/vendor/async-aws/core/src/Response.php(127): Symfony\Component\HttpClient\Response\AsyncResponse->getStatusCode()
#15 /app/vendor/async-aws/core/src/Response.php(104): AsyncAws\Core\Response->resolve()
#16 /app/vendor/league/flysystem-async-aws-s3/AsyncAwsS3Adapter.php(329): AsyncAws\Core\Response->__destruct()
#17 /app/vendor/league/flysystem-async-aws-s3/AsyncAwsS3Adapter.php(131): League\Flysystem\AsyncAwsS3\AsyncAwsS3Adapter->upload('851/1228/239958...', '\xFF\xD8\xFF\xE0\x00\x10JFIF\x00\x01\x01\x01\x00...', Object(League\Flysystem\Config))
#18 /app/vendor/league/flysystem/src/Filesystem.php(44): League\Flysystem\AsyncAwsS3\AsyncAwsS3Adapter->write('851/1228/239958...', '\xFF\xD8\xFF\xE0\x00\x10JFIF\x00\x01\x01\x01\x00...', Object(League\Flysystem\Config))

[Sarke]

I think I was able to catch this one. A TransportException is thrown, which is caught and ignored in RetryableHttpClient. The relevant code is this, I believe:

https://github.com/symfony/http-client/blob/d2464f48482223c7c6826cd8c6ed7929d1ce6093/RetryableHttpClient.php#L59-L69

I only looked at it briefly, but I think it's because this retry mechanism doesn't always yield or return. I'm not sure if that is a bug in symfony/http-client or if it's just how it interacts with the AsyncAWS' IterableStream.

Error stack:

ErrorChunk.php:53, Symfony\Component\HttpClient\Chunk\ErrorChunk->isTimeout()
RetryableHttpClient.php:62, Symfony\Component\HttpClient\RetryableHttpClient->Symfony\Component\HttpClient\{closure:/app/vendor/symfony/http-client/RetryableHttpClient.php:59-145}()
AsyncResponse.php:342, Generator->valid()
AsyncResponse.php:342, Symfony\Component\HttpClient\Response\AsyncResponse::passthruStream()
AsyncResponse.php:286, Symfony\Component\HttpClient\Response\AsyncResponse::stream()
AsyncResponse.php:66, Symfony\Component\HttpClient\Response\AsyncResponse::Symfony\Component\HttpClient\Response\{closure:/app/vendor/symfony/http-client/Response/AsyncResponse.php:60-84}()
CommonResponseTrait.php:152, Symfony\Component\HttpClient\Response\AsyncResponse::initialize()
AsyncResponse.php:93, Symfony\Component\HttpClient\Response\AsyncResponse->getStatusCode()
Response.php:127, AsyncAws\Core\Response->resolve()
Response.php:104, AsyncAws\Core\Response->__destruct()
AsyncAwsS3Adapter.php:329, AsyncAws\S3\S3Client->putObject()
AsyncAwsS3Adapter.php:329, League\Flysystem\AsyncAwsS3\AsyncAwsS3Adapter->upload()
AsyncAwsS3Adapter.php:131, League\Flysystem\AsyncAwsS3\AsyncAwsS3Adapter->write()
Filesystem.php:44, League\Flysystem\Filesystem->write()