SimpleS3: cURL error "API function called from within callback" when using streams

[derrabus]

Hello 👋🏻

When I use PHP streams to copy a file from one bucket to another, the SimpleS3Client crashes with a very cryptic exception message:

PHP Fatal error:  Uncaught Symfony\Component\HttpClient\Exception\TransportException: API function called from within callback in /path/to/vendor/symfony/http-client/Response/CurlResponse.php:300

The problematic line is this one:

$client->upload('two', 'test.txt', $client->download('one', 'test.txt')->getContentAsResource());

This problem can also be triggered through Flysystem with the AsyncAWS adapter:

$filesystemTwo->writeStream('test.php', $filesystemOne->readStream('test.php'));

Full Reproducer

<?php

declare(strict_types=1);

use AsyncAws\Core\Configuration;
use AsyncAws\S3\Exception\BucketAlreadyExistsException;
use AsyncAws\S3\Exception\BucketAlreadyOwnedByYouException;
use AsyncAws\SimpleS3\SimpleS3Client;

require __DIR__ . '/vendor/autoload.php';

$client = new SimpleS3Client([
    Configuration::OPTION_ENDPOINT => 'http://localhost:9000',
    Configuration::OPTION_ACCESS_KEY_ID => 'minioadmin',
    Configuration::OPTION_SECRET_ACCESS_KEY => 'minioadmin',
    Configuration::OPTION_PATH_STYLE_ENDPOINT => true,
]);

foreach (['one', 'two'] as $bucket) {
    try {
        $client->createBucket(['Bucket' => $bucket]);
    } catch (BucketAlreadyExistsException|BucketAlreadyOwnedByYouException) {
    }
}

$client->upload('one', 'test.txt', 'Hello World');
$client->upload('two', 'test.txt', $client->download('one', 'test.txt')->getContentAsResource());

Full stack trace

PHP Fatal error:  Uncaught Symfony\Component\HttpClient\Exception\TransportException: API function called from within callback in /Volumes/Projects/async-aws-stream-copy-reproducer/vendor/symfony/http-client/Response/CurlResponse.php:300
Stack trace:
#0 /Volumes/Projects/async-aws-stream-copy-reproducer/vendor/symfony/http-client/Response/TransportResponseTrait.php(172): Symfony\Component\HttpClient\Response\CurlResponse::perform(Object(Symfony\Component\HttpClient\Internal\CurlClientState), Array)
#1 [internal function]: Symfony\Component\HttpClient\Response\CurlResponse::stream(Array, 0.0)
#2 /Volumes/Projects/async-aws-stream-copy-reproducer/vendor/symfony/http-client/Response/ResponseStream.php(45): Generator->rewind()
#3 /Volumes/Projects/async-aws-stream-copy-reproducer/vendor/symfony/http-client/Response/AsyncResponse.php(265): Symfony\Component\HttpClient\Response\ResponseStream->rewind()
#4 /Volumes/Projects/async-aws-stream-copy-reproducer/vendor/symfony/http-client/Response/StreamWrapper.php(123): Symfony\Component\HttpClient\Response\AsyncResponse::stream(Array, 0.0)
#5 [internal function]: Symfony\Component\HttpClient\Response\StreamWrapper->stream_read(8192)
#6 /Volumes/Projects/async-aws-stream-copy-reproducer/vendor/async-aws/core/src/Stream/ResourceStream.php(75): fread(Resource id #128, 65536)
#7 /Volumes/Projects/async-aws-stream-copy-reproducer/vendor/symfony/http-client/HttpClientTrait.php(485): AsyncAws\Core\Stream\ResourceStream->getIterator()
#8 [internal function]: Symfony\Component\HttpClient\CurlHttpClient::{closure:Symfony\Component\HttpClient\HttpClientTrait::normalizeBody():485}(Object(AsyncAws\Core\Stream\ResourceStream))
#9 /Volumes/Projects/async-aws-stream-copy-reproducer/vendor/symfony/http-client/HttpClientTrait.php(468): Generator->valid()
#10 /Volumes/Projects/async-aws-stream-copy-reproducer/vendor/symfony/http-client/CurlHttpClient.php(396): Symfony\Component\HttpClient\CurlHttpClient::{closure:{closure:Symfony\Component\HttpClient\HttpClientTrait::normalizeBody():467}:467}(11)
#11 /Volumes/Projects/async-aws-stream-copy-reproducer/vendor/symfony/http-client/CurlHttpClient.php(252): Symfony\Component\HttpClient\CurlHttpClient::readRequestBody(11, Object(Closure), '', false)
#12 [internal function]: Symfony\Component\HttpClient\CurlHttpClient::{closure:Symfony\Component\HttpClient\CurlHttpClient::request():248}(Object(CurlHandle), NULL, 11)
#13 /Volumes/Projects/async-aws-stream-copy-reproducer/vendor/symfony/http-client/CurlHttpClient.php(347): curl_multi_exec(Object(CurlMultiHandle), 0)
#14 /Volumes/Projects/async-aws-stream-copy-reproducer/vendor/symfony/http-client/Response/AsyncResponse.php(265): Symfony\Component\HttpClient\CurlHttpClient->stream(Array, -0.0)
#15 /Volumes/Projects/async-aws-stream-copy-reproducer/vendor/symfony/http-client/Response/AsyncResponse.php(66): Symfony\Component\HttpClient\Response\AsyncResponse::stream(Array, -0.0)
#16 /Volumes/Projects/async-aws-stream-copy-reproducer/vendor/symfony/http-client/Response/CommonResponseTrait.php(144): Symfony\Component\HttpClient\Response\AsyncResponse::{closure:Symfony\Component\HttpClient\Response\AsyncResponse::__construct():60}(Object(Symfony\Component\HttpClient\Response\AsyncResponse), -0.0)
#17 /Volumes/Projects/async-aws-stream-copy-reproducer/vendor/symfony/http-client/Response/AsyncResponse.php(97): Symfony\Component\HttpClient\Response\AsyncResponse::initialize(Object(Symfony\Component\HttpClient\Response\AsyncResponse))
#18 /Volumes/Projects/async-aws-stream-copy-reproducer/vendor/async-aws/core/src/Response.php(147): Symfony\Component\HttpClient\Response\AsyncResponse->getStatusCode()
#19 /Volumes/Projects/async-aws-stream-copy-reproducer/vendor/async-aws/core/src/Response.php(124): AsyncAws\Core\Response->resolve()
#20 /Volumes/Projects/async-aws-stream-copy-reproducer/vendor/async-aws/simple-s3/src/SimpleS3Client.php(200): AsyncAws\Core\Response->__destruct()
#21 /Volumes/Projects/async-aws-stream-copy-reproducer/vendor/async-aws/simple-s3/src/SimpleS3Client.php(92): AsyncAws\SimpleS3\SimpleS3Client->doSmallFileUpload(Array, 'two', 'test.txt', Resource id #128)
#22 /Volumes/Projects/async-aws-stream-copy-reproducer/test.php(27): AsyncAws\SimpleS3\SimpleS3Client->upload('two', 'test.txt', Resource id #128)
#23 {main}

[jderusse]

I think there is a bug in the Symfony's HttpClient curl's implementation (or a bug in curl).

When streaming the body of the UPLOAD request, curl fails to stream the response from the DOWLOAD request.

see symfony/symfony#61716 to track the bug.

The workarround could be to remove these lines

aws/src/Core/src/Response.php

Lines 372 to 374 in 2f6755b

	if (\is_callable([$this->httpResponse, 'toStream'])) {
	return new ResponseBodyResourceStream($this->httpResponse->toStream());
	}

And use the ResponseBodyStream at line 381 that avoids the bug by:

1. stream the original resource in a temp file
2. use that file to provide the body of the request

aws/src/Core/src/Stream/ResponseBodyStream.php

Lines 92 to 99 in 2f6755b

	if (null === $this->fallback) {
	// Use getChunks() to read stream content to $this->fallback
	foreach ($this->getChunks() as $chunk) {
	}
	\assert(null !== $this->fallback);
	}
	return $this->fallback->getContentAsResource();

any opinion @nicolas-grekas

[nicolas-grekas]

That's a limitation of the curl extension. curl would be able to do that but the API exposed to us via the extension doesn't give any options.
Putting stuff in a temporary buffer is the way to go indeed.
An alternative is to use AmphpHttpClient, which can do that.

[derrabus]

I feel like this is a trap. The cURL implementation is picked automatically for me. In fact, I'm using the Flysystem-Adapter for AsyncAWS, so the issue happens two abstraction layers below my actual application code. And the exception that I get is absolutely cryptic and does not at all hint at a possible soloution.

So there's no way we can detect this situation in the cURL client so that we could either work around the issue or raise a more meaningful exception?

[jderusse]

I've opened a PR to remove usage of Response's stream and fix the issue (see #1947)

Drawback: the response is first streamed in a temporary file. so operation like upload(download()->toStream()) would be slowest.