S3 multipart upload always fails silently (v2)

[by225]

Trying to upload a 7G file to S3 using SDK v2.8.12 on PHP 5.4.40. File uploads less than 5G work great, but multipart uploads have never worked. They always end with no message or error at all (bypassing register_shutdown_function), usually just before the upload should complete.

Appears to be a problem with cURL multi. Usually, execution ends just before uploading the last group of parts.

I have full S3 access.

Code is nothing special:

$uploader = UploadBuilder::newInstance()
    ->setBucket($bucket_nm)
    ->setKey($key)
    ->setMinPartSize(100 * 1024 * 1024)
    ->setConcurrency(3)
    ->setSource($src_path)
    ->setClient($s3)
    ->build();
try {
    $uploader->getEventDispatcher()->addListener(
    $uploader::AFTER_PART_UPLOAD,
      function($event) {
            $msg = $event['state']->count() . ' parts uploaded.';
            echo "$msg<br />";
            WriteToLog($msg);
      }
    );
    $uploader->upload();
    $msg = 'Upload complete.';
} catch (MultipartUploadException $e) {
    $uploader->abort();
    $msg = 'Upload failed. ' . $e->getMessage() . '.';
}
echo "$msg<br />";
WriteToLog($msg);

[jeremeamia]

Thanks for reporting this. It may take us some time to look into it and try to reproduce the issue. If we have any follow up questions, we'll let you know.

[jeremeamia]

Also, if you have the time and desire to try out the multipart uploader in Version 3, that would help us determine if the issue is isolated to just Version 2. The V3 docs for this feature are here.

[by225]

Ended up using the Ruby SDK to upload the original 7G file. It worked the first time, though apparently without concurrency.

Just tested PHP SDK v3 on a different server with PHP 5.5.

With the region an empty string, I got: An exception occurred while initiating a multipart upload.

With the region set explicitly to the default value, the script exits with a fatal error: Uncaught exception 'GuzzleHttp\Promise\RejectionException' with message 'The promise was rejected with reason: Invoking the wait callback did not resolve the promise' in .../GuzzleHttp/Promise/functions.php:111

I'm not calling promise() directly. Just using basic upload from: http://docs.aws.amazon.com/aws-sdk-php/v3/guide/service/s3-multipart-upload.html

[jeremeamia]

Thanks for giving that a shot. We'll have to do some research into this.

[onethumb]

We're also seeing the GuzzleHttp\Promise\RejectionException issue intermittently and haven't yet determined the root cause.

[teseo]

It is also happening to v3 and with pretty small files (in my case 3KB file). Exception:
An exception occurred while completing a multipart upload.
I'm using (3.3.0) of aws/aws-sdk-php

[jeskew]

@teseo can you post the full exception message?

[teseo]

Exception message:

An exception occurred while completing a multipart upload.

Exception Trace:

#0 /path/vendor/guzzlehttp/promises/src/Promise.php(199): Aws\Multipart\AbstractUploader->Aws\Multipart\{closure}(Object(Aws\S3\Exception\S3Exception))
#1 /path/vendor/guzzlehttp/promises/src/Promise.php(152): GuzzleHttp\Promise\Promise::callHandler(2, Object(Aws\S3\Exception\S3Exception), Array)
#2 /path/vendor/guzzlehttp/promises/src/TaskQueue.php(60): GuzzleHttp\Promise\Promise::GuzzleHttp\Promise\{closure}()
#3 /path/vendor/guzzlehttp/guzzle/src/Handler/CurlMultiHandler.php(96): GuzzleHttp\Promise\TaskQueue->run()
#4 /path/vendor/guzzlehttp/guzzle/src/Handler/CurlMultiHandler.php(123): GuzzleHttp\Handler\CurlMultiHandler->tick()
#5 /path/vendor/guzzlehttp/promises/src/Promise.php(240): GuzzleHttp\Handler\CurlMultiHandler->execute(true)
#6 /path/vendor/guzzlehttp/promises/src/Promise.php(217): GuzzleHttp\Promise\Promise->invokeWaitFn()
#7 /path/vendor/guzzlehttp/promises/src/Promise.php(261): GuzzleHttp\Promise\Promise->waitIfPending()
#8 /path/vendor/guzzlehttp/promises/src/Promise.php(219): GuzzleHttp\Promise\Promise->invokeWaitList()
#9 /path/vendor/guzzlehttp/promises/src/Promise.php(261): GuzzleHttp\Promise\Promise->waitIfPending()
#10 /path/vendor/guzzlehttp/promises/src/Promise.php(219): GuzzleHttp\Promise\Promise->invokeWaitList()
#11 /path/vendor/guzzlehttp/promises/src/Promise.php(62): GuzzleHttp\Promise\Promise->waitIfPending()
#12 /path/vendor/aws/aws-sdk-php/src/Multipart/AbstractUploader.php(86): GuzzleHttp\Promise\Promise->wait()
#13 /path/vendor/path/Job/MoveToAwsJob.php(121): Aws\Multipart\AbstractUploader->upload()
#14 /path/vendor/slm/queue-beanstalkd/src/SlmQueueBeanstalkd/Worker/BeanstalkdWorker.php(33): Module\Job\MoveToAwsJob->execute()
#15 /path/vendor/slm/queue/src/SlmQueue/Strategy/ProcessQueueStrategy.php(70): SlmQueueBeanstalkd\Worker\BeanstalkdWorker->processJob(Object(Module\Job\MoveToAwsJob), Object(SlmQueueBeanstalkd\Queue\BeanstalkdQueue))
#16 [internal function]: SlmQueue\Strategy\ProcessQueueStrategy->onJobProcess(Object(SlmQueue\Worker\WorkerEvent))
#17 /path/vendor/zendframework/zend-eventmanager/src/EventManager.php(444): call_user_func(Array, Object(SlmQueue\Worker\WorkerEvent))
#18 /path/vendor/zendframework/zend-eventmanager/src/EventManager.php(205): Zend\EventManager\EventManager->triggerListeners('process.job', Object(SlmQueue\Worker\WorkerEvent), Array)
#19 /path/vendor/slm/queue/src/SlmQueue/Strategy/ProcessQueueStrategy.php(55): Zend\EventManager\EventManager->trigger('process.job', Object(SlmQueue\Worker\WorkerEvent))
#20 [internal function]: SlmQueue\Strategy\ProcessQueueStrategy->onJobPop(Object(SlmQueue\Worker\WorkerEvent))
#21 /path/vendor/zendframework/zend-eventmanager/src/EventManager.php(444): call_user_func(Array, Object(SlmQueue\Worker\WorkerEvent))
#22 /path/vendor/zendframework/zend-eventmanager/src/EventManager.php(205): Zend\EventManager\EventManager->triggerListeners('process.queue', Object(SlmQueue\Worker\WorkerEvent), Array)
#23 /path/vendor/slm/queue/src/SlmQueue/Worker/AbstractWorker.php(46): Zend\EventManager\EventManager->trigger('process.queue', Object(SlmQueue\Worker\WorkerEvent))
#24 /path/vendor/slm/queue/src/SlmQueue/Controller/AbstractWorkerController.php(49): SlmQueue\Worker\AbstractWorker->processQueue(Object(SlmQueueBeanstalkd\Queue\BeanstalkdQueue), Array)
#25 /path/vendor/zendframework/zend-mvc/src/Controller/AbstractActionController.php(82): SlmQueue\Controller\AbstractWorkerController->processAction()
#26 [internal function]: Zend\Mvc\Controller\AbstractActionController->onDispatch(Object(Zend\Mvc\MvcEvent))
#27 /path/vendor/zendframework/zend-eventmanager/src/EventManager.php(444): call_user_func(Array, Object(Zend\Mvc\MvcEvent))
#28 /path/vendor/zendframework/zend-eventmanager/src/EventManager.php(205): Zend\EventManager\EventManager->triggerListeners('dispatch', Object(Zend\Mvc\MvcEvent), Object(Closure))
#29 /path/vendor/zendframework/zend-mvc/src/Controller/AbstractController.php(118): Zend\EventManager\EventManager->trigger('dispatch', Object(Zend\Mvc\MvcEvent), Object(Closure))
#30 /path/vendor/zendframework/zend-mvc/src/DispatchListener.php(93): Zend\Mvc\Controller\AbstractController->dispatch(Object(Zend\Console\Request), Object(Zend\Console\Response))
#31 [internal function]: Zend\Mvc\DispatchListener->onDispatch(Object(Zend\Mvc\MvcEvent))
#32 /path/vendor/zendframework/zend-eventmanager/src/EventManager.php(444): call_user_func(Array, Object(Zend\Mvc\MvcEvent))
#33 /path/vendor/zendframework/zend-eventmanager/src/EventManager.php(205): Zend\EventManager\EventManager->triggerListeners('dispatch', Object(Zend\Mvc\MvcEvent), Object(Closure))
#34 /path/vendor/zendframework/zend-mvc/src/Application.php(314): Zend\EventManager\EventManager->trigger('dispatch', Object(Zend\Mvc\MvcEvent), Object(Closure))
#35 /path/public/index.php(17): Zend\Mvc\Application->run()
#36 {main}"

[jeskew]

@teseo It looks like one of the parts did not upload successfully within the configured retry limits on your client. You can keep track of upload state and retry the unsuccessful uploads:

$source = '/path/to/large/file.zip';
$uploader = new MultipartUploader($s3Client, $source, [
    'bucket' => 'your-bucket',
    'key'    => 'my-file.zip',
]);

do {
    try {
        $result = $uploader->upload();
    } catch (MultipartUploadException $e) {
        $uploader = new MultipartUploader($s3Client, $source, [
            'state' => $e->getState(),
        ]);
    }
} while (!isset($result));

It sounds like you're using a multipart upload for very small files, which would make your code more complex than it needs to be. If you're not sure whether an object is large enough to warrant a multipart upload, you could try using the upload method on Aws\S3\S3Client, which will make the decision for you.

[teseo]

Hello Jeskew,

Thank you very much for the responses. My current code looks like this:

php
$options = array(
'params' => array(
'ServerSideEncryption' => 'aws:kms',
'SSEKMSKeyId' => $keyIdAlias,
));
$options['before_initiate'] = function ($command) use ($options) {
foreach ($options['params'] as $k => $v) {
$command[$k] = $v;
}
};

            $uploader = new MultipartUploader($s3Client, $fileName, [
                    'Bucket' => $bucket,
                    'Key'    =>  $keyName,
                    'acl'    => 'private'
                ] + $options);


            try{
                echo "3 Let's upload the file ..." . PHP_EOL;

                $result = $uploader->upload();

                if(!empty($result['ObjectURL'])){
                    echo "4 object uploaded!...." . PHP_EOL;


                } else {
                    echo "3 Transfer failed!...." . PHP_EOL;
                }

            }catch (\Exception $e){
                echo "Failed in step 3" . PHP_EOL;
                var_dump($e->getMessage());
            }

The reason I'm doing this and not with the regular S3Client and upload is that when I was sending big files it actually did not enter the if where the multipart upload works, in the other hand the uploaded file was a text file with the big file location. I decided to use Multipart Upload as I didn't know this makes a difference with big and small files. I also found easier to integrate my KMS system with S3 on v3 this way.

[teseo]

I can see int the S3Client upload method the next:

     * @param mixed  $body    Object data to upload. Can be a
     *                        StreamInterface, PHP stream resource, or a
     *                        string of data to upload.

When I used it and I was passing the path to a large file, it understood I tried to send a text file with the location of the large file as content.

[teseo]

I can also see that MultipartUploader extends from AbstractUploader class and this class uses the method determineSource to manipulate the $source variable like this:

php
/**
* Turns the provided source into a stream and stores it.
*
* If a string is provided, it is assumed to be a filename, otherwise, it
* passes the value directly to Psr7\stream_for().
*
* @param mixed $source
*
* @return Stream
*/
private function determineSource($source)
{
// Use the contents of a file as the data source.
if (is_string($source)) {
$source = Psr7\try_fopen($source, 'r');
}

    // Create a source stream.
    $stream = Psr7\stream_for($source);
    if (!$stream->isReadable()) {
        throw new IAE('Source stream must be readable.');
    }

    return $stream;
}

While S3Client directly goes for 

``` php```
        // Perform the required operations to upload the S3 Object.
        $body = Psr7\stream_for($body);

Without determining what source it is. I don't know if this is for a reason or is a bug.

[jeskew]

@teseo The multipart uploader will accept a path for backwards-compatibility reasons, but there was no such BC concern on S3Client's upload method.

[jeskew]

I believe this was fixed in 2b43a45. @Beejer, are you still experiencing this issue?

[teseo]

@jeskew from your response, what would be the ideal way to upload a few Gigs file, through
S3Client->upload() and passing a string with the file location?

[jeskew]

@teseo You should call fopen on the location and pass the handle to S3Client->upload().

[by225]

I still get the same results with 2.8.19: the last set of parts does not upload, and the operation ends with no error.

[jeskew]

Ah I didn't realize you were still on v2.

[teseo]

@jeskew passing fopen to S3Client worked perfectly for me. It could be great to specify in the documentation fopen('/path-to-file.ext') insteand of '/path-to-file.ext' when calling S3Client.

Thanks for your support!

[jeskew]

After doing some more research, I was unable to reproduce the error exactly as described, but I did observe a multipart upload entering a "hanging" state wherein a curl_multi_select call would never resolve. (Something similar is described in this SO post.) I was unable to force the v3 multipart uploader to trigger a similar error after extensive testing, and the RejectionException reported above appears to have been addressed by 2b43a45.

register_shutdown_function being bypassed is a strong indicator that the issue is not caused by userland code like the SDK but rather by curl_multi. @Beejer if you're still experiencing this error, is there a chance that the PHP process is hanging and being killed by an external process?

[jeskew]

Closing, as there hasn't been any recent activity and all signs point to the initial diagnosis (something going on with curl_multi) being correct. The "hanging" described above only happened after I downgraded to PHP 5.4 and went away when I reinstalled a current version of PHP, so that could be tried if the issue pops up again.

Please feel free to reopen if you have any questions or concerns.