S3 as primary storage : empty files are impossible to upload #7585

ArnY · 2017-12-20T14:23:08Z

Steps to reproduce

set up a nextcloud instance with a s3 bucket as primary storage
create an empty file on your hard drive
try to upload the empty file or drop the file in your synch'ed folder

Expected behaviour

the file should be uploaded to the nextcloud instance, on the s3 bucket.

Actual behaviour

in both case (upload from web or using the synchronization client), the upload will fail with the following errors

cloudprod3.cprv.univ-nantes.prive/syslog:Dec 20 14:28:10 cloudprod3.cprv.univ-nantes.prive ownCloud[2272]: {objectstore} Could not create object urn:oid:5857935 for files/truc.txt.ocTransferId1292085902.part: {"Exception":"Aws\\S3\\Exception\\S3MultipartUploadException","Message":"An exception occurred while completing a multipart upload: Error executing \"CompleteMultipartUpload\" on \"http:\/\/cloudrgw.ha.univ-nantes.prive\/cloudprod\/urn%3Aoid%3A5857935?uploadId=2~8qS66aBMfjoVHA2bLLWdZstr9wQGnVx\"; AWS HTTP error: Client error response [url] http:\/\/cloudrgw.ha.univ-nantes.prive\/cloudprod\/urn%3Aoid%3A5857935?uploadId=2~8qS66aBMfjoVHA2bLLWdZstr9wQGnVx [status code] 400 [reason phrase] Bad Request MalformedXML (client):  - <?xml version=\"1.0\" encoding=\"UTF-8\"?><Error><Code>MalformedXML<\/Code><BucketName>cloudprod<\/BucketName><RequestId>tx000000000000000120f00-005a3a656a-3727e-default<\/RequestId><HostId>3727e-default-default<\/HostId><\/Error>","Code":0,"Trace":"#0 \/var\/www\/nextcloud\/3rdparty\/guzzlehttp\/promises\/src\/Promise.php(203): Aws\\Multipart\\AbstractUploadManager->Aws\\Multipart\\{closure}(Object(Aws\\S3\\Exception\\S3Exception))\n#1 \/var\/www\/nextcloud\/3rdparty\/guzzlehttp\/promises\/src\/Promise.php(156): GuzzleHttp\\Promise\\Promise::callHandler(2, Object(Aws\\S3\\Exception\\S3Exception), Array)\n#2 \/var\/www\/nextcloud\/3rdparty\/guzzlehttp\/promises\/src\/TaskQueue.php(47): GuzzleHttp\\Promise\\Promise::GuzzleHttp\\Promise\\{closure}()\n#3 \/var\/www\/nextcloud\/3rdparty\/guzzlehttp\/promises\/src\/Promise.php(234): GuzzleHttp\\Promise\\TaskQueue->run()\n#4 \/var\/www\/nextcloud\/3rdparty\/guzzlehttp\/promises\/src\/Promise.php(267): GuzzleHttp\\Promise\\Promise->waitIfPending()\n#5 \/var\/www\/nextcloud\/3rdparty\/guzzlehttp\/promises\/src\/Promise.php(225): GuzzleHttp\\Promise\\Promise->invokeWaitList()\n#6 \/var\/www\/nextcloud\/3rdparty\/guzzlehttp\/promises\/src\/Promise.php(267): GuzzleHttp\\Promise\\Promise->waitIfPending()\n#7 \/var\/www\/nextcloud\/3rdparty\/guzzlehttp\/promises\/src\/Promise.php(225): GuzzleHttp\\Promise\\Promise->invokeWaitList()\n#8 \/var\/www\/nextcloud\/3rdparty\/guzzlehttp\/promises\/src\/Promise.php(62): GuzzleHttp\\Promise\\Promise->waitIfPending()\n#9 \/var\/www\/nextcloud\/3rdparty\/guzzlehttp\/promises\/src\/Coroutine.php(65): GuzzleHttp\\Promise\\Promise->wait()\n#10 \/var\/www\/nextcloud\/3rdparty\/guzzlehttp\/promises\/src\/Promise.php(246): GuzzleHttp\\Promise\\Coroutine->GuzzleHttp\\Promise\\{closure}(true)\n#11 \/var\/www\/nextcloud\/3rdparty\/guzzlehttp\/promises\/src\/Promise.php(223): GuzzleHttp\\Promise\\Promise->invokeWaitFn()\n#12 \/var\/www\/nextcloud\/3rdparty\/guzzlehttp\/promises\/src\/Promise.php(267): GuzzleHttp\\Promise\\Promise->waitIfPending()\n#13 \/var\/www\/nextcloud\/3rdparty\/guzzlehttp\/promises\/src\/Promise.php(225): GuzzleHttp\\Promise\\Promise->invokeWaitList()\n#14 \/var\/www\/nextcloud\/3rdparty\/guzzlehttp\/promises\/src\/Promise.php(62): GuzzleHttp\\Promise\\Promise->waitIfPending()\n#15 \/var\/www\/nextcloud\/3rdparty\/aws\/aws-sdk-php\/src\/Multipart\/AbstractUploadManager.php(83): GuzzleHttp\\Promise\\Promise->wait()\n#16 \/var\/www\/nextcloud\/lib\/private\/Files\/ObjectStore\/S3ObjectTrait.php(105): Aws\\Multipart\\AbstractUploadManager->upload()\n#17 \/var\/www\/nextcloud\/lib\/private\/Files\/ObjectStore\/S3ObjectTrait.php(81): OC\\Files\\ObjectStore\\S3->multiPartUpload('urn:oid:5857935', Resource id #19)\n#18 \/var\/www\/nextcloud\/lib\/private\/Files\/ObjectStore\/ObjectStoreStorage.php(406): OC\\Files\\ObjectStore\\S3->writeObject('urn:oid:5857935', Resource id #19)\n#19 \/var\/www\/nextcloud\/lib\/private\/Files\/ObjectStore\/ObjectStoreStorage.php(306): OC\\Files\\ObjectStore\\ObjectStoreStorage->writeBack('\/uncloud-tmp\/ph...', 'files\/truc.txt....')\n#20 [internal function]: OC\\Files\\ObjectStore\\ObjectStoreStorage->OC\\Files\\ObjectStore\\{closure}()\n#21 \/var\/www\/nextcloud\/3rdparty\/icewind\/streams\/src\/CallbackWrapper.php(109): call_user_func(Object(Closure))\n#22 [internal function]: Icewind\\Streams\\CallbackWrapper->stream_close()\n#23 \/var\/www\/nextcloud\/apps\/dav\/lib\/Connector\/Sabre\/File.php(135): fclose(Resource id #17)\n#24 \/var\/www\/nextcloud\/apps\/dav\/lib\/Connector\/Sabre\/Directory.php(151): OCA\\DAV\\Connector\\Sabre\\File->put(Resource id #13)\n#25 \/var\/www\/nextcloud\/3rdparty\/sabre\/dav\/lib\/DAV\/Server.php(1096): OCA\\DAV\\Connector\\Sabre\\Directory->createFile('truc.txt', Resource id #13)\n#26 \/var\/www\/nextcloud\/3rdparty\/sabre\/dav\/lib\/DAV\/CorePlugin.php(525): Sabre\\DAV\\Server->createFile('files\/abelard-a...', Resource id #13, NULL)\n#27 [internal function]: Sabre\\DAV\\CorePlugin->httpPut(Object(Sabre\\HTTP\\Request), Object(Sabre\\HTTP\\Response))\n#28 \/var\/www\/nextcloud\/3rdparty\/sabre\/event\/lib\/EventEmitterTrait.php(105): call_user_func_array(Array, Array)\n#29 \/var\/www\/nextcloud\/3rdparty\/sabre\/dav\/lib\/DAV\/Server.php(479): Sabre\\Event\\EventEmitter->emit('method:PUT', Array)\n#30 \/var\/www\/nextcloud\/3rdparty\/sabre\/dav\/lib\/DAV\/Server.php(254): Sabre\\DAV\\Server->invokeMethod(Object(Sabre\\HTTP\\Request), Object(Sabre\\HTTP\\Response))\n#31 \/var\/www\/nextcloud\/apps\/dav\/lib\/Server.php(258): Sabre\\DAV\\Server->exec()\n#32 \/var\/www\/nextcloud\/apps\/dav\/appinfo\/v2\/remote.php(33): OCA\\DAV\\Server->exec()\n#33 \/var\/www\/nextcloud\/remote.php(162): require_once('\/var\/www\/nextcl...')\n#34 {main}","File":"\/var\/www\/nextcloud\/3rdparty\/aws\/aws-sdk-php\/src\/Multipart\/AbstractUploadManager.php","Line":136}
cloudprod3.cprv.univ-nantes.prive/syslog:Dec 20 14:28:10 cloudprod3.cprv.univ-nantes.prive ownCloud[2272]: {webdav} Exception: {"Exception":"Sabre\\DAV\\Exception","Message":"An exception occurred while completing a multipart upload: Error executing \"CompleteMultipartUpload\" on \"http:\/\/cloudrgw.ha.univ-nantes.prive\/cloudprod\/urn%3Aoid%3A5857935?uploadId=2~8qS66aBMfjoVHA2bLLWdZstr9wQGnVx\"; AWS HTTP error: Client error response [url] http:\/\/cloudrgw.ha.univ-nantes.prive\/cloudprod\/urn%3Aoid%3A5857935?uploadId=2~8qS66aBMfjoVHA2bLLWdZstr9wQGnVx [status code] 400 [reason phrase] Bad Request MalformedXML (client):  - <?xml version=\"1.0\" encoding=\"UTF-8\"?><Error><Code>MalformedXML<\/Code><BucketName>cloudprod<\/BucketName><RequestId>tx000000000000000120f00-005a3a656a-3727e-default<\/RequestId><HostId>3727e-default-default<\/HostId><\/Error>","Code":0,"Trace":"#0 \/var\/www\/nextcloud\/apps\/dav\/lib\/Connector\/Sabre\/File.php(159): OCA\\DAV\\Connector\\Sabre\\File->convertToSabreException(Object(Aws\\S3\\Exception\\S3MultipartUploadException))\n#1 \/var\/www\/nextcloud\/apps\/dav\/lib\/Connector\/Sabre\/Directory.php(151): OCA\\DAV\\Connector\\Sabre\\File->put(Resource id #13)\n#2 \/var\/www\/nextcloud\/3rdparty\/sabre\/dav\/lib\/DAV\/Server.php(1096): OCA\\DAV\\Connector\\Sabre\\Directory->createFile('truc.txt', Resource id #13)\n#3 \/var\/www\/nextcloud\/3rdparty\/sabre\/dav\/lib\/DAV\/CorePlugin.php(525): Sabre\\DAV\\Server->createFile('files\/abelard-a...', Resource id #13, NULL)\n#4 [internal function]: Sabre\\DAV\\CorePlugin->httpPut(Object(Sabre\\HTTP\\Request), Object(Sabre\\HTTP\\Response))\n#5 \/var\/www\/nextcloud\/3rdparty\/sabre\/event\/lib\/EventEmitterTrait.php(105): call_user_func_array(Array, Array)\n#6 \/var\/www\/nextcloud\/3rdparty\/sabre\/dav\/lib\/DAV\/Server.php(479): Sabre\\Event\\EventEmitter->emit('method:PUT', Array)\n#7 \/var\/www\/nextcloud\/3rdparty\/sabre\/dav\/lib\/DAV\/Server.php(254): Sabre\\DAV\\Server->invokeMethod(Object(Sabre\\HTTP\\Request), Object(Sabre\\HTTP\\Response))\n#8 \/var\/www\/nextcloud\/apps\/dav\/lib\/Server.php(258): Sabre\\DAV\\Server->exec()\n#9 \/var\/www\/nextcloud\/apps\/dav\/appinfo\/v2\/remote.php(33): OCA\\DAV\\Server->exec()\n#10 \/var\/www\/nextcloud\/remote.php(162): require_once('\/var\/www\/nextcl...')\n#11 {main}","File":"\/var\/www\/nextcloud\/apps\/dav\/lib\/Connector\/Sabre\/File.php","Line":567}

Adding just one letter in the file fixes the problem.

I first suspected our radosgw cluster, but it doesn't seem to be at fault since uploading the same empty file with s3cmd succeeds:

~$ s3cmd put test.txt s3://cloudprod
upload: 'toto' -> 's3://cloudprod/test.txt'  [1 of 1]
 0 of 0     0% in    0s     0.00 B/s  done
~$ s3cmd ls s3://cloudprod/test.txt 
2017-12-20 14:02         0   s3://cloudprod/test.txt
~$ s3cmd del s3://cloudprod/test.txt
delete: 's3://cloudprod/test.txt'

Server configuration

Operating system: Debian 9.1

Web server: apache 2.4

Database: mariadb 10.1 + galera 3

PHP version: 7.0

Nextcloud version: 12.0.4 and 13beta1

Updated from an older Nextcloud/ownCloud or fresh install: updated from

Where did you install Nextcloud from: nextcloud.com's download servers

Signing status:

Signing status

Login as admin user into your Nextcloud and access 
http://example.com/index.php/settings/integrity/failed 
paste the results here.

"No errors have been found."

List of activated apps:

App list

If you have access to your command line run e.g.:
sudo -u www-data php occ app:list
from within your Nextcloud installation folder

Enabled:

activity: 2.5.2
apporder: 0.4.1
bruteforcesettings: 1.0.2
comments: 1.2.0
dav: 1.3.0
external: 2.0.3
federatedfilesharing: 1.2.0
federation: 1.2.0
files: 1.7.2
files_pdfviewer: 1.1.1
files_sharing: 1.4.0
files_texteditor: 2.4.1
files_trashbin: 1.2.0
files_versions: 1.5.0
files_videoplayer: 1.1.0
firstrunwizard: 2.1
gallery: 17.0.0
impersonate: 1.0.1
logreader: 2.0.0
lookup_server_connector: 1.0.0
nextcloud_announcements: 1.1
notifications: 2.0.0
oauth2: 1.0.5
password_policy: 1.2.2
provisioning_api: 1.2.0
quota_warning: 1.1.1
serverinfo: 1.2.0
sharebymail: 1.2.0
survey_client: 1.0.0
twofactor_backupcodes: 1.1.1
twofactor_totp: 1.3.1
twofactor_u2f: 1.4.0
updatenotification: 1.2.0
user_ldap: 1.2.1
workflowengine: 1.2.0
Disabled:
admin_audit
encryption
files_external
systemtags
theming
user_external

Nextcloud configuration:

Config report

If you have access to your command line run e.g.:
sudo -u www-data php occ config:list system
from within your Nextcloud installation folder

or 

Insert your config.php content here. 
Make sure to remove all sensitive content such as passwords. (e.g. database password, passwordsalt, secret, smtp password, …)

...
"objectstore": {
"class": "OC\Files\ObjectStore\S3",
"arguments": {
"bucket": "cloudprod",
"autocreate": true,
"key": "XXXXXXXXXXXXXXXXXX",
"secret": "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX",
"hostname": "cloudrgw.ha.univ-nantes.prive",
"port": 80,
"use_ssl": false,
"region": "optional",
"use_path_style": true
}
},
...

Are you using external storage, if yes which one: S3 as primary storage

Are you using encryption: no

Are you using an external user-backend, if yes which one: LDAP

The text was updated successfully, but these errors were encountered:

MorrisJobke · 2017-12-22T10:42:10Z

cc @icewind1991

icewind1991 · 2017-12-22T10:53:23Z

This is a limitation of amazon and not something we can do much about

ArnY · 2017-12-22T17:47:05Z

Actually, i'm not using amazon's s3 but radosgw, not sure this makes a difference. But I also showed in the ticket that s3cmd does upload the empty files without any trouble so it doesn't looks the problem is on the storage side but indeed in nextcloud.

Marlinc · 2017-12-24T17:14:04Z

It appears I've ran into the same issue with radosgw. It only happened after upgrading to Nextcloud 12.0.4

KroniK907 · 2017-12-29T21:13:31Z

I have also run into this issue with S3 as my primary storage. Appears to be new since my upgrade to 12.0.4

simie7 · 2018-01-08T09:23:28Z

I confirm that we have same issue after upgrade to 12.0.4. We use CEPH as primary S3 storage.

ibr123 · 2018-01-21T13:02:22Z

Hi,
I have the same issue, I have and my server is linked to Amazon S3, I downloaded the latest of NextCloud server which is after the date of this issue, yet the problem still occurs
Please refer to this issue
Thanks

icewind1991 · 2018-01-24T16:24:22Z

Please see if #8033 fixes the problem for you

ArnY · 2018-01-25T08:32:22Z

It does indeed fix the problem! Thanks. Can we expect that commit to be backported into the next 12.0 version?

icewind1991 · 2018-01-29T15:13:39Z

Fix should be available in 12.0.6 and 13.0.1

ibr123 · 2018-01-29T15:38:43Z

Hi, from where I can download these versions?

MorrisJobke · 2018-01-29T16:49:07Z

They are not released yet. Planned release is mid march.

MorrisJobke · 2018-02-08T13:30:35Z

Fixed via #8033 (master) and there are also the backport to 12.0.6 and 13.0.1 linked.

MorrisJobke added bug feature: filesystem labels Dec 22, 2017

MorrisJobke closed this as completed Dec 22, 2017

ibr123 mentioned this issue Jan 21, 2018

Multipart Exception at Amazon S3 #7919

Closed

icewind1991 reopened this Jan 24, 2018

icewind1991 mentioned this issue Jan 24, 2018

Use S3Client::upload instead of splitting single/multipart upload ourselves #8033

Merged

MorrisJobke added this to the Nextcloud 14 milestone Jan 24, 2018

icewind1991 closed this as completed Jan 29, 2018

ibr123 mentioned this issue Feb 11, 2018

AWS - MalformedPOSTRequest #7910

Closed

nextcloud-bot mentioned this issue Jun 27, 2018

S3 Primary storage backup #10022

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

S3 as primary storage : empty files are impossible to upload #7585

S3 as primary storage : empty files are impossible to upload #7585

ArnY commented Dec 20, 2017 •

edited

MorrisJobke commented Dec 22, 2017

icewind1991 commented Dec 22, 2017

ArnY commented Dec 22, 2017

Marlinc commented Dec 24, 2017

KroniK907 commented Dec 29, 2017

simie7 commented Jan 8, 2018

ibr123 commented Jan 21, 2018

icewind1991 commented Jan 24, 2018

ArnY commented Jan 25, 2018

icewind1991 commented Jan 29, 2018

ibr123 commented Jan 29, 2018

MorrisJobke commented Jan 29, 2018

MorrisJobke commented Feb 8, 2018

S3 as primary storage : empty files are impossible to upload #7585

S3 as primary storage : empty files are impossible to upload #7585

Comments

ArnY commented Dec 20, 2017 • edited

Steps to reproduce

Expected behaviour

Actual behaviour

Server configuration

MorrisJobke commented Dec 22, 2017

icewind1991 commented Dec 22, 2017

ArnY commented Dec 22, 2017

Marlinc commented Dec 24, 2017

KroniK907 commented Dec 29, 2017

simie7 commented Jan 8, 2018

ibr123 commented Jan 21, 2018

icewind1991 commented Jan 24, 2018

ArnY commented Jan 25, 2018

icewind1991 commented Jan 29, 2018

ibr123 commented Jan 29, 2018

MorrisJobke commented Jan 29, 2018

MorrisJobke commented Feb 8, 2018

ArnY commented Dec 20, 2017 •

edited