TempManager prevents large files from being uploaded

[cortho]

ownCloud 10.2.0.4
fresh install

On a system with slow disk io (for whatever reason) large uploads cannot be finished since TempManager is cleaning the directory before the upload was completed. The chunks are uploaded completely and the temporary file in the tmp directory has got the correct size.

In my setup I have a disk read of 20M/sec which would allow to read ~72G in one hour.

Before TempManager is cleaning, the tmp directory looks like this:
-rw------- 1 apache apache 0 11. Sep 13:51 oc_tmp_07t3vr
-rw------- 1 apache apache 102400000000 11. Sep 14:11 oc_tmp_07t3vr-.bin
-rw------- 1 apache apache 6553600 11. Sep 13:51 phppEsIzF

Since there is a hard coded timeout of 3600 seconds at https://github.com/owncloud/core/blob/master/lib/private/TempManager.php#L186, the cleanup mechanism is running, even when the upload was not yet completed.

So the with first cron at 00 the files oc_tmp_07t3vr and phppEsIzF were removed and with a second at 15 the last file oc_tmp_07t3vr-.bin was removed as well.

lsof and iotopshow, that httpd is still accessing the .bin-file

Since the upload process still was in progress, ownCloud threw the obvious exception that the file could not be opened:

{
  "reqId": "XXjfu-vbNg7stBCxogkOOAAAACk",
  "level": 3,
  "time": "2019-09-11T13:15:03+00:00",
  "remoteAddr": "127.0.0.1",
  "user": "uploadtest",
  "app": "webdav",
  "method": "PUT",
  "url": "/remote.php/webdav/testdir2019-09-11_13-22/100gb.bin-chunking-1568200952-1954-1953",
  "message": "Caused by: {"Exception":"RuntimeException","Message":"Unable to open /scratch/tmp/oc_tmp_07t3vr-.bin using mode r: fopen(/scratch/tmp/oc_tmp_07t3vr-.bin): failed to open stream: No such file or directory","Code":0,"Trace":"
#0 [internal function]: GuzzleHttp\Psr7\{closure}(2, 'fopen(/scratch/...', '/var/www/ownclo...', 311, Array)                                                         
#1 /var/www/owncloud/apps-external/files_primary_s3/vendor/guzzlehttp/psr7/src/functions.php(311): fopen('/scratch/tmp/oc...', 'r')                                  
#2 /var/www/owncloud/apps-external/files_primary_s3/vendor/guzzlehttp/psr7/src/LazyOpenStream.php(37): GuzzleHttp\Psr7  ry_fopen('/scratch/tmp/oc...', 'r')          
#3 /var/www/owncloud/apps-external/files_primary_s3/vendor/guzzlehttp/psr7/src/StreamDecoratorTrait.php(31): GuzzleHttp\Psr7\LazyOpenStream->createStream()          
#4 /var/www/owncloud/apps-external/files_primary_s3/vendor/guzzlehttp/psr7/src/StreamDecoratorTrait.php(101): GuzzleHttp\Psr7\LazyOpenStream->__get('stream')        
#5 /var/www/owncloud/apps-external/files_primary_s3/vendor/guzzlehttp/psr7/src/LimitStream.php(110): GuzzleHttp\Psr7\LazyOpenStream->tell()                          
#6 /var/www/owncloud/apps-external/files_primary_s3/vendor/guzzlehttp/psr7/src/LimitStream.php(34): GuzzleHttp\Psr7\LimitStream->setOffset(82575360000)              
#7 /var/www/owncloud/apps-external/files_primary_s3/vendor/aws/aws-sdk-php/src/Multipart/AbstractUploader.php(39): GuzzleHttp\Psr7\LimitStream->__construct(Object(GuzzleHttp\Psr7\LazyOpenStream), 10240000, 82575360000)                                                                                                                
#8 /var/www/owncloud/apps-external/files_primary_s3/vendor/aws/aws-sdk-php/src/S3/MultipartUploader.php(110): Aws\Multipart\AbstractUploader->limitPartStream(Object(GuzzleHttp\Psr7\LazyOpenStream))                                                                                                                                     
#9 /var/www/owncloud/apps-external/files_primary_s3/vendor/aws/aws-sdk-php/src/Multipart/AbstractUploader.php(53): Aws\S3\MultipartUploader->createPart(true, 8065)  
#10 /var/www/owncloud/apps-external/files_primary_s3/vendor/aws/aws-sdk-php/src/CommandPool.php(55): Aws\Multipart\AbstractUploader->getUploadCommands(Object(Closure))
#11 [internal function]: Aws\CommandPool->Aws\{closure}(Object(Generator))
#12 /var/www/owncloud/apps-external/files_primary_s3/vendor/guzzlehttp/promises/src/EachPromise.php(187): Generator->next()
#13 /var/www/owncloud/apps-external/files_primary_s3/vendor/guzzlehttp/promises/src/EachPromise.php(213): GuzzleHttp\Promise\EachPromise->advanceIterator()
#14 /var/www/owncloud/apps-external/files_primary_s3/vendor/guzzlehttp/promises/src/EachPromise.php(161): GuzzleHttp\Promise\EachPromise->step(8061)
#15 /var/www/owncloud/apps-external/files_primary_s3/vendor/guzzlehttp/promises/src/Promise.php(203): GuzzleHttp\Promise\EachPromise->GuzzleHttp\Promise\{closure}(Object(Aws\Result))
#16 /var/www/owncloud/apps-external/files_primary_s3/vendor/guzzlehttp/promises/src/Promise.php(156): GuzzleHttp\Promise\Promise::callHandler(1, Object(Aws\Result), Array)
#17 /var/www/owncloud/apps-external/files_primary_s3/vendor/guzzlehttp/promises/src/TaskQueue.php(47): GuzzleHttp\Promise\Promise::GuzzleHttp\Promise\{closure}()
#18 /var/www/owncloud/apps-external/files_primary_s3/vendor/guzzlehttp/promises/src/Promise.php(246): GuzzleHttp\Promise\TaskQueue->run(true)
#19 /var/www/owncloud/apps-external/files_primary_s3/vendor/guzzlehttp/promises/src/Promise.php(223): GuzzleHttp\Promise\Promise->invokeWaitFn()
#20 /var/www/owncloud/apps-external/files_primary_s3/vendor/guzzlehttp/promises/src/Promise.php(267): GuzzleHttp\Promise\Promise->waitIfPending()
#21 /var/www/owncloud/apps-external/files_primary_s3/vendor/guzzlehttp/promises/src/Promise.php(225): GuzzleHttp\Promise\Promise->invokeWaitList()
#22 /var/www/owncloud/apps-external/files_primary_s3/vendor/guzzlehttp/promises/src/Promise.php(62): GuzzleHttp\Promise\Promise->waitIfPending()
#23 /var/www/owncloud/apps-external/files_primary_s3/vendor/guzzlehttp/promises/src/Coroutine.php(65): GuzzleHttp\Promise\Promise->wait()
#24 /var/www/owncloud/apps-external/files_primary_s3/vendor/guzzlehttp/promises/src/Promise.php(246): GuzzleHttp\Promise\Coroutine->GuzzleHttp\Promise\{closure}(true)
#25 /var/www/owncloud/apps-external/files_primary_s3/vendor/guzzlehttp/promises/src/Promise.php(223): GuzzleHttp\Promise\Promise->invokeWaitFn()
#26 /var/www/owncloud/apps-external/files_primary_s3/vendor/guzzlehttp/promises/src/Promise.php(267): GuzzleHttp\Promise\Promise->waitIfPending()
#27 /var/www/owncloud/apps-external/files_primary_s3/vendor/guzzlehttp/promises/src/Promise.php(225): GuzzleHttp\Promise\Promise->invokeWaitList()
#28 /var/www/owncloud/apps-external/files_primary_s3/vendor/guzzlehttp/promises/src/Promise.php(62): GuzzleHttp\Promise\Promise->waitIfPending()
#29 /var/www/owncloud/apps-external/files_primary_s3/vendor/aws/aws-sdk-php/src/S3/ObjectUploader.php(90): GuzzleHttp\Promise\Promise->wait()
#30 /var/www/owncloud/apps-external/files_primary_s3/lib/s3storage.php(129): Aws\S3\ObjectUploader->upload()
#31 /var/www/owncloud/lib/private/Files/ObjectStore/ObjectStoreStorage.php(426): OCA\Files_Primary_S3\S3Storage->writeObject('urn:oid:75326', Resource id #2566)
#32 /var/www/owncloud/lib/private/Files/Stream/Close.php(104): OC\Files\ObjectStore\ObjectStoreStorage->writeBack('/scratch/tmp/oc...')
#33 [internal function]: OC\Files\Stream\Close->stream_close()
#34 /var/www/owncloud/apps/files_external/3rdparty/icewind/streams/src/Wrapper.php(124): fclose(Resource id #604)
#35 /var/www/owncloud/lib/private/Files/Stream/Checksum.php(171): Icewind\Streams\Wrapper->stream_close()
#36 [internal function]: OC\Files\Stream\Checksum->stream_close()
#37 /var/www/owncloud/lib/private/legacy/filechunking.php(182): fclose(Resource id #611)
#38 /var/www/owncloud/apps/dav/lib/Connector/Sabre/File.php(558): OC_FileChunking->file_assemble(Object(OCA\Files_Trashbin\Storage), 'files/testdir20...')
#39 /var/www/owncloud/apps/dav/lib/Connector/Sabre/File.php(152): OCA\DAV\Connector\Sabre\File->createFileChunked(Resource id #434)
#40 /var/www/owncloud/apps/dav/lib/Connector/Sabre/Directory.php(172): OCA\DAV\Connector\Sabre\File->put(Resource id #434)
#41 /var/www/owncloud/lib/composer/sabre/dav/lib/DAV/Server.php(1096): OCA\DAV\Connector\Sabre\Directory->createFile('100gb.bin-chunk...', Resource id #434)
#42 /var/www/owncloud/lib/composer/sabre/dav/lib/DAV/CorePlugin.php(525): Sabre\DAV\Server->createFile('testdir2019-09-...', Resource id #434, NULL)
#43 [internal function]: Sabre\DAV\CorePlugin->httpPut(Object(Sabre\HTTP\Request), Object(Sabre\HTTP\Response))
#44 /var/www/owncloud/lib/composer/sabre/event/lib/EventEmitterTrait.php(105): call_user_func_array(Array, Array)
#45 /var/www/owncloud/lib/composer/sabre/dav/lib/DAV/Server.php(479): Sabre\Event\EventEmitter->emit('method:PUT', Array)
#46 /var/www/owncloud/lib/composer/sabre/dav/lib/DAV/Server.php(254): Sabre\DAV\Server->invokeMethod(Object(Sabre\HTTP\Request), Object(Sabre\HTTP\Response))
#47 /var/www/owncloud/apps/dav/appinfo/v1/webdav.php(65): Sabre\DAV\Server->exec()
#48 /var/www/owncloud/remote.php(165): require_once('/var/www/ownclo...')
#49 {main}","File":"/var/www/owncloud/apps-external/files_primary_s3/vendor/guzzlehttp/psr7/src/functions.php","Line":303}"
}

This issue should not cover the slow io issue, but the fact, that TempManager is arbitrarily removing files using a hard coded value, without any check, if a file is in use.

[DeepDiver1975]

Option 1: add config parameter to allow larger timeout
Option 2: add check if the file is in use - as proposed by OP

[enbrnz]

Hi @cortho, how would I be able to reproduce the issue?
Does the upload just need to take longer than 60 minutes? Or is it only in combination with files_primary_s3?

[cortho]

The upload was basically finished, as I stated in my OP. You would have to throttle hd throughput so assembling the chunks would take more than an hour.

Anyway, I think the issue is clear. There is an arbitrary 1h timeout AND files are removed, even when they are in use