Handling files which require Swift large object storage #24606
Conversation
|
By analyzing the blame information on this pull request, we identified @icewind1991, @butonic, @berendt and @Xenopathic to be potential reviewers |
|
CLA here #19004 (comment) |
|
It works with my local setup. But I'm not that familiar with SWIFT.... |
|
I don't know enough about Swift to be able to make a decision here. |
|
👍 and should be added to https://github.com/owncloud/core/blob/master/lib/private/Files/ObjectStore/Swift.php#L111 as well. in a separate PR |
|
Hmmmshit |
PVince81
force-pushed
the
swift-largeobjectsupport
branch
from
883872e
to
1982d87
Compare
|
Rebased. Had to fix the This fixes the SWIFT tests: |
| $objects = $this->getContainer()->objectList(array( | ||
| 'path' => $defaultSegmentPath | ||
| )); | ||
| $objectPaths = []; |
There was a problem hiding this comment.
return array_map(function($object){
return $object->getName();
}, $objects);😄
|
Needs unit tests (is there a way to force it to use chunking for smaller files?) Also I would expect custom logic for reading files. |
| $this->objectCache->remove($path2 . '/'); | ||
| } else { | ||
| // Large object file | ||
| // TODO: Implement after PHP OpenCloud is upgraded |
There was a problem hiding this comment.
imo it would make more sense to update opencloud before we merge this
merging a half working solution can only lead to unexpected behaviour
There was a problem hiding this comment.
yeah, indeed I also have some concerns about this.
The problem is that he opencloud lib we need for large object storage only supports PHP > 5.4 so we can't update. (from what I heard)
Thought about this a bit. It would mean we'd need a test that uploads a virtual 4-5 Gb file, maybe generated from random data as we don't want to check in such file in git. Due to the many concerns voiced here I'll move this to 9.2 as it seems this isn't ready. |
|
(and with a bit of luck maybe we can drop support of PHP 5.4 in 9.2 😉) |
|
9.2 will only support PHP >= 5.6, so there's a chance to update the lib maybe ? |
|
update php-opencloud #26111 |
At a particular threshold, php-opencloud DLO support is used to upload the object in segments. Large object files are detected when copying, renaming and deleting. Deletions remove any segments that follow the naming convention used by php-opencloud. For now, copy/rename simply fails for large files. The best solution for implementing it involves upgrading php-opencloud.
DeepDiver1975
force-pushed
the
swift-largeobjectsupport
branch
from
1982d87
to
560efc8
Compare
|
to be reconsidered in separate SWIFT app |
|
This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs. |
Thanks @tjdett for this commit, from #19004
At a particular threshold, php-opencloud DLO support is used to upload
the object in segments.
Large object files are detected when copying, renaming and deleting.
Deletions remove any segments that follow the naming convention used by
php-opencloud.
For now, copy/rename simply fails for large files. The best solution for
implementing it involves upgrading php-opencloud.