Skip to content

Comments

Implement chunked uploads#39

Merged
claucambra merged 86 commits intomainfrom
feature/chunked-upload
Jan 15, 2025
Merged

Implement chunked uploads#39
claucambra merged 86 commits intomainfrom
feature/chunked-upload

Conversation

@claucambra
Copy link
Contributor

@claucambra claucambra commented Jan 9, 2025

@claucambra claucambra self-assigned this Jan 9, 2025
@claucambra
Add RemoteFileChunk type
23c9dfd 
Signed-off-by: Claudio Cambra <developer@claudiocambra.com>
@claucambra
Add chunkedUpload method to RemoteInterface protocol
b650cec 
Signed-off-by: Claudio Cambra <developer@claudiocambra.com>
@claucambra
Implement chunkedUpload in nextcloudkit extension
ebff7db 
Signed-off-by: Claudio Cambra <developer@claudiocambra.com>
@claucambra
Make the progressHandler for chunked upload match other methods
9640f58 
Signed-off-by: Claudio Cambra <developer@claudiocambra.com>
@claucambra
Add simple starter test for upload result struct
9ea7e3d 
Signed-off-by: Claudio Cambra <developer@claudiocambra.com>
@claucambra
Add basic chunked upload implementation for mock remote interface
9f29aed 
Signed-off-by: Claudio Cambra <developer@claudiocambra.com>
@claucambra
Add logging when no valid root item on mock remote interface
246b124 
Signed-off-by: Claudio Cambra <developer@claudiocambra.com>
@claucambra
Fix remaining file size in chunked upload implementation of MRI
319e004 
Signed-off-by: Claudio Cambra <developer@claudiocambra.com>
@claucambra
Fix warning about unused response var
5ad26d5 
Signed-off-by: Claudio Cambra <developer@claudiocambra.com>
@claucambra
Add convenience method to get a root mock remote item
85a53b4 
Signed-off-by: Claudio Cambra <developer@claudiocambra.com>

f identifier

Signed-off-by: Claudio Cambra <developer@claudiocambra.com>
@claucambra
Use convenience root item method in tests
8cc9751 
Signed-off-by: Claudio Cambra <developer@claudiocambra.com>

f use rootitem

Signed-off-by: Claudio Cambra <developer@claudiocambra.com>
@claucambra
Add convenience method to create root trash item
94cd66e 
Signed-off-by: Claudio Cambra <developer@claudiocambra.com>
@claucambra
Use convenience method to create root trash item in tests
fac3db1 
Signed-off-by: Claudio Cambra <developer@claudiocambra.com>
@claucambra
Add UploadResult convenience struct
9c0f6cb 
Signed-off-by: Claudio Cambra <developer@claudiocambra.com>
@claucambra
Add starter convenience upload function to perform standard/chunked u…
fa7437c 
…pload automatically

Signed-off-by: Claudio Cambra <developer@claudiocambra.com>
@claucambra
Add basic test to check standard upload in upload convenience func
6cb61f1 
Signed-off-by: Claudio Cambra <developer@claudiocambra.com>
@claucambra
Use correct chunkSize value in uplaod convenience func
a59bd09 
Signed-off-by: Claudio Cambra <developer@claudiocambra.com>
@claucambra
Add simple test for upload convenience func chunked upload
9dc801f 
Signed-off-by: Claudio Cambra <developer@claudiocambra.com>
@claucambra
Start chunk name from 1 imitating the real server
05e446b 
Signed-off-by: Claudio Cambra <developer@claudiocambra.com>
@claucambra
Make chunk upload id an argument in convenience upload func
dbe7c19 
Signed-off-by: Claudio Cambra <developer@claudiocambra.com>
@claucambra
Set chunks at initialisation time in MRI
e7fa1e0 
Signed-off-by: Claudio Cambra <developer@claudiocambra.com>
@claucambra
Call chunkUploadStartedHandler in MRI
d9f6167 
Signed-off-by: Claudio Cambra <developer@claudiocambra.com>
@claucambra
Pass all relevant arguments to inner upload during chunk upload in MRI
fb4f3d9 
Signed-off-by: Claudio Cambra <developer@claudiocambra.com>
@claucambra
Call chunkUploadCompleteHandler in MRI
b5b35ae 
Signed-off-by: Claudio Cambra <developer@claudiocambra.com>
@claucambra
Allow passing in chunkUploadCompleteHandler
1bf5c2b 
Signed-off-by: Claudio Cambra <developer@claudiocambra.com>
@claucambra
Only call chunkUploadCompleteHandler once in MRI
e9feb47 
Signed-off-by: Claudio Cambra <developer@claudiocambra.com>
@claucambra
Add RemoteFileChunk database object type
9ae04df 
Signed-off-by: Claudio Cambra <developer@claudiocambra.com>
@claucambra
Remove previous RemoteFileChunk typealias
04e42f3 
Signed-off-by: Claudio Cambra <developer@claudiocambra.com>
@claucambra
Add convenience to get array of remotefilechunk from the nckit file c…
49275d0 
…hunk tuples

Signed-off-by: Claudio Cambra <developer@claudiocambra.com>
@claucambra
Add test for resumed chunked upload of newly created item
0ea0931 
Signed-off-by: Claudio Cambra <developer@claudiocambra.com>
@claucambra
Modify RemoteInterface to directly take a local path and a remote pat…
1766104 
…h for chunked uploads

This makes the API sensible and consistent with the default upload
method used in NCFPK.

Note that this requires upstream changes in NextcloudKit:
nextcloud/NextcloudKit#116

Signed-off-by: Claudio Cambra <developer@claudiocambra.com>
@claucambra
Delete unused ItemUploadTests
effa38c 
Signed-off-by: Claudio Cambra <developer@claudiocambra.com>
@claucambra
Add a test for chunked upload of modified item contents
258c998 
Signed-off-by: Claudio Cambra <developer@claudiocambra.com>
@claucambra
Add test for modified item chunked upload, resumed
7505ee8 
Signed-off-by: Claudio Cambra <developer@claudiocambra.com>
@claucambra
Properly employ remainingChunks argument in mock remote interface
379513a 
Signed-off-by: Claudio Cambra <developer@claudiocambra.com>
@claucambra
Add expected remaining chunks to database in upload tests for resumed…
9a0cb4f 
… chunked upload

Signed-off-by: Claudio Cambra <developer@claudiocambra.com>
@claucambra
Fix state of remaining chunks in database in resumed chunked upload o…
566ac13 
…f modified item contents

Signed-off-by: Claudio Cambra <developer@claudiocambra.com>
@claucambra
Fix state of remaining chunks in database in resumed chunked upload o…
bc2e5ea 
…f newly created file

Signed-off-by: Claudio Cambra <developer@claudiocambra.com>
@claucambra
Add test for chunked upload of MRI
916b34d 
Signed-off-by: Claudio Cambra <developer@claudiocambra.com>
@claucambra
Temporarily use nckit branch with chunk destination changes
76bd2e1 
Signed-off-by: Claudio Cambra <developer@claudiocambra.com>
@claucambra
Add some logging for chunked upload
704428a 
Signed-off-by: Claudio Cambra <developer@claudiocambra.com>
@claucambra
Ensure we remove illegal chars from file provider item id used as chu…
cceffe1 
…nk upload id

Signed-off-by: Claudio Cambra <developer@claudiocambra.com>
@claucambra
Update item create test with chunked upload to check handling of ille…
e5503e2 
…gal chunk upload ID

Signed-off-by: Claudio Cambra <developer@claudiocambra.com>
@claucambra
Remove period from upload logging
3cc3ac0 
Signed-off-by: Claudio Cambra <developer@claudiocambra.com>
@claucambra
Set custom file chunks directory when using nckit chunked upload
c2b6876 
Signed-off-by: Claudio Cambra <developer@claudiocambra.com>
@claucambra
Add additional logging when starting nckit chunked upload
3e580de 
Signed-off-by: Claudio Cambra <developer@claudiocambra.com>
@claucambra
Ensure chunk upload using nckit occurs on non-main queue
73dd1c8 
Internally this method uses a dispatch semaphore to wait for each chunk
upload to be complete; if this is done on the main queue the process
locks up

Signed-off-by: Claudio Cambra <developer@claudiocambra.com>
@claucambra
Fix issues with urls used in nckit extension for chunked uploads
f2f2d3c 
Signed-off-by: Claudio Cambra <developer@claudiocambra.com>
@claucambra
Remove over-active chunk progress logging
31cbb27 
Signed-off-by: Claudio Cambra <developer@claudiocambra.com>
@claucambra
Simplify returns for upload function
c3bd5bf 
Signed-off-by: Claudio Cambra <developer@claudiocambra.com>
@claucambra
Use setStatusForItemMetadata when setting post-upload metadata state …
a4b2e82 
…in Item content modification

Signed-off-by: Claudio Cambra <developer@claudiocambra.com>
@claucambra claucambra force-pushed the feature/chunked-upload branch from 7250b2f to a4b2e82 Compare January 9, 2025 08:54
@claucambra claucambra added the enhancement New feature or request label Jan 9, 2025
@claucambra claucambra added this to the 2.0.0 milestone Jan 9, 2025
@codecov-commenter
Copy link

codecov-commenter commented Jan 9, 2025 •
edited
Loading

⚠️ Please install the 'codecov app svg image' to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

Attention: Patch coverage is 80.44554% with 158 lines in your changes missing coverage. Please review.

Project coverage is 78.50%. Comparing base (c55db0a) to head (07b82ad).
Report is 87 commits behind head on main.

Files with missing lines Patch % Lines
...erKit/Interface/NextcloudKit+RemoteInterface.swift 0.00% 107 Missing ⚠️
...es/NextcloudFileProviderKit/Utilities/Upload.swift 80.37% 21 Missing ⚠️
...loudFileProviderKit/Metadata/RemoteFileChunk.swift 25.00% 18 Missing ⚠️
Tests/Interface/MockRemoteInterface.swift 86.84% 10 Missing ⚠️
...ileProviderKit/Database/FilesDatabaseManager.swift 75.00% 1 Missing ⚠️
...es/NextcloudFileProviderKit/Item/Item+Modify.swift 96.42% 1 Missing ⚠️

❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files 
@@            Coverage Diff             @@
##             main      #39      +/-   ##
==========================================
- Coverage   78.57%   78.50%   -0.08%     
==========================================
  Files          46       49       +3     
  Lines       10027    10719     +692     
==========================================
+ Hits         7879     8415     +536     
- Misses       2148     2304     +156

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@claucambra
Do not add chunks to database if we have chunkes logged in db
65d913a 
Signed-off-by: Claudio Cambra <developer@claudiocambra.com>
@claucambra
Switch back to develop NextcloudKit branch
07b82ad 
Signed-off-by: Claudio Cambra <developer@claudiocambra.com>
@claucambra claucambra merged commit 5075002 into main Jan 15, 2025
3 checks passed
@Rello Rello deleted the feature/chunked-upload branch August 15, 2025 07:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

@claucambra claucambra

Labels

enhancement New feature or request

Projects

None yet

Milestone

2.0.0

Development

Successfully merging this pull request may close these issues.

2 participants

@claucambra @codecov-commenter