use tusd datastorages #4148

butonic · 2023-09-01T14:13:47Z

We are currently ... misusing tusd. Misusing because we implemented our own tusd DataStore. This prevents us from using the existing tusd datastore implementations, eg. the filestore or s3store.

We already require calling InitializeUpload via cs3 before being able to send PATCH requests to the dataprovider. To integrate with tusd we should instead use the PreUploadCreateCallback and PreFinishResponseCallback. Actually, only the latter. The dataprovider never receive a POST directly, so the PreUploadCreateCallback is never called. Instead we need to make InitializeUpload call the same steps as the tusd unrouted handler. Then whe can hand off the complete tus process to tusd and wait for it to make the PreFinishResponseCallback call, where we then calculate checksums and create the actual decomposedfs node.

For this to work we cannot use tusds FileInfo.Storage because it is overwritten when calling NewUpload. So we have to put any metadata into the FileInfo.Metadata map. The latter is also persisted as X-Amz-Meta- in s3.

This PoC works and allows uploading directly to a minio s3 bucket ... with hardcoded credentials ... well ... it is a PoC
TODO

make s3 options configurable
fix tests
add key prefix option to be able to store uploads and blobs in same bucket
move callbacks to upload package and delete unused code
added metadata is exposed in HEAD requests, filter it out? eg the dir reveals the full path even for share recipients. this is already a problem on edge
prefix all incoming metadata in InitiateUpload()
check if storing metadata in X-Amz-Meta- is a problem?
not if metadata is filtered when it leaves ocis, eg. in the response of HEAD requests
pass trace id via metadata
test performance in kubernetes

Noteworthy changes

We need to return a fileid after initiating an upload. Currently, we roll the new uuid for new files and put it in the uplod metadata. ...
The problem: If there are two concurrent requests trying to create the same child they would receive different fileids.

We can no longer store the new nodeid, postprocessing status or any other metadata in the tus upload metadata. After creating the upload we have to use the node metadata itself (which actually makes sense because all bytes have been transferred).

This requires us to already roll the node id and create a file node in InitializeUpload, and mark it as in processing.
This in turn causes new files to show up in directory listings even if not all bytes have been transferred, yet.

hmmm ... how does the curren tcode deal with the problem?

it stores info.Storage["NodeExists"] = "false"
this causes initNewNode to be called
which would then fail with an AlreadyExists errtype when trying to create the parent symlink

we could do the same in InitiateUpload ... using the symlink creation for atomicity instead of having to use a filelock

simple upload needs to initialize the file upload with the filesize

We currently have a way to decide how to initialize the SizeDeferred flag of a tus upload. We cannot use 0 because empty files do exist. This causes the sample upload to fail ...

InitiateFileUploadRequest does not even have a size proerty ...

update-docs · 2023-09-01T14:13:51Z

Thanks for opening this pull request! The maintainers of this repository would appreciate it if you would create a changelog item based on your changes.

butonic · 2023-09-19T10:40:08Z

@micbar so the tests expect only one version to exist when a file is uploaded with the same mtime. but the code now keeps track of every revision:

  Scenario: upload the same file twice with the same mtime and a version is available                                                                # /drone/src/tmp/testrunner/tests/acceptance/features/coreApiVersions/fileVersions.feature:251
    Given user "Alice" has uploaded file "filesForUpload/textfile.txt" to "file.txt" with mtime "Thu, 08 Aug 2019 04:18:13 GMT" using the WebDAV API # FeatureContext::userHasUploadedFileToWithMtimeUsingTheWebdavApi()
    When user "Alice" uploads file "filesForUpload/textfile.txt" to "file.txt" with mtime "Thu, 08 Aug 2019 04:18:13 GMT" using the WebDAV API       # FeatureContext::userUploadsFileToWithMtimeUsingTheWebdavApi()
    Then the HTTP status code should be "204"                                                                                                        # FeatureContext::thenTheHTTPStatusCodeShouldBe()
    And the version folder of file "/file.txt" for user "Alice" should contain "1" element                                                           # FilesVersionsContext::theVersionFolderOfFileShouldContainElements()
      could not find 1 version element(s) in 
      <?xml version="1.0"?>
      <d:multistatus xmlns:s="http://sabredav.org/ns" xmlns:d="DAV:" xmlns:oc="http://owncloud.org/ns"><d:response><d:href>/remote.php/dav/meta/1284d238-aa92-42ce-bdc4-0b0000009157$88f60b76-eb21-103d-957c-43ed0cdaa8aa%21a0cc9d8f-5794-41e5-9e27-282999ba5df3/</d:href><d:propstat><d:prop><d:getetag/></d:prop><d:status>HTTP/1.1 404 Not Found</d:status></d:propstat></d:response><d:response><d:href>/remote.php/dav/meta/1284d238-aa92-42ce-bdc4-0b0000009157$88f60b76-eb21-103d-957c-43ed0cdaa8aa%21a0cc9d8f-5794-41e5-9e27-282999ba5df3/v/a0cc9d8f-5794-41e5-9e27-282999ba5df3.REV.2023-09-19T10:17:51.960713985Z</d:href><d:propstat><d:prop><d:getetag>"b1e23aeabf59d10f5daadc021b988fd5"</d:getetag></d:prop><d:status>HTTP/1.1 200 OK</d:status></d:propstat></d:response><d:response><d:href>/remote.php/dav/meta/1284d238-aa92-42ce-bdc4-0b0000009157$88f60b76-eb21-103d-957c-43ed0cdaa8aa%21a0cc9d8f-5794-41e5-9e27-282999ba5df3/v/a0cc9d8f-5794-41e5-9e27-282999ba5df3.REV.2023-09-19T10:17:51.926984925Z</d:href><d:propstat><d:prop><d:getetag>"a431c21b9be6bd1327e1f06b40c74d0b"</d:getetag></d:prop><d:status>HTTP/1.1 200 OK</d:status></d:propstat></d:response></d:multistatus>
 
      Failed asserting that 2 matches expected 1.

  Scenario: upload the same file more than twice with the same mtime and only one version is available                                               # /drone/src/tmp/testrunner/tests/acceptance/features/coreApiVersions/fileVersions.feature:258
    Given user "Alice" has uploaded file "filesForUpload/textfile.txt" to "file.txt" with mtime "Thu, 08 Aug 2019 04:18:13 GMT" using the WebDAV API # FeatureContext::userHasUploadedFileToWithMtimeUsingTheWebdavApi()
    And user "Alice" has uploaded file "filesForUpload/textfile.txt" to "file.txt" with mtime "Thu, 08 Aug 2019 04:18:13 GMT" using the WebDAV API   # FeatureContext::userHasUploadedFileToWithMtimeUsingTheWebdavApi()
    When user "Alice" uploads file "filesForUpload/textfile.txt" to "file.txt" with mtime "Thu, 08 Aug 2019 04:18:13 GMT" using the WebDAV API       # FeatureContext::userUploadsFileToWithMtimeUsingTheWebdavApi()
    Then the HTTP status code should be "204"                                                                                                        # FeatureContext::thenTheHTTPStatusCodeShouldBe()
    And the version folder of file "/file.txt" for user "Alice" should contain "1" element                                                           # FilesVersionsContext::theVersionFolderOfFileShouldContainElements()
      could not find 1 version element(s) in 
      <?xml version="1.0"?>
      <d:multistatus xmlns:s="http://sabredav.org/ns" xmlns:d="DAV:" xmlns:oc="http://owncloud.org/ns"><d:response><d:href>/remote.php/dav/meta/1284d238-aa92-42ce-bdc4-0b0000009157$8911b43e-eb21-103d-957f-43ed0cdaa8aa%214cfb2093-1f4c-485a-bcff-69c601d5d93c/</d:href><d:propstat><d:prop><d:getetag/></d:prop><d:status>HTTP/1.1 404 Not Found</d:status></d:propstat></d:response><d:response><d:href>/remote.php/dav/meta/1284d238-aa92-42ce-bdc4-0b0000009157$8911b43e-eb21-103d-957f-43ed0cdaa8aa%214cfb2093-1f4c-485a-bcff-69c601d5d93c/v/4cfb2093-1f4c-485a-bcff-69c601d5d93c.REV.2023-09-19T10:17:52.178843967Z</d:href><d:propstat><d:prop><d:getetag>"9b010889f9a3f345b2d872d6c8abe030"</d:getetag></d:prop><d:status>HTTP/1.1 200 OK</d:status></d:propstat></d:response><d:response><d:href>/remote.php/dav/meta/1284d238-aa92-42ce-bdc4-0b0000009157$8911b43e-eb21-103d-957f-43ed0cdaa8aa%214cfb2093-1f4c-485a-bcff-69c601d5d93c/v/4cfb2093-1f4c-485a-bcff-69c601d5d93c.REV.2023-09-19T10:17:52.144869863Z</d:href><d:propstat><d:prop><d:getetag>"7349ebd02bc0ba490ece6bab4c95ac15"</d:getetag></d:prop><d:status>HTTP/1.1 200 OK</d:status></d:propstat></d:response><d:response><d:href>/remote.php/dav/meta/1284d238-aa92-42ce-bdc4-0b0000009157$8911b43e-eb21-103d-957f-43ed0cdaa8aa%214cfb2093-1f4c-485a-bcff-69c601d5d93c/v/4cfb2093-1f4c-485a-bcff-69c601d5d93c.REV.2023-09-19T10:17:52.108406347Z</d:href><d:propstat><d:prop><d:getetag>"90d1fa707994f4f6a53232b538bb1bd9"</d:getetag></d:prop><d:status>HTTP/1.1 200 OK</d:status></d:propstat></d:response></d:multistatus>
     
      Failed asserting that 3 matches expected 1.

  Scenario: upload the same file twice with the same mtime and no version after restoring                                                            # /drone/src/tmp/testrunner/tests/acceptance/features/coreApiVersions/fileVersions.feature:266
    Given user "Alice" has uploaded file "filesForUpload/textfile.txt" to "file.txt" with mtime "Thu, 08 Aug 2019 04:18:13 GMT" using the WebDAV API # FeatureContext::userHasUploadedFileToWithMtimeUsingTheWebdavApi()
    And user "Alice" has uploaded file "filesForUpload/textfile.txt" to "file.txt" with mtime "Thu, 08 Aug 2019 04:18:13 GMT" using the WebDAV API   # FeatureContext::userHasUploadedFileToWithMtimeUsingTheWebdavApi()
    When user "Alice" restores version index "1" of file "/file.txt" using the WebDAV API                                                            # FilesVersionsContext::userRestoresVersionIndexOfFile()
    Then the HTTP status code should be "204"                                                                                                        # FeatureContext::thenTheHTTPStatusCodeShouldBe()
    And the version folder of file "/file.txt" for user "Alice" should contain "0" element                                                           # FilesVersionsContext::theVersionFolderOfFileShouldContainElements()
      could not find 0 version element(s) in 
      <?xml version="1.0"?>
      <d:multistatus xmlns:s="http://sabredav.org/ns" xmlns:d="DAV:" xmlns:oc="http://owncloud.org/ns"><d:response><d:href>/remote.php/dav/meta/1284d238-aa92-42ce-bdc4-0b0000009157$89344148-eb21-103d-9582-43ed0cdaa8aa%21546c80fa-3ed2-4cea-ae9c-0e33a552c7b2/</d:href><d:propstat><d:prop><d:getetag/></d:prop><d:status>HTTP/1.1 404 Not Found</d:status></d:propstat></d:response><d:response><d:href>/remote.php/dav/meta/1284d238-aa92-42ce-bdc4-0b0000009157$89344148-eb21-103d-9582-43ed0cdaa8aa%21546c80fa-3ed2-4cea-ae9c-0e33a552c7b2/v/546c80fa-3ed2-4cea-ae9c-0e33a552c7b2.REV.2023-09-19T10:17:52.333974461Z</d:href><d:propstat><d:prop><d:getetag>"e32940ea5bae7aa1271e92cf6d38fde2"</d:getetag></d:prop><d:status>HTTP/1.1 200 OK</d:status></d:propstat></d:response><d:response><d:href>/remote.php/dav/meta/1284d238-aa92-42ce-bdc4-0b0000009157$89344148-eb21-103d-9582-43ed0cdaa8aa%21546c80fa-3ed2-4cea-ae9c-0e33a552c7b2/v/546c80fa-3ed2-4cea-ae9c-0e33a552c7b2.REV.2019-08-08T04:18:13Z</d:href><d:propstat><d:prop><d:getetag>"b2516215c4b4930ea43a608875ac06f4"</d:getetag></d:prop><d:status>HTTP/1.1 200 OK</d:status></d:propstat></d:response></d:multistatus>
      
      Failed asserting that 2 matches expected 0.

I'm not sure if we want this keep this arguably broken revision tracking?

butonic · 2023-09-19T11:29:14Z

Thesize diff calculation might be racy:

there is an existing file with size 10
upload A is started and writes a 5 byte file - this will set the file size to 5 when all bytes are uploaded
upload B is started and writes a 15 byte file - this will set the file size to 15 when all bytes are uploaded
all bytes for B arrived. size is 15 postprocessing B is started
all bytes for A arrived. size is 5 postprocessing A is started
postprocessing B fails, so the sizediff of 15 is subtracted ... an the file now has size -10?

micbar · 2023-09-19T11:50:47Z

postprocessing B fails, so the sizediff of 15 is subtracted ... an the file now has size -10?

No, the result is 15

10 + 5 + 15 -15 = 15

butonic · 2023-09-20T18:45:53Z

so the error seems to be that the revision has no blobid attribute set ...

{"level":"error","error":"error reading blobid xattr: xattr.get /drone/src/tmp/reva/data/spaces/28/175508-ec25-103d-820e-9b9f4a005ed7/nodes/bd/1b/06/73/-b87d-49f8-bcab-535b42cc37dc.REV.2023-09-20T17:16:18.608228057Z.mlock user.ocis.blobid: no data available","revision":"/drone/src/tmp/reva/data/spaces/28/175508-ec25-103d-820e-9b9f4a005ed7/nodes/bd/1b/06/73/-b87d-49f8-bcab-535b42cc37dc.REV.2023-09-20T17:16:18.608228057Z.mlock","time":"2023-09-20T17:16:26.941046467Z","message":"error reading blobid attribute"}
�[90m2023-09-20 17:16:26.941�[0m �[1m�[31mERR�[0m�[0m �[1m../../../internal/grpc/services/storageprovider/storageprovider.go:981�[0m�[36m >�[0m failed to empty recycle �[36merror=�[0m�[31m"error reading blobid xattr: xattr.get /drone/src/tmp/reva/data/spaces/28/175508-ec25-103d-820e-9b9f4a005ed7/nodes/bd/1b/06/73/-b87d-49f8-bcab-535b42cc37dc.REV.2023-09-20T17:16:18.608228057Z.mlock user.ocis.blobid: no data available"�[0m �[36mkey=�[0m/ �[36mpid=�[0m12 �[36mpkg=�[0mrgrpc �[36mreference=�[0m{"path":".","resource_id":{"opaque_id":"28175508-ec25-103d-820e-9b9f4a005ed7","space_id":"28175508-ec25-103d-820e-9b9f4a005ed7","storage_id":"1284d238-aa92-42ce-bdc4-0b0000009157"}} �[36mstatus=�[0m{"code":15,"message":"error emptying recycle:error reading blobid xattr: xattr.get /drone/src/tmp/reva/data/spaces/28/175508-ec25-103d-820e-9b9f4a005ed7/nodes/bd/1b/06/73/-b87d-49f8-bcab-535b42cc37dc.REV.2023-09-20T17:16:18.608228057Z.mlock user.ocis.blobid: no data available","trace":"038deb2cd375cbe1eccb9e2e361a3c9b"} �[36mtraceid=�[0ma675d2a0acf71398263689c8bda6c54c
�[90m2023-09-20 17:16:26.941�[0m �[33mDBG�[0m �[1m../../../internal/grpc/interceptors/log/log.go:69�[0m�[36m >�[0m unary �[36mcode=�[0mOK �[36mend=�[0m"20/Sep/2023:17:16:26 +0000" �[36mfrom=�[0mtcp://127.0.0.1:53086 �[36mpid=�[0m12 �[36mpkg=�[0mrgrpc �[36mstart=�[0m"20/Sep/2023:17:16:26 +0000" �[36mtime_ns=�[0m10453361 �[36mtraceid=�[0ma675d2a0acf71398263689c8bda6c54c �[36muri=�[0m/cs3.storage.provider.v1beta1.ProviderAPI/PurgeRecycle �[36muser-agent=�[0mGuzzleHttp/7
�[90m2023-09-20 17:16:26.941�[0m �[33mDBG�[0m �[1m../../../internal/grpc/interceptors/log/log.go:69�[0m�[36m >�[0m unary �[36mcode=�[0mOK �[36mend=�[0m"20/Sep/2023:17:16:26 +0000" �[36mfrom=�[0mtcp://127.0.0.1:44532 �[36mpid=�[0m10 �[36mpkg=�[0mrgrpc �[36mstart=�[0m"20/Sep/2023:17:16:26 +0000" �[36mtime_ns=�[0m11953954 �[36mtraceid=�[0m3e1880e0396e5774c6c5ad2c2f55a4fd �[36muri=�[0m/cs3.gateway.v1beta1.GatewayAPI/PurgeRecycle �[36muser-agent=�[0mGuzzleHttp/7
�[90m2023-09-20 17:16:26.942�[0m �[1m�[31mERR�[0m�[0m �[1m../../../internal/http/services/owncloud/ocdav/errors/error.go:174�[0m�[36m >�[0m Internal Server Error �[36mcode=�[0m500 �[36mitem_path=�[0m/ �[36mkey=�[0m �[36mpid=�[0m9 �[36mpkg=�[0mrhttp �[36mreference=�[0m{"path":".","resource_id":{"opaque_id":"28175508-ec25-103d-820e-9b9f4a005ed7","space_id":"28175508-ec25-103d-820e-9b9f4a005ed7","storage_id":"1284d238-aa92-42ce-bdc4-0b0000009157"}} �[36mstatus=�[0m{"code":15,"message":"error emptying recycle:error reading blobid xattr: xattr.get /drone/src/tmp/reva/data/spaces/28/175508-ec25-103d-820e-9b9f4a005ed7/nodes/bd/1b/06/73/-b87d-49f8-bcab-535b42cc37dc.REV.2023-09-20T17:16:18.608228057Z.mlock user.ocis.blobid: no data available","trace":"038deb2cd375cbe1eccb9e2e361a3c9b"} �[36mtraceid=�[0m038deb2cd375cbe1eccb9e2e361a3c9b
�[90m2023-09-20 17:16:26.942�[0m �[1m�[31mERR�[0m�[0m �[1m../../../internal/http/interceptors/log/log.go:112�[0m�[36m >�[0m http �[36mend=�[0m"20/Sep/2023:17:16:26 +0000" �[36mhost=�[0m192.168.29.5 �[36mmethod=�[0mDELETE �[36mpid=�[0m9 �[36mpkg=�[0mrhttp �[36mproto=�[0mHTTP/1.1 �[36msize=�[0m0 �[36mstart=�[0m"20/Sep/2023:17:16:26 +0000" �[36mstatus=�[0m500 �[36mtime_ns=�[0m29996616 �[36mtraceid=�[0m038deb2cd375cbe1eccb9e2e361a3c9b �[36muri=�[0m/remote.php/dav/trash-bin/Alice �[36murl=�[0m/

butonic · 2023-10-16T09:16:40Z

we should move the code to get the list of uploads from decompesedfs to a tus package ... uploads are not storagedriver specific, but tus datastore specific. we might contribute them upstream if they want them.

Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de> filter metadata Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de> calculate sizediff on demand Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de> fix empty bucketid Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de> add changelog Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de> drop unnecessary flag from cleanup Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de> fix tests Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de> drop unused variable Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de> fix unit tests Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de> move Finalize to upload package Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de> move more functions to upload package Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de> propagate sizediff on BytesReady and Revert on failure Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de> reuse tus datastore from fs Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de> always set file length when initiating uploads Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de> log error if we cannot terminate upload Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de> make linter happy Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de> always send upload length Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de> use s3ng config for datastore as well Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de> allow configuring upload object prefixes and temp dir Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de> fix s3ng test config Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de> fix chunking v1 Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de> fix chunking v1 --sign open existing file Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de> restore size diff calculation Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de> treat 0 byte uploads as size deferred Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de> return empty reader for 0 byte files Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de> comment change that always sends filesize in upload opaque Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de> remove commented code Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de> fix legacy chunking on s3 Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de> comment size to check if that causes nc tests te fail Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de> run single failing test Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de> drop unused option Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de> ignore mlock files when iterating revisions Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>

butonic · 2023-11-01T14:26:26Z

I rebased on top of latest reva. no big difference, looking at the current errors.

I don't know what is going on with the S3NG driver. IT does not seem to change the checksum ... in the log all cheksums are the same:

2023-11-01 13:40:45.882 DBG ../../../pkg/storage/utils/decomposedfs/node/node.go:836 > AsResourceInfo node=091bc15c-f6d4-42fb-841e-d9e8d468431a pid=11 pkg=rgrpc ri={"arbitrary_metadata":{"metadata":{"http://owncloud.org/ns/favorite":""}},"checksum":{"sum":"da39a3ee5e6b4b0d3255bfef95601890afd80709","type":4},"etag":"\"1112d204aeb097963bb7355bd420ec19\"","id":{"opaque_id":"091bc15c-f6d4-42fb-841e-d9e8d468431a","space_id":"ddc2004c-0977-11eb-9d3f-a793888cd0f8"},"mime_type":"text/plain","mtime":{"nanos":237217658,"seconds":1698846039},"name":"testfile.txt","opaque":{"map":{"adler32":{"decoder":"plain","value":"MDAwMDAwMDE="},"md5":{"decoder":"plain","value":"ZDQxZDhjZDk4ZjAwYjIwNGU5ODAwOTk4ZWNmODQyN2U="}}},"owner":{"idp":"http://localhost:20080","opaque_id":"ddc2004c-0977-11eb-9d3f-a793888cd0f8","type":1},"parent_id":{"opaque_id":"88934515-d122-4682-b09c-69253e277d3c","space_id":"ddc2004c-0977-11eb-9d3f-a793888cd0f8"},"path":"testfile.txt","permission_set":{"add_grant":true,"create_container":true,"delete":true,"deny_grant":true,"get_path":true,"get_quota":true,"initiate_file_download":true,"initiate_file_upload":true,"list_container":true,"list_file_versions":true,"list_grants":true,"list_recycle":true,"move":true,"purge_recycle":true,"remove_grant":true,"restore_file_version":true,"restore_recycle_item":true,"stat":true,"update_grant":true},"size":4,"type":1} traceid=16a577dc4a5dfad5934a582c4f03c6d8
2023-11-01 13:40:46.39 DBG ../../../pkg/storage/utils/decomposedfs/node/node.go:836 > AsResourceInfo node=4123baa3-aad8-4a65-8583-cc51b29c67ae pid=11 pkg=rgrpc ri={"arbitrary_metadata":{"metadata":{"http://owncloud.org/ns/favorite":""}},"checksum":{"sum":"da39a3ee5e6b4b0d3255bfef95601890afd80709","type":4},"etag":"\"8646f95ab819510f114ec92492ae34cc\"","id":{"opaque_id":"4123baa3-aad8-4a65-8583-cc51b29c67ae","space_id":"ddc2004c-0977-11eb-9d3f-a793888cd0f8"},"mime_type":"text/plain","mtime":{"nanos":983751921,"seconds":1698846045},"name":"testfile.txt","opaque":{"map":{"adler32":{"decoder":"plain","value":"MDAwMDAwMDE="},"md5":{"decoder":"plain","value":"ZDQxZDhjZDk4ZjAwYjIwNGU5ODAwOTk4ZWNmODQyN2U="}}},"owner":{"idp":"http://localhost:20080","opaque_id":"ddc2004c-0977-11eb-9d3f-a793888cd0f8","type":1},"parent_id":{"opaque_id":"c47dce33-4955-415c-ae8a-398ad423e36e","space_id":"ddc2004c-0977-11eb-9d3f-a793888cd0f8"},"path":"testfile.txt","permission_set":{"add_grant":true,"create_container":true,"delete":true,"deny_grant":true,"get_path":true,"get_quota":true,"initiate_file_download":true,"initiate_file_upload":true,"list_container":true,"list_file_versions":true,"list_grants":true,"list_recycle":true,"move":true,"purge_recycle":true,"remove_grant":true,"restore_file_version":true,"restore_recycle_item":true,"stat":true,"update_grant":true},"size":4,"type":1} traceid=21a0e077cfa1b71aa48fe53b147175ea
2023-11-01 13:40:46.921 DBG ../../../pkg/storage/utils/decomposedfs/node/node.go:836 > AsResourceInfo node=cd5b25f4-49e7-4318-9bb7-3c3523073646 pid=11 pkg=rgrpc ri={"arbitrary_metadata":{"metadata":{"http://owncloud.org/ns/favorite":""}},"checksum":{"sum":"da39a3ee5e6b4b0d3255bfef95601890afd80709","type":4},"etag":"\"d31836684af382b6ccb39310494b7b51\"","id":{"opaque_id":"cd5b25f4-49e7-4318-9bb7-3c3523073646","space_id":"ddc2004c-0977-11eb-9d3f-a793888cd0f8"},"mime_type":"text/plain","mtime":{"nanos":508119371,"seconds":1698846046},"name":"testfile.txt","opaque":{"map":{"adler32":{"decoder":"plain","value":"MDAwMDAwMDE="},"md5":{"decoder":"plain","value":"ZDQxZDhjZDk4ZjAwYjIwNGU5ODAwOTk4ZWNmODQyN2U="}}},"owner":{"idp":"http://localhost:20080","opaque_id":"ddc2004c-0977-11eb-9d3f-a793888cd0f8","type":1},"parent_id":{"opaque_id":"a461a9a3-edfc-43a8-bb5b-2ea198921f46","space_id":"ddc2004c-0977-11eb-9d3f-a793888cd0f8"},"path":"testfile.txt","permission_set":{"add_grant":true,"create_container":true,"delete":true,"deny_grant":true,"get_path":true,"get_quota":true,"initiate_file_download":true,"initiate_file_upload":true,"list_container":true,"list_file_versions":true,"list_grants":true,"list_recycle":true,"move":true,"purge_recycle":true,"remove_grant":true,"restore_file_version":true,"restore_recycle_item":true,"stat":true,"update_grant":true},"size":4,"type":1} traceid=2f98d91cc260627d96f20b35c726f832
2023-11-01 13:40:46.956 DBG ../../../pkg/storage/utils/decomposedfs/node/node.go:836 > AsResourceInfo node=cd5b25f4-49e7-4318-9bb7-3c3523073646 pid=11 pkg=rgrpc ri={"arbitrary_metadata":{"metadata":{"http://owncloud.org/ns/favorite":""}},"checksum":{"sum":"da39a3ee5e6b4b0d3255bfef95601890afd80709","type":4},"etag":"\"d31836684af382b6ccb39310494b7b51\"","id":{"opaque_id":"cd5b25f4-49e7-4318-9bb7-3c3523073646","space_id":"ddc2004c-0977-11eb-9d3f-a793888cd0f8"},"mime_type":"text/plain","mtime":{"nanos":508119371,"seconds":1698846046},"name":"testfile.txt","opaque":{"map":{"adler32":{"decoder":"plain","value":"MDAwMDAwMDE="},"md5":{"decoder":"plain","value":"ZDQxZDhjZDk4ZjAwYjIwNGU5ODAwOTk4ZWNmODQyN2U="}}},"owner":{"idp":"http://localhost:20080","opaque_id":"ddc2004c-0977-11eb-9d3f-a793888cd0f8","type":1},"parent_id":{"opaque_id":"a461a9a3-edfc-43a8-bb5b-2ea198921f46","space_id":"ddc2004c-0977-11eb-9d3f-a793888cd0f8"},"path":"testfile.txt","permission_set":{"add_grant":true,"create_container":true,"delete":true,"deny_grant":true,"get_path":true,"get_quota":true,"initiate_file_download":true,"initiate_file_upload":true,"list_container":true,"list_file_versions":true,"list_grants":true,"list_recycle":true,"move":true,"purge_recycle":true,"remove_grant":true,"restore_file_version":true,"restore_recycle_item":true,"stat":true,"update_grant":true},"size":4,"type":1} traceid=0073648fc16c532e76b63bd839e14249
2023-11-01 13:40:47.457 DBG ../../../pkg/storage/utils/decomposedfs/node/node.go:836 > AsResourceInfo node=269512c8-fc14-403a-90a7-80ea27e5acb9 pid=11 pkg=rgrpc ri={"arbitrary_metadata":{"metadata":{"http://owncloud.org/ns/favorite":""}},"checksum":{"sum":"da39a3ee5e6b4b0d3255bfef95601890afd80709","type":4},"etag":"\"c6b07d58f65560996be6104130000cd1\"","id":{"opaque_id":"269512c8-fc14-403a-90a7-80ea27e5acb9","space_id":"ddc2004c-0977-11eb-9d3f-a793888cd0f8"},"mime_type":"text/plain","mtime":{"nanos":56569403,"seconds":1698846047},"name":"testfile.txt","opaque":{"map":{"adler32":{"decoder":"plain","value":"MDAwMDAwMDE="},"md5":{"decoder":"plain","value":"ZDQxZDhjZDk4ZjAwYjIwNGU5ODAwOTk4ZWNmODQyN2U="}}},"owner":{"idp":"http://localhost:20080","opaque_id":"ddc2004c-0977-11eb-9d3f-a793888cd0f8","type":1},"parent_id":{"opaque_id":"0dee8e04-c092-422d-a9ee-84e27a167e5c","space_id":"ddc2004c-0977-11eb-9d3f-a793888cd0f8"},"path":"testfile.txt","permission_set":{"add_grant":true,"create_container":true,"delete":true,"deny_grant":true,"get_path":true,"get_quota":true,"initiate_file_download":true,"initiate_file_upload":true,"list_container":true,"list_file_versions":true,"list_grants":true,"list_recycle":true,"move":true,"purge_recycle":true,"remove_grant":true,"restore_file_version":true,"restore_recycle_item":true,"stat":true,"update_grant":true},"size":4,"type":1} traceid=4e739bd3e6f86fdeb04ac1d433d8bd8e
2023-11-01 13:40:47.933 DBG ../../../pkg/storage/utils/decomposedfs/node/node.go:836 > AsResourceInfo node=b3bcba28-1e94-4b05-9b47-79f61a8ace07 pid=11 pkg=rgrpc ri={"arbitrary_metadata":{"metadata":{"http://owncloud.org/ns/favorite":""}},"checksum":{"sum":"da39a3ee5e6b4b0d3255bfef95601890afd80709","type":4},"etag":"\"50f41d31401bec2f8194e15e871c00ae\"","id":{"opaque_id":"b3bcba28-1e94-4b05-9b47-79f61a8ace07","space_id":"ddc2004c-0977-11eb-9d3f-a793888cd0f8"},"mime_type":"text/plain","mtime":{"nanos":471408166,"seconds":1698846047},"name":"testfile2.txt","opaque":{"map":{"adler32":{"decoder":"plain","value":"MDAwMDAwMDE="},"md5":{"decoder":"plain","value":"ZDQxZDhjZDk4ZjAwYjIwNGU5ODAwOTk4ZWNmODQyN2U="}}},"owner":{"idp":"http://localhost:20080","opaque_id":"ddc2004c-0977-11eb-9d3f-a793888cd0f8","type":1},"parent_id":{"opaque_id":"0dee8e04-c092-422d-a9ee-84e27a167e5c","space_id":"ddc2004c-0977-11eb-9d3f-a793888cd0f8"},"path":"testfile2.txt","permission_set":{"add_grant":true,"create_container":true,"delete":true,"deny_grant":true,"get_path":true,"get_quota":true,"initiate_file_download":true,"initiate_file_upload":true,"list_container":true,"list_file_versions":true,"list_grants":true,"list_recycle":true,"move":true,"purge_recycle":true,"remove_grant":true,"restore_file_version":true,"restore_recycle_item":true,"stat":true,"update_grant":true},"size":4,"type":1} traceid=38da573821a8243502f25f5e4a75ff7b
2023-11-01 13:40:48.417 DBG ../../../pkg/storage/utils/decomposedfs/node/node.go:836 > AsResourceInfo node=705c459d-ccb6-44d1-bd3f-db5d97f6f5e4 pid=11 pkg=rgrpc ri={"arbitrary_metadata":{"metadata":{"http://owncloud.org/ns/favorite":""}},"checksum":{"sum":"da39a3ee5e6b4b0d3255bfef95601890afd80709","type":4},"etag":"\"4aa0ca37b7bfa01b8afb0f9c27ffcc29\"","id":{"opaque_id":"705c459d-ccb6-44d1-bd3f-db5d97f6f5e4","space_id":"ddc2004c-0977-11eb-9d3f-a793888cd0f8"},"mime_type":"text/plain","mtime":{"nanos":11574237,"seconds":1698846048},"name":"testfile.txt","opaque":{"map":{"adler32":{"decoder":"plain","value":"MDAwMDAwMDE="},"md5":{"decoder":"plain","value":"ZDQxZDhjZDk4ZjAwYjIwNGU5ODAwOTk4ZWNmODQyN2U="}}},"owner":{"idp":"http://localhost:20080","opaque_id":"ddc2004c-0977-11eb-9d3f-a793888cd0f8","type":1},"parent_id":{"opaque_id":"32859e68-58d1-47a9-aca5-f276b683a418","space_id":"ddc2004c-0977-11eb-9d3f-a793888cd0f8"},"path":"testfile.txt","permission_set":{"add_grant":true,"create_container":true,"delete":true,"deny_grant":true,"get_path":true,"get_quota":true,"initiate_file_download":true,"initiate_file_upload":true,"list_container":true,"list_file_versions":true,"list_grants":true,"list_recycle":true,"move":true,"purge_recycle":true,"remove_grant":true,"restore_file_version":true,"restore_recycle_item":true,"stat":true,"update_grant":true},"size":4,"type":1} traceid=84b47d669f9d2fe0382cff1220fe8940
2023-11-01 13:40:48.885 DBG ../../../pkg/storage/utils/decomposedfs/node/node.go:836 > AsResourceInfo node=9f0b42ea-2c07-4050-b0d8-8ba42d1b0db5 pid=11 pkg=rgrpc ri={"arbitrary_metadata":{"metadata":{"http://owncloud.org/ns/favorite":""}},"checksum":{"sum":"da39a3ee5e6b4b0d3255bfef95601890afd80709","type":4},"etag":"\"aa0dc1e084ca9527ebd086e4d2e03205\"","id":{"opaque_id":"9f0b42ea-2c07-4050-b0d8-8ba42d1b0db5","space_id":"ddc2004c-0977-11eb-9d3f-a793888cd0f8"},"mime_type":"text/plain","mtime":{"nanos":431815619,"seconds":1698846048},"name":"testfile2.txt","opaque":{"map":{"adler32":{"decoder":"plain","value":"MDAwMDAwMDE="},"md5":{"decoder":"plain","value":"ZDQxZDhjZDk4ZjAwYjIwNGU5ODAwOTk4ZWNmODQyN2U="}}},"owner":{"idp":"http://localhost:20080","opaque_id":"ddc2004c-0977-11eb-9d3f-a793888cd0f8","type":1},"parent_id":{"opaque_id":"20ab60f8-fe41-4726-a6e3-dac22683c1ab","space_id":"ddc2004c-0977-11eb-9d3f-a793888cd0f8"},"path":"testfile2.txt","permission_set":{"add_grant":true,"create_container":true,"delete":true,"deny_grant":true,"get_path":true,"get_quota":true,"initiate_file_download":true,"initiate_file_upload":true,"list_container":true,"list_file_versions":true,"list_grants":true,"list_recycle":true,"move":true,"purge_recycle":true,"remove_grant":true,"restore_file_version":true,"restore_recycle_item":true,"stat":true,"update_grant":true},"size":4,"type":1} traceid=df112e33d2e066628951bae7f13c065a
2023-11-01 13:40:49.365 DBG ../../../pkg/storage/utils/decomposedfs/node/node.go:836 > AsResourceInfo node=cfeb456b-d0a9-4c10-81a9-f966b1c9de51 pid=11 pkg=rgrpc ri={"arbitrary_metadata":{"metadata":{"http://owncloud.org/ns/favorite":""}},"checksum":{"sum":"da39a3ee5e6b4b0d3255bfef95601890afd80709","type":4},"etag":"\"f682bba2167bfa6798bdbdcc079d3f96\"","id":{"opaque_id":"cfeb456b-d0a9-4c10-81a9-f966b1c9de51","space_id":"ddc2004c-0977-11eb-9d3f-a793888cd0f8"},"mime_type":"text/plain","mtime":{"nanos":965061958,"seconds":1698846048},"name":"testfile.txt","opaque":{"map":{"adler32":{"decoder":"plain","value":"MDAwMDAwMDE="},"md5":{"decoder":"plain","value":"ZDQxZDhjZDk4ZjAwYjIwNGU5ODAwOTk4ZWNmODQyN2U="}}},"owner":{"idp":"http://localhost:20080","opaque_id":"ddc2004c-0977-11eb-9d3f-a793888cd0f8","type":1},"parent_id":{"opaque_id":"6b890791-0f16-436b-a34f-52d1d8b44d00","space_id":"ddc2004c-0977-11eb-9d3f-a793888cd0f8"},"path":"testfile.txt","permission_set":{"add_grant":true,"create_container":true,"delete":true,"deny_grant":true,"get_path":true,"get_quota":true,"initiate_file_download":true,"initiate_file_upload":true,"list_container":true,"list_file_versions":true,"list_grants":true,"list_recycle":true,"move":true,"purge_recycle":true,"remove_grant":true,"restore_file_version":true,"restore_recycle_item":true,"stat":true,"update_grant":true},"size":4,"type":1} traceid=ff3d93a60eef7a0160d0cc2edd3c8d4c
2023-11-01 13:40:49.368 DBG ../../../pkg/storage/utils/decomposedfs/node/node.go:836 > AsResourceInfo node=cfeb456b-d0a9-4c10-81a9-f966b1c9de51 pid=11 pkg=rgrpc ri={"arbitrary_metadata":{"metadata":{"http://owncloud.org/ns/favorite":""}},"checksum":{"sum":"da39a3ee5e6b4b0d3255bfef95601890afd80709","type":4},"etag":"\"f682bba2167bfa6798bdbdcc079d3f96\"","id":{"opaque_id":"cfeb456b-d0a9-4c10-81a9-f966b1c9de51","space_id":"ddc2004c-0977-11eb-9d3f-a793888cd0f8"},"mime_type":"text/plain","mtime":{"nanos":965061958,"seconds":1698846048},"name":"testfile.txt","opaque":{"map":{"adler32":{"decoder":"plain","value":"MDAwMDAwMDE="},"md5":{"decoder":"plain","value":"ZDQxZDhjZDk4ZjAwYjIwNGU5ODAwOTk4ZWNmODQyN2U="}}},"owner":{"idp":"http://localhost:20080","opaque_id":"ddc2004c-0977-11eb-9d3f-a793888cd0f8","type":1},"parent_id":{"opaque_id":"6b890791-0f16-436b-a34f-52d1d8b44d00","space_id":"ddc2004c-0977-11eb-9d3f-a793888cd0f8"},"path":"testfile.txt","permission_set":{"add_grant":true,"create_container":true,"delete":true,"deny_grant":true,"get_path":true,"get_quota":true,"initiate_file_download":true,"initiate_file_upload":true,"list_container":true,"list_file_versions":true,"list_grants":true,"list_recycle":true,"move":true,"purge_recycle":true,"remove_grant":true,"restore_file_version":true,"restore_recycle_item":true,"stat":true,"update_grant":true},"size":4,"type":1} traceid=6031ad50ca49f982bc19113d9e222e8d
2023-11-01 13:40:49.377 DBG ../../../pkg/storage/utils/decomposedfs/node/node.go:836 > AsResourceInfo node=cfeb456b-d0a9-4c10-81a9-f966b1c9de51 pid=11 pkg=rgrpc ri={"arbitrary_metadata":{"metadata":{"http://owncloud.org/ns/favorite":""}},"checksum":{"sum":"da39a3ee5e6b4b0d3255bfef95601890afd80709","type":4},"etag":"\"f682bba2167bfa6798bdbdcc079d3f96\"","id":{"opaque_id":"cfeb456b-d0a9-4c10-81a9-f966b1c9de51","space_id":"ddc2004c-0977-11eb-9d3f-a793888cd0f8"},"mime_type":"text/plain","mtime":{"nanos":965061958,"seconds":1698846048},"name":"testfile.txt","opaque":{"map":{"adler32":{"decoder":"plain","value":"MDAwMDAwMDE="},"md5":{"decoder":"plain","value":"ZDQxZDhjZDk4ZjAwYjIwNGU5ODAwOTk4ZWNmODQyN2U="}}},"owner":{"idp":"http://localhost:20080","opaque_id":"ddc2004c-0977-11eb-9d3f-a793888cd0f8","type":1},"parent_id":{"opaque_id":"6b890791-0f16-436b-a34f-52d1d8b44d00","space_id":"ddc2004c-0977-11eb-9d3f-a793888cd0f8"},"path":"testfile.txt","permission_set":{"add_grant":true,"create_container":true,"delete":true,"deny_grant":true,"get_path":true,"get_quota":true,"initiate_file_download":true,"initiate_file_upload":true,"list_container":true,"list_file_versions":true,"list_grants":true,"list_recycle":true,"move":true,"purge_recycle":true,"remove_grant":true,"restore_file_version":true,"restore_recycle_item":true,"stat":true,"update_grant":true},"size":4,"type":1} traceid=d76c0244e3bcb631f5f82c4c8e20d601
2023-11-01 13:40:49.38 DBG ../../../pkg/storage/utils/decomposedfs/node/node.go:836 > AsResourceInfo node=cfeb456b-d0a9-4c10-81a9-f966b1c9de51 pid=11 pkg=rgrpc ri={"arbitrary_metadata":{"metadata":{"http://owncloud.org/ns/favorite":""}},"checksum":{"sum":"da39a3ee5e6b4b0d3255bfef95601890afd80709","type":4},"etag":"\"f682bba2167bfa6798bdbdcc079d3f96\"","id":{"opaque_id":"cfeb456b-d0a9-4c10-81a9-f966b1c9de51","space_id":"ddc2004c-0977-11eb-9d3f-a793888cd0f8"},"mime_type":"text/plain","mtime":{"nanos":965061958,"seconds":1698846048},"name":"testfile.txt","opaque":{"map":{"adler32":{"decoder":"plain","value":"MDAwMDAwMDE="},"md5":{"decoder":"plain","value":"ZDQxZDhjZDk4ZjAwYjIwNGU5ODAwOTk4ZWNmODQyN2U="}}},"owner":{"idp":"http://localhost:20080","opaque_id":"ddc2004c-0977-11eb-9d3f-a793888cd0f8","type":1},"parent_id":{"opaque_id":"6b890791-0f16-436b-a34f-52d1d8b44d00","space_id":"ddc2004c-0977-11eb-9d3f-a793888cd0f8"},"path":"testfile.txt","permission_set":{"add_grant":true,"create_container":true,"delete":true,"deny_grant":true,"get_path":true,"get_quota":true,"initiate_file_download":true,"initiate_file_upload":true,"list_container":true,"list_file_versions":true,"list_grants":true,"list_recycle":true,"move":true,"purge_recycle":true,"remove_grant":true,"restore_file_version":true,"restore_recycle_item":true,"stat":true,"update_grant":true},"size":4,"type":1} traceid=d001a07f2e78c477ab93ef971b963b05
2023-11-01 13:40:49.822 DBG ../../../pkg/storage/utils/decomposedfs/node/node.go:836 > AsResourceInfo node=cfeb456b-d0a9-4c10-81a9-f966b1c9de51 pid=11 pkg=rgrpc ri={"arbitrary_metadata":{"metadata":{"http://owncloud.org/ns/favorite":""}},"checksum":{"sum":"da39a3ee5e6b4b0d3255bfef95601890afd80709","type":4},"etag":"\"b408cf3de57cd61ecb815fb51b1a9876\"","id":{"opaque_id":"cfeb456b-d0a9-4c10-81a9-f966b1c9de51","space_id":"ddc2004c-0977-11eb-9d3f-a793888cd0f8"},"mime_type":"text/plain","mtime":{"nanos":381622079,"seconds":1698846049},"name":"testfile.txt","opaque":{"map":{"adler32":{"decoder":"plain","value":"MDAwMDAwMDE="},"md5":{"decoder":"plain","value":"ZDQxZDhjZDk4ZjAwYjIwNGU5ODAwOTk4ZWNmODQyN2U="}}},"owner":{"idp":"http://localhost:20080","opaque_id":"ddc2004c-0977-11eb-9d3f-a793888cd0f8","type":1},"parent_id":{"opaque_id":"6b890791-0f16-436b-a34f-52d1d8b44d00","space_id":"ddc2004c-0977-11eb-9d3f-a793888cd0f8"},"path":"testfile.txt","permission_set":{"add_grant":true,"create_container":true,"delete":true,"deny_grant":true,"get_path":true,"get_quota":true,"initiate_file_download":true,"initiate_file_upload":true,"list_container":true,"list_file_versions":true,"list_grants":true,"list_recycle":true,"move":true,"purge_recycle":true,"remove_grant":true,"restore_file_version":true,"restore_recycle_item":true,"stat":true,"update_grant":true},"size":8,"type":1} traceid=1f26f82f40ecb8d764fe86c86b93dbaf
2023-11-01 13:40:49.826 DBG ../../../pkg/storage/utils/decomposedfs/node/node.go:836 > AsResourceInfo node=cfeb456b-d0a9-4c10-81a9-f966b1c9de51 pid=11 pkg=rgrpc ri={"arbitrary_metadata":{"metadata":{"http://owncloud.org/ns/favorite":""}},"checksum":{"sum":"da39a3ee5e6b4b0d3255bfef95601890afd80709","type":4},"etag":"\"b408cf3de57cd61ecb815fb51b1a9876\"","id":{"opaque_id":"cfeb456b-d0a9-4c10-81a9-f966b1c9de51","space_id":"ddc2004c-0977-11eb-9d3f-a793888cd0f8"},"mime_type":"text/plain","mtime":{"nanos":381622079,"seconds":1698846049},"name":"testfile.txt","opaque":{"map":{"adler32":{"decoder":"plain","value":"MDAwMDAwMDE="},"md5":{"decoder":"plain","value":"ZDQxZDhjZDk4ZjAwYjIwNGU5ODAwOTk4ZWNmODQyN2U="}}},"owner":{"idp":"http://localhost:20080","opaque_id":"ddc2004c-0977-11eb-9d3f-a793888cd0f8","type":1},"parent_id":{"opaque_id":"6b890791-0f16-436b-a34f-52d1d8b44d00","space_id":"ddc2004c-0977-11eb-9d3f-a793888cd0f8"},"path":"testfile.txt","permission_set":{"add_grant":true,"create_container":true,"delete":true,"deny_grant":true,"get_path":true,"get_quota":true,"initiate_file_download":true,"initiate_file_upload":true,"list_container":true,"list_file_versions":true,"list_grants":true,"list_recycle":true,"move":true,"purge_recycle":true,"remove_grant":true,"restore_file_version":true,"restore_recycle_item":true,"stat":true,"update_grant":true},"size":8,"type":1} traceid=065bddf142c59768e31cc32b926581db
2023-11-01 13:40:50.293 DBG ../../../pkg/storage/utils/decomposedfs/node/node.go:836 > AsResourceInfo node=45c54439-7432-4f6d-bfab-74ed7c87c3db pid=11 pkg=rgrpc ri={"arbitrary_metadata":{"metadata":{"http://owncloud.org/ns/favorite":""}},"checksum":{"sum":"da39a3ee5e6b4b0d3255bfef95601890afd80709","type":4},"etag":"\"2ed0a9ef485e09fa89a62701562a11c2\"","id":{"opaque_id":"45c54439-7432-4f6d-bfab-74ed7c87c3db","space_id":"f7fbf8c8-139b-4376-b307-cf0a8c2d0d9c"},"mime_type":"text/plain","mtime":{"nanos":886834954,"seconds":1698846049},"name":"testfile.txt","opaque":{"map":{"adler32":{"decoder":"plain","value":"MDAwMDAwMDE="},"md5":{"decoder":"plain","value":"ZDQxZDhjZDk4ZjAwYjIwNGU5ODAwOTk4ZWNmODQyN2U="}}},"owner":{"idp":"http://localhost:20080","opaque_id":"f7fbf8c8-139b-4376-b307-cf0a8c2d0d9c","type":1},"parent_id":{"opaque_id":"f7fbf8c8-139b-4376-b307-cf0a8c2d0d9c","space_id":"f7fbf8c8-139b-4376-b307-cf0a8c2d0d9c"},"path":"testfile.txt","permission_set":{"add_grant":true,"create_container":true,"delete":true,"deny_grant":true,"get_path":true,"get_quota":true,"initiate_file_download":true,"initiate_file_upload":true,"list_container":true,"list_file_versions":true,"list_grants":true,"list_recycle":true,"move":true,"purge_recycle":true,"remove_grant":true,"restore_file_version":true,"restore_recycle_item":true,"stat":true,"update_grant":true},"size":11,"type":1} traceid=7accb4fa50787572b4c07ef911cb7ba8
2023-11-01 13:40:50.296 DBG ../../../pkg/storage/utils/decomposedfs/node/node.go:836 > AsResourceInfo node=45c54439-7432-4f6d-bfab-74ed7c87c3db pid=11 pkg=rgrpc ri={"arbitrary_metadata":{"metadata":{"http://owncloud.org/ns/favorite":""}},"checksum":{"sum":"da39a3ee5e6b4b0d3255bfef95601890afd80709","type":4},"etag":"\"2ed0a9ef485e09fa89a62701562a11c2\"","id":{"opaque_id":"45c54439-7432-4f6d-bfab-74ed7c87c3db","space_id":"f7fbf8c8-139b-4376-b307-cf0a8c2d0d9c"},"mime_type":"text/plain","mtime":{"nanos":886834954,"seconds":1698846049},"name":"testfile.txt","opaque":{"map":{"adler32":{"decoder":"plain","value":"MDAwMDAwMDE="},"md5":{"decoder":"plain","value":"ZDQxZDhjZDk4ZjAwYjIwNGU5ODAwOTk4ZWNmODQyN2U="}}},"owner":{"idp":"http://localhost:20080","opaque_id":"f7fbf8c8-139b-4376-b307-cf0a8c2d0d9c","type":1},"parent_id":{"opaque_id":"f7fbf8c8-139b-4376-b307-cf0a8c2d0d9c","space_id":"f7fbf8c8-139b-4376-b307-cf0a8c2d0d9c"},"path":"testfile.txt","permission_set":{"add_grant":true,"create_container":true,"delete":true,"deny_grant":true,"get_path":true,"get_quota":true,"initiate_file_download":true,"initiate_file_upload":true,"list_container":true,"list_file_versions":true,"list_grants":true,"list_recycle":true,"move":true,"purge_recycle":true,"remove_grant":true,"restore_file_version":true,"restore_recycle_item":true,"stat":true,"update_grant":true},"size":11,"type":1} traceid=330e875e4c51f0f5fe8c9894a38b9e0f
2023-11-01 13:40:50.304 DBG ../../../pkg/storage/utils/decomposedfs/node/node.go:836 > AsResourceInfo node=45c54439-7432-4f6d-bfab-74ed7c87c3db pid=11 pkg=rgrpc ri={"arbitrary_metadata":{"metadata":{"http://owncloud.org/ns/favorite":""}},"checksum":{"sum":"da39a3ee5e6b4b0d3255bfef95601890afd80709","type":4},"etag":"\"2ed0a9ef485e09fa89a62701562a11c2\"","id":{"opaque_id":"45c54439-7432-4f6d-bfab-74ed7c87c3db","space_id":"f7fbf8c8-139b-4376-b307-cf0a8c2d0d9c"},"mime_type":"text/plain","mtime":{"nanos":886834954,"seconds":1698846049},"name":"testfile.txt","opaque":{"map":{"adler32":{"decoder":"plain","value":"MDAwMDAwMDE="},"md5":{"decoder":"plain","value":"ZDQxZDhjZDk4ZjAwYjIwNGU5ODAwOTk4ZWNmODQyN2U="}}},"owner":{"idp":"http://localhost:20080","opaque_id":"f7fbf8c8-139b-4376-b307-cf0a8c2d0d9c","type":1},"parent_id":{"opaque_id":"f7fbf8c8-139b-4376-b307-cf0a8c2d0d9c","space_id":"f7fbf8c8-139b-4376-b307-cf0a8c2d0d9c"},"path":"testfile.txt","permission_set":{"add_grant":true,"create_container":true,"delete":true,"deny_grant":true,"get_path":true,"get_quota":true,"initiate_file_download":true,"initiate_file_upload":true,"list_container":true,"list_file_versions":true,"list_grants":true,"list_recycle":true,"move":true,"purge_recycle":true,"remove_grant":true,"restore_file_version":true,"restore_recycle_item":true,"stat":true,"update_grant":true},"size":11,"type":1} traceid=d0ce5537c920a4bf2c78be0fdbb26bf0
2023-11-01 13:40:50.769 DBG ../../../pkg/storage/utils/decomposedfs/node/node.go:836 > AsResourceInfo node=113e1828-4f71-4f93-acee-a973e40bb42b pid=11 pkg=rgrpc ri={"arbitrary_metadata":{"metadata":{"http://owncloud.org/ns/favorite":""}},"checksum":{"sum":"da39a3ee5e6b4b0d3255bfef95601890afd80709","type":4},"etag":"\"f3e2b5268d351aeefe9a2ab61c5bdcec\"","id":{"opaque_id":"113e1828-4f71-4f93-acee-a973e40bb42b","space_id":"f7fbf8c8-139b-4376-b307-cf0a8c2d0d9c"},"mime_type":"text/plain","mtime":{"nanos":369177028,"seconds":1698846050},"name":"testfile.txt","opaque":{"map":{"adler32":{"decoder":"plain","value":"MDAwMDAwMDE="},"md5":{"decoder":"plain","value":"ZDQxZDhjZDk4ZjAwYjIwNGU5ODAwOTk4ZWNmODQyN2U="}}},"owner":{"idp":"http://localhost:20080","opaque_id":"f7fbf8c8-139b-4376-b307-cf0a8c2d0d9c","type":1},"parent_id":{"opaque_id":"82524ddc-0490-4585-9aa4-fa266af7929c","space_id":"f7fbf8c8-139b-4376-b307-cf0a8c2d0d9c"},"path":"testfile.txt","permission_set":{"add_grant":true,"create_container":true,"delete":true,"deny_grant":true,"get_path":true,"get_quota":true,"initiate_file_download":true,"initiate_file_upload":true,"list_container":true,"list_file_versions":true,"list_grants":true,"list_recycle":true,"move":true,"purge_recycle":true,"remove_grant":true,"restore_file_version":true,"restore_recycle_item":true,"stat":true,"update_grant":true},"size":11,"type":1} traceid=c8b90e7ee8e305c5a6546ce985999bcd
2023-11-01 13:40:51.237 DBG ../../../pkg/storage/utils/decomposedfs/node/node.go:836 > AsResourceInfo node=92c89c24-f949-4b7b-8c53-f705610edbe8 pid=11 pkg=rgrpc ri={"arbitrary_metadata":{"metadata":{"http://owncloud.org/ns/favorite":""}},"checksum":{"sum":"da39a3ee5e6b4b0d3255bfef95601890afd80709","type":4},"etag":"\"5e3e055a09498a46cc4a3c607955a99a\"","id":{"opaque_id":"92c89c24-f949-4b7b-8c53-f705610edbe8","space_id":"f7fbf8c8-139b-4376-b307-cf0a8c2d0d9c"},"mime_type":"text/plain","mtime":{"nanos":834508195,"seconds":1698846050},"name":"testfile.txt","opaque":{"map":{"adler32":{"decoder":"plain","value":"MDAwMDAwMDE="},"md5":{"decoder":"plain","value":"ZDQxZDhjZDk4ZjAwYjIwNGU5ODAwOTk4ZWNmODQyN2U="}}},"owner":{"idp":"http://localhost:20080","opaque_id":"f7fbf8c8-139b-4376-b307-cf0a8c2d0d9c","type":1},"parent_id":{"opaque_id":"f7fbf8c8-139b-4376-b307-cf0a8c2d0d9c","space_id":"f7fbf8c8-139b-4376-b307-cf0a8c2d0d9c"},"path":"testfile.txt","permission_set":{"add_grant":true,"create_container":true,"delete":true,"deny_grant":true,"get_path":true,"get_quota":true,"initiate_file_download":true,"initiate_file_upload":true,"list_container":true,"list_file_versions":true,"list_grants":true,"list_recycle":true,"move":true,"purge_recycle":true,"remove_grant":true,"restore_file_version":true,"restore_recycle_item":true,"stat":true,"update_grant":true},"size":11,"type":1} traceid=1d3330c8dcafb1f3c0cc2b7ef5a32bfd
2023-11-01 13:40:51.24 DBG ../../../pkg/storage/utils/decomposedfs/node/node.go:836 > AsResourceInfo node=92c89c24-f949-4b7b-8c53-f705610edbe8 pid=11 pkg=rgrpc ri={"arbitrary_metadata":{"metadata":{"http://owncloud.org/ns/favorite":""}},"checksum":{"sum":"da39a3ee5e6b4b0d3255bfef95601890afd80709","type":4},"etag":"\"5e3e055a09498a46cc4a3c607955a99a\"","id":{"opaque_id":"92c89c24-f949-4b7b-8c53-f705610edbe8","space_id":"f7fbf8c8-139b-4376-b307-cf0a8c2d0d9c"},"mime_type":"text/plain","mtime":{"nanos":834508195,"seconds":1698846050},"name":"testfile.txt","opaque":{"map":{"adler32":{"decoder":"plain","value":"MDAwMDAwMDE="},"md5":{"decoder":"plain","value":"ZDQxZDhjZDk4ZjAwYjIwNGU5ODAwOTk4ZWNmODQyN2U="}}},"owner":{"idp":"http://localhost:20080","opaque_id":"f7fbf8c8-139b-4376-b307-cf0a8c2d0d9c","type":1},"parent_id":{"opaque_id":"f7fbf8c8-139b-4376-b307-cf0a8c2d0d9c","space_id":"f7fbf8c8-139b-4376-b307-cf0a8c2d0d9c"},"path":"testfile.txt","permission_set":{"add_grant":true,"create_container":true,"delete":true,"deny_grant":true,"get_path":true,"get_quota":true,"initiate_file_download":true,"initiate_file_upload":true,"list_container":true,"list_file_versions":true,"list_grants":true,"list_recycle":true,"move":true,"purge_recycle":true,"remove_grant":true,"restore_file_version":true,"restore_recycle_item":true,"stat":true,"update_grant":true},"size":11,"type":1} traceid=fe992a99fa98e538ad33187f91b14774
2023-11-01 13:40:51.258 DBG ../../../pkg/storage/utils/decomposedfs/node/node.go:836 > AsResourceInfo node=92c89c24-f949-4b7b-8c53-f705610edbe8 pid=11 pkg=rgrpc ri={"arbitrary_metadata":{"metadata":{"http://owncloud.org/ns/favorite":""}},"checksum":{"sum":"da39a3ee5e6b4b0d3255bfef95601890afd80709","type":4},"etag":"\"5e3e055a09498a46cc4a3c607955a99a\"","id":{"opaque_id":"92c89c24-f949-4b7b-8c53-f705610edbe8","space_id":"f7fbf8c8-139b-4376-b307-cf0a8c2d0d9c"},"mime_type":"text/plain","mtime":{"nanos":834508195,"seconds":1698846050},"name":"Test-File.txt","opaque":{"map":{"adler32":{"decoder":"plain","value":"MDAwMDAwMDE="},"md5":{"decoder":"plain","value":"ZDQxZDhjZDk4ZjAwYjIwNGU5ODAwOTk4ZWNmODQyN2U="}}},"owner":{"idp":"http://localhost:20080","opaque_id":"f7fbf8c8-139b-4376-b307-cf0a8c2d0d9c","type":1},"parent_id":{"opaque_id":"f7fbf8c8-139b-4376-b307-cf0a8c2d0d9c","space_id":"f7fbf8c8-139b-4376-b307-cf0a8c2d0d9c"},"path":"Test-File.txt","permission_set":{"add_grant":true,"create_container":true,"delete":true,"deny_grant":true,"get_path":true,"get_quota":true,"initiate_file_download":true,"initiate_file_upload":true,"list_container":true,"list_file_versions":true,"list_grants":true,"list_recycle":true,"move":true,"purge_recycle":true,"remove_grant":true,"restore_file_version":true,"restore_recycle_item":true,"stat":true,"update_grant":true},"size":11,"type":1} traceid=536b2b4bcc6c0a43906d3d325f9b9bf6
2023-11-01 13:40:51.26 DBG ../../../pkg/storage/utils/decomposedfs/node/node.go:836 > AsResourceInfo node=92c89c24-f949-4b7b-8c53-f705610edbe8 pid=11 pkg=rgrpc ri={"arbitrary_metadata":{"metadata":{"http://owncloud.org/ns/favorite":""}},"checksum":{"sum":"da39a3ee5e6b4b0d3255bfef95601890afd80709","type":4},"etag":"\"5e3e055a09498a46cc4a3c607955a99a\"","id":{"opaque_id":"92c89c24-f949-4b7b-8c53-f705610edbe8","space_id":"f7fbf8c8-139b-4376-b307-cf0a8c2d0d9c"},"mime_type":"text/plain","mtime":{"nanos":834508195,"seconds":1698846050},"name":"Test-File.txt","opaque":{"map":{"adler32":{"decoder":"plain","value":"MDAwMDAwMDE="},"md5":{"decoder":"plain","value":"ZDQxZDhjZDk4ZjAwYjIwNGU5ODAwOTk4ZWNmODQyN2U="}}},"owner":{"idp":"http://localhost:20080","opaque_id":"f7fbf8c8-139b-4376-b307-cf0a8c2d0d9c","type":1},"parent_id":{"opaque_id":"f7fbf8c8-139b-4376-b307-cf0a8c2d0d9c","space_id":"f7fbf8c8-139b-4376-b307-cf0a8c2d0d9c"},"path":"Test-File.txt","permission_set":{"add_grant":true,"create_container":true,"delete":true,"deny_grant":true,"get_path":true,"get_quota":true,"initiate_file_download":true,"initiate_file_upload":true,"list_container":true,"list_file_versions":true,"list_grants":true,"list_recycle":true,"move":true,"purge_recycle":true,"remove_grant":true,"restore_file_version":true,"restore_recycle_item":true,"stat":true,"update_grant":true},"size":11,"type":1} traceid=ccb80074b528c5a3011998b40ca01b2e
2023-11-01 13:40:51.27 DBG ../../../pkg/storage/utils/decomposedfs/node/node.go:836 > AsResourceInfo node=92c89c24-f949-4b7b-8c53-f705610edbe8 pid=11 pkg=rgrpc ri={"arbitrary_metadata":{"metadata":{"http://owncloud.org/ns/favorite":""}},"checksum":{"sum":"da39a3ee5e6b4b0d3255bfef95601890afd80709","type":4},"etag":"\"5e3e055a09498a46cc4a3c607955a99a\"","id":{"opaque_id":"92c89c24-f949-4b7b-8c53-f705610edbe8","space_id":"f7fbf8c8-139b-4376-b307-cf0a8c2d0d9c"},"mime_type":"text/plain","mtime":{"nanos":834508195,"seconds":1698846050},"name":"Test-File.txt","opaque":{"map":{"adler32":{"decoder":"plain","value":"MDAwMDAwMDE="},"md5":{"decoder":"plain","value":"ZDQxZDhjZDk4ZjAwYjIwNGU5ODAwOTk4ZWNmODQyN2U="}}},"owner":{"idp":"http://localhost:20080","opaque_id":"f7fbf8c8-139b-4376-b307-cf0a8c2d0d9c","type":1},"parent_id":{"opaque_id":"f7fbf8c8-139b-4376-b307-cf0a8c2d0d9c","space_id":"f7fbf8c8-139b-4376-b307-cf0a8c2d0d9c"},"path":"Test-File.txt","permission_set":{"add_grant":true,"create_container":true,"delete":true,"deny_grant":true,"get_path":true,"get_quota":true,"initiate_file_download":true,"initiate_file_upload":true,"list_container":true,"list_file_versions":true,"list_grants":true,"list_recycle":true,"move":true,"purge_recycle":true,"remove_grant":true,"restore_file_version":true,"restore_recycle_item":true,"stat":true,"update_grant":true},"size":11,"type":1} traceid=88ffc6f81f3a39362f5eca32d173415f
2023-11-01 13:40:51.868 DBG ../../../pkg/storage/utils/decomposedfs/node/node.go:836 > AsResourceInfo node=f4ea75a0-c09b-440e-a56f-57f11996f468 pid=11 pkg=rgrpc ri={"arbitrary_metadata":{"metadata":{"http://owncloud.org/ns/favorite":""}},"checksum":{"sum":"da39a3ee5e6b4b0d3255bfef95601890afd80709","type":4},"etag":"\"635d826ac686d62df2da532257813ed8\"","id":{"opaque_id":"f4ea75a0-c09b-440e-a56f-57f11996f468","space_id":"ddc2004c-0977-11eb-9d3f-a793888cd0f8"},"mime_type":"text/plain","mtime":{"nanos":419090839,"seconds":1698846051},"name":"test.txt","opaque":{"map":{"adler32":{"decoder":"plain","value":"MDAwMDAwMDE="},"md5":{"decoder":"plain","value":"ZDQxZDhjZDk4ZjAwYjIwNGU5ODAwOTk4ZWNmODQyN2U="}}},"owner":{"idp":"http://localhost:20080","opaque_id":"ddc2004c-0977-11eb-9d3f-a793888cd0f8","type":1},"parent_id":{"opaque_id":"129cdf0d-de72-463e-8ef6-5ade7582391e","space_id":"ddc2004c-0977-11eb-9d3f-a793888cd0f8"},"path":"/public-office-documents/test.txt","permission_set":{"add_grant":true,"create_container":true,"delete":true,"deny_grant":true,"get_path":true,"get_quota":true,"initiate_file_download":true,"initiate_file_upload":true,"list_container":true,"list_file_versions":true,"list_grants":true,"list_recycle":true,"move":true,"purge_recycle":true,"remove_grant":true,"restore_file_version":true,"restore_recycle_item":true,"stat":true,"update_grant":true},"type":1} traceid=8ad8dab675f3667d6586476f084bc77e
2023-11-01 13:40:52.376 DBG ../../../pkg/storage/utils/decomposedfs/node/node.go:836 > AsResourceInfo node=f4ea75a0-c09b-440e-a56f-57f11996f468 pid=11 pkg=rgrpc ri={"arbitrary_metadata":{"metadata":{"http://owncloud.org/ns/favorite":""}},"checksum":{"sum":"da39a3ee5e6b4b0d3255bfef95601890afd80709","type":4},"etag":"\"635d826ac686d62df2da532257813ed8\"","id":{"opaque_id":"f4ea75a0-c09b-440e-a56f-57f11996f468","space_id":"ddc2004c-0977-11eb-9d3f-a793888cd0f8"},"mime_type":"text/plain","mtime":{"nanos":419090839,"seconds":1698846051},"name":"test.txt","opaque":{"map":{"adler32":{"decoder":"plain","value":"MDAwMDAwMDE="},"md5":{"decoder":"plain","value":"ZDQxZDhjZDk4ZjAwYjIwNGU5ODAwOTk4ZWNmODQyN2U="}}},"owner":{"idp":"http://localhost:20080","opaque_id":"ddc2004c-0977-11eb-9d3f-a793888cd0f8","type":1},"parent_id":{"opaque_id":"129cdf0d-de72-463e-8ef6-5ade7582391e","space_id":"ddc2004c-0977-11eb-9d3f-a793888cd0f8"},"path":"/public-office-documents/test.txt","permission_set":{"add_grant":true,"create_container":true,"delete":true,"deny_grant":true,"get_path":true,"get_quota":true,"initiate_file_download":true,"initiate_file_upload":true,"list_container":true,"list_file_versions":true,"list_grants":true,"list_recycle":true,"move":true,"purge_recycle":true,"remove_grant":true,"restore_file_version":true,"restore_recycle_item":true,"stat":true,"update_grant":true},"type":1} traceid=ae3a962c9990e05460388bcf1e269589
2023-11-01 13:40:52.376 DBG ../../../pkg/storage/utils/decomposedfs/node/node.go:836 > AsResourceInfo node=d5a1aaa5-4f8a-4ee1-9c9d-bf6eabd41d70 pid=11 pkg=rgrpc ri={"arbitrary_metadata":{"metadata":{"http://owncloud.org/ns/favorite":""}},"checksum":{"sum":"da39a3ee5e6b4b0d3255bfef95601890afd80709","type":4},"etag":"\"9943e40c65cd0380ea7526ee77d79d81\"","id":{"opaque_id":"d5a1aaa5-4f8a-4ee1-9c9d-bf6eabd41d70","space_id":"ddc2004c-0977-11eb-9d3f-a793888cd0f8"},"mime_type":"text/plain","mtime":{"nanos":892700322,"seconds":1698846051},"name":"test2.txt","opaque":{"map":{"adler32":{"decoder":"plain","value":"MDAwMDAwMDE="},"md5":{"decoder":"plain","value":"ZDQxZDhjZDk4ZjAwYjIwNGU5ODAwOTk4ZWNmODQyN2U="}}},"owner":{"idp":"http://localhost:20080","opaque_id":"ddc2004c-0977-11eb-9d3f-a793888cd0f8","type":1},"parent_id":{"opaque_id":"129cdf0d-de72-463e-8ef6-5ade7582391e","space_id":"ddc2004c-0977-11eb-9d3f-a793888cd0f8"},"path":"/public-office-documents/test2.txt","permission_set":{"add_grant":true,"create_container":true,"delete":true,"deny_grant":true,"get_path":true,"get_quota":true,"initiate_file_download":true,"initiate_file_upload":true,"list_container":true,"list_file_versions":true,"list_grants":true,"list_recycle":true,"move":true,"purge_recycle":true,"remove_grant":true,"restore_file_version":true,"restore_recycle_item":true,"stat":true,"update_grant":true},"type":1} traceid=ae3a962c9990e05460388bcf1e269589

must have something to do with the s3 backend ... the tests use the xattr metadata_backend ... will check that 👀

Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>

butonic · 2023-11-01T14:47:10Z

da39a3ee5e6b4b0d3255bfef95601890afd80709 is the hash for the empty string ... maybe reading back the upload from ceph does not work?

Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>

butonic · 2023-11-01T20:01:45Z

Indeed. The added log line appears: internal error: mismatching upload length. expected 4, could only copy 0

👀

2023-11-01 15:13:14.018 DBG ../../../pkg/rhttp/rhttp.go:248 > http routing: head=data tail=/simple/ef186ae941dd9250bf46163afe2772b6+2~PxXm3ze5qcW3AfYzPWkgfjAmKQmrtNG svc=data pid=11 pkg=rhttp
2023-11-01 15:13:14.018 DBG ../../../internal/http/services/dataprovider/dataprovider.go:179 > dataprovider routing: path=/simple/ef186ae941dd9250bf46163afe2772b6+2~PxXm3ze5qcW3AfYzPWkgfjAmKQmrtNG pid=11 pkg=rhttp traceid=af187840ef124bd7d3ae7577f64b1218
2023-11-01 15:13:14.381 ERR ../../../pkg/rhttp/datatx/manager/simple/simple.go:140 > error uploading file error="internal error: mismatching upload length. expected 4, could only copy 0" datatx=simple pid=11 pkg=rhttp traceid=af187840ef124bd7d3ae7577f64b1218
2023-11-01 15:13:14.381 ERR ../../../internal/http/interceptors/log/log.go:112 > http end="01/Nov/2023:15:13:14 +0000" host=192.168.3.4 method=PUT pid=11 pkg=rhttp proto=HTTP/1.1 size=0 start="01/Nov/2023:15:13:14 +0000" status=500 time_ns=362901001 traceid=af187840ef124bd7d3ae7577f64b1218 uri=/data/simple/ef186ae941dd9250bf46163afe2772b6+2~PxXm3ze5qcW3AfYzPWkgfjAmKQmrtNG url=/ef186ae941dd9250bf46163afe2772b6+2~PxXm3ze5qcW3AfYzPWkgfjAmKQmrtNG

🤔 why are the uploads done via /simple/{uploadid} and not /tus/{uploadid} ... hm ... might be the cs3 validator ... locally the tus upload via web ui works fine ... and it uses /tus/{uploadid}

👀

Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>

aduffeck · 2023-12-05T07:27:00Z

internal/grpc/services/storageprovider/storageprovider.go

@@ -338,6 +338,15 @@ func (s *service) InitiateFileUpload(ctx context.Context, req *provider.Initiate
 		}, nil
 	}

+	// FIXME: This is a hack to transport more metadata to the storage.FS InitiateUpload implementation


I know, we already talked about it, but just for the record 😛 I would rather track the FIXME issues and thought-comments on github or drop them entirely. Either they are important and should be tracked and prioritized properly or they aren't, in which case the FIXMEs in the code are just ballast we'll carry on indefinitely.

pkg/rhttp/datatx/manager/tus/tus.go

aduffeck · 2023-12-05T07:47:40Z

pkg/storage/fs/s3ng/blobstore/blobstore.go

@@ -63,6 +63,22 @@ func New(endpoint, region, bucket, accessKey, secretKey string) (*Blobstore, err
 	}, nil
 }

+func (bs *Blobstore) MoveBlob(node *node.Node, source, bucket, key string) error {
+
+	_, err := bs.client.CopyObject(context.Background(), minio.CopyDestOptions{


I think we also need to delete the source after copying it.

pkg/storage/utils/decomposedfs/tree/tree.go

pkg/storage/utils/decomposedfs/upload/upload.go

Co-authored-by: Andre Duffeck <aduffeck@users.noreply.github.com>

Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>

Everything will be stored in the upload session which we already have for our own metadata anyway.

Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>

butonic · 2024-07-29T14:46:04Z

We evolved our own DataStore into a OcisSession that also covers postprocessing.

butonic requested review from a team, labkode and ishank011 as code owners September 1, 2023 14:13

butonic self-assigned this Sep 1, 2023

butonic force-pushed the use-tus-data-storage branch 3 times, most recently from 0289878 to 6b85fe4 Compare September 7, 2023 14:30

butonic marked this pull request as draft September 8, 2023 10:19

butonic force-pushed the use-tus-data-storage branch from dd5f382 to 171c9af Compare September 8, 2023 15:55

butonic force-pushed the use-tus-data-storage branch from eac4bc5 to 4e34553 Compare September 18, 2023 14:04

butonic mentioned this pull request Oct 16, 2023

use tusd datastorages owncloud/ocis#7499

Closed

butonic force-pushed the use-tus-data-storage branch from 0d70283 to 1293660 Compare November 1, 2023 13:33

butonic added 3 commits November 1, 2023 15:31

add comments

7d2a741

Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>

test msgpack metadata

27936aa

Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>

run all tests again

11ebb76

Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>

test copy

80cad58

Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>

butonic added 4 commits November 1, 2023 21:14

log error when trying to read upload info

21301b4

Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>

log upload info details when copying fails

6d1eb80

Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>

log as error

aa14928

Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>

reinitialze logger

eaf5632

Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>

butonic added 2 commits December 4, 2023 16:10

remove debug logging

89f9606

Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>

more cleanup

a5edbe5

Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>

aduffeck reviewed Dec 5, 2023

View reviewed changes

butonic and others added 13 commits December 5, 2023 10:32

Apply suggestions from code review

d99a424

Co-authored-by: Andre Duffeck <aduffeck@users.noreply.github.com>

use log from context

2d95dcc

Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>

simplify if conditions

87916ac

Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>

fix typo

5053013

Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>

fix listing and purging upload sessions

1a35958

Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>

drop comment

ea559e1

Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>

fix upload progress creation and read offset from tus upload

829df6d

Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>

read uploads concurrently

747f8f7

Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>

fix loading upload sessions

b643432

Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>

Purge both the metadata and the upload info when finalizing

5c9beca

Use Size when reporting the size

9c2ed70

Implement tus datastores which support cleaning up metadata only

9167e4b

Directly write blob to the right location in s3

086e321

aduffeck force-pushed the use-tus-data-storage branch from e1c897e to 13fdc36 Compare December 7, 2023 07:12

Clarify naming

c11ca05

wkloucek mentioned this pull request Dec 7, 2023

update s3 bucket permissions for oCIS 5 owncloud/ocis-charts#446

Merged

11 tasks

WIP: Get rid of separate tus fileinfo metadata

be393b3

Everything will be stored in the upload session which we already have for our own metadata anyway.

aduffeck force-pushed the use-tus-data-storage branch from 13fdc36 to be393b3 Compare December 7, 2023 08:41

make filestore upload session compatible

f8264b5

Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>

butonic force-pushed the use-tus-data-storage branch from 637142d to f8264b5 Compare December 7, 2023 12:48

aduffeck added 3 commits December 7, 2023 14:34

Do not read the session we just created just to get the ID

7d3ae90

Simplify finishing the upload

d41ec1e

Retrieve the upload offset from the session instead of reading the parts

e3ba91d

butonic mentioned this pull request Dec 12, 2023

use dedicated upload session struct #4402

Merged

butonic marked this pull request as draft December 14, 2023 10:56

butonic closed this Jul 29, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

use tusd datastorages #4148

use tusd datastorages #4148

butonic commented Sep 1, 2023 •

edited

Loading

update-docs bot commented Sep 1, 2023

butonic commented Sep 19, 2023

butonic commented Sep 19, 2023

micbar commented Sep 19, 2023 •

edited

Loading

butonic commented Sep 20, 2023

butonic commented Oct 16, 2023 •

edited

Loading

butonic commented Nov 1, 2023 •

edited

Loading

butonic commented Nov 1, 2023

butonic commented Nov 1, 2023 •

edited

Loading

aduffeck Dec 5, 2023

aduffeck Dec 5, 2023

butonic commented Jul 29, 2024

use tusd datastorages #4148

use tusd datastorages #4148

Conversation

butonic commented Sep 1, 2023 • edited Loading

Noteworthy changes

simple upload needs to initialize the file upload with the filesize

update-docs bot commented Sep 1, 2023

butonic commented Sep 19, 2023

butonic commented Sep 19, 2023

micbar commented Sep 19, 2023 • edited Loading

butonic commented Sep 20, 2023

butonic commented Oct 16, 2023 • edited Loading

butonic commented Nov 1, 2023 • edited Loading

butonic commented Nov 1, 2023

butonic commented Nov 1, 2023 • edited Loading

aduffeck Dec 5, 2023

Choose a reason for hiding this comment

aduffeck Dec 5, 2023

Choose a reason for hiding this comment

butonic commented Jul 29, 2024

butonic commented Sep 1, 2023 •

edited

Loading

micbar commented Sep 19, 2023 •

edited

Loading

butonic commented Oct 16, 2023 •

edited

Loading

butonic commented Nov 1, 2023 •

edited

Loading

butonic commented Nov 1, 2023 •

edited

Loading