Already pass size difference to the cache updater to avoid calculation of the folder size to update #28310
Conversation
juliushaertl
force-pushed
the
enh/s3-size-calc
branch
2 times, most recently
from
eb540ed
to
720fc91
Compare
juliushaertl
force-pushed
the
enh/s3-size-calc
branch
from
720fc91
to
7e93284
Compare
|
Pushed a fix for the comment and an additional commit for another case where we can speed up updating the size: Additionally the sizeDifference is also passed in the View when creating directories (which always have a difference of 0) and when calling file_put_contents where we know the size difference from the storage return value. This avoids an unnecessary size calculation when creating user avatars where it would have calculated the size of the appdata_*/avatar folder for each user creation call which would cause a high read load on databases with lots of users. |
juliushaertl
force-pushed
the
enh/s3-size-calc
branch
from
7e93284
to
56ca54a
Compare
|
CI is unhappy |
szaimen
added
2. developing
juliushaertl
force-pushed
the
enh/s3-size-calc
branch
from
56ca54a
to
dc4f59c
Compare
juliushaertl
added
3. to review
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
| @@ -266,7 +267,7 @@ | |||
| // because we have no clue about the cause we can only throw back a 500/Internal Server Error | |||
| throw new Exception($this->l10n->t('Could not write file contents')); | |||
| } | |||
| [$count, $result] = \OC_Helper::streamCopy($data, $target); | |||
| [$writtenByteCount, $result] = \OC_Helper::streamCopy($data, $target); | |||
Check notice
Code scanning / Psalm
PossiblyInvalidArgument
| try { | ||
| $count = $partStorage->writeStream($internalPartPath, $wrappedData); | ||
| $writtenByteCount = $partStorage->writeStream($internalPartPath, $wrappedData); |
Check notice
Code scanning / Psalm
PossiblyInvalidArgument
|
Still CI failures... |
|
Restarted drone as I cannot see the failure locally and also can't see a relation to the change. |
|
failing tests look related to me... |
|
Sorry, read the output wrong, will have a look |
…lation on s3 Signed-off-by: Julius Härtl <jus@bitgrid.net>
Signed-off-by: Julius Härtl <jus@bitgrid.net>
Signed-off-by: Julius Härtl <jus@bitgrid.net>
…p correcting the folder size Signed-off-by: Julius Härtl <jus@bitgrid.net>
juliushaertl
force-pushed
the
enh/s3-size-calc
branch
from
cca0b2b
to
cc7c467
Compare
|
I haven't found a proper way yet to pass this along. I'll close this for now, but will keep a note to probably discuss further for future performance optimizations. |
With S3 the scanner will never return anything so the updater always falls back to a full size calculation. Since we know the transferred filesize during upload, we can pass it as an optional parameter then in order to directly update the parent folder sizes.
Additionally the sizeDifference is also passed in the View when creating directories (which always have a difference of 0) and when calling file_put_contents where we know the size difference from the storage return value.