Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

New cache state #38804

merged 7 commits into from Jun 11, 2021

New cache state #38804

merged 7 commits into from Jun 11, 2021


Copy link


Add a new state in the cache to know the folders that have been shallow scanned to avoid re-scanning them

Related Issue

Motivation and Context

A partial scan implies that a folder might have been scanned on its first level but not deeper. Except for the size, the rest of the information can be reused and there is no need to scan the folder again (if there is no change detected).
This PR improves this scenario and prevents hitting the backend storage if there is no change

How Has This Been Tested?

Checked with an slow external storage

  1. Create folders "/ext/A/B/C/D" in the backend
  2. Ensure "/ext/A" contains plenty of files and folders
  3. Enter in "/ext/A". A scan of the backend should happen
  4. Reload the page

Page will load faster because we won't hit the external storage.

Screenshots (if appropriate):

Types of changes

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Database schema changes (next release will require increase of minor version instead of patch)
  • Breaking change (fix or feature that would cause existing functionality to change)
  • Technical debt
  • Tests only (no source changes)


  • Code changes
  • Unit tests added
  • Acceptance tests added
  • Documentation ticket raised:
  • Changelog item, see TEMPLATE

Copy link

update-docs bot commented Jun 3, 2021

Thanks for opening this pull request! The maintainers of this repository would appreciate it if you would create a changelog item based on your changes.

@@ -1412,6 +1413,8 @@ private function getCacheEntry($storage, $internalPath, $relativePath) {
$storage->getPropagator()->propagateChange($internalPath, \time());
$data = $cache->get($internalPath);
$this->unlockFile($relativePath, ILockingProvider::LOCK_SHARED);
} elseif ($data['size'] === IScanner::SIZE_SHALLOW_SCANNED) {
$cache->correctFolderSize($internalPath, $data);
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Lock shouldn't be needed because we don't hit the storage... should we propagate the change using the propagator?

@jvillafanez jvillafanez marked this pull request as ready for review June 4, 2021 07:28
Copy link

@JammingBen JammingBen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added input regarding the getIncomplete() method. Other than that it looks good to me, but changelog is missing.

lib/private/Files/Cache/Cache.php Show resolved Hide resolved
Copy link
Member Author

This is ready.
The only open question is #38804 (comment) but I don't think it's critical. Size should be properly update in the DB in every directory, and client should perform a propfind and update the size of the requested folders. No need to force a change in the etag to force change detection just for the size of a folder.

Copy link

💥 Acceptance tests webUIAddUsers-chrome-mariadb10.2-php7.4 failed. The build is cancelled...

Copy link
Fatal error: Call to undefined method GuzzleHttp\Exception\ConnectException::getResponse() (Behat\Testwork\Call\Exception\FatalThrowableError)
Some problem with mailhog - I will restart CI.

Copy link

sonarcloud bot commented Jun 11, 2021

Kudos, SonarCloud Quality Gate passed!

Bug A 0 Bugs
Vulnerability A 0 Vulnerabilities
Security Hotspot A 0 Security Hotspots
Code Smell A 2 Code Smells

No Coverage information No Coverage information
No Duplication information No Duplication information

@jvillafanez jvillafanez merged commit cba1e6b into master Jun 11, 2021
@delete-merged-branch delete-merged-branch bot deleted the new_cache_state branch June 11, 2021 08:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
None yet

Successfully merging this pull request may close these issues.

None yet

4 participants