[stable-34.0] Eliminate O(N²) Realm work in large-folder enumeration - #10469
Merged
Conversation
…ation Opening a large remote folder (e.g. /Talk, ~7000 items) for the first time was pathologically slow: the non-paginated depth-1 write used by change / working-set enumeration (depth1ReadUpdateItemMetadatas) took ~10 min, and once the folder was fully cached did not complete within 19 min — ~11 items/sec pinning a single core. The cost was per-item O(N²) work over a flat folder: - The logical-address dedup/lookup queries (RealmItemMetadata.hasLocation) key on the normalized columns. normalizedServerUrl is indexed but is non-selective for a flat folder (every child shares the parent URL), so each per-item eviction/lookup scanned all siblings. The selective column, normalizedFileName, was not indexed. - processItemMetadatasToDelete / processItemMetadatasToUpdate diffed the existing and incoming sets with in-memory .contains(where:) / .first(where:) linear scans — O(existing × updated) — which no index can help. Changes: - Index normalizedFileName (@persisted(indexed: true)); bump the Realm schema to 204 (addedNormalizedFileNameIndexToRealmItemMetadata). The index is rebuilt automatically on open; no data migration. - Rewrite the two set-diff loops to O(N): a Set of incoming ocIds and an ocId-keyed dictionary of existing rows replace the linear scans, drop a redundant per-item Realm re-open, and cache inheritedKeepDownloaded per parent serverUrl. Delete/rename, #9923/#10054 keep-downloaded preservation and ticket-96101301 dedup semantics are unchanged. - Add os_signpost intervals and opt-in wall-clock PERF logging (performanceLoggingEnabled user default, off by default) across the enumeration path, and gate FileProviderLogger.debug before its Task dispatch so disabled debug logging no longer allocates a Task per item. Measured on production (~6982-item folder): the offending write drops from 635 s / >19 min to ~1.1 s (~6200 items/sec). Full package suite green (325 tests). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Iva Horn <iva.horn@nextcloud.com>
backportbot
Bot
requested review from
claucambra,
i2h3,
mgallien and
nilsding
as code owners
July 28, 2026 10:45
backportbot
Bot
added
os: 🍎 macOS
Contributor
|
Artifact containing the AppImage: nextcloud-appimage-pr-10469.zip Digest: To test this change/fix you can download the above artifact file, unzip it, and run it. Please make sure to quit your existing Nextcloud app and backup your data. |
|
8 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Backport of PR #10466