Eliminate O(N²) Realm work in large-folder enumeration - #10466
Merged
Conversation
…ation Opening a large remote folder (e.g. /Talk, ~7000 items) for the first time was pathologically slow: the non-paginated depth-1 write used by change / working-set enumeration (depth1ReadUpdateItemMetadatas) took ~10 min, and once the folder was fully cached did not complete within 19 min — ~11 items/sec pinning a single core. The cost was per-item O(N²) work over a flat folder: - The logical-address dedup/lookup queries (RealmItemMetadata.hasLocation) key on the normalized columns. normalizedServerUrl is indexed but is non-selective for a flat folder (every child shares the parent URL), so each per-item eviction/lookup scanned all siblings. The selective column, normalizedFileName, was not indexed. - processItemMetadatasToDelete / processItemMetadatasToUpdate diffed the existing and incoming sets with in-memory .contains(where:) / .first(where:) linear scans — O(existing × updated) — which no index can help. Changes: - Index normalizedFileName (@persisted(indexed: true)); bump the Realm schema to 204 (addedNormalizedFileNameIndexToRealmItemMetadata). The index is rebuilt automatically on open; no data migration. - Rewrite the two set-diff loops to O(N): a Set of incoming ocIds and an ocId-keyed dictionary of existing rows replace the linear scans, drop a redundant per-item Realm re-open, and cache inheritedKeepDownloaded per parent serverUrl. Delete/rename, #9923/#10054 keep-downloaded preservation and ticket-96101301 dedup semantics are unchanged. - Add os_signpost intervals and opt-in wall-clock PERF logging (performanceLoggingEnabled user default, off by default) across the enumeration path, and gate FileProviderLogger.debug before its Task dispatch so disabled debug logging no longer allocates a Task per item. Measured on production (~6982-item folder): the offending write drops from 635 s / >19 min to ~1.1 s (~6200 items/sec). Full package suite green (325 tests). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Iva Horn <iva.horn@nextcloud.com>
i2h3
requested review from
camilasan,
claucambra,
mgallien and
nilsding
as code owners
July 28, 2026 09:49
i2h3
added
os: 🍎 macOS
github-project-automation
Bot
moved this to 🧭 Planning evaluation (don't pick)
in 💻 Desktop Clients team
Collaborator
Author
|
/backport to stable-34.0 |
i2h3
moved this from 🧭 Planning evaluation (don't pick)
to 🏗️ In progress
in 💻 Desktop Clients team
Contributor
|
Artifact containing the AppImage: nextcloud-appimage-pr-10466.zip Digest: To test this change/fix you can download the above artifact file, unzip it, and run it. Please make sure to quit your existing Nextcloud app and backup your data. |
github-project-automation
Bot
moved this from 🏗️ In progress
to ☑️ Done
in 💻 Desktop Clients team
|
8 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Another one for @jospoortvliet which should reduce the whole problem to a pure network/server issue (very slow WebDAV responses).
This PR fixes pathological slowness when first enumerating a large remote folder in the macOS File Provider extension: the non-paginated depth-1 write used by change/working-set enumeration took over ten minutes for a ~7,000-item folder and could fail to complete within 19 minutes, running at ~11 items/sec on a single core. The cost was per-item O(N²) work over a flat folder — logical-address dedup/lookup queries that fell back to scanning every sibling because the selective normalizedFileName column was unindexed, plus two in-memory set-diff loops (.contains(where:)/.first(where:)) that no index can help. The fix indexes normalizedFileName (Realm schema 204, rebuilt automatically on open, no data migration) and rewrites the set-diff to O(N) using an ocId-keyed set and dictionary while preserving all existing delete/rename, keep-downloaded (#9923/#10054) and dedup (ticket-96101301) semantics; it also adds os_signpost intervals and opt-in wall-clock instrumentation that were used to diagnose the issue. Measured on production, the offending write drops from ~635 s (or >19 min) to ~1.1 s (~6,200 items/sec), and the full package test suite (325 tests) remains green.
Checklist
AI (if applicable)