fix: reduce memory spike from DirectoryFetcher and SessionContext clone#11606
Draft
warp-dev-github-integration[bot] wants to merge 1 commit into
Draft
fix: reduce memory spike from DirectoryFetcher and SessionContext clone#11606warp-dev-github-integration[bot] wants to merge 1 commit into
warp-dev-github-integration[bot] wants to merge 1 commit into
Conversation
Two changes to address excessive memory usage (9.06GB) observed in Sentry issue 7259255054: 1. Wrap cached_directory_entries DashMap in Arc in SessionContext so that cloning SessionContext is O(1) instead of deep-copying the entire directory cache. The heap profile showed SessionContext::clone() allocating ~2.36GB from DashMap deep-copy. 2. Cap DirectoryFetcher::fetch_files_async at 10,000 entries to prevent unbounded Vec<DirectoryItem> allocation. The heap profile showed Vec<DirectoryItem>::from_iter consuming ~4.71GB (52% of total). Sentry: https://sentry.io/organizations/warpdotdev/issues/7259255054/ Co-Authored-By: Oz <oz-agent@warp.dev>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Fix excessive memory usage (9.06GB) caused by two issues in the context chips directory fetcher and session context path completion cache:
SessionContext::clone()deep-copies the entire DashMap cache (~2.36GB): Thecached_directory_entriesfield is aDashMap<TypedPathBuf, Arc<Vec<EngineDirEntry>>>. SinceSessionContextderivesClone, every clone deep-copies all keys and values. This happens on everyrefetch_directorycall and whenever context chips are reset. Wrapping the DashMap inArcmakes cloning O(1) while sharing the same underlying cache.DirectoryFetcher::fetch_files_asynccreates unboundedVec<DirectoryItem>(~4.71GB): When a directory contains an extremely large number of files, all entries are converted toDirectoryItemobjects with no limit. Adding a cap of 10,000 entries prevents the unbounded allocation while still being more than sufficient for UI display purposes.Heap profile evidence (event 1e605502188449c19fa61c71475beb66):
Vec<DirectoryItem>::from_iterviaDirectoryFetcher::fetch_files_async: 4.71GB (52%)DashMap::cloneviaSessionContext::clone: 2.36GB (26%)Linked Issue
Testing
cargo check -p warp --libpassescargo clippy -p warp --lib -- -D warningspassescargo fmt -- --checkpassesAgent Mode
Conversation: https://staging.warp.dev/conversation/444c97c2-e0e9-4f82-8ee8-70e5bf987717
Run: https://oz.staging.warp.dev/runs/019e55a1-427e-7d3e-9b1d-e054750250ec
This PR was generated with Oz.