Skip to content

Reduce per-worker memory of long-lived caches#5969

Merged
ondrejmirtes merged 2 commits into
2.2.xfrom
reduce-worker-memory
Jul 2, 2026
Merged

Reduce per-worker memory of long-lived caches#5969
ondrejmirtes merged 2 commits into
2.2.xfrom
reduce-worker-memory

Conversation

@ondrejmirtes

Copy link
Copy Markdown
Member

Two independent reductions of per-worker retained memory, found by profiling self-analysis (make phpstan reports the sum of per-worker peak memory):

1. Restore value sharing in name scope maps hydrated from the file cache

FileTypeMapper persists name scope maps with var_export(), which cannot represent shared references. A freshly created map shares the uses/constUses arrays and parent IntermediaryNameScope objects across all scopes of a file (COW / object references), but a map hydrated from the cache carries a private copy of everything in every scope — e.g. MutatingScope.php hydrates as ~200 scopes, each with its own copy of the same ~150-entry uses array and its own duplicate of the whole parent chain (~1.7 MB for that one file). Interning the hydrated scopes through a per-load pool restores the sharing: the retained size of FileTypeMapper::$memoryCache in an average worker drops from ~70 MB to ~7 MB.

2. Cap CachedParser by total cached source size

The AST of a parsed file takes roughly 50–60× more memory than its source text, and the parser cache limit was entry-count-based only — a few very large files (in self-analysis e.g. NodeScopeResolver.php, TypeCombinatorTest.php) each pinned tens of MB of AST in a worker for the whole run. The cache now also evicts by total source bytes of the cached entries (CACHED_SOURCE_BYTES_LIMIT, 512 KB of source ≈ ~30 MB of AST), on top of the existing count limit and LRU order. Eviction happens before insertion, so the entry being inserted is never evicted, and a single oversized file is still cached while it is being analysed.

Results (self-analysis, warm caches, 10-core machine / 8 workers):

Used memory
2.2.x 2.72 GB
with this PR 2.30 GB

Per-worker peaks drop from ~340 MB to ~280 MB; elapsed time is unchanged. Full test suite passes.

🤖 Generated with Claude Code

https://claude.ai/code/session_017DjgMHgfvwhD4ogdTaRtLp

ondrejmirtes and others added 2 commits July 2, 2026 08:29
The name scope maps are persisted with var_export() which cannot
represent shared references. A hydrated map of a file with many members
carried its own copy of the same uses/constUses arrays and a duplicate
of the whole parent scope chain in every single scope, taking up many
times more memory than a freshly created map where these values are
shared. Interning the hydrated scopes restores the sharing.

In self-analysis this shrinks the retained size of
FileTypeMapper::$memoryCache in an average worker process
from ~70 MB to ~7 MB.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017DjgMHgfvwhD4ogdTaRtLp
The AST of a parsed file takes up roughly 50-60x more memory than the
source code itself, and the cache limit was entry-count-based only, so
a handful of very large files could pin tens of megabytes of AST in
each worker process for the whole run. Evict by total source bytes of
the cached entries in addition to the entry count.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017DjgMHgfvwhD4ogdTaRtLp
@ondrejmirtes ondrejmirtes merged commit bf4d513 into 2.2.x Jul 2, 2026
665 of 668 checks passed
@ondrejmirtes ondrejmirtes deleted the reduce-worker-memory branch July 2, 2026 06:38
@staabm staabm mentioned this pull request Jul 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant