[2.x] Intern directory item conversions in MappedFileConverter

[hoangmaihuy]

Disclaimer: Developed with Claude Code and human review in the loop

Fixes #1750

Problem

MappedFileConverter.toVirtualFile expands a directory classpath entry into one
MappedVirtualFile per contained file. Because every consumer converts its classpath
independently and the FileConverter is shared build-wide, a directory that appears on many
consumers' classpaths is re-materialized once per consumer — producing large numbers of
value-identical MappedVirtualFile instances (encoded-path strings included) retained
simultaneously. See #1750 for the full write-up.

Change

• Cache per-item conversions in MappedFileConverter, keyed by the file's
  (lastModified, size), mirroring the metadata-keyed caching already used by
  ClasspathCache. Consumers sharing a classpath now share one MappedVirtualFile per
  contained file instead of each re-materializing a copy.
• Directory conversions are still rebuilt from a fresh listing on every call — only the
  items are shared. Directory membership is therefore always current (added/removed files
  are reflected), and each item is revalidated against its (lastModified, size) before reuse.
• Items are held through a SoftReference: while any live conversion references an item the
  reference cannot clear (simultaneously-held conversions keep sharing one instance); once no
  conversion holds it, the GC may reclaim it under memory pressure, bounding the retention of a
  long-lived shared converter. A reclaimed item is simply rebuilt on the next conversion.
• Top-level conversions are intentionally left uncached. Consumers hold a source
  conversion across edits and rely on its lazy contentHash reading the content of first
  access; interning those would fuse instances whose laziness must stay independent and break
  incremental source change detection.

Why it's safe

• Interned item hashes are never consumed by change detection. Source stamps read
  contentHash only on top-level source files (fresh instances). Classpath/library change
  detection stamps by path (forHashInRootPaths → FarmHash.ofPath), re-resolving class
  files rather than reading MappedDirectory.items — even for directory classpath entries.
  So sharing item instances does not affect incremental correctness.
• A cached item is served only while its (lastModified, size) is unchanged, so a memoized
  contentHash/contentHashStr is never served across a detected content change (the same
  heuristic ClasspathCache already relies on).
• The cache is an instance field of the converter (not a global singleton), so its retention
  is bounded by the converter's lifetime, and further bounded under pressure by the soft
  references.

Results

Micro-benchmark (FileConverterBenchmark, JMH; 2000-file directory, 100 consumers holding
conversions simultaneously): ~12% faster average conversion and ~4.5× less retained heap
(26.8 MB → 5.9 MB) for a shared converter vs a fresh converter per consumer.

Macro (real clean compile — fresh empty action cache, wiped target + boot, ~300-subproject
build, 132,362 class files):

Build	wall time	in-JVM peak	jstat peak	post-GC	vs stock (peak)
Stock (no interning)	968 s	17,984 MB	16.41 GB	3,795 MB	—
Directory-item interning	927 s (−4.2%)	16,944 MB	15.48 GB	3,884 MB	−1,040 MB (−5.8%)

jstat sample distribution corroborates the ordering (the interning build never exceeded
16 GB; stock touched 16.4 GB).

Caveat: one run per build; wall time and momentary heap peaks are both timing-noisy (treat
exact figures as ±a few percent), but the direction — lower peak heap and lower wall time — is
consistent across in-JVM peak, jstat peak, the distribution, and wall clock.

Tests

MappedFileConverterInterningSpec covers:

• items shared across repeated directory conversions;
• an item re-converted fresh once its file is rewritten (same size, bumped mtime);
• directory content changes (add) reflected in a fresh conversion;
• a missing directory that later exists;
• top-level conversions keeping fresh-instance semantics.

🤖 Generated with Claude Code

[eed3si9n]

Thanks for the contribution!

[eed3si9n]

Thanks!