Skip to content

Commit

Permalink
Intern repository mapping entries
Browse files Browse the repository at this point in the history
All repositories generated by a module extension have the same repository mapping entries. Without interning, if a module extension generates N repositories, each such repository would have its own copy with (more than) N entries, resulting in overall memory usage quadratic in N.

We intern the entries, not the `RepositoryMapping` itself, since the latter also includes the owner repo, which differs between extension repos.

Output of `bazel info used-heap-size-after-gc` after running a build with a synthetic module extension generating N + 1 repos and requesting all of them:

```
before
N=100:     32MB
N=1000:    77MB
N=3000:   371MB
N=5000:   961MB
N=10000: 3614MB

after
N=100:     32MB
N=1000:    44MB
N=3000:    71MB
N=5000:    91MB
N=10000:  158MB
```

Closes #19269.

PiperOrigin-RevId: 558940840
Change-Id: I07402f203b5f11bf448a1ae9e9ee4637ad4c536d
  • Loading branch information
fmeum authored and Copybara-Service committed Aug 22, 2023
1 parent 8337dd7 commit 74aadb2
Showing 1 changed file with 13 additions and 1 deletion.
Expand Up @@ -17,6 +17,8 @@
import com.google.auto.value.AutoValue;
import com.google.common.base.Preconditions;
import com.google.common.collect.ImmutableMap;
import com.google.common.collect.Interner;
import com.google.devtools.build.lib.concurrent.BlazeInterners;
import java.util.HashMap;
import java.util.Map;
import java.util.Map.Entry;
Expand All @@ -34,6 +36,15 @@
@AutoValue
public abstract class RepositoryMapping {

// All repositories generated by a module extension have the same repository mapping
// entries. Without interning, if a module extension generates N repositories, each such
// repository would have its own copy with (more than) N entries, resulting in memory usage that
// is quadratic in N.
// Note: We intern the entries, not the RepositoryMapping itself, because the latter also includes
// the owner repo, which differs between extension repos.
private static final Interner<ImmutableMap<String, RepositoryName>> ENTRIES_INTERNER =
BlazeInterners.newWeakInterner();

// Always fallback to the requested name
public static final RepositoryMapping ALWAYS_FALLBACK = createAllowingFallback(ImmutableMap.of());

Expand All @@ -60,7 +71,8 @@ public static RepositoryMapping createAllowingFallback(Map<String, RepositoryNam

private static RepositoryMapping createInternal(
Map<String, RepositoryName> entries, RepositoryName ownerRepo) {
return new AutoValue_RepositoryMapping(ImmutableMap.copyOf(entries), ownerRepo);
return new AutoValue_RepositoryMapping(
ENTRIES_INTERNER.intern(ImmutableMap.copyOf(entries)), ownerRepo);
}

/**
Expand Down

0 comments on commit 74aadb2

Please sign in to comment.