fix: erase persistent_term leak in GRPC.Client.Connection on disconnect#509
Merged
sleipnir merged 1 commit intoMar 11, 2026
Merged
Conversation
Each connect/disconnect cycle leaks one persistent_term entry because the entry is never erased. Since persistent_term is not garbage-collected, this causes unbounded memory growth on long-running nodes.
sleipnir
approved these changes
Mar 11, 2026
Collaborator
|
Thank you @ryochin |
cgreeno
added a commit
to cgreeno/grpc
that referenced
this pull request
Apr 28, 2026
…ycles Mirrors the regression pattern from PR elixir-grpc#509, which added a 100-cycle test to prove the persistent_term leak fix. Same idea, applied to the tables the refactor introduced: the shared Registry and the per-LB ETS tables. Snapshots :ets.all() and the Registry size before the loop, cycles 500 connect+disconnect pairs, asserts the Registry returns to its starting size and no more than a handful of new tables exist (VM may create a few incidentally during the run).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
GRPC.Client.Connection.init/1stores a persistent_term entry keyed by the channel's ref:On disconnect,
handle_call({:disconnect, _})drops thevirtual_channelfrom state viaMap.drop/2, then stops the GenServer via{:continue, :stop}:However, the persistent_term entry is never erased -- neither in the disconnect handler nor in
terminate/2(which is a no-op:def terminate(_reason, _state), do: :ok).By default, each
connect/2call generates a fresh ref viamake_ref(), every connect/disconnect cycle permanently leaks one persistent_term entry.Impact
Applications that create short-lived connections (e.g. connect -> RPC -> disconnect per request) accumulate persistent_term entries indefinitely. Unlike regular process memory, persistent_term entries are not garbage-collected and persist for the lifetime of the BEAM node, causing steady memory growth with no upper bound.
Proposed fix
Erase the persistent_term entry in
handle_call({:disconnect, ...}), beforeMap.dropremoves the ref from state:Additionally,
terminate/2should erase the entry as a safety net for abnormal termination paths wheredisconnectis never called:Note:
terminate/2alone is insufficient becauseMap.drop(state, [:real_channels, :virtual_channel])removes the ref from state beforeterminateis called in the normal disconnect path. Both locations are needed.Reproduction
Requires a gRPC server listening on
localhost:50051(any service will do).