Skip to content

feat(grpc-gcp): prime scaled channels before publish - #14232

Merged
rahul2393 merged 2 commits into
mainfrom
fm/dcp-split-5-channel-primer
Sep 2, 2026
Merged

feat(grpc-gcp): prime scaled channels before publish#14232
rahul2393 merged 2 commits into
mainfrom
fm/dcp-split-5-channel-primer

Conversation

@rahul2393

Copy link
Copy Markdown
Contributor

With dynamic scaling enabled, a freshly built channel was published to the pool the moment its delegate was constructed. Its first real RPCs then paid for connection establishment, TLS, and any service-side warm-up, so every scale-up event injected a latency spike into live traffic right when the pool was already under load.

Change

  • Optional channel primer. New GcpChannelPrimer hook (setChannelPrimer) issues a cheap end-to-end RPC on each newly built scale-up channel before it is published; for Cloud Spanner this can be a SELECT 1. Channels in a scale-up batch are primed concurrently and each one is published as soon as its own prime succeeds, so one slow channel never holds back the rest. A null primer (the default) keeps the existing publish-on-build path. Only dynamic scale-up channels are primed; the initial pool is not.
  • Bounded attempts. Each attempt is bounded by channelPrimeTimeout (default 10s) and retried up to channelPrimeMaxAttempts (default 3) with exponential backoff (100ms doubling, capped at 5s). The timeout covers the whole attempt, including the synchronous prime() call. A primer still blocked inside prime() at the timeout fails the channel without a retry, since nothing can stop that call; primer invocations run on a dedicated executor so a blocked primer cannot starve the shared scheduler that drives scale-up, draining, and timeouts. Exhausted or timed-out channels are closed and counted in the new scale_up_prime_failures metric.
  • Scale-up accounting. Channels still priming count toward both the pool size cap and the desired size, so a second scale-up during a slow prime batch cannot rebuild the same shortfall or construct delegates past maxSize. A prime abandoned because its call never returned keeps its slot until the call does, so a stuck primer can pin at most maxSize slots rather than one thread per scale-up event.
  • Lifecycle. Unpublished priming channels are never visible to pickers; shutdown cancels every pending prime and closes its delegate. Per-RPC pick and stream-accounting paths are unchanged, with no new hot-path allocations.
  • Configuration. setChannelPrimeTimeout rejects negative or non-nanosecond-representable durations (zero uses the default); setChannelPrimeMaxAttempts rejects negatives (zero uses the default).

@rahul2393
rahul2393 requested review from a team as code owners September 1, 2026 19:00

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a channel priming mechanism (GcpChannelPrimer) to warm up newly built delegate channels during dynamic scale-up before they are published to the channel pool. It adds options to configure the primer, timeout, and maximum retry attempts, along with a dedicated cached thread pool to execute priming tasks asynchronously without blocking the shared background scheduler. Additionally, it updates the dynamic scale-up logic to account for in-flight priming channels to prevent over-provisioning, integrates a new metric (scale_up_prime_failures) to track priming failures, and includes comprehensive unit tests validating the priming lifecycle, timeouts, retries, and shutdown behavior. There are no review comments, so I have no feedback to provide.

Comment thread grpc-gcp-java/src/main/java/com/google/cloud/grpc/GcpManagedChannel.java Outdated
Comment thread grpc-gcp-java/src/main/java/com/google/cloud/grpc/GcpManagedChannel.java Outdated
Comment thread grpc-gcp-java/src/main/java/com/google/cloud/grpc/GcpManagedChannel.java Outdated
@rahul2393

Copy link
Copy Markdown
Contributor Author

/gemini review

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a channel priming mechanism for channels built via dynamic scale-up in GcpManagedChannel. It adds GcpChannelPrimeController and the GcpChannelPrimer interface to allow warming up channels with a cheap, read-only RPC before they serve live traffic. Configuration options for priming timeout, maximum attempts, and exponential backoff with jitter are added to GcpManagedChannelOptions. Additionally, a new metric scale_up_prime_failures is introduced to track priming failures. There are no review comments to evaluate, so I have no feedback to provide.

@rahul2393
rahul2393 merged commit dd75645 into main Sep 2, 2026
214 checks passed
@rahul2393
rahul2393 deleted the fm/dcp-split-5-channel-primer branch September 2, 2026 16:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants