feat(grpc-gcp): prime scaled channels before publish - #14232
Conversation
There was a problem hiding this comment.
Code Review
This pull request introduces a channel priming mechanism (GcpChannelPrimer) to warm up newly built delegate channels during dynamic scale-up before they are published to the channel pool. It adds options to configure the primer, timeout, and maximum retry attempts, along with a dedicated cached thread pool to execute priming tasks asynchronously without blocking the shared background scheduler. Additionally, it updates the dynamic scale-up logic to account for in-flight priming channels to prevent over-provisioning, integrates a new metric (scale_up_prime_failures) to track priming failures, and includes comprehensive unit tests validating the priming lifecycle, timeouts, retries, and shutdown behavior. There are no review comments, so I have no feedback to provide.
|
/gemini review |
There was a problem hiding this comment.
Code Review
This pull request introduces a channel priming mechanism for channels built via dynamic scale-up in GcpManagedChannel. It adds GcpChannelPrimeController and the GcpChannelPrimer interface to allow warming up channels with a cheap, read-only RPC before they serve live traffic. Configuration options for priming timeout, maximum attempts, and exponential backoff with jitter are added to GcpManagedChannelOptions. Additionally, a new metric scale_up_prime_failures is introduced to track priming failures. There are no review comments to evaluate, so I have no feedback to provide.
With dynamic scaling enabled, a freshly built channel was published to the pool the moment its delegate was constructed. Its first real RPCs then paid for connection establishment, TLS, and any service-side warm-up, so every scale-up event injected a latency spike into live traffic right when the pool was already under load.
Change
GcpChannelPrimerhook (setChannelPrimer) issues a cheap end-to-end RPC on each newly built scale-up channel before it is published; for Cloud Spanner this can be aSELECT 1. Channels in a scale-up batch are primed concurrently and each one is published as soon as its own prime succeeds, so one slow channel never holds back the rest. Anullprimer (the default) keeps the existing publish-on-build path. Only dynamic scale-up channels are primed; the initial pool is not.channelPrimeTimeout(default 10s) and retried up tochannelPrimeMaxAttempts(default 3) with exponential backoff (100ms doubling, capped at 5s). The timeout covers the whole attempt, including the synchronousprime()call. A primer still blocked insideprime()at the timeout fails the channel without a retry, since nothing can stop that call; primer invocations run on a dedicated executor so a blocked primer cannot starve the shared scheduler that drives scale-up, draining, and timeouts. Exhausted or timed-out channels are closed and counted in the newscale_up_prime_failuresmetric.maxSize. A prime abandoned because its call never returned keeps its slot until the call does, so a stuck primer can pin at mostmaxSizeslots rather than one thread per scale-up event.setChannelPrimeTimeoutrejects negative or non-nanosecond-representable durations (zero uses the default);setChannelPrimeMaxAttemptsrejects negatives (zero uses the default).