From 72acbf769a39ff3d439dc580de9a2f4b3ae70f4b Mon Sep 17 00:00:00 2001 From: Robert Lucian Chiriac Date: Tue, 31 Mar 2020 14:34:51 +0300 Subject: [PATCH 1/3] Update autoscaling.md --- docs/deployments/autoscaling.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/deployments/autoscaling.md b/docs/deployments/autoscaling.md index ecf31872e3..ccfbaacbfb 100644 --- a/docs/deployments/autoscaling.md +++ b/docs/deployments/autoscaling.md @@ -30,7 +30,7 @@ Cortex autoscales your web services based on your configuration. * `max_replica_concurrency` (default: 1024): This is the maximum number of in-flight requests per replica before requests are rejected with HTTP error code 503. `max_replica_concurrency` includes requests that are currently being processed as well as requests that are waiting in the replica's queue (a replica can actively process `workers_per_replica` * `threads_per_worker` requests concurrently, and will hold any additional requests in a local queue). Decreasing `max_replica_concurrency` and configuring the client to retry when it receives 503 responses will improve queue fairness by preventing requests from sitting in long queues. - Note (if `workers_per_replica` > 1): Because requests are randomly assigned to workers within a replica (which leads to unbalanced worker queues), clients may receive 503 responses before reaching `max_replica_concurrency`. For example, if you set `workers_per_replica: 2` and `max_replica_concurrency: 100`, each worker will have a maximum queue length of 50 requests. If your replica receives 90 requests, there is a possibility that more than 50 requests are routed to 1 worker, therefore each additional request beyond the 50 requests are responded with a 503. + *Note (if `workers_per_replica` > 1): Because requests are randomly assigned to workers within a replica (which leads to unbalanced worker queues), clients may receive 503 responses before reaching `max_replica_concurrency`. For example, if you set `workers_per_replica: 2` and `max_replica_concurrency: 100`, each worker will be allowed to handle 50 requests concurrently. If your replica receives 90 requests, there is a possibility that more than 50 requests are routed to 1 worker, therefore each additional request beyond the 50 requests are responded with a 503.* * `window` (default: 60s): The time over which to average the API wide in-flight requests (which is the sum of in-flight requests in each replica). The longer the window, the slower the autoscaler will react to changes in API wide in-flight requests, since it is averaged over the `window`. API wide in-flight requests is calculated every 10 seconds, so `window` must be a multiple of 10 seconds. From 7d763dd6f7daded724b997b44ff1e78fcb624653 Mon Sep 17 00:00:00 2001 From: Robert Lucian Chiriac Date: Tue, 31 Mar 2020 15:10:28 +0300 Subject: [PATCH 2/3] Update autoscaling.md --- docs/deployments/autoscaling.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/deployments/autoscaling.md b/docs/deployments/autoscaling.md index ccfbaacbfb..58a6cf44f3 100644 --- a/docs/deployments/autoscaling.md +++ b/docs/deployments/autoscaling.md @@ -30,7 +30,7 @@ Cortex autoscales your web services based on your configuration. * `max_replica_concurrency` (default: 1024): This is the maximum number of in-flight requests per replica before requests are rejected with HTTP error code 503. `max_replica_concurrency` includes requests that are currently being processed as well as requests that are waiting in the replica's queue (a replica can actively process `workers_per_replica` * `threads_per_worker` requests concurrently, and will hold any additional requests in a local queue). Decreasing `max_replica_concurrency` and configuring the client to retry when it receives 503 responses will improve queue fairness by preventing requests from sitting in long queues. - *Note (if `workers_per_replica` > 1): Because requests are randomly assigned to workers within a replica (which leads to unbalanced worker queues), clients may receive 503 responses before reaching `max_replica_concurrency`. For example, if you set `workers_per_replica: 2` and `max_replica_concurrency: 100`, each worker will be allowed to handle 50 requests concurrently. If your replica receives 90 requests, there is a possibility that more than 50 requests are routed to 1 worker, therefore each additional request beyond the 50 requests are responded with a 503.* + *Note (if `workers_per_replica` > 1): Because requests are randomly assigned to workers within a replica (which leads to unbalanced worker queues), clients may receive 503 responses before reaching `max_replica_concurrency`. For example, if you set `workers_per_replica: 2` and `max_replica_concurrency: 100`, each worker will be allowed to handle 50 requests concurrently. If your replica receives 90 requests that take the same amount of time to process, there is a 24.6% possibility that more than 50 requests are routed to 1 worker, therefore each additional request beyond the 50 requests are responded with a 503. To prevent that, increasing the `max_replica_concurrency` is recommended to minimize the probability of getting 503 responses.* * `window` (default: 60s): The time over which to average the API wide in-flight requests (which is the sum of in-flight requests in each replica). The longer the window, the slower the autoscaler will react to changes in API wide in-flight requests, since it is averaged over the `window`. API wide in-flight requests is calculated every 10 seconds, so `window` must be a multiple of 10 seconds. From 157820a6b96351261ff5ced168a8775d1aab4217 Mon Sep 17 00:00:00 2001 From: David Eliahu Date: Tue, 31 Mar 2020 13:24:59 -0700 Subject: [PATCH 3/3] Update autoscaling.md --- docs/deployments/autoscaling.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/deployments/autoscaling.md b/docs/deployments/autoscaling.md index 58a6cf44f3..39edfaaa6d 100644 --- a/docs/deployments/autoscaling.md +++ b/docs/deployments/autoscaling.md @@ -30,7 +30,7 @@ Cortex autoscales your web services based on your configuration. * `max_replica_concurrency` (default: 1024): This is the maximum number of in-flight requests per replica before requests are rejected with HTTP error code 503. `max_replica_concurrency` includes requests that are currently being processed as well as requests that are waiting in the replica's queue (a replica can actively process `workers_per_replica` * `threads_per_worker` requests concurrently, and will hold any additional requests in a local queue). Decreasing `max_replica_concurrency` and configuring the client to retry when it receives 503 responses will improve queue fairness by preventing requests from sitting in long queues. - *Note (if `workers_per_replica` > 1): Because requests are randomly assigned to workers within a replica (which leads to unbalanced worker queues), clients may receive 503 responses before reaching `max_replica_concurrency`. For example, if you set `workers_per_replica: 2` and `max_replica_concurrency: 100`, each worker will be allowed to handle 50 requests concurrently. If your replica receives 90 requests that take the same amount of time to process, there is a 24.6% possibility that more than 50 requests are routed to 1 worker, therefore each additional request beyond the 50 requests are responded with a 503. To prevent that, increasing the `max_replica_concurrency` is recommended to minimize the probability of getting 503 responses.* + *Note (if `workers_per_replica` > 1): Because requests are randomly assigned to workers within a replica (which leads to unbalanced worker queues), clients may receive 503 responses before reaching `max_replica_concurrency`. For example, if you set `workers_per_replica: 2` and `max_replica_concurrency: 100`, each worker will be allowed to handle 50 requests concurrently. If your replica receives 90 requests that take the same amount of time to process, there is a 24.6% possibility that more than 50 requests are routed to 1 worker, and each request that is routed to that worker above 50 is responded to with a 503. To address this, it is recommended to implement client retries for 503 errors, or to increase `max_replica_concurrency` to minimize the probability of getting 503 responses.* * `window` (default: 60s): The time over which to average the API wide in-flight requests (which is the sum of in-flight requests in each replica). The longer the window, the slower the autoscaler will react to changes in API wide in-flight requests, since it is averaged over the `window`. API wide in-flight requests is calculated every 10 seconds, so `window` must be a multiple of 10 seconds.