From 143a760b2118645425037f47d4cd68334c30ca91 Mon Sep 17 00:00:00 2001 From: jsteemann Date: Tue, 21 Sep 2021 11:16:12 +0200 Subject: [PATCH 1/4] docs for APM-164 --- 3.9/http/general.md | 44 +++++++++++++++++++++++------ 3.9/programs-arangod-http.md | 17 ++++++++++- 3.9/release-notes-api-changes39.md | 29 +++++++++++++++++++ 3.9/release-notes-new-features39.md | 30 ++++++++++++++++++++ 4 files changed, 110 insertions(+), 10 deletions(-) diff --git a/3.9/http/general.md b/3.9/http/general.md index fe572e2473..4c9f47f71e 100644 --- a/3.9/http/general.md +++ b/3.9/http/general.md @@ -26,7 +26,7 @@ format or ArangoDB's custom [VelocyPack](https://github.com/arangodb/velocypack) binary format. Details on the expected format and JSON attributes can be found in the documentation of the individual API endpoints. -Clients sending requests to ArangoDB must use either HTTP 1.0, HTTP 1.1, HTTP 2 +Clients sending requests to ArangoDB must use either HTTP 1.1, HTTP 2 or VelocyStream. Other HTTP versions or protocols are not supported by ArangoDB. Clients are required to include the `Content-Length` HTTP header with the @@ -39,19 +39,16 @@ HTTP Keep-Alive --------------- ArangoDB supports HTTP keep-alive. If the client does not send a `Connection` -header in its request, and the client uses HTTP version 1.1, ArangoDB will assume -the client wants to keep alive the connection. +header in its request, ArangoDB will assume the client wants to keep alive the +connection. If clients do not wish to use the keep-alive feature, they should explicitly indicate that by sending a `Connection: Close` HTTP header in the request. -ArangoDB will close connections automatically for clients that send requests -using HTTP 1.0, except if they send an `Connection: Keep-Alive` header. - The default Keep-Alive timeout can be specified at server start using the `--http.keep-alive-timeout` startup option. -Establishing TCP connections is expensive, since it takes several ping pongs +Establishing TCP connections is expensive, since it takes several roundtrips between the communication parties. Therefore you can use connection keep-alive to send several HTTP request over one TCP-connection; each request is treated independently by definition. You can use this feature @@ -540,14 +537,13 @@ The following APIs may use request forwarding: - `/_api/control_pregel` - `/_api/cursor` -- `/_api/document` - `/_api/job` - `/_api/replication` - `/_api/query` - `/_api/tasks` - `/_api/transaction` -Note: since forwarding such requests require an additional cluster-internal HTTP +Note: since forwarding such requests requires an additional cluster-internal HTTP request, they should be avoided when possible for best performance. Typically this is accomplished either by directing the requests to the correct Coordinator at a client-level or by enabling request "stickiness" on a load balancer. Since @@ -557,3 +553,33 @@ request forwarding as a fall-back solution. Note: some endpoints which return "global" data, such as `GET /_api/tasks` will only return data corresponding to the server on which the request is executed. These endpoints will generally not work well with load-balancers. + +Overload control +---------------- + +Since version 3.9.0 arangod will return an `x-arango-queue-time-seconds` HTTP +header with all responses. This header contains the most recent request +queueing/dequeing time (in seconds) as tracked by the server's scheduler. +This value can be used by client applications and drivers to detect server +overload and react on it. +The arangod startup option `--http.return-queue-time-header` can be set to +`false` to suppress these headers in responses sent by arangod. + +In a cluster, the value returned in the `x-arango-queue-time-seconds` header +is the most recent queueing/dequeing request time of the coordinator the +request was sent to, except if the request is forwarded by the coordinator to +another coordinator. In that case, the value will indicate the current +queueing/dequeing time of the forwarded-to coordinator. + +In addition, client applications and drivers can optionally augment the +requests they send to arangod with the header `x-arango-queue-time-seconds`. +If set, the value of the header should contain the maximum server-side +queuing time (in seconds) that the client application is willing to accept. +If the header is set in an incoming request, arangod will compare the current +dequeing time from its scheduler with the maximum queue time value contained +in the request header. If the current quequeing time exceeds the value set +in the header, arangod will reject the request and return HTTP 412 +(precondition failed) with the error code 21004 (queue time violated). + +In a cluster, the `x-arango-queue-time-seconds` request header will be +checked on the receiving coordinator, before any request forwarding. diff --git a/3.9/programs-arangod-http.md b/3.9/programs-arangod-http.md index 199362ab94..62ed023cf8 100644 --- a/3.9/programs-arangod-http.md +++ b/3.9/programs-arangod-http.md @@ -14,6 +14,22 @@ Idle keep-alive connections will be closed by the server automatically when the timeout is reached. A keep-alive-timeout value 0 will disable the keep alive feature entirely. +## Queue time header + +Introduced in: v3.9.0 + +`--http.return-queue-time-header` + +If *true*, the server will return the `x-arango-queue-time-seconds` HTTP +header with all responses. The value contained in this header indicates the +current queueing/dequeing time for requests in the scheduler (in seconds). +Client applications and drivers can use this value to control the server +load and also react on overload. +Setting the option to *false* will make arangod not return the HTTP header +in reponses. + +The default value is *true*. + ## Hide Product header `--http.hide-product-header` @@ -24,7 +40,6 @@ responses. The default is *false*. - ## Allow method override `--http.allow-method-override` diff --git a/3.9/release-notes-api-changes39.md b/3.9/release-notes-api-changes39.md index 854270b3d8..c2ac5ad978 100644 --- a/3.9/release-notes-api-changes39.md +++ b/3.9/release-notes-api-changes39.md @@ -41,6 +41,35 @@ should be prepared for this feature. Also see [Database Naming Conventions](data-modeling-naming-conventions-database-names.html). +### Overload control + +ArangoDB version 3.9.0 will return an `x-arango-queue-time-seconds` HTTP +header with all responses. This header contains the most recent request +queueing/dequeing time (in seconds) as tracked by the server's scheduler. +This value can be used by client applications and drivers to detect server +overload and react on it. +The arangod startup option `--http.return-queue-time-header` can be set to +`false` to suppress these headers in responses sent by arangod. + +In a cluster, the value returned in the `x-arango-queue-time-seconds` header +is the most recent queueing/dequeing request time of the coordinator the +request was sent to, except if the request is forwarded by the coordinator to +another coordinator. In that case, the value will indicate the current +queueing/dequeing time of the forwarded-to coordinator. + +In addition, client applications and drivers can optionally augment the +requests they send to arangod with the header `x-arango-queue-time-seconds`. +If set, the value of the header should contain the maximum server-side +queuing time (in seconds) that the client application is willing to accept. +If the header is set in an incoming request, arangod will compare the current +dequeing time from its scheduler with the maximum queue time value contained +in the request header. If the current quequeing time exceeds the value set +in the header, arangod will reject the request and return HTTP 412 +(precondition failed) with the error code 21004 (queue time violated). + +In a cluster, the `x-arango-queue-time-seconds` request header will be +checked on the receiving coordinator, before any request forwarding. + ### Privilege changes ### Endpoint return value changes diff --git a/3.9/release-notes-new-features39.md b/3.9/release-notes-new-features39.md index 2ea28fdb36..5cd64ecb81 100644 --- a/3.9/release-notes-new-features39.md +++ b/3.9/release-notes-new-features39.md @@ -271,6 +271,36 @@ A pseudo log topic `"all"` was added. Setting the log level for the "all" log topic will adjust the log level for **all existing log topics**. For example, `--log.level all=debug` will set all log topics to log level "debug". +Overload control +---------------- + +ArangoDB version 3.9.0 will return an `x-arango-queue-time-seconds` HTTP +header with all responses. This header contains the most recent request +queueing/dequeing time (in seconds) as tracked by the server's scheduler. +This value can be used by client applications and drivers to detect server +overload and react on it. +The arangod startup option `--http.return-queue-time-header` can be set to +`false` to suppress these headers in responses sent by arangod. + +In a cluster, the value returned in the `x-arango-queue-time-seconds` header +is the most recent queueing/dequeing request time of the coordinator the +request was sent to, except if the request is forwarded by the coordinator to +another coordinator. In that case, the value will indicate the current +queueing/dequeing time of the forwarded-to coordinator. + +In addition, client applications and drivers can optionally augment the +requests they send to arangod with the header `x-arango-queue-time-seconds`. +If set, the value of the header should contain the maximum server-side +queuing time (in seconds) that the client application is willing to accept. +If the header is set in an incoming request, arangod will compare the current +dequeing time from its scheduler with the maximum queue time value contained +in the request header. If the current quequeing time exceeds the value set +in the header, arangod will reject the request and return HTTP 412 +(precondition failed) with the error code 21004 (queue time violated). + +In a cluster, the `x-arango-queue-time-seconds` request header will be +checked on the receiving coordinator, before any request forwarding. + Support info API ---------------- From e096d0213e1affeed88377b3a17a2b68aff92e1e Mon Sep 17 00:00:00 2001 From: Simran Spiller Date: Wed, 22 Sep 2021 11:17:00 +0200 Subject: [PATCH 2/4] Review --- 3.9/http/general.md | 36 +++++++++++++++-------------- 3.9/programs-arangod-http.md | 9 ++++---- 3.9/release-notes-api-changes39.md | 34 +++++++++++++-------------- 3.9/release-notes-new-features39.md | 30 ++++++++++++------------ 4 files changed, 56 insertions(+), 53 deletions(-) diff --git a/3.9/http/general.md b/3.9/http/general.md index 4c9f47f71e..02d8e22217 100644 --- a/3.9/http/general.md +++ b/3.9/http/general.md @@ -557,29 +557,31 @@ These endpoints will generally not work well with load-balancers. Overload control ---------------- -Since version 3.9.0 arangod will return an `x-arango-queue-time-seconds` HTTP -header with all responses. This header contains the most recent request -queueing/dequeing time (in seconds) as tracked by the server's scheduler. -This value can be used by client applications and drivers to detect server +Introduced in: v3.9.0 + +_arangod_ returns an `x-arango-queue-time-seconds` HTTP +header with all responses. This header contains the most recent request +queueing/dequeuing time (in seconds) as tracked by the server's scheduler. +This value can be used by client applications and drivers to detect server overload and react on it. + The arangod startup option `--http.return-queue-time-header` can be set to `false` to suppress these headers in responses sent by arangod. In a cluster, the value returned in the `x-arango-queue-time-seconds` header -is the most recent queueing/dequeing request time of the coordinator the -request was sent to, except if the request is forwarded by the coordinator to -another coordinator. In that case, the value will indicate the current -queueing/dequeing time of the forwarded-to coordinator. +is the most recent queueing/dequeuing request time of the Coordinator the +request was sent to, except if the request is forwarded by the Coordinator to +another Coordinator. In that case, the value will indicate the current +queueing/dequeuing time of the forwarded-to Coordinator. In addition, client applications and drivers can optionally augment the requests they send to arangod with the header `x-arango-queue-time-seconds`. -If set, the value of the header should contain the maximum server-side -queuing time (in seconds) that the client application is willing to accept. -If the header is set in an incoming request, arangod will compare the current -dequeing time from its scheduler with the maximum queue time value contained -in the request header. If the current quequeing time exceeds the value set -in the header, arangod will reject the request and return HTTP 412 +If set, the value of the header should contain the maximum server-side +queuing time (in seconds) that the client application is willing to accept. +If the header is set in an incoming request, arangod will compare the current +dequeuing time from its scheduler with the maximum queue time value contained +in the request header. If the current queueing time exceeds the value set +in the header, arangod will reject the request and return HTTP 412 (precondition failed) with the error code 21004 (queue time violated). - -In a cluster, the `x-arango-queue-time-seconds` request header will be -checked on the receiving coordinator, before any request forwarding. +In a cluster, the `x-arango-queue-time-seconds` request header will be +checked on the receiving Coordinator, before any request forwarding. diff --git a/3.9/programs-arangod-http.md b/3.9/programs-arangod-http.md index 62ed023cf8..62331fdf6f 100644 --- a/3.9/programs-arangod-http.md +++ b/3.9/programs-arangod-http.md @@ -22,11 +22,12 @@ alive feature entirely. If *true*, the server will return the `x-arango-queue-time-seconds` HTTP header with all responses. The value contained in this header indicates the -current queueing/dequeing time for requests in the scheduler (in seconds). -Client applications and drivers can use this value to control the server +current queueing/dequeuing time for requests in the scheduler (in seconds). +Client applications and drivers can use this value to control the server load and also react on overload. -Setting the option to *false* will make arangod not return the HTTP header -in reponses. + +Setting the option to `false` will make arangod not return the HTTP header +in responses. The default value is *true*. diff --git a/3.9/release-notes-api-changes39.md b/3.9/release-notes-api-changes39.md index c2ac5ad978..5286c881b9 100644 --- a/3.9/release-notes-api-changes39.md +++ b/3.9/release-notes-api-changes39.md @@ -43,32 +43,32 @@ Also see [Database Naming Conventions](data-modeling-naming-conventions-database ### Overload control -ArangoDB version 3.9.0 will return an `x-arango-queue-time-seconds` HTTP -header with all responses. This header contains the most recent request -queueing/dequeing time (in seconds) as tracked by the server's scheduler. -This value can be used by client applications and drivers to detect server +Starting with version 3.9.0, ArangoDB returns an `x-arango-queue-time-seconds` +HTTP header with all responses. This header contains the most recent request +queueing/dequeuing time (in seconds) as tracked by the server's scheduler. +This value can be used by client applications and drivers to detect server overload and react on it. + The arangod startup option `--http.return-queue-time-header` can be set to `false` to suppress these headers in responses sent by arangod. In a cluster, the value returned in the `x-arango-queue-time-seconds` header -is the most recent queueing/dequeing request time of the coordinator the -request was sent to, except if the request is forwarded by the coordinator to -another coordinator. In that case, the value will indicate the current -queueing/dequeing time of the forwarded-to coordinator. +is the most recent queueing/dequeuing request time of the Coordinator the +request was sent to, except if the request is forwarded by the Coordinator to +another Coordinator. In that case, the value will indicate the current +queueing/dequeuing time of the forwarded-to Coordinator. In addition, client applications and drivers can optionally augment the requests they send to arangod with the header `x-arango-queue-time-seconds`. -If set, the value of the header should contain the maximum server-side -queuing time (in seconds) that the client application is willing to accept. -If the header is set in an incoming request, arangod will compare the current -dequeing time from its scheduler with the maximum queue time value contained -in the request header. If the current quequeing time exceeds the value set -in the header, arangod will reject the request and return HTTP 412 +If set, the value of the header should contain the maximum server-side +queuing time (in seconds) that the client application is willing to accept. +If the header is set in an incoming request, arangod will compare the current +dequeuing time from its scheduler with the maximum queue time value contained +in the request header. If the current queueing time exceeds the value set +in the header, arangod will reject the request and return HTTP 412 (precondition failed) with the error code 21004 (queue time violated). - -In a cluster, the `x-arango-queue-time-seconds` request header will be -checked on the receiving coordinator, before any request forwarding. +In a cluster, the `x-arango-queue-time-seconds` request header will be +checked on the receiving Coordinator, before any request forwarding. ### Privilege changes diff --git a/3.9/release-notes-new-features39.md b/3.9/release-notes-new-features39.md index 5cd64ecb81..1a744799ca 100644 --- a/3.9/release-notes-new-features39.md +++ b/3.9/release-notes-new-features39.md @@ -274,32 +274,32 @@ topic will adjust the log level for **all existing log topics**. For example, Overload control ---------------- -ArangoDB version 3.9.0 will return an `x-arango-queue-time-seconds` HTTP -header with all responses. This header contains the most recent request -queueing/dequeing time (in seconds) as tracked by the server's scheduler. +Starting with version 3.9.0, ArangoDB returns an `x-arango-queue-time-seconds` +HTTP header with all responses. This header contains the most recent request +queueing/dequeuing time (in seconds) as tracked by the server's scheduler. This value can be used by client applications and drivers to detect server overload and react on it. + The arangod startup option `--http.return-queue-time-header` can be set to `false` to suppress these headers in responses sent by arangod. In a cluster, the value returned in the `x-arango-queue-time-seconds` header -is the most recent queueing/dequeing request time of the coordinator the -request was sent to, except if the request is forwarded by the coordinator to -another coordinator. In that case, the value will indicate the current -queueing/dequeing time of the forwarded-to coordinator. +is the most recent queueing/dequeuing request time of the Coordinator the +request was sent to, except if the request is forwarded by the Coordinator to +another Coordinator. In that case, the value will indicate the current +queueing/dequeuing time of the forwarded-to Coordinator. In addition, client applications and drivers can optionally augment the requests they send to arangod with the header `x-arango-queue-time-seconds`. -If set, the value of the header should contain the maximum server-side -queuing time (in seconds) that the client application is willing to accept. -If the header is set in an incoming request, arangod will compare the current -dequeing time from its scheduler with the maximum queue time value contained -in the request header. If the current quequeing time exceeds the value set -in the header, arangod will reject the request and return HTTP 412 +If set, the value of the header should contain the maximum server-side +queuing time (in seconds) that the client application is willing to accept. +If the header is set in an incoming request, arangod will compare the current +dequeuing time from its scheduler with the maximum queue time value contained +in the request header. If the current queueing time exceeds the value set +in the header, arangod will reject the request and return HTTP 412 (precondition failed) with the error code 21004 (queue time violated). - In a cluster, the `x-arango-queue-time-seconds` request header will be -checked on the receiving coordinator, before any request forwarding. +checked on the receiving Coordinator, before any request forwarding. Support info API ---------------- From 978f1e771032e3054acb33168128fa4ac5c5be96 Mon Sep 17 00:00:00 2001 From: Jan Date: Wed, 22 Sep 2021 17:12:10 +0200 Subject: [PATCH 3/4] Update 3.9/http/general.md --- 3.9/http/general.md | 19 +++++++++++++++++-- 1 file changed, 17 insertions(+), 2 deletions(-) diff --git a/3.9/http/general.md b/3.9/http/general.md index 02d8e22217..62bb17e120 100644 --- a/3.9/http/general.md +++ b/3.9/http/general.md @@ -582,6 +582,21 @@ If the header is set in an incoming request, arangod will compare the current dequeuing time from its scheduler with the maximum queue time value contained in the request header. If the current queueing time exceeds the value set in the header, arangod will reject the request and return HTTP 412 -(precondition failed) with the error code 21004 (queue time violated). +(precondition failed) with the error code 21004 (queue time violated). +Using a value of 0 or a non-numeric value in the header will lead to the +header value being ignored by arangod. + +There is also a new metric `arangodb_scheduler_queue_time_violations_total` +that is increased whenever a request is dropped because of the requested +queue time not being satisfiable. Administrators can use this metric to monitor +overload situations. Although all instance types will expose this metric, +it will likely always be `0` on DB servers and agency instances because the +`x-arango-queue-time-seconds` header is not used in cluster-internal requests. + In a cluster, the `x-arango-queue-time-seconds` request header will be -checked on the receiving Coordinator, before any request forwarding. +checked on the receiving Coordinator, before any request forwarding. If the +request is forwarded by the Coordinator to a different Coordinator, the +receiving Coordinator will also check the header on its own. +Apart from that, the header will not be included in cluster-internal requests +executed by the Coordinator, e.g. when the Coordinator issues sub-requests +to DB servers or agency instances. From 51df7f63249436c93fd9d1dde10c5925f847ab7a Mon Sep 17 00:00:00 2001 From: Simran Spiller Date: Mon, 27 Sep 2021 18:39:39 +0200 Subject: [PATCH 4/4] Spelling --- 3.9/http/general.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/3.9/http/general.md b/3.9/http/general.md index 62bb17e120..a5229bef50 100644 --- a/3.9/http/general.md +++ b/3.9/http/general.md @@ -586,17 +586,17 @@ in the header, arangod will reject the request and return HTTP 412 Using a value of 0 or a non-numeric value in the header will lead to the header value being ignored by arangod. -There is also a new metric `arangodb_scheduler_queue_time_violations_total` +There is also a metric `arangodb_scheduler_queue_time_violations_total` that is increased whenever a request is dropped because of the requested queue time not being satisfiable. Administrators can use this metric to monitor overload situations. Although all instance types will expose this metric, -it will likely always be `0` on DB servers and agency instances because the +it will likely always be `0` on DB-Servers and agency instances because the `x-arango-queue-time-seconds` header is not used in cluster-internal requests. In a cluster, the `x-arango-queue-time-seconds` request header will be checked on the receiving Coordinator, before any request forwarding. If the -request is forwarded by the Coordinator to a different Coordinator, the +request is forwarded by the Coordinator to a different Coordinator, the receiving Coordinator will also check the header on its own. Apart from that, the header will not be included in cluster-internal requests executed by the Coordinator, e.g. when the Coordinator issues sub-requests -to DB servers or agency instances. +to DB-Servers or Agency instances.