From f3bcf7b4187beefbc1a1e1d898ac11914d1652a0 Mon Sep 17 00:00:00 2001 From: Simran Spiller Date: Tue, 7 Nov 2023 15:57:25 +0100 Subject: [PATCH 01/13] Projection optimizer rule changes --- .../release-notes/version-3.12/api-changes-in-3-12.md | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/site/content/3.12/release-notes/version-3.12/api-changes-in-3-12.md b/site/content/3.12/release-notes/version-3.12/api-changes-in-3-12.md index 71b5b0071a..e8e05d289c 100644 --- a/site/content/3.12/release-notes/version-3.12/api-changes-in-3-12.md +++ b/site/content/3.12/release-notes/version-3.12/api-changes-in-3-12.md @@ -62,6 +62,16 @@ Note that it is still forbidden to use `_id` as a top-level attribute or sub-attribute in `fields` of persistent indexes. On the other hand, inverted indexes have been allowing to index and store the `_id` system attribute. +#### Optimizer rule changes + +The `remove-unnecessary-projections` AQL optimizer rule has been renamed to +`optimize-projections` and now includes an additional optimization. + +Moreover, a `remove-unnecessary-calculations-4` rule has been added. + +The affected endpoints are `POST /_api/cursor`, `POST /_api/explain`, and +`GET /_api/query/rules`. + ### Privilege changes From a931e48d9925545c3af93e3cc4ed66597a87d4a9 Mon Sep 17 00:00:00 2001 From: Simran Spiller Date: Fri, 10 Nov 2023 11:52:38 +0100 Subject: [PATCH 02/13] Shorten menu title --- site/content/3.10/arangograph/arangograph-changelog.md | 2 +- site/content/3.11/arangograph/arangograph-changelog.md | 2 +- site/content/3.12/arangograph/arangograph-changelog.md | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/site/content/3.10/arangograph/arangograph-changelog.md b/site/content/3.10/arangograph/arangograph-changelog.md index f8c53b06fb..0a3daf5c2a 100644 --- a/site/content/3.10/arangograph/arangograph-changelog.md +++ b/site/content/3.10/arangograph/arangograph-changelog.md @@ -1,6 +1,6 @@ --- title: ArangoGraph Changelog -menuTitle: ArangoGraph Changelog +menuTitle: Changelog weight: 75 description: >- The ArangoGraph changelog covers notable changes in behavior or new diff --git a/site/content/3.11/arangograph/arangograph-changelog.md b/site/content/3.11/arangograph/arangograph-changelog.md index f8c53b06fb..0a3daf5c2a 100644 --- a/site/content/3.11/arangograph/arangograph-changelog.md +++ b/site/content/3.11/arangograph/arangograph-changelog.md @@ -1,6 +1,6 @@ --- title: ArangoGraph Changelog -menuTitle: ArangoGraph Changelog +menuTitle: Changelog weight: 75 description: >- The ArangoGraph changelog covers notable changes in behavior or new diff --git a/site/content/3.12/arangograph/arangograph-changelog.md b/site/content/3.12/arangograph/arangograph-changelog.md index f8c53b06fb..0a3daf5c2a 100644 --- a/site/content/3.12/arangograph/arangograph-changelog.md +++ b/site/content/3.12/arangograph/arangograph-changelog.md @@ -1,6 +1,6 @@ --- title: ArangoGraph Changelog -menuTitle: ArangoGraph Changelog +menuTitle: Changelog weight: 75 description: >- The ArangoGraph changelog covers notable changes in behavior or new From 32bf9ebad5dfcd3f1580933550d29e70be01e321 Mon Sep 17 00:00:00 2001 From: Simran Spiller Date: Mon, 13 Nov 2023 12:48:20 +0100 Subject: [PATCH 03/13] API changes, formatting --- .../3.10/develop/http-api/databases.md | 5 +- .../version-3.10/api-changes-in-3-10.md | 29 ++++++++ .../3.11/develop/http-api/databases.md | 5 +- .../version-3.11/api-changes-in-3-11.md | 63 ++++++++++++++++ .../version-3.11/whats-new-in-3-11.md | 2 +- .../3.12/develop/http-api/databases.md | 5 +- .../version-3.12/api-changes-in-3-12.md | 73 ++++++++++++++++++- .../version-3.12/whats-new-in-3-12.md | 12 +-- 8 files changed, 177 insertions(+), 17 deletions(-) diff --git a/site/content/3.10/develop/http-api/databases.md b/site/content/3.10/develop/http-api/databases.md index 631beb835a..2e90265c6f 100644 --- a/site/content/3.10/develop/http-api/databases.md +++ b/site/content/3.10/develop/http-api/databases.md @@ -281,8 +281,9 @@ paths: is returned if the database was created successfully. '400': description: | - is returned if the request parameters are invalid or if a database with the - specified name already exists. + is returned if the request parameters are invalid, if a database with the + specified name already exists, or if the configured limit to the number + of databases has been reached. '403': description: | is returned if the request was not executed in the `_system` database. diff --git a/site/content/3.10/release-notes/version-3.10/api-changes-in-3-10.md b/site/content/3.10/release-notes/version-3.10/api-changes-in-3-10.md index b5b466f428..a7e690d0a5 100644 --- a/site/content/3.10/release-notes/version-3.10/api-changes-in-3-10.md +++ b/site/content/3.10/release-notes/version-3.10/api-changes-in-3-10.md @@ -126,6 +126,23 @@ if the request body was an empty array. Example: Now, a request like this succeeds and returns an empty array as response. +#### Limit to the number of databases in a deployment + +Introduced in: v3.10.10 + +The new `--database.max-databases` startup option can cap the number of databases +and creating databases using the `POST /_api/database` endpoint can thus now fail +for this reason if your deployment is at or above the configured maximum. Example: + +```json +{ + "code": 400, + "error": true, + "errorMessage": "unable to create additional database because it would exceed the configured maximum number of databases (2)", + "errorNum": 32 +} +``` + ### Endpoint return value changes - Since ArangoDB 3.8, there have been two APIs for retrieving the metrics in two @@ -807,6 +824,18 @@ The following metrics have been added: | `arangodb_file_descriptors_limit` | System limit for the number of open files for the arangod process. | | `arangodb_file_descriptors_current` | Number of file descriptors currently opened by the arangod process. | +--- + +Introduced in: v3.10.11 + +The following metrics have been added to improve the observability of in-memory +cache subsystem: + +- `rocksdb_cache_free_memory_tasks_total` +- `rocksdb_cache_free_memory_tasks_duration_total` +- `rocksdb_cache_migrate_tasks_total` +- `rocksdb_cache_migrate_tasks_duration_total` + #### Pregel API When loading the graph data into memory, a `"loading"` state is now returned by diff --git a/site/content/3.11/develop/http-api/databases.md b/site/content/3.11/develop/http-api/databases.md index 631beb835a..2e90265c6f 100644 --- a/site/content/3.11/develop/http-api/databases.md +++ b/site/content/3.11/develop/http-api/databases.md @@ -281,8 +281,9 @@ paths: is returned if the database was created successfully. '400': description: | - is returned if the request parameters are invalid or if a database with the - specified name already exists. + is returned if the request parameters are invalid, if a database with the + specified name already exists, or if the configured limit to the number + of databases has been reached. '403': description: | is returned if the request was not executed in the `_system` database. diff --git a/site/content/3.11/release-notes/version-3.11/api-changes-in-3-11.md b/site/content/3.11/release-notes/version-3.11/api-changes-in-3-11.md index e18b17d9ed..3f22ad2132 100644 --- a/site/content/3.11/release-notes/version-3.11/api-changes-in-3-11.md +++ b/site/content/3.11/release-notes/version-3.11/api-changes-in-3-11.md @@ -262,6 +262,23 @@ Disabled: arangodb_agency_cache_callback_number{role="SINGLE"}0 ``` +#### Limit to the number of databases in a deployment + +Introduced in: v3.10.10, v3.11.2 + +The new `--database.max-databases` startup option can cap the number of databases +and creating databases using the `POST /_api/database` endpoint can thus now fail +for this reason if your deployment is at or above the configured maximum. Example: + +```json +{ + "code": 400, + "error": true, + "errorMessage": "unable to create additional database because it would exceed the configured maximum number of databases (2)", + "errorNum": 32 +} +``` + ### Endpoint return value changes Introduced in: v3.8.8, v3.9.4, v3.10.1 @@ -599,6 +616,52 @@ The following metric stores the peak value of the `rocksdb_cache_allocated` metr |:------|:------------| | `rocksdb_cache_peak_allocated` | Global peak memory allocation of ArangoDB in-memory caches. | +--- + +Introduced in: v3.11.2 + +The following metrics have been added about the LZ4 compression for values in +the in-memory edge cache: + +- `rocksdb_cache_edge_inserts_effective_entries_size_total` +- `rocksdb_cache_edge_inserts_uncompressed_entries_size_total` +- `rocksdb_cache_edge_compression_ratio` + +--- + +Introduced in: v3.10.11, v3.11.4 + +The following metrics have been added to improve the observability of in-memory +cache subsystem: + +- `rocksdb_cache_free_memory_tasks_total` +- `rocksdb_cache_free_memory_tasks_duration_total` +- `rocksdb_cache_migrate_tasks_total` +- `rocksdb_cache_migrate_tasks_duration_total` + +--- + +Introduced in: v3.11.4 + +The following metrics have been added to improve the observability of in-memory +edge cache: + +- `rocksdb_cache_edge_compressed_inserts_total` +- `rocksdb_cache_edge_empty_inserts_total` +- `rocksdb_cache_edge_inserts_total` + +--- + +Introduced in: v3.11.5 + +The following metrics have been added to monitor and detect temporary or +permanent connectivity issues as well as how many scheduler threads are in the +detached state: + +- `arangodb_network_connectivity_failures_coordinators` +- `arangodb_network_connectivity_failures_dbservers_total` +- `arangodb_scheduler_num_detached_threads` + #### Log level API Introduced in: v3.10.2 diff --git a/site/content/3.11/release-notes/version-3.11/whats-new-in-3-11.md b/site/content/3.11/release-notes/version-3.11/whats-new-in-3-11.md index 78c3b93999..9e82670fad 100644 --- a/site/content/3.11/release-notes/version-3.11/whats-new-in-3-11.md +++ b/site/content/3.11/release-notes/version-3.11/whats-new-in-3-11.md @@ -515,7 +515,7 @@ Query Statistics: ### New stage in query profiling output -Introduced in: v3.10.3, v3.11.0 +Introduced in: v3.10.3 The query profiling output has a new `instantiating executors` stage. The time spent in this stage is the time needed to create the query executors diff --git a/site/content/3.12/develop/http-api/databases.md b/site/content/3.12/develop/http-api/databases.md index 631beb835a..2e90265c6f 100644 --- a/site/content/3.12/develop/http-api/databases.md +++ b/site/content/3.12/develop/http-api/databases.md @@ -281,8 +281,9 @@ paths: is returned if the database was created successfully. '400': description: | - is returned if the request parameters are invalid or if a database with the - specified name already exists. + is returned if the request parameters are invalid, if a database with the + specified name already exists, or if the configured limit to the number + of databases has been reached. '403': description: | is returned if the request was not executed in the `_system` database. diff --git a/site/content/3.12/release-notes/version-3.12/api-changes-in-3-12.md b/site/content/3.12/release-notes/version-3.12/api-changes-in-3-12.md index e09d1b73da..418a0d0c1e 100644 --- a/site/content/3.12/release-notes/version-3.12/api-changes-in-3-12.md +++ b/site/content/3.12/release-notes/version-3.12/api-changes-in-3-12.md @@ -69,6 +69,23 @@ Moreover, a `remove-unnecessary-calculations-4` rule has been added. The affected endpoints are `POST /_api/cursor`, `POST /_api/explain`, and `GET /_api/query/rules`. +#### Limit to the number of databases in a deployment + +Introduced in: v3.10.10, v3.11.2 + +The new `--database.max-databases` startup option can cap the number of databases +and creating databases using the `POST /_api/database` endpoint can thus now fail +for this reason if your deployment is at or above the configured maximum. Example: + +```json +{ + "code": 400, + "error": true, + "errorMessage": "unable to create additional database because it would exceed the configured maximum number of databases (2)", + "errorNum": 32 +} +``` + ### Privilege changes @@ -125,11 +142,59 @@ produced no warnings. #### Metrics API -The metrics endpoint includes the following new metric: +The metrics endpoint includes the following new metrics about AQL queries and +ongoing dumps: + +- `arangodb_aql_cursors_active` +- `arangodb_dump_memory_usage` +- `arangodb_dump_ongoing` +- `arangodb_dump_threads_blocked_total` + +--- + +Introduced in: v3.11.2 + +The following metrics have been added about the LZ4 compression for values in +the in-memory edge cache: + +- `rocksdb_cache_edge_inserts_effective_entries_size_total` +- `rocksdb_cache_edge_inserts_uncompressed_entries_size_total` +- `rocksdb_cache_edge_compression_ratio` + +--- + +Introduced in: v3.10.11, v3.11.4 + +The following metrics have been added to improve the observability of in-memory +cache subsystem: + +- `rocksdb_cache_free_memory_tasks_total` +- `rocksdb_cache_free_memory_tasks_duration_total` +- `rocksdb_cache_migrate_tasks_total` +- `rocksdb_cache_migrate_tasks_duration_total` + +--- + +Introduced in: v3.11.4 + +The following metrics have been added to improve the observability of in-memory +edge cache: + +- `rocksdb_cache_edge_compressed_inserts_total` +- `rocksdb_cache_edge_empty_inserts_total` +- `rocksdb_cache_edge_inserts_total` + +--- + +Introduced in: v3.11.5 + +The following metrics have been added to monitor and detect temporary or +permanent connectivity issues as well as how many scheduler threads are in the +detached state: -| Label | Description | -|:------|:------------| -| `arangodb_aql_cursors_active` | Current number of active AQL query cursors. | +- `arangodb_network_connectivity_failures_coordinators` +- `arangodb_network_connectivity_failures_dbservers_total` +- `arangodb_scheduler_num_detached_threads` ### Endpoints moved diff --git a/site/content/3.12/release-notes/version-3.12/whats-new-in-3-12.md b/site/content/3.12/release-notes/version-3.12/whats-new-in-3-12.md index baa2d5586e..613b7df9f4 100644 --- a/site/content/3.12/release-notes/version-3.12/whats-new-in-3-12.md +++ b/site/content/3.12/release-notes/version-3.12/whats-new-in-3-12.md @@ -77,7 +77,7 @@ indexes have been allowing to index and store the `_id` system attribute. ### LZ4 compression for values in the in-memory edge cache -Introduced in: v3.11.2, v3.12.0 +Introduced in: v3.11.2 LZ4 compression of edge index cache values allows to store more data in main memory than without compression, so the available memory can be used more @@ -115,7 +115,7 @@ cache, but not decreased when data gets evicted from the cache. ### Limit the number of databases in a deployment -Introduced in: v3.10.10, v3.11.2, v3.12.0 +Introduced in: v3.10.10, v3.11.2 The `--database.max-databases` startup option allows you to limit the number of databases that can exist in parallel in a deployment. You can use this @@ -147,7 +147,7 @@ in the meantime should be removed manually. ### Cluster-internal connectivity checks -Introduced in: v3.11.5, v.3.12.0 +Introduced in: v3.11.5 This feature makes Coordinators and DB-Servers in a cluster periodically send check requests to each other, in order to see if all nodes can connect to @@ -183,7 +183,7 @@ collections to include. ### In-memory edge cache startup options and metrics -Introduced in: v3.11.4, v3.12.0 +Introduced in: v3.11.4 The following startup options have been added: @@ -204,9 +204,9 @@ The following metrics have been added: ### Observability of in-memory cache subsystem -Introduced in: v3.10.11, v.3.11.4, v.3.12.0 +Introduced in: v3.10.11, v3.11.4 -The following metrics have been added to improve the observability of in-memory +The following metrics have been added to improve the observability of the in-memory cache subsystem: - `rocksdb_cache_free_memory_tasks_total`: Total number of free memory tasks that were scheduled by the in-memory edge cache subsystem. This metric will From 677fc9d0c1b681546b3d4a01016288689ee559ed Mon Sep 17 00:00:00 2001 From: Simran Spiller Date: Mon, 13 Nov 2023 13:42:33 +0100 Subject: [PATCH 04/13] DOC-597 | Improvement to Hot Backups --- .../version-3.10/whats-new-in-3-10.md | 9 ++ .../version-3.11/whats-new-in-3-11.md | 9 ++ .../version-3.12/whats-new-in-3-12.md | 85 ++++++++++--------- 3 files changed, 65 insertions(+), 38 deletions(-) diff --git a/site/content/3.10/release-notes/version-3.10/whats-new-in-3-10.md b/site/content/3.10/release-notes/version-3.10/whats-new-in-3-10.md index 30b0c414e3..fcd46756de 100644 --- a/site/content/3.10/release-notes/version-3.10/whats-new-in-3-10.md +++ b/site/content/3.10/release-notes/version-3.10/whats-new-in-3-10.md @@ -1574,6 +1574,15 @@ The following system metrics have been added: | `arangodb_file_descriptors_limit` | System limit for the number of open files for the arangod process. | | `arangodb_file_descriptors_current` | Number of file descriptors currently opened by the arangod process. | +### More instant Hot Backups + +Introduced in: v3.10.10, v3.11.3 + +Cluster deployments no longer wait for all in-progress transactions to get +committed when a user requests a Hot Backup. The waiting could cause deadlocks +and thus Hot Backups to fail, in particular in ArangoGraph. Now, Hot Backups are +created immediately and commits have to wait until the backup process is done. + ## Client tools ### arangobench diff --git a/site/content/3.11/release-notes/version-3.11/whats-new-in-3-11.md b/site/content/3.11/release-notes/version-3.11/whats-new-in-3-11.md index 9e82670fad..ff1730f8c7 100644 --- a/site/content/3.11/release-notes/version-3.11/whats-new-in-3-11.md +++ b/site/content/3.11/release-notes/version-3.11/whats-new-in-3-11.md @@ -1221,6 +1221,15 @@ The following system metrics have been added: | `arangodb_file_descriptors_limit` | System limit for the number of open files for the arangod process. | | `arangodb_file_descriptors_current` | Number of file descriptors currently opened by the arangod process. | +### More instant Hot Backups + +Introduced in: v3.10.10, v3.11.3 + +Cluster deployments no longer wait for all in-progress transactions to get +committed when a user requests a Hot Backup. The waiting could cause deadlocks +and thus Hot Backups to fail, in particular in ArangoGraph. Now, Hot Backups are +created immediately and commits have to wait until the backup process is done. + ### In-memory edge cache startup options and metrics Introduced in: v3.11.4 diff --git a/site/content/3.12/release-notes/version-3.12/whats-new-in-3-12.md b/site/content/3.12/release-notes/version-3.12/whats-new-in-3-12.md index 613b7df9f4..820552ebc1 100644 --- a/site/content/3.12/release-notes/version-3.12/whats-new-in-3-12.md +++ b/site/content/3.12/release-notes/version-3.12/whats-new-in-3-12.md @@ -181,6 +181,53 @@ collections to include. ## Miscellaneous changes +### Active AQL query cursors metric + +The `arangodb_aql_cursors_active` metric has been added and shows the number +of active AQL query cursors. + +AQL query cursors are created for queries that produce more results than +specified in the `batchSize` query option (default value: `1000`). Such results +can be fetched incrementally by client operations in chunks. +As it is unclear if and when a client will fetch any remaining data from a +cursor, every cursor has a server-side timeout value (TTL) after which it is +considered inactive and garbage-collected. + +### RocksDB .sst file partitioning (experimental) + +The following experimental startup options for RockDB .sst file partitioning +have been added: + +- `--rocksdb.partition-files-for-documents` +- `--rocksdb.partition-files-for-primary-index` +- `--rocksdb.partition-files-for-edge-index` +- `--rocksdb.partition-files-for-persistent-index` + +Enabling any of these options makes RocksDB's compaction write the +data for different collections/shards/indexes into different .sst files. +Otherwise, the document data from different collections/shards/indexes +can be mixed and written into the same .sst files. + +When these options are enabled, the RocksDB compaction is more efficient since +a lot of different collections/shards/indexes are written to in parallel. +The disadvantage of enabling these options is that there can be more .sst +files than when the option is turned off, and the disk space used by +these .sst files can be higher. +In particular, on deployments with many collections/shards/indexes +this can lead to a very high number of .sst files, with the potential +of outgrowing the maximum number of file descriptors the ArangoDB process +can open. Thus, these options should only be enabled on deployments with a +limited number of collections/shards/indexes. + +### More instant Hot Backups + +Introduced in: v3.10.10, v3.11.3 + +Cluster deployments no longer wait for all in-progress transactions to get +committed when a user requests a Hot Backup. The waiting could cause deadlocks +and thus Hot Backups to fail, in particular in ArangoGraph. Now, Hot Backups are +created immediately and commits have to wait until the backup process is done. + ### In-memory edge cache startup options and metrics Introduced in: v3.11.4 @@ -226,44 +273,6 @@ cache subsystem: by the cache subsystem to migrate existing cache hash tables to a bigger or smaller table. -### RocksDB .sst file partitioning (experimental) - -The following experimental startup options for RockDB .sst file partitioning -have been added: - -- `--rocksdb.partition-files-for-documents` -- `--rocksdb.partition-files-for-primary-index` -- `--rocksdb.partition-files-for-edge-index` -- `--rocksdb.partition-files-for-persistent-index` - -Enabling any of these options makes RocksDB's compaction write the -data for different collections/shards/indexes into different .sst files. -Otherwise, the document data from different collections/shards/indexes -can be mixed and written into the same .sst files. - -When these options are enabled, the RocksDB compaction is more efficient since -a lot of different collections/shards/indexes are written to in parallel. -The disadvantage of enabling these options is that there can be more .sst -files than when the option is turned off, and the disk space used by -these .sst files can be higher. -In particular, on deployments with many collections/shards/indexes -this can lead to a very high number of .sst files, with the potential -of outgrowing the maximum number of file descriptors the ArangoDB process -can open. Thus, these options should only be enabled on deployments with a -limited number of collections/shards/indexes. - -### Active AQL query cursors metric - -The `arangodb_aql_cursors_active` metric has been added and shows the number -of active AQL query cursors. - -AQL query cursors are created for queries that produce more results than -specified in the `batchSize` query option (default value: `1000`). Such results -can be fetched incrementally by client operations in chunks. -As it is unclear if and when a client will fetch any remaining data from a -cursor, every cursor has a server-side timeout value (TTL) after which it is -considered inactive and garbage-collected. - ### Detached scheduler threads Introduced in: v3.11.5 From 7004e629cb7ac801bacc15e590c8cab1ff8d4622 Mon Sep 17 00:00:00 2001 From: Simran Spiller Date: Mon, 13 Nov 2023 13:52:52 +0100 Subject: [PATCH 05/13] --log.max-queued-entries startup option --- .../version-3.10/whats-new-in-3-10.md | 14 ++++++++++++++ .../version-3.11/whats-new-in-3-11.md | 14 ++++++++++++++ .../version-3.12/whats-new-in-3-12.md | 14 ++++++++++++++ 3 files changed, 42 insertions(+) diff --git a/site/content/3.10/release-notes/version-3.10/whats-new-in-3-10.md b/site/content/3.10/release-notes/version-3.10/whats-new-in-3-10.md index fcd46756de..2c29b72196 100644 --- a/site/content/3.10/release-notes/version-3.10/whats-new-in-3-10.md +++ b/site/content/3.10/release-notes/version-3.10/whats-new-in-3-10.md @@ -1384,6 +1384,20 @@ attempt to create an additional database fails with error if other databases are dropped first. The default value for this option is unlimited, so an arbitrary amount of databases can be created. +### Configurable maximum for queued log entries + +Introduced in: v3.10.12 + +The new `--log.max-queued-entries` startup option lets you configure how many +log entries are queued in a background thread. + +Log entries are pushed on a queue for asynchronous writing unless you enable the +`--log.force-direct` startup option. If you use a slow log output (e.g. syslog), +the queue might grow and eventually overflow. + +You can configure the upper bound of the queue with this option. If the queue is +full, log entries are written synchronously until the queue has space again. + ## Miscellaneous changes ### Optimizer rules endpoint diff --git a/site/content/3.11/release-notes/version-3.11/whats-new-in-3-11.md b/site/content/3.11/release-notes/version-3.11/whats-new-in-3-11.md index ff1730f8c7..33d738aac8 100644 --- a/site/content/3.11/release-notes/version-3.11/whats-new-in-3-11.md +++ b/site/content/3.11/release-notes/version-3.11/whats-new-in-3-11.md @@ -1036,6 +1036,20 @@ permanent connectivity issues: - `arangodb_network_connectivity_failures_dbservers_total`: Number of failed connectivity check requests sent to DB-Servers. +### Configurable maximum for queued log entries + +Introduced in: v3.10.12, v3.11.5 + +The new `--log.max-queued-entries` startup option lets you configure how many +log entries are queued in a background thread. + +Log entries are pushed on a queue for asynchronous writing unless you enable the +`--log.force-direct` startup option. If you use a slow log output (e.g. syslog), +the queue might grow and eventually overflow. + +You can configure the upper bound of the queue with this option. If the queue is +full, log entries are written synchronously until the queue has space again. + ## Miscellaneous changes ### Write-write conflict improvements diff --git a/site/content/3.12/release-notes/version-3.12/whats-new-in-3-12.md b/site/content/3.12/release-notes/version-3.12/whats-new-in-3-12.md index 820552ebc1..2c3f38a108 100644 --- a/site/content/3.12/release-notes/version-3.12/whats-new-in-3-12.md +++ b/site/content/3.12/release-notes/version-3.12/whats-new-in-3-12.md @@ -169,6 +169,20 @@ permanent connectivity issues: - `arangodb_network_connectivity_failures_dbservers_total`: Number of failed connectivity check requests sent to DB-Servers. +### Configurable maximum for queued log entries + +Introduced in: v3.10.12, v3.11.5 + +The new `--log.max-queued-entries` startup option lets you configure how many +log entries are queued in a background thread. + +Log entries are pushed on a queue for asynchronous writing unless you enable the +`--log.force-direct` startup option. If you use a slow log output (e.g. syslog), +the queue might grow and eventually overflow. + +You can configure the upper bound of the queue with this option. If the queue is +full, log entries are written synchronously until the queue has space again. + ## Client tools ### arangodump From 74d6a9636dc32e829c8603412ca0b36a8e943210 Mon Sep 17 00:00:00 2001 From: Simran Spiller Date: Mon, 13 Nov 2023 13:58:27 +0100 Subject: [PATCH 06/13] AQL operation title formatting --- site/content/3.10/aql/high-level-operations/collect.md | 2 +- site/content/3.10/aql/high-level-operations/filter.md | 2 +- site/content/3.10/aql/high-level-operations/for.md | 2 +- site/content/3.10/aql/high-level-operations/insert.md | 2 +- site/content/3.10/aql/high-level-operations/let.md | 2 +- site/content/3.10/aql/high-level-operations/limit.md | 2 +- site/content/3.10/aql/high-level-operations/remove.md | 2 +- site/content/3.10/aql/high-level-operations/replace.md | 2 +- site/content/3.10/aql/high-level-operations/return.md | 2 +- site/content/3.10/aql/high-level-operations/search.md | 2 +- site/content/3.10/aql/high-level-operations/sort.md | 2 +- site/content/3.10/aql/high-level-operations/update.md | 2 +- site/content/3.10/aql/high-level-operations/upsert.md | 2 +- site/content/3.10/aql/high-level-operations/window.md | 2 +- site/content/3.10/aql/high-level-operations/with.md | 2 +- site/content/3.11/aql/high-level-operations/collect.md | 2 +- site/content/3.11/aql/high-level-operations/filter.md | 2 +- site/content/3.11/aql/high-level-operations/for.md | 2 +- site/content/3.11/aql/high-level-operations/insert.md | 2 +- site/content/3.11/aql/high-level-operations/let.md | 2 +- site/content/3.11/aql/high-level-operations/limit.md | 2 +- site/content/3.11/aql/high-level-operations/remove.md | 2 +- site/content/3.11/aql/high-level-operations/replace.md | 2 +- site/content/3.11/aql/high-level-operations/return.md | 2 +- site/content/3.11/aql/high-level-operations/search.md | 2 +- site/content/3.11/aql/high-level-operations/sort.md | 2 +- site/content/3.11/aql/high-level-operations/update.md | 2 +- site/content/3.11/aql/high-level-operations/upsert.md | 2 +- site/content/3.11/aql/high-level-operations/window.md | 2 +- site/content/3.11/aql/high-level-operations/with.md | 2 +- site/content/3.12/aql/high-level-operations/collect.md | 2 +- site/content/3.12/aql/high-level-operations/filter.md | 2 +- site/content/3.12/aql/high-level-operations/for.md | 2 +- site/content/3.12/aql/high-level-operations/insert.md | 2 +- site/content/3.12/aql/high-level-operations/let.md | 2 +- site/content/3.12/aql/high-level-operations/limit.md | 2 +- site/content/3.12/aql/high-level-operations/remove.md | 2 +- site/content/3.12/aql/high-level-operations/replace.md | 2 +- site/content/3.12/aql/high-level-operations/return.md | 2 +- site/content/3.12/aql/high-level-operations/search.md | 2 +- site/content/3.12/aql/high-level-operations/sort.md | 2 +- site/content/3.12/aql/high-level-operations/update.md | 2 +- site/content/3.12/aql/high-level-operations/upsert.md | 2 +- site/content/3.12/aql/high-level-operations/window.md | 2 +- site/content/3.12/aql/high-level-operations/with.md | 2 +- 45 files changed, 45 insertions(+), 45 deletions(-) diff --git a/site/content/3.10/aql/high-level-operations/collect.md b/site/content/3.10/aql/high-level-operations/collect.md index 291d7d7e8a..7f773e8f07 100644 --- a/site/content/3.10/aql/high-level-operations/collect.md +++ b/site/content/3.10/aql/high-level-operations/collect.md @@ -1,5 +1,5 @@ --- -title: COLLECT operation in AQL +title: '`COLLECT` operation in AQL' menuTitle: COLLECT weight: 40 description: >- diff --git a/site/content/3.10/aql/high-level-operations/filter.md b/site/content/3.10/aql/high-level-operations/filter.md index fb829925f6..8a7d851a28 100644 --- a/site/content/3.10/aql/high-level-operations/filter.md +++ b/site/content/3.10/aql/high-level-operations/filter.md @@ -1,5 +1,5 @@ --- -title: FILTER operation in AQL +title: '`FILTER` operation in AQL' menuTitle: FILTER weight: 15 description: >- diff --git a/site/content/3.10/aql/high-level-operations/for.md b/site/content/3.10/aql/high-level-operations/for.md index 6bead8b68e..e0d4088ec3 100644 --- a/site/content/3.10/aql/high-level-operations/for.md +++ b/site/content/3.10/aql/high-level-operations/for.md @@ -1,5 +1,5 @@ --- -title: FOR operation in AQL +title: '`FOR` operation in AQL' menuTitle: FOR weight: 5 description: >- diff --git a/site/content/3.10/aql/high-level-operations/insert.md b/site/content/3.10/aql/high-level-operations/insert.md index 6a596c272e..ab7843c4e5 100644 --- a/site/content/3.10/aql/high-level-operations/insert.md +++ b/site/content/3.10/aql/high-level-operations/insert.md @@ -1,5 +1,5 @@ --- -title: INSERT operation in AQL +title: '`INSERT` operation in AQL' menuTitle: INSERT weight: 65 description: >- diff --git a/site/content/3.10/aql/high-level-operations/let.md b/site/content/3.10/aql/high-level-operations/let.md index 185ef04205..6e3fd5b3ca 100644 --- a/site/content/3.10/aql/high-level-operations/let.md +++ b/site/content/3.10/aql/high-level-operations/let.md @@ -1,5 +1,5 @@ --- -title: LET operation in AQL +title: '`LET` operation in AQL' menuTitle: LET weight: 35 description: >- diff --git a/site/content/3.10/aql/high-level-operations/limit.md b/site/content/3.10/aql/high-level-operations/limit.md index 6ddb4eaeb4..c16f2b44b1 100644 --- a/site/content/3.10/aql/high-level-operations/limit.md +++ b/site/content/3.10/aql/high-level-operations/limit.md @@ -1,5 +1,5 @@ --- -title: LIMIT operation in AQL +title: '`LIMIT` operation in AQL' menuTitle: LIMIT weight: 30 description: >- diff --git a/site/content/3.10/aql/high-level-operations/remove.md b/site/content/3.10/aql/high-level-operations/remove.md index 5ee8ccede5..cdf743bdff 100644 --- a/site/content/3.10/aql/high-level-operations/remove.md +++ b/site/content/3.10/aql/high-level-operations/remove.md @@ -1,5 +1,5 @@ --- -title: REMOVE operation in AQL +title: '`REMOVE` operation in AQL' menuTitle: REMOVE weight: 50 description: >- diff --git a/site/content/3.10/aql/high-level-operations/replace.md b/site/content/3.10/aql/high-level-operations/replace.md index a9a7bef299..0071d439e8 100644 --- a/site/content/3.10/aql/high-level-operations/replace.md +++ b/site/content/3.10/aql/high-level-operations/replace.md @@ -1,5 +1,5 @@ --- -title: REPLACE operation in AQL +title: '`REPLACE` operation in AQL' menuTitle: REPLACE weight: 60 description: >- diff --git a/site/content/3.10/aql/high-level-operations/return.md b/site/content/3.10/aql/high-level-operations/return.md index 8502f3ceac..0418c5f9ac 100644 --- a/site/content/3.10/aql/high-level-operations/return.md +++ b/site/content/3.10/aql/high-level-operations/return.md @@ -1,5 +1,5 @@ --- -title: RETURN operation in AQL +title: '`RETURN` operation in AQL' menuTitle: RETURN weight: 10 description: >- diff --git a/site/content/3.10/aql/high-level-operations/search.md b/site/content/3.10/aql/high-level-operations/search.md index b382f94d00..b09b921457 100644 --- a/site/content/3.10/aql/high-level-operations/search.md +++ b/site/content/3.10/aql/high-level-operations/search.md @@ -1,5 +1,5 @@ --- -title: SEARCH operation in AQL +title: '`SEARCH` operation in AQL' menuTitle: SEARCH weight: 20 description: >- diff --git a/site/content/3.10/aql/high-level-operations/sort.md b/site/content/3.10/aql/high-level-operations/sort.md index 2e267c3ba6..f1644443ea 100644 --- a/site/content/3.10/aql/high-level-operations/sort.md +++ b/site/content/3.10/aql/high-level-operations/sort.md @@ -1,5 +1,5 @@ --- -title: SORT operation in AQL +title: '`SORT` operation in AQL' menuTitle: SORT weight: 25 description: >- diff --git a/site/content/3.10/aql/high-level-operations/update.md b/site/content/3.10/aql/high-level-operations/update.md index aa9f6f38e0..5ad06e5180 100644 --- a/site/content/3.10/aql/high-level-operations/update.md +++ b/site/content/3.10/aql/high-level-operations/update.md @@ -1,5 +1,5 @@ --- -title: UPDATE operation in AQL +title: '`UPDATE` operation in AQL' menuTitle: UPDATE weight: 55 description: >- diff --git a/site/content/3.10/aql/high-level-operations/upsert.md b/site/content/3.10/aql/high-level-operations/upsert.md index 7a81d63603..f8faa16c33 100644 --- a/site/content/3.10/aql/high-level-operations/upsert.md +++ b/site/content/3.10/aql/high-level-operations/upsert.md @@ -1,5 +1,5 @@ --- -title: UPSERT operation in AQL +title: '`UPSERT` operation in AQL' menuTitle: UPSERT weight: 70 description: >- diff --git a/site/content/3.10/aql/high-level-operations/window.md b/site/content/3.10/aql/high-level-operations/window.md index 3a44e07f9b..42a076bda0 100644 --- a/site/content/3.10/aql/high-level-operations/window.md +++ b/site/content/3.10/aql/high-level-operations/window.md @@ -1,5 +1,5 @@ --- -title: WINDOW operation in AQL +title: '`WINDOW` operation in AQL' menuTitle: WINDOW weight: 45 description: >- diff --git a/site/content/3.10/aql/high-level-operations/with.md b/site/content/3.10/aql/high-level-operations/with.md index ace18e02e6..c8a55ddaff 100644 --- a/site/content/3.10/aql/high-level-operations/with.md +++ b/site/content/3.10/aql/high-level-operations/with.md @@ -1,5 +1,5 @@ --- -title: WITH operation in AQL +title: '`WITH` operation in AQL' menuTitle: WITH weight: 75 description: >- diff --git a/site/content/3.11/aql/high-level-operations/collect.md b/site/content/3.11/aql/high-level-operations/collect.md index 4e8123b781..3f3aec8a49 100644 --- a/site/content/3.11/aql/high-level-operations/collect.md +++ b/site/content/3.11/aql/high-level-operations/collect.md @@ -1,5 +1,5 @@ --- -title: COLLECT operation in AQL +title: '`COLLECT` operation in AQL' menuTitle: COLLECT weight: 40 description: >- diff --git a/site/content/3.11/aql/high-level-operations/filter.md b/site/content/3.11/aql/high-level-operations/filter.md index fb829925f6..8a7d851a28 100644 --- a/site/content/3.11/aql/high-level-operations/filter.md +++ b/site/content/3.11/aql/high-level-operations/filter.md @@ -1,5 +1,5 @@ --- -title: FILTER operation in AQL +title: '`FILTER` operation in AQL' menuTitle: FILTER weight: 15 description: >- diff --git a/site/content/3.11/aql/high-level-operations/for.md b/site/content/3.11/aql/high-level-operations/for.md index 6bead8b68e..e0d4088ec3 100644 --- a/site/content/3.11/aql/high-level-operations/for.md +++ b/site/content/3.11/aql/high-level-operations/for.md @@ -1,5 +1,5 @@ --- -title: FOR operation in AQL +title: '`FOR` operation in AQL' menuTitle: FOR weight: 5 description: >- diff --git a/site/content/3.11/aql/high-level-operations/insert.md b/site/content/3.11/aql/high-level-operations/insert.md index 9a219cc8db..dcb41ad8fe 100644 --- a/site/content/3.11/aql/high-level-operations/insert.md +++ b/site/content/3.11/aql/high-level-operations/insert.md @@ -1,5 +1,5 @@ --- -title: INSERT operation in AQL +title: '`INSERT` operation in AQL' menuTitle: INSERT weight: 65 description: >- diff --git a/site/content/3.11/aql/high-level-operations/let.md b/site/content/3.11/aql/high-level-operations/let.md index 185ef04205..6e3fd5b3ca 100644 --- a/site/content/3.11/aql/high-level-operations/let.md +++ b/site/content/3.11/aql/high-level-operations/let.md @@ -1,5 +1,5 @@ --- -title: LET operation in AQL +title: '`LET` operation in AQL' menuTitle: LET weight: 35 description: >- diff --git a/site/content/3.11/aql/high-level-operations/limit.md b/site/content/3.11/aql/high-level-operations/limit.md index 6ddb4eaeb4..c16f2b44b1 100644 --- a/site/content/3.11/aql/high-level-operations/limit.md +++ b/site/content/3.11/aql/high-level-operations/limit.md @@ -1,5 +1,5 @@ --- -title: LIMIT operation in AQL +title: '`LIMIT` operation in AQL' menuTitle: LIMIT weight: 30 description: >- diff --git a/site/content/3.11/aql/high-level-operations/remove.md b/site/content/3.11/aql/high-level-operations/remove.md index 4805b4fe1b..0cbc721511 100644 --- a/site/content/3.11/aql/high-level-operations/remove.md +++ b/site/content/3.11/aql/high-level-operations/remove.md @@ -1,5 +1,5 @@ --- -title: REMOVE operation in AQL +title: '`REMOVE` operation in AQL' menuTitle: REMOVE weight: 50 description: >- diff --git a/site/content/3.11/aql/high-level-operations/replace.md b/site/content/3.11/aql/high-level-operations/replace.md index b70dc19cb8..90f2baef4b 100644 --- a/site/content/3.11/aql/high-level-operations/replace.md +++ b/site/content/3.11/aql/high-level-operations/replace.md @@ -1,5 +1,5 @@ --- -title: REPLACE operation in AQL +title: '`REPLACE` operation in AQL' menuTitle: REPLACE weight: 60 description: >- diff --git a/site/content/3.11/aql/high-level-operations/return.md b/site/content/3.11/aql/high-level-operations/return.md index 8502f3ceac..0418c5f9ac 100644 --- a/site/content/3.11/aql/high-level-operations/return.md +++ b/site/content/3.11/aql/high-level-operations/return.md @@ -1,5 +1,5 @@ --- -title: RETURN operation in AQL +title: '`RETURN` operation in AQL' menuTitle: RETURN weight: 10 description: >- diff --git a/site/content/3.11/aql/high-level-operations/search.md b/site/content/3.11/aql/high-level-operations/search.md index 997a5e9188..afa2f180d6 100644 --- a/site/content/3.11/aql/high-level-operations/search.md +++ b/site/content/3.11/aql/high-level-operations/search.md @@ -1,5 +1,5 @@ --- -title: SEARCH operation in AQL +title: '`SEARCH` operation in AQL' menuTitle: SEARCH weight: 20 description: >- diff --git a/site/content/3.11/aql/high-level-operations/sort.md b/site/content/3.11/aql/high-level-operations/sort.md index 2e267c3ba6..f1644443ea 100644 --- a/site/content/3.11/aql/high-level-operations/sort.md +++ b/site/content/3.11/aql/high-level-operations/sort.md @@ -1,5 +1,5 @@ --- -title: SORT operation in AQL +title: '`SORT` operation in AQL' menuTitle: SORT weight: 25 description: >- diff --git a/site/content/3.11/aql/high-level-operations/update.md b/site/content/3.11/aql/high-level-operations/update.md index 5ea7f8d069..d2a4c0c72e 100644 --- a/site/content/3.11/aql/high-level-operations/update.md +++ b/site/content/3.11/aql/high-level-operations/update.md @@ -1,5 +1,5 @@ --- -title: UPDATE operation in AQL +title: '`UPDATE` operation in AQL' menuTitle: UPDATE weight: 55 description: >- diff --git a/site/content/3.11/aql/high-level-operations/upsert.md b/site/content/3.11/aql/high-level-operations/upsert.md index 7a81d63603..f8faa16c33 100644 --- a/site/content/3.11/aql/high-level-operations/upsert.md +++ b/site/content/3.11/aql/high-level-operations/upsert.md @@ -1,5 +1,5 @@ --- -title: UPSERT operation in AQL +title: '`UPSERT` operation in AQL' menuTitle: UPSERT weight: 70 description: >- diff --git a/site/content/3.11/aql/high-level-operations/window.md b/site/content/3.11/aql/high-level-operations/window.md index 3a44e07f9b..42a076bda0 100644 --- a/site/content/3.11/aql/high-level-operations/window.md +++ b/site/content/3.11/aql/high-level-operations/window.md @@ -1,5 +1,5 @@ --- -title: WINDOW operation in AQL +title: '`WINDOW` operation in AQL' menuTitle: WINDOW weight: 45 description: >- diff --git a/site/content/3.11/aql/high-level-operations/with.md b/site/content/3.11/aql/high-level-operations/with.md index ace18e02e6..c8a55ddaff 100644 --- a/site/content/3.11/aql/high-level-operations/with.md +++ b/site/content/3.11/aql/high-level-operations/with.md @@ -1,5 +1,5 @@ --- -title: WITH operation in AQL +title: '`WITH` operation in AQL' menuTitle: WITH weight: 75 description: >- diff --git a/site/content/3.12/aql/high-level-operations/collect.md b/site/content/3.12/aql/high-level-operations/collect.md index 4e8123b781..3f3aec8a49 100644 --- a/site/content/3.12/aql/high-level-operations/collect.md +++ b/site/content/3.12/aql/high-level-operations/collect.md @@ -1,5 +1,5 @@ --- -title: COLLECT operation in AQL +title: '`COLLECT` operation in AQL' menuTitle: COLLECT weight: 40 description: >- diff --git a/site/content/3.12/aql/high-level-operations/filter.md b/site/content/3.12/aql/high-level-operations/filter.md index fb829925f6..8a7d851a28 100644 --- a/site/content/3.12/aql/high-level-operations/filter.md +++ b/site/content/3.12/aql/high-level-operations/filter.md @@ -1,5 +1,5 @@ --- -title: FILTER operation in AQL +title: '`FILTER` operation in AQL' menuTitle: FILTER weight: 15 description: >- diff --git a/site/content/3.12/aql/high-level-operations/for.md b/site/content/3.12/aql/high-level-operations/for.md index 6bead8b68e..e0d4088ec3 100644 --- a/site/content/3.12/aql/high-level-operations/for.md +++ b/site/content/3.12/aql/high-level-operations/for.md @@ -1,5 +1,5 @@ --- -title: FOR operation in AQL +title: '`FOR` operation in AQL' menuTitle: FOR weight: 5 description: >- diff --git a/site/content/3.12/aql/high-level-operations/insert.md b/site/content/3.12/aql/high-level-operations/insert.md index 9a219cc8db..dcb41ad8fe 100644 --- a/site/content/3.12/aql/high-level-operations/insert.md +++ b/site/content/3.12/aql/high-level-operations/insert.md @@ -1,5 +1,5 @@ --- -title: INSERT operation in AQL +title: '`INSERT` operation in AQL' menuTitle: INSERT weight: 65 description: >- diff --git a/site/content/3.12/aql/high-level-operations/let.md b/site/content/3.12/aql/high-level-operations/let.md index 185ef04205..6e3fd5b3ca 100644 --- a/site/content/3.12/aql/high-level-operations/let.md +++ b/site/content/3.12/aql/high-level-operations/let.md @@ -1,5 +1,5 @@ --- -title: LET operation in AQL +title: '`LET` operation in AQL' menuTitle: LET weight: 35 description: >- diff --git a/site/content/3.12/aql/high-level-operations/limit.md b/site/content/3.12/aql/high-level-operations/limit.md index 6ddb4eaeb4..c16f2b44b1 100644 --- a/site/content/3.12/aql/high-level-operations/limit.md +++ b/site/content/3.12/aql/high-level-operations/limit.md @@ -1,5 +1,5 @@ --- -title: LIMIT operation in AQL +title: '`LIMIT` operation in AQL' menuTitle: LIMIT weight: 30 description: >- diff --git a/site/content/3.12/aql/high-level-operations/remove.md b/site/content/3.12/aql/high-level-operations/remove.md index 4805b4fe1b..0cbc721511 100644 --- a/site/content/3.12/aql/high-level-operations/remove.md +++ b/site/content/3.12/aql/high-level-operations/remove.md @@ -1,5 +1,5 @@ --- -title: REMOVE operation in AQL +title: '`REMOVE` operation in AQL' menuTitle: REMOVE weight: 50 description: >- diff --git a/site/content/3.12/aql/high-level-operations/replace.md b/site/content/3.12/aql/high-level-operations/replace.md index b70dc19cb8..90f2baef4b 100644 --- a/site/content/3.12/aql/high-level-operations/replace.md +++ b/site/content/3.12/aql/high-level-operations/replace.md @@ -1,5 +1,5 @@ --- -title: REPLACE operation in AQL +title: '`REPLACE` operation in AQL' menuTitle: REPLACE weight: 60 description: >- diff --git a/site/content/3.12/aql/high-level-operations/return.md b/site/content/3.12/aql/high-level-operations/return.md index 8502f3ceac..0418c5f9ac 100644 --- a/site/content/3.12/aql/high-level-operations/return.md +++ b/site/content/3.12/aql/high-level-operations/return.md @@ -1,5 +1,5 @@ --- -title: RETURN operation in AQL +title: '`RETURN` operation in AQL' menuTitle: RETURN weight: 10 description: >- diff --git a/site/content/3.12/aql/high-level-operations/search.md b/site/content/3.12/aql/high-level-operations/search.md index e3897dabb0..030c0bbab7 100644 --- a/site/content/3.12/aql/high-level-operations/search.md +++ b/site/content/3.12/aql/high-level-operations/search.md @@ -1,5 +1,5 @@ --- -title: SEARCH operation in AQL +title: '`SEARCH` operation in AQL' menuTitle: SEARCH weight: 20 description: >- diff --git a/site/content/3.12/aql/high-level-operations/sort.md b/site/content/3.12/aql/high-level-operations/sort.md index 2e267c3ba6..f1644443ea 100644 --- a/site/content/3.12/aql/high-level-operations/sort.md +++ b/site/content/3.12/aql/high-level-operations/sort.md @@ -1,5 +1,5 @@ --- -title: SORT operation in AQL +title: '`SORT` operation in AQL' menuTitle: SORT weight: 25 description: >- diff --git a/site/content/3.12/aql/high-level-operations/update.md b/site/content/3.12/aql/high-level-operations/update.md index 5ea7f8d069..d2a4c0c72e 100644 --- a/site/content/3.12/aql/high-level-operations/update.md +++ b/site/content/3.12/aql/high-level-operations/update.md @@ -1,5 +1,5 @@ --- -title: UPDATE operation in AQL +title: '`UPDATE` operation in AQL' menuTitle: UPDATE weight: 55 description: >- diff --git a/site/content/3.12/aql/high-level-operations/upsert.md b/site/content/3.12/aql/high-level-operations/upsert.md index 7a81d63603..f8faa16c33 100644 --- a/site/content/3.12/aql/high-level-operations/upsert.md +++ b/site/content/3.12/aql/high-level-operations/upsert.md @@ -1,5 +1,5 @@ --- -title: UPSERT operation in AQL +title: '`UPSERT` operation in AQL' menuTitle: UPSERT weight: 70 description: >- diff --git a/site/content/3.12/aql/high-level-operations/window.md b/site/content/3.12/aql/high-level-operations/window.md index 3a44e07f9b..42a076bda0 100644 --- a/site/content/3.12/aql/high-level-operations/window.md +++ b/site/content/3.12/aql/high-level-operations/window.md @@ -1,5 +1,5 @@ --- -title: WINDOW operation in AQL +title: '`WINDOW` operation in AQL' menuTitle: WINDOW weight: 45 description: >- diff --git a/site/content/3.12/aql/high-level-operations/with.md b/site/content/3.12/aql/high-level-operations/with.md index ace18e02e6..c8a55ddaff 100644 --- a/site/content/3.12/aql/high-level-operations/with.md +++ b/site/content/3.12/aql/high-level-operations/with.md @@ -1,5 +1,5 @@ --- -title: WITH operation in AQL +title: '`WITH` operation in AQL' menuTitle: WITH weight: 75 description: >- From 54a631bea080ffd433a758fb48acdeda4744f1c8 Mon Sep 17 00:00:00 2001 From: Simran Spiller Date: Thu, 16 Nov 2023 14:19:30 +0100 Subject: [PATCH 07/13] Add Read from followers in clusters to EE features --- site/content/3.10/about-arangodb/features/_index.md | 1 + .../3.10/about-arangodb/features/enterprise-edition.md | 4 ++++ .../3.10/about-arangodb/features/highlights-by-version.md | 4 ++++ .../3.10/release-notes/version-3.10/whats-new-in-3-10.md | 7 ++++++- site/content/3.11/about-arangodb/features/_index.md | 1 + .../3.11/about-arangodb/features/enterprise-edition.md | 4 ++++ .../3.11/about-arangodb/features/highlights-by-version.md | 4 ++++ .../3.11/release-notes/version-3.10/whats-new-in-3-10.md | 7 ++++++- site/content/3.12/about-arangodb/features/_index.md | 1 + .../3.12/about-arangodb/features/enterprise-edition.md | 4 ++++ .../3.12/about-arangodb/features/highlights-by-version.md | 4 ++++ .../3.12/release-notes/version-3.10/whats-new-in-3-10.md | 7 ++++++- 12 files changed, 45 insertions(+), 3 deletions(-) diff --git a/site/content/3.10/about-arangodb/features/_index.md b/site/content/3.10/about-arangodb/features/_index.md index c782536143..6704ed1d28 100644 --- a/site/content/3.10/about-arangodb/features/_index.md +++ b/site/content/3.10/about-arangodb/features/_index.md @@ -63,6 +63,7 @@ See all [Enterprise Edition Features](enterprise-edition.md). {{% /comment %}} | Only regular cluster deployments | **OneShard** deployment option to store all collections of a database on a single cluster node, to combine the performance of a single server and ACID semantics with a fault-tolerant cluster setup | | ACID transactions for multi-document / multi-collection queries on single servers, for single document operations in clusters, and for multi-document queries in clusters for collections with a single shard | In addition, ACID transactions for multi-collection queries using the OneShard feature | +| Always read from leader shards in clusters | Optionally allow dirty reads to **read from followers** to scale reads | | TLS key and certificate rotation | In addition, **key rotation for JWT secrets** and **server name indication** (SNI) | | Built-in user management and authentication | Additional **LDAP authentication** option | | Only server logs | **Audit log** of server interactions | diff --git a/site/content/3.10/about-arangodb/features/enterprise-edition.md b/site/content/3.10/about-arangodb/features/enterprise-edition.md index 5e07530902..6370df0205 100644 --- a/site/content/3.10/about-arangodb/features/enterprise-edition.md +++ b/site/content/3.10/about-arangodb/features/enterprise-edition.md @@ -68,6 +68,10 @@ features outlined below. For additional information, see stored values, primary sort columns, and primary key columns in memory to improve the performance of Views and inverted indexes. +- [**Read from followers in clusters**](../../develop/http-api/documents.md#read-from-followers): + Allow dirty reads so that Coordinators can read from any shard replica and not + only from the leader, for scaling reads. + ## Querying - [**Pregel in Cluster**](../../data-science/pregel/_index.md#prerequisites): diff --git a/site/content/3.10/about-arangodb/features/highlights-by-version.md b/site/content/3.10/about-arangodb/features/highlights-by-version.md index 0269642d92..c62ff81bf6 100644 --- a/site/content/3.10/about-arangodb/features/highlights-by-version.md +++ b/site/content/3.10/about-arangodb/features/highlights-by-version.md @@ -65,6 +65,10 @@ aliases: Optimized data loading for AQL traversal queries if only a few document attributes are accessed. +- [**Read from followers in clusters**](../../develop/http-api/documents.md#read-from-followers): + Allow dirty reads so that Coordinators can read from any shard replica and not + only from the leader, for scaling reads. + Also see [What's New in 3.10](../../release-notes/version-3.10/whats-new-in-3-10.md). ## Version 3.9 diff --git a/site/content/3.10/release-notes/version-3.10/whats-new-in-3-10.md b/site/content/3.10/release-notes/version-3.10/whats-new-in-3-10.md index 2c29b72196..179d40431f 100644 --- a/site/content/3.10/release-notes/version-3.10/whats-new-in-3-10.md +++ b/site/content/3.10/release-notes/version-3.10/whats-new-in-3-10.md @@ -1048,7 +1048,12 @@ that have copies of the data. Therefore, the read throughput is higher. This feature is only available in the Enterprise Edition. -For more information, see [Read from followers](../../develop/http-api/documents.md#read-from-followers). +For more information, see [Read from followers](../../develop/http-api/documents.md#read-from-followers) +in the HTTP API documentation. + +The JavaScript API supports an `allowDirtyReads` option for +[AQL queries](../../aql/how-to-invoke-aql/with-arangosh.md#allowdirtyreads) and +[reading documents](../../develop/javascript-api/@arangodb/collection-object.md#collectiondocumentobject--options). ## Improved shard rebalancing diff --git a/site/content/3.11/about-arangodb/features/_index.md b/site/content/3.11/about-arangodb/features/_index.md index c782536143..6704ed1d28 100644 --- a/site/content/3.11/about-arangodb/features/_index.md +++ b/site/content/3.11/about-arangodb/features/_index.md @@ -63,6 +63,7 @@ See all [Enterprise Edition Features](enterprise-edition.md). {{% /comment %}} | Only regular cluster deployments | **OneShard** deployment option to store all collections of a database on a single cluster node, to combine the performance of a single server and ACID semantics with a fault-tolerant cluster setup | | ACID transactions for multi-document / multi-collection queries on single servers, for single document operations in clusters, and for multi-document queries in clusters for collections with a single shard | In addition, ACID transactions for multi-collection queries using the OneShard feature | +| Always read from leader shards in clusters | Optionally allow dirty reads to **read from followers** to scale reads | | TLS key and certificate rotation | In addition, **key rotation for JWT secrets** and **server name indication** (SNI) | | Built-in user management and authentication | Additional **LDAP authentication** option | | Only server logs | **Audit log** of server interactions | diff --git a/site/content/3.11/about-arangodb/features/enterprise-edition.md b/site/content/3.11/about-arangodb/features/enterprise-edition.md index 5e07530902..6370df0205 100644 --- a/site/content/3.11/about-arangodb/features/enterprise-edition.md +++ b/site/content/3.11/about-arangodb/features/enterprise-edition.md @@ -68,6 +68,10 @@ features outlined below. For additional information, see stored values, primary sort columns, and primary key columns in memory to improve the performance of Views and inverted indexes. +- [**Read from followers in clusters**](../../develop/http-api/documents.md#read-from-followers): + Allow dirty reads so that Coordinators can read from any shard replica and not + only from the leader, for scaling reads. + ## Querying - [**Pregel in Cluster**](../../data-science/pregel/_index.md#prerequisites): diff --git a/site/content/3.11/about-arangodb/features/highlights-by-version.md b/site/content/3.11/about-arangodb/features/highlights-by-version.md index 79ff0130cb..acf64d6438 100644 --- a/site/content/3.11/about-arangodb/features/highlights-by-version.md +++ b/site/content/3.11/about-arangodb/features/highlights-by-version.md @@ -91,6 +91,10 @@ Also see [What's New in 3.11](../../release-notes/version-3.11/whats-new-in-3-11 Optimized data loading for AQL traversal queries if only a few document attributes are accessed. +- [**Read from followers in clusters**](../../develop/http-api/documents.md#read-from-followers): + Allow dirty reads so that Coordinators can read from any shard replica and not + only from the leader, for scaling reads. + Also see [What's New in 3.10](../../release-notes/version-3.10/whats-new-in-3-10.md). ## Version 3.9 diff --git a/site/content/3.11/release-notes/version-3.10/whats-new-in-3-10.md b/site/content/3.11/release-notes/version-3.10/whats-new-in-3-10.md index 30b0c414e3..e222eaa16a 100644 --- a/site/content/3.11/release-notes/version-3.10/whats-new-in-3-10.md +++ b/site/content/3.11/release-notes/version-3.10/whats-new-in-3-10.md @@ -1048,7 +1048,12 @@ that have copies of the data. Therefore, the read throughput is higher. This feature is only available in the Enterprise Edition. -For more information, see [Read from followers](../../develop/http-api/documents.md#read-from-followers). +For more information, see [Read from followers](../../develop/http-api/documents.md#read-from-followers) +in the HTTP API documentation. + +The JavaScript API supports an `allowDirtyReads` option for +[AQL queries](../../aql/how-to-invoke-aql/with-arangosh.md#allowdirtyreads) and +[reading documents](../../develop/javascript-api/@arangodb/collection-object.md#collectiondocumentobject--options). ## Improved shard rebalancing diff --git a/site/content/3.12/about-arangodb/features/_index.md b/site/content/3.12/about-arangodb/features/_index.md index c782536143..6704ed1d28 100644 --- a/site/content/3.12/about-arangodb/features/_index.md +++ b/site/content/3.12/about-arangodb/features/_index.md @@ -63,6 +63,7 @@ See all [Enterprise Edition Features](enterprise-edition.md). {{% /comment %}} | Only regular cluster deployments | **OneShard** deployment option to store all collections of a database on a single cluster node, to combine the performance of a single server and ACID semantics with a fault-tolerant cluster setup | | ACID transactions for multi-document / multi-collection queries on single servers, for single document operations in clusters, and for multi-document queries in clusters for collections with a single shard | In addition, ACID transactions for multi-collection queries using the OneShard feature | +| Always read from leader shards in clusters | Optionally allow dirty reads to **read from followers** to scale reads | | TLS key and certificate rotation | In addition, **key rotation for JWT secrets** and **server name indication** (SNI) | | Built-in user management and authentication | Additional **LDAP authentication** option | | Only server logs | **Audit log** of server interactions | diff --git a/site/content/3.12/about-arangodb/features/enterprise-edition.md b/site/content/3.12/about-arangodb/features/enterprise-edition.md index 90ea02ca64..bfc2758c8f 100644 --- a/site/content/3.12/about-arangodb/features/enterprise-edition.md +++ b/site/content/3.12/about-arangodb/features/enterprise-edition.md @@ -72,6 +72,10 @@ features outlined below. For additional information, see Retrieve search results for the highest-ranking matches from Views faster by defining a list of sort expressions to optimize. +- [**Read from followers in clusters**](../../develop/http-api/documents.md#read-from-followers): + Allow dirty reads so that Coordinators can read from any shard replica and not + only from the leader, for scaling reads. + ## Querying - [**Pregel in Cluster**](../../data-science/pregel/_index.md#prerequisites): diff --git a/site/content/3.12/about-arangodb/features/highlights-by-version.md b/site/content/3.12/about-arangodb/features/highlights-by-version.md index 76de63b6db..e997c56cb7 100644 --- a/site/content/3.12/about-arangodb/features/highlights-by-version.md +++ b/site/content/3.12/about-arangodb/features/highlights-by-version.md @@ -105,6 +105,10 @@ Also see [What's New in 3.11](../../release-notes/version-3.11/whats-new-in-3-11 Optimized data loading for AQL traversal queries if only a few document attributes are accessed. +- [**Read from followers in clusters**](../../develop/http-api/documents.md#read-from-followers): + Allow dirty reads so that Coordinators can read from any shard replica and not + only from the leader, for scaling reads. + Also see [What's New in 3.10](../../release-notes/version-3.10/whats-new-in-3-10.md). ## Version 3.9 diff --git a/site/content/3.12/release-notes/version-3.10/whats-new-in-3-10.md b/site/content/3.12/release-notes/version-3.10/whats-new-in-3-10.md index 30b0c414e3..e222eaa16a 100644 --- a/site/content/3.12/release-notes/version-3.10/whats-new-in-3-10.md +++ b/site/content/3.12/release-notes/version-3.10/whats-new-in-3-10.md @@ -1048,7 +1048,12 @@ that have copies of the data. Therefore, the read throughput is higher. This feature is only available in the Enterprise Edition. -For more information, see [Read from followers](../../develop/http-api/documents.md#read-from-followers). +For more information, see [Read from followers](../../develop/http-api/documents.md#read-from-followers) +in the HTTP API documentation. + +The JavaScript API supports an `allowDirtyReads` option for +[AQL queries](../../aql/how-to-invoke-aql/with-arangosh.md#allowdirtyreads) and +[reading documents](../../develop/javascript-api/@arangodb/collection-object.md#collectiondocumentobject--options). ## Improved shard rebalancing From a15a662b3604bf8e1f845d03e4633b5eec5c4ee4 Mon Sep 17 00:00:00 2001 From: Simran Spiller Date: Thu, 16 Nov 2023 14:21:22 +0100 Subject: [PATCH 08/13] Sync release notes between versions --- .../version-3.10/api-changes-in-3-10.md | 29 +++++++++ .../version-3.10/whats-new-in-3-10.md | 23 +++++++ .../version-3.10/api-changes-in-3-10.md | 29 +++++++++ .../version-3.10/whats-new-in-3-10.md | 23 +++++++ .../version-3.11/api-changes-in-3-11.md | 63 +++++++++++++++++++ .../version-3.11/whats-new-in-3-11.md | 57 ++++++++++++----- 6 files changed, 207 insertions(+), 17 deletions(-) diff --git a/site/content/3.11/release-notes/version-3.10/api-changes-in-3-10.md b/site/content/3.11/release-notes/version-3.10/api-changes-in-3-10.md index b5b466f428..a7e690d0a5 100644 --- a/site/content/3.11/release-notes/version-3.10/api-changes-in-3-10.md +++ b/site/content/3.11/release-notes/version-3.10/api-changes-in-3-10.md @@ -126,6 +126,23 @@ if the request body was an empty array. Example: Now, a request like this succeeds and returns an empty array as response. +#### Limit to the number of databases in a deployment + +Introduced in: v3.10.10 + +The new `--database.max-databases` startup option can cap the number of databases +and creating databases using the `POST /_api/database` endpoint can thus now fail +for this reason if your deployment is at or above the configured maximum. Example: + +```json +{ + "code": 400, + "error": true, + "errorMessage": "unable to create additional database because it would exceed the configured maximum number of databases (2)", + "errorNum": 32 +} +``` + ### Endpoint return value changes - Since ArangoDB 3.8, there have been two APIs for retrieving the metrics in two @@ -807,6 +824,18 @@ The following metrics have been added: | `arangodb_file_descriptors_limit` | System limit for the number of open files for the arangod process. | | `arangodb_file_descriptors_current` | Number of file descriptors currently opened by the arangod process. | +--- + +Introduced in: v3.10.11 + +The following metrics have been added to improve the observability of in-memory +cache subsystem: + +- `rocksdb_cache_free_memory_tasks_total` +- `rocksdb_cache_free_memory_tasks_duration_total` +- `rocksdb_cache_migrate_tasks_total` +- `rocksdb_cache_migrate_tasks_duration_total` + #### Pregel API When loading the graph data into memory, a `"loading"` state is now returned by diff --git a/site/content/3.11/release-notes/version-3.10/whats-new-in-3-10.md b/site/content/3.11/release-notes/version-3.10/whats-new-in-3-10.md index e222eaa16a..179d40431f 100644 --- a/site/content/3.11/release-notes/version-3.10/whats-new-in-3-10.md +++ b/site/content/3.11/release-notes/version-3.10/whats-new-in-3-10.md @@ -1389,6 +1389,20 @@ attempt to create an additional database fails with error if other databases are dropped first. The default value for this option is unlimited, so an arbitrary amount of databases can be created. +### Configurable maximum for queued log entries + +Introduced in: v3.10.12 + +The new `--log.max-queued-entries` startup option lets you configure how many +log entries are queued in a background thread. + +Log entries are pushed on a queue for asynchronous writing unless you enable the +`--log.force-direct` startup option. If you use a slow log output (e.g. syslog), +the queue might grow and eventually overflow. + +You can configure the upper bound of the queue with this option. If the queue is +full, log entries are written synchronously until the queue has space again. + ## Miscellaneous changes ### Optimizer rules endpoint @@ -1579,6 +1593,15 @@ The following system metrics have been added: | `arangodb_file_descriptors_limit` | System limit for the number of open files for the arangod process. | | `arangodb_file_descriptors_current` | Number of file descriptors currently opened by the arangod process. | +### More instant Hot Backups + +Introduced in: v3.10.10, v3.11.3 + +Cluster deployments no longer wait for all in-progress transactions to get +committed when a user requests a Hot Backup. The waiting could cause deadlocks +and thus Hot Backups to fail, in particular in ArangoGraph. Now, Hot Backups are +created immediately and commits have to wait until the backup process is done. + ## Client tools ### arangobench diff --git a/site/content/3.12/release-notes/version-3.10/api-changes-in-3-10.md b/site/content/3.12/release-notes/version-3.10/api-changes-in-3-10.md index b5b466f428..a7e690d0a5 100644 --- a/site/content/3.12/release-notes/version-3.10/api-changes-in-3-10.md +++ b/site/content/3.12/release-notes/version-3.10/api-changes-in-3-10.md @@ -126,6 +126,23 @@ if the request body was an empty array. Example: Now, a request like this succeeds and returns an empty array as response. +#### Limit to the number of databases in a deployment + +Introduced in: v3.10.10 + +The new `--database.max-databases` startup option can cap the number of databases +and creating databases using the `POST /_api/database` endpoint can thus now fail +for this reason if your deployment is at or above the configured maximum. Example: + +```json +{ + "code": 400, + "error": true, + "errorMessage": "unable to create additional database because it would exceed the configured maximum number of databases (2)", + "errorNum": 32 +} +``` + ### Endpoint return value changes - Since ArangoDB 3.8, there have been two APIs for retrieving the metrics in two @@ -807,6 +824,18 @@ The following metrics have been added: | `arangodb_file_descriptors_limit` | System limit for the number of open files for the arangod process. | | `arangodb_file_descriptors_current` | Number of file descriptors currently opened by the arangod process. | +--- + +Introduced in: v3.10.11 + +The following metrics have been added to improve the observability of in-memory +cache subsystem: + +- `rocksdb_cache_free_memory_tasks_total` +- `rocksdb_cache_free_memory_tasks_duration_total` +- `rocksdb_cache_migrate_tasks_total` +- `rocksdb_cache_migrate_tasks_duration_total` + #### Pregel API When loading the graph data into memory, a `"loading"` state is now returned by diff --git a/site/content/3.12/release-notes/version-3.10/whats-new-in-3-10.md b/site/content/3.12/release-notes/version-3.10/whats-new-in-3-10.md index e222eaa16a..179d40431f 100644 --- a/site/content/3.12/release-notes/version-3.10/whats-new-in-3-10.md +++ b/site/content/3.12/release-notes/version-3.10/whats-new-in-3-10.md @@ -1389,6 +1389,20 @@ attempt to create an additional database fails with error if other databases are dropped first. The default value for this option is unlimited, so an arbitrary amount of databases can be created. +### Configurable maximum for queued log entries + +Introduced in: v3.10.12 + +The new `--log.max-queued-entries` startup option lets you configure how many +log entries are queued in a background thread. + +Log entries are pushed on a queue for asynchronous writing unless you enable the +`--log.force-direct` startup option. If you use a slow log output (e.g. syslog), +the queue might grow and eventually overflow. + +You can configure the upper bound of the queue with this option. If the queue is +full, log entries are written synchronously until the queue has space again. + ## Miscellaneous changes ### Optimizer rules endpoint @@ -1579,6 +1593,15 @@ The following system metrics have been added: | `arangodb_file_descriptors_limit` | System limit for the number of open files for the arangod process. | | `arangodb_file_descriptors_current` | Number of file descriptors currently opened by the arangod process. | +### More instant Hot Backups + +Introduced in: v3.10.10, v3.11.3 + +Cluster deployments no longer wait for all in-progress transactions to get +committed when a user requests a Hot Backup. The waiting could cause deadlocks +and thus Hot Backups to fail, in particular in ArangoGraph. Now, Hot Backups are +created immediately and commits have to wait until the backup process is done. + ## Client tools ### arangobench diff --git a/site/content/3.12/release-notes/version-3.11/api-changes-in-3-11.md b/site/content/3.12/release-notes/version-3.11/api-changes-in-3-11.md index e18b17d9ed..3f22ad2132 100644 --- a/site/content/3.12/release-notes/version-3.11/api-changes-in-3-11.md +++ b/site/content/3.12/release-notes/version-3.11/api-changes-in-3-11.md @@ -262,6 +262,23 @@ Disabled: arangodb_agency_cache_callback_number{role="SINGLE"}0 ``` +#### Limit to the number of databases in a deployment + +Introduced in: v3.10.10, v3.11.2 + +The new `--database.max-databases` startup option can cap the number of databases +and creating databases using the `POST /_api/database` endpoint can thus now fail +for this reason if your deployment is at or above the configured maximum. Example: + +```json +{ + "code": 400, + "error": true, + "errorMessage": "unable to create additional database because it would exceed the configured maximum number of databases (2)", + "errorNum": 32 +} +``` + ### Endpoint return value changes Introduced in: v3.8.8, v3.9.4, v3.10.1 @@ -599,6 +616,52 @@ The following metric stores the peak value of the `rocksdb_cache_allocated` metr |:------|:------------| | `rocksdb_cache_peak_allocated` | Global peak memory allocation of ArangoDB in-memory caches. | +--- + +Introduced in: v3.11.2 + +The following metrics have been added about the LZ4 compression for values in +the in-memory edge cache: + +- `rocksdb_cache_edge_inserts_effective_entries_size_total` +- `rocksdb_cache_edge_inserts_uncompressed_entries_size_total` +- `rocksdb_cache_edge_compression_ratio` + +--- + +Introduced in: v3.10.11, v3.11.4 + +The following metrics have been added to improve the observability of in-memory +cache subsystem: + +- `rocksdb_cache_free_memory_tasks_total` +- `rocksdb_cache_free_memory_tasks_duration_total` +- `rocksdb_cache_migrate_tasks_total` +- `rocksdb_cache_migrate_tasks_duration_total` + +--- + +Introduced in: v3.11.4 + +The following metrics have been added to improve the observability of in-memory +edge cache: + +- `rocksdb_cache_edge_compressed_inserts_total` +- `rocksdb_cache_edge_empty_inserts_total` +- `rocksdb_cache_edge_inserts_total` + +--- + +Introduced in: v3.11.5 + +The following metrics have been added to monitor and detect temporary or +permanent connectivity issues as well as how many scheduler threads are in the +detached state: + +- `arangodb_network_connectivity_failures_coordinators` +- `arangodb_network_connectivity_failures_dbservers_total` +- `arangodb_scheduler_num_detached_threads` + #### Log level API Introduced in: v3.10.2 diff --git a/site/content/3.12/release-notes/version-3.11/whats-new-in-3-11.md b/site/content/3.12/release-notes/version-3.11/whats-new-in-3-11.md index 01dd8ba116..65eed12913 100644 --- a/site/content/3.12/release-notes/version-3.11/whats-new-in-3-11.md +++ b/site/content/3.12/release-notes/version-3.11/whats-new-in-3-11.md @@ -515,7 +515,7 @@ Query Statistics: ### New stage in query profiling output -Introduced in: v3.10.3, v3.11.0 +Introduced in: v3.10.3 The query profiling output has a new `instantiating executors` stage. The time spent in this stage is the time needed to create the query executors @@ -1036,26 +1036,19 @@ permanent connectivity issues: - `arangodb_network_connectivity_failures_dbservers_total`: Number of failed connectivity check requests sent to DB-Servers. -### Detached scheduler threads +### Configurable maximum for queued log entries -Introduced in: v3.11.5 - -A scheduler thread now has the capability to detach itself from the scheduler -if it observes the need to perform a potentially long running task, like waiting -for a lock. This allows a new scheduler thread to be started and prevents -scenarios where all threads are blocked waiting for a lock, which has previously -led to deadlock situations. +Introduced in: v3.10.12, v3.11.5 -Threads waiting for more than 1 second on a collection lock will detach -themselves. +The new `--log.max-queued-entries` startup option lets you configure how many +log entries are queued in a background thread. -The following startup option has been added: -- `--server.max-number-detached-threads`: The maximum number of detached scheduler - threads. +Log entries are pushed on a queue for asynchronous writing unless you enable the +`--log.force-direct` startup option. If you use a slow log output (e.g. syslog), +the queue might grow and eventually overflow. -The following metric has been added: -- `arangodb_scheduler_num_detached_threads`: The number of worker threads - currently started and detached from the scheduler. +You can configure the upper bound of the queue with this option. If the queue is +full, log entries are written synchronously until the queue has space again. ## Miscellaneous changes @@ -1242,6 +1235,15 @@ The following system metrics have been added: | `arangodb_file_descriptors_limit` | System limit for the number of open files for the arangod process. | | `arangodb_file_descriptors_current` | Number of file descriptors currently opened by the arangod process. | +### More instant Hot Backups + +Introduced in: v3.10.10, v3.11.3 + +Cluster deployments no longer wait for all in-progress transactions to get +committed when a user requests a Hot Backup. The waiting could cause deadlocks +and thus Hot Backups to fail, in particular in ArangoGraph. Now, Hot Backups are +created immediately and commits have to wait until the backup process is done. + ### In-memory edge cache startup options and metrics Introduced in: v3.11.4 @@ -1287,6 +1289,27 @@ cache subsystem: by the cache subsystem to migrate existing cache hash tables to a bigger or smaller table. +### Detached scheduler threads + +Introduced in: v3.11.5 + +A scheduler thread now has the capability to detach itself from the scheduler +if it observes the need to perform a potentially long running task, like waiting +for a lock. This allows a new scheduler thread to be started and prevents +scenarios where all threads are blocked waiting for a lock, which has previously +led to deadlock situations. + +Threads waiting for more than 1 second on a collection lock will detach +themselves. + +The following startup option has been added: +- `--server.max-number-detached-threads`: The maximum number of detached scheduler + threads. + +The following metric has been added: +- `arangodb_scheduler_num_detached_threads`: The number of worker threads + currently started and detached from the scheduler. + ## Client tools ### arangodump From f234352c2fa474d08988f65795b94f1a9ce6ab66 Mon Sep 17 00:00:00 2001 From: Simran Spiller Date: Thu, 16 Nov 2023 15:42:24 +0100 Subject: [PATCH 09/13] Adjustable Stream Transactions size --- .../3.10/develop/transactions/stream-transactions.md | 4 ++-- .../3.11/develop/transactions/stream-transactions.md | 4 ++-- .../3.12/develop/transactions/stream-transactions.md | 8 ++++++-- .../release-notes/version-3.12/api-changes-in-3-12.md | 8 ++++++++ .../version-3.12/incompatible-changes-in-3-12.md | 8 ++++++++ .../3.12/release-notes/version-3.12/whats-new-in-3-12.md | 6 ++++++ 6 files changed, 32 insertions(+), 6 deletions(-) diff --git a/site/content/3.10/develop/transactions/stream-transactions.md b/site/content/3.10/develop/transactions/stream-transactions.md index f5c5743997..521114e5cc 100644 --- a/site/content/3.10/develop/transactions/stream-transactions.md +++ b/site/content/3.10/develop/transactions/stream-transactions.md @@ -45,7 +45,7 @@ on the Coordinator to ensure that abandoned transactions cannot block the cluster from operating properly: - Maximum idle timeout of up to **120 seconds** between operations. -- Maximum transaction size of **128 MB** per DB-Server. +- Maximum transaction size of **128 MB** (per DB-Server in clusters). These limits are also enforced for Stream Transactions on single servers. @@ -103,7 +103,7 @@ Additionally, `options` can have the following optional attributes: waiting on collection locks. This option is only meaningful when using `exclusive` locks. If not specified, a default value is used. Setting `lockTimeout` to `0` makes ArangoDB not time out waiting for a lock. -- `maxTransactionSize`: Transaction size limit in bytes. +- `maxTransactionSize`: Transaction size limit in bytes. Can be at most 128 MiB. The method returns an object that lets you run supported operations as part of the transactions, get the status information, and commit or abort the transaction. diff --git a/site/content/3.11/develop/transactions/stream-transactions.md b/site/content/3.11/develop/transactions/stream-transactions.md index f5c5743997..521114e5cc 100644 --- a/site/content/3.11/develop/transactions/stream-transactions.md +++ b/site/content/3.11/develop/transactions/stream-transactions.md @@ -45,7 +45,7 @@ on the Coordinator to ensure that abandoned transactions cannot block the cluster from operating properly: - Maximum idle timeout of up to **120 seconds** between operations. -- Maximum transaction size of **128 MB** per DB-Server. +- Maximum transaction size of **128 MB** (per DB-Server in clusters). These limits are also enforced for Stream Transactions on single servers. @@ -103,7 +103,7 @@ Additionally, `options` can have the following optional attributes: waiting on collection locks. This option is only meaningful when using `exclusive` locks. If not specified, a default value is used. Setting `lockTimeout` to `0` makes ArangoDB not time out waiting for a lock. -- `maxTransactionSize`: Transaction size limit in bytes. +- `maxTransactionSize`: Transaction size limit in bytes. Can be at most 128 MiB. The method returns an object that lets you run supported operations as part of the transactions, get the status information, and commit or abort the transaction. diff --git a/site/content/3.12/develop/transactions/stream-transactions.md b/site/content/3.12/develop/transactions/stream-transactions.md index f5c5743997..2fefb6768e 100644 --- a/site/content/3.12/develop/transactions/stream-transactions.md +++ b/site/content/3.12/develop/transactions/stream-transactions.md @@ -45,10 +45,13 @@ on the Coordinator to ensure that abandoned transactions cannot block the cluster from operating properly: - Maximum idle timeout of up to **120 seconds** between operations. -- Maximum transaction size of **128 MB** per DB-Server. +- Maximum transaction size with a default of **128 MiB** (per DB-Server in clusters). These limits are also enforced for Stream Transactions on single servers. +The maximum size for a single Stream Transaction can be adjusted with the +`--transaction.streaming-max-transaction-size` startup option. + The default maximum idle timeout is **60 seconds** between operations in a single Stream Transaction. The maximum value can be bumped up to at most 120 seconds by setting the `--transaction.streaming-idle-timeout` startup option. @@ -103,7 +106,8 @@ Additionally, `options` can have the following optional attributes: waiting on collection locks. This option is only meaningful when using `exclusive` locks. If not specified, a default value is used. Setting `lockTimeout` to `0` makes ArangoDB not time out waiting for a lock. -- `maxTransactionSize`: Transaction size limit in bytes. +- `maxTransactionSize`: Transaction size limit in bytes. Can be at most the + value of the `--transaction.streaming-max-transaction-size` startup option. The method returns an object that lets you run supported operations as part of the transactions, get the status information, and commit or abort the transaction. diff --git a/site/content/3.12/release-notes/version-3.12/api-changes-in-3-12.md b/site/content/3.12/release-notes/version-3.12/api-changes-in-3-12.md index 23686001be..fd0921ba2a 100644 --- a/site/content/3.12/release-notes/version-3.12/api-changes-in-3-12.md +++ b/site/content/3.12/release-notes/version-3.12/api-changes-in-3-12.md @@ -86,6 +86,14 @@ for this reason if your deployment is at or above the configured maximum. Exampl } ``` +#### Adjustable Stream Transaction size + +The [Stream Transactions HTTP API](../../develop/http-api/transactions/stream-transactions.md) +may now allow larger transactions or be limited to smaller transactions because +the maximum transaction size can now be configured with the +`--transaction.streaming-max-transaction-size` startup option. +The default value remains 128 MiB. + ### Privilege changes diff --git a/site/content/3.12/release-notes/version-3.12/incompatible-changes-in-3-12.md b/site/content/3.12/release-notes/version-3.12/incompatible-changes-in-3-12.md index 7a5a8aba50..0250cd75fc 100644 --- a/site/content/3.12/release-notes/version-3.12/incompatible-changes-in-3-12.md +++ b/site/content/3.12/release-notes/version-3.12/incompatible-changes-in-3-12.md @@ -79,6 +79,14 @@ The second option is the recommended one, as it signals the intent more clearly, and makes the cache behave "as expected", i.e. use up to the configured memory limit and not just 56% of it. +#### Adjustable Stream Transaction size + +[Stream Transactions](../../develop/transactions/stream-transactions.md) may +now be limited to smaller transaction sizes because the maximum transaction size +can now be configured with the `--transaction.streaming-max-transaction-size` +startup option. The default value remains 128 MiB but configuring a lower limit +can cause previously working Stream Transactions to fail. + ## Client tools ### jslint feature in arangosh diff --git a/site/content/3.12/release-notes/version-3.12/whats-new-in-3-12.md b/site/content/3.12/release-notes/version-3.12/whats-new-in-3-12.md index 93a9a7fdad..568c14cb78 100644 --- a/site/content/3.12/release-notes/version-3.12/whats-new-in-3-12.md +++ b/site/content/3.12/release-notes/version-3.12/whats-new-in-3-12.md @@ -112,6 +112,12 @@ indexes have been allowing to index and store the `_id` system attribute. ## Server options +### Adjustable Stream Transaction size + +The previously fixed limit of 128 MiB for [Stream Transactions](../../develop/transactions/stream-transactions.md) +can now be configured with the new `--transaction.streaming-max-transaction-size` +startup option. The default value remains 128 MiB. + ### LZ4 compression for values in the in-memory edge cache Introduced in: v3.11.2 From bf8ca24ea438dc976bd0e920f72866757779145a Mon Sep 17 00:00:00 2001 From: Simran Spiller Date: Thu, 16 Nov 2023 15:43:06 +0100 Subject: [PATCH 10/13] ArangoSearch file descriptor metric, headline tweak --- .../3.12/release-notes/version-3.12/whats-new-in-3-12.md | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/site/content/3.12/release-notes/version-3.12/whats-new-in-3-12.md b/site/content/3.12/release-notes/version-3.12/whats-new-in-3-12.md index 568c14cb78..5b69fc607d 100644 --- a/site/content/3.12/release-notes/version-3.12/whats-new-in-3-12.md +++ b/site/content/3.12/release-notes/version-3.12/whats-new-in-3-12.md @@ -276,6 +276,13 @@ of outgrowing the maximum number of file descriptors the ArangoDB process can open. Thus, these options should only be enabled on deployments with a limited number of collections/shards/indexes. +### ArangoSearch file descriptor metric + +The following metric has been added: + +- `arangodb_search_file_descriptors`: + Amount of memory in bytes that is used for consolidating this index. + ### More instant Hot Backups Introduced in: v3.10.10, v3.11.3 @@ -389,7 +396,7 @@ with large shards. the network is slow or its capacity is maxed out. The data is decompressed on the client side and recompressed if you enable the `--compress-output` option. -#### Resource usage limits and metrics +#### Server-side resource usage limits and metrics The following `arangod` startup options can be used to limit the resource usage of parallel _arangodump_ invocations: From 4557a2f69b4c1eb8b35da1a2aa7f7919d51bbb03 Mon Sep 17 00:00:00 2001 From: Simran Spiller Date: Mon, 20 Nov 2023 11:36:45 +0100 Subject: [PATCH 11/13] Fix metric help string, add to API changes --- .../3.12/release-notes/version-3.12/api-changes-in-3-12.md | 3 ++- .../3.12/release-notes/version-3.12/whats-new-in-3-12.md | 2 +- 2 files changed, 3 insertions(+), 2 deletions(-) diff --git a/site/content/3.12/release-notes/version-3.12/api-changes-in-3-12.md b/site/content/3.12/release-notes/version-3.12/api-changes-in-3-12.md index fd0921ba2a..8eae47a756 100644 --- a/site/content/3.12/release-notes/version-3.12/api-changes-in-3-12.md +++ b/site/content/3.12/release-notes/version-3.12/api-changes-in-3-12.md @@ -151,13 +151,14 @@ produced no warnings. #### Metrics API The metrics endpoint includes the following new metrics about AQL queries, -ongoing dumps, and ArangoSearch parallelism: +ongoing dumps, ArangoSearch parallelism and used file descriptors: - `arangodb_aql_cursors_active` - `arangodb_dump_memory_usage` - `arangodb_dump_ongoing` - `arangodb_dump_threads_blocked_total` - `arangodb_search_execution_threads_demand` +- `arangodb_search_file_descriptors` --- diff --git a/site/content/3.12/release-notes/version-3.12/whats-new-in-3-12.md b/site/content/3.12/release-notes/version-3.12/whats-new-in-3-12.md index 5b69fc607d..21f8a763e9 100644 --- a/site/content/3.12/release-notes/version-3.12/whats-new-in-3-12.md +++ b/site/content/3.12/release-notes/version-3.12/whats-new-in-3-12.md @@ -281,7 +281,7 @@ limited number of collections/shards/indexes. The following metric has been added: - `arangodb_search_file_descriptors`: - Amount of memory in bytes that is used for consolidating this index. + Current count of opened file descriptors for this index. ### More instant Hot Backups From f8bf63b78cea4fafe2053de77b9b02da5d10aea2 Mon Sep 17 00:00:00 2001 From: Simran Spiller Date: Mon, 20 Nov 2023 12:01:08 +0100 Subject: [PATCH 12/13] Clarify CircleCI upstream references --- CIRCLECI.md | 41 +++++++++++++++++++++++++++++++++-------- 1 file changed, 33 insertions(+), 8 deletions(-) diff --git a/CIRCLECI.md b/CIRCLECI.md index eaed1daa19..a2a2e800d1 100644 --- a/CIRCLECI.md +++ b/CIRCLECI.md @@ -75,9 +75,9 @@ arguments are invoked: | Parameter type | Name | Value | |:---------------|:-----|:------| | string | `workflow` | `generate` | -| string | `arangodb-3_10` | `{string in PR Template at 3.10}` | -| string | `arangodb-3_11` | `{string in PR Template at 3.11}` | -| string | `arangodb-3_12` | `{string in PR Template at 3.12}` | +| string | `arangodb-3_10` | [Upstream reference](#upstream-reference) for 3.10 | +| string | `arangodb-3_11` | [Upstream reference](#upstream-reference) for 3.11 | +| string | `arangodb-3_12` | [Upstream reference](#upstream-reference) for 3.12 | | string | `generators` | `examples` | | string | `deploy-url` | `deploy-preview-{PR_NUMBER}` | @@ -93,9 +93,9 @@ arguments are invoked: | Parameter type | Name | Value | |:---------------|:-----|:------| | string | `workflow` | `generate` | -| string | `arangodb-3_10` | `{string in PR Template at 3.10}` | -| string | `arangodb-3_11` | `{string in PR Template at 3.11}` | -| string | `arangodb-3_12` | `{string in PR Template at 3.12}` | +| string | `arangodb-3_10` | [Upstream reference](#upstream-reference) for 3.10 | +| string | `arangodb-3_11` | [Upstream reference](#upstream-reference) for 3.11 | +| string | `arangodb-3_12` | [Upstream reference](#upstream-reference) for 3.12 | | string | `generators` | `examples` | | string | `deploy-url` | `deploy-preview-{PR_NUMBER}` | | boolean | `commit-generated` | `true` | @@ -121,13 +121,38 @@ or for multiple versions. | Parameter type | Name | Value | |:---------------|:-----|:------| | string | `workflow` | `generate` | -| string | `arangodb-3_10` | `{string in PR Template at 3.10}` | -| string | `arangodb-3_11` | `{string in PR Template at 3.11}` | +| string | `arangodb-3_10` | [Upstream reference](#upstream-references) for 3.10 | +| string | `arangodb-3_11` | [Upstream reference](#upstream-references) for 3.11 | | string | `generators` | `examples` | | boolean | `commit-generated` | `true` | | string | `deploy-url` | `deploy-preview-{PR_NUMBER}` | | string | `override` | `http,^aql.*` | +### Upstream references + +Documentation pull requests specify upstream references like so: + +```markdown +- 3.10: +- 3.11: https://github.com/arangodb/arangodb/pull/12345 +- 3.12: arangodb/enterprise-preview:devel-nightly +``` + +The above example indicates that ArangoDB versions 3.11 and 3.12 contain changes +relevant to the docs PR, but 3.10 does not. Relevant changes are typically +behavior changes of _arangod_ that will be visible in documentation examples. + +For 3.11, a link to a PR in the `arangodb/arangodb` repository is given. It is +used by the GitHub integration to determine the feature branch to compile and +use for generating examples. Do not specify a link when manually triggering a +pipeline in CircleCI but the **branch name** (like `feature/new-aql-function`)! + +For 3.12, an ArangoDB Enterprise Edition image hosted on +[Docker Hub](https://hub.docker.com/) is specified. Using Docker images has the +advantage that the compilation of ArangoDB can be skipped, making the example +generation faster. Of course, this requires that an image containing relevant +changes to ArangoDB exists. + ## Release workflow for ArangoDB releases To run a release job for a new ArangoDB patch release (e.g. 3.11.4), From b6f8dcf43216a22a2ab8bd8c0ce453b78ffeaaa4 Mon Sep 17 00:00:00 2001 From: Simran Spiller Date: Wed, 22 Nov 2023 09:47:05 +0100 Subject: [PATCH 13/13] Clarify --- .../3.12/release-notes/version-3.12/whats-new-in-3-12.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/site/content/3.12/release-notes/version-3.12/whats-new-in-3-12.md b/site/content/3.12/release-notes/version-3.12/whats-new-in-3-12.md index 21f8a763e9..a54f18987e 100644 --- a/site/content/3.12/release-notes/version-3.12/whats-new-in-3-12.md +++ b/site/content/3.12/release-notes/version-3.12/whats-new-in-3-12.md @@ -281,7 +281,7 @@ limited number of collections/shards/indexes. The following metric has been added: - `arangodb_search_file_descriptors`: - Current count of opened file descriptors for this index. + Current count of opened file descriptors for an ArangoSearch index. ### More instant Hot Backups