From 528b48682132fc3cd8bc6dc001f4e4861f361eb2 Mon Sep 17 00:00:00 2001 From: Simran Spiller Date: Thu, 16 Nov 2023 15:55:56 +0100 Subject: [PATCH 1/2] Release notes for memory accounting metrics and memory usage improvements --- .../version-3.12/api-changes-in-3-12.md | 18 +++++++++- .../incompatible-changes-in-3-12.md | 20 +++++++++++ .../version-3.12/whats-new-in-3-12.md | 33 +++++++++++++++++++ 3 files changed, 70 insertions(+), 1 deletion(-) diff --git a/site/content/3.12/release-notes/version-3.12/api-changes-in-3-12.md b/site/content/3.12/release-notes/version-3.12/api-changes-in-3-12.md index 2ae11ed92a..49c3719f4c 100644 --- a/site/content/3.12/release-notes/version-3.12/api-changes-in-3-12.md +++ b/site/content/3.12/release-notes/version-3.12/api-changes-in-3-12.md @@ -143,7 +143,7 @@ produced no warnings. #### Metrics API The metrics endpoint includes the following new metrics about AQL queries, -ongoing dumps, and ArangoSearch execution threads: +ongoing dumps, and ArangoSearch: - `arangodb_aql_cursors_active` - `arangodb_dump_memory_usage` @@ -151,6 +151,22 @@ ongoing dumps, and ArangoSearch execution threads: - `arangodb_dump_threads_blocked_total` - `arangodb_search_execution_threads_demand` +The following new metrics for improved memory observability have been added: + +- `arangodb_agency_node_memory_usage` +- `arangodb_index_estimates_memory_usage` +- `arangodb_internal_cluster_info_memory_usage` +- `arangodb_requests_memory_usage` +- `arangodb_revision_tree_buffered_memory_usage` +- `arangodb_scheduler_queue_memory_usage` +- `arangodb_scheduler_stack_memory_usage` +- `arangodb_search_consolidations_memory_usage` +- `arangodb_search_mapped_memory` +- `arangodb_search_readers_memory_usage` +- `arangodb_search_writers_memory_usage` +- `arangodb_transactions_internal_memory_usage` +- `arangodb_transactions_rest_memory_usage` + --- Introduced in: v3.11.2 diff --git a/site/content/3.12/release-notes/version-3.12/incompatible-changes-in-3-12.md b/site/content/3.12/release-notes/version-3.12/incompatible-changes-in-3-12.md index 7a5a8aba50..6771c2101e 100644 --- a/site/content/3.12/release-notes/version-3.12/incompatible-changes-in-3-12.md +++ b/site/content/3.12/release-notes/version-3.12/incompatible-changes-in-3-12.md @@ -79,6 +79,26 @@ The second option is the recommended one, as it signals the intent more clearly, and makes the cache behave "as expected", i.e. use up to the configured memory limit and not just 56% of it. +## Higher reported memory usage for AQL queries + +Due to the [improved memory accounting in v3.12](whats-new-in-3-12.md#improved-memory-accounting), +certain AQL queries may now get aborted because they exceed the defined +memory limit but didn't get killed in previous versions. This is because of the +more accurate memory tracking that reports a higher (actual) usage now. It allows +ArangoDB to more reliably detect and kill queries that go over the per-query and +global query memory limit, potentially preventing out-of-memory crashes of +_arangod_ processes. + +In particular, AQL queries that perform write operations now report a +significantly higher `peakMemoryUsage` in cursors than before. This is also +reflected in the `arangodb_aql_global_memory_usage` metric. Memory used for +ArangoSearch `SEARCH` operations is now also accounted for in the metric. + +You may need to adjust affected queries to use less memory or increase the +per-query limit with the [`memoryLimit` query option](../../aql/how-to-invoke-aql/with-arangosh.md#memorylimit) +or its default using the `--query.memory-limit` startup option. You can adjust +the global limit with the `--query.global-memory-limit` startup option. + ## Client tools ### jslint feature in arangosh diff --git a/site/content/3.12/release-notes/version-3.12/whats-new-in-3-12.md b/site/content/3.12/release-notes/version-3.12/whats-new-in-3-12.md index 93a9a7fdad..c05e9483f0 100644 --- a/site/content/3.12/release-notes/version-3.12/whats-new-in-3-12.md +++ b/site/content/3.12/release-notes/version-3.12/whats-new-in-3-12.md @@ -54,6 +54,39 @@ for details. ## Analyzers +## Improved memory accounting and usage + +Version 3.12 features multiple improvements to observability of ArangoDB +deployments. Memory usage is more accurately tracked and additional metrics have +been added for monitoring the memory consumption. + +AQL queries may now report a higher memory usage and thus run into memory limits +sooner, see [Higher reported memory usage for AQL queries](incompatible-changes-in-3-12.md#higher-reported-memory-usage-for-aql-queries). + +Furthermore, the memory usage of some subsystems has been optimized. When +dropping a database, all contained collections are now marked as dropped +immediately. Ongoing operations on these collections can be stopped earlier, and +memory for the underlying collections and indexes can be reclaimed sooner. +Memory used for index selectively estimates is now also released early. +ArangoSearch has a smaller memory footprint for removal operations now. + +The following new metrics have been added for memory observability: + +| Label | Description | +|:------|:------------| +| `arangodb_agency_node_memory_usage` | Memory used by Agency store/cache. | +| `arangodb_index_estimates_memory_usage` | Total memory usage of all index selectivity estimates. | +| `arangodb_internal_cluster_info_memory_usage` | Amount of memory spent in ClusterInfo. | +| `arangodb_requests_memory_usage` | Memory consumed by incoming, queued, and currently processed requests. | +| `arangodb_revision_tree_buffered_memory_usage` | Total memory usage of buffered updates for all revision trees. | +| `arangodb_scheduler_queue_memory_usage` | Number of bytes allocated for tasks in the scheduler queue. | +| `arangodb_scheduler_stack_memory_usage` | Approximate stack memory usage of worker threads. | +| `arangodb_search_consolidations_memory_usage` | Current count of opened file descriptors for this index. | +| `arangodb_search_mapped_memory` | Amount of memory in bytes that is mapped. | +| `arangodb_search_readers_memory_usage` | Amount of memory in bytes that is used for reading from this index. | +| `arangodb_search_writers_memory_usage` | Amount of memory in bytes that is used for writing to this index. | +| `arangodb_transactions_internal_memory_usage` | Total memory usage of internal transactions. | +| `arangodb_transactions_rest_memory_usage` | Total memory usage of user transactions (excluding top-level AQL queries). | ## Web interface From f4689c509a829171ee62f5fe644ca693b5ee7c48 Mon Sep 17 00:00:00 2001 From: Simran Spiller Date: Wed, 22 Nov 2023 09:31:22 +0100 Subject: [PATCH 2/2] Review --- .../version-3.12/incompatible-changes-in-3-12.md | 2 +- .../3.12/release-notes/version-3.12/whats-new-in-3-12.md | 8 ++++---- 2 files changed, 5 insertions(+), 5 deletions(-) diff --git a/site/content/3.12/release-notes/version-3.12/incompatible-changes-in-3-12.md b/site/content/3.12/release-notes/version-3.12/incompatible-changes-in-3-12.md index 6771c2101e..5280c837de 100644 --- a/site/content/3.12/release-notes/version-3.12/incompatible-changes-in-3-12.md +++ b/site/content/3.12/release-notes/version-3.12/incompatible-changes-in-3-12.md @@ -90,7 +90,7 @@ global query memory limit, potentially preventing out-of-memory crashes of _arangod_ processes. In particular, AQL queries that perform write operations now report a -significantly higher `peakMemoryUsage` in cursors than before. This is also +significantly higher `peakMemoryUsage` than before. This is also reflected in the `arangodb_aql_global_memory_usage` metric. Memory used for ArangoSearch `SEARCH` operations is now also accounted for in the metric. diff --git a/site/content/3.12/release-notes/version-3.12/whats-new-in-3-12.md b/site/content/3.12/release-notes/version-3.12/whats-new-in-3-12.md index c05e9483f0..90c0008dad 100644 --- a/site/content/3.12/release-notes/version-3.12/whats-new-in-3-12.md +++ b/site/content/3.12/release-notes/version-3.12/whats-new-in-3-12.md @@ -81,10 +81,10 @@ The following new metrics have been added for memory observability: | `arangodb_revision_tree_buffered_memory_usage` | Total memory usage of buffered updates for all revision trees. | | `arangodb_scheduler_queue_memory_usage` | Number of bytes allocated for tasks in the scheduler queue. | | `arangodb_scheduler_stack_memory_usage` | Approximate stack memory usage of worker threads. | -| `arangodb_search_consolidations_memory_usage` | Current count of opened file descriptors for this index. | -| `arangodb_search_mapped_memory` | Amount of memory in bytes that is mapped. | -| `arangodb_search_readers_memory_usage` | Amount of memory in bytes that is used for reading from this index. | -| `arangodb_search_writers_memory_usage` | Amount of memory in bytes that is used for writing to this index. | +| `arangodb_search_consolidations_memory_usage` | Amount of memory in bytes that is used for consolidating an ArangoSearch index. | +| `arangodb_search_mapped_memory` | Amount of memory in bytes that is mapped for an ArangoSearch index. | +| `arangodb_search_readers_memory_usage` | Amount of memory in bytes that is used for reading from an ArangoSearch index. | +| `arangodb_search_writers_memory_usage` | Amount of memory in bytes that is used for writing to an ArangoSearch index. | | `arangodb_transactions_internal_memory_usage` | Total memory usage of internal transactions. | | `arangodb_transactions_rest_memory_usage` | Total memory usage of user transactions (excluding top-level AQL queries). |