From d37cf2068f23d3d0dfb5158471d6a659ea169f93 Mon Sep 17 00:00:00 2001 From: Stef Nestor <26751266+stefnestor@users.noreply.github.com> Date: Wed, 3 Jul 2024 18:31:33 -0600 Subject: [PATCH 1/9] (Doc+) Error "number of documents in the index can't exceed" MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 👋 howdy, team! This adds resolution outline for error ... which induces ongoing, lowkey support ``` Number of documents in the index can't exceed [2147483519] ``` --- .../how-to/size-your-shards.asciidoc | 47 +++++++++++++++++++ 1 file changed, 47 insertions(+) diff --git a/docs/reference/how-to/size-your-shards.asciidoc b/docs/reference/how-to/size-your-shards.asciidoc index 56e5fbbf15c77..b1c1d52ec9493 100644 --- a/docs/reference/how-to/size-your-shards.asciidoc +++ b/docs/reference/how-to/size-your-shards.asciidoc @@ -544,3 +544,50 @@ PUT _cluster/settings } } ---- + +[discrete] +==== Number of documents in the index can't exceed [2147483519] + +Elasticsearch shards reflect Lucene's underlying https://github.com/apache/lucene/issues/5176[index +`MAX_DOC` hard limit] of 2,147,483,519 (`(2^31)-1`) docs. This figure calulates +from the sum of `docs.count` plus `docs.deleted` as reported by <> +per shard. Attempting to surpass this limit, e.g. via ingestion's <>, +will cause error + +[source, txt] +---- +Elasticsearch exception [type=illegal_argument_exception, reason=Number of documents in the index can't exceed [2147483519]] +---- + +or this error's sister stack trace will appear + +[source, txt] +---- +Caused by: java.lang.IllegalArgumentException: number of documents in the index cannot exceed 2147483519 +---- + +Noting that in this error message, `index` is referencing a Lucene index which correlates to an +<>. + +TIP: This calculation works off all docs within the shard, including nested docs. +Therefore, it's results are not guaranteed to line up to the <> +which masks nested docs. + +Encountering this error usually suggests an issue with sharding strategy being +out of compliance with the above recommendations on this page, specifically at least +<>. + +Frequently, the `docs.deleted` for the index will be elevated that +<> can work as a short-term stop gap to regain +ability to admin the index, e.g. for `my-index-000001` this command would appear + +[source,console] +---- +POST my-index-000001/_forcemerge?only_expunge_deletes=true +---- + +This will kick off an asynchronous task which can be monitored via the <>. + +Longer-term, this error can usually be resolved by <> +from the index or by <> or <> the index with +corrected shard strategy settings. From a8010cdd903709b6fe52a12c5af7ba1b42b4dfea Mon Sep 17 00:00:00 2001 From: Stef Nestor <26751266+stefnestor@users.noreply.github.com> Date: Thu, 4 Jul 2024 09:43:15 -0600 Subject: [PATCH 2/9] feedback --- docs/reference/how-to/size-your-shards.asciidoc | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/docs/reference/how-to/size-your-shards.asciidoc b/docs/reference/how-to/size-your-shards.asciidoc index b1c1d52ec9493..1fd48d6c85903 100644 --- a/docs/reference/how-to/size-your-shards.asciidoc +++ b/docs/reference/how-to/size-your-shards.asciidoc @@ -546,17 +546,20 @@ PUT _cluster/settings ---- [discrete] -==== Number of documents in the index can't exceed [2147483519] +==== Number of documents in the index shard cannot exceed [2147483519] + +NOTE: Historically, this error reported `Number of documents in the index shard cannot exceed [2147483519]` where +`index` was referencing a Lucene index which correlates to an <>. Elasticsearch shards reflect Lucene's underlying https://github.com/apache/lucene/issues/5176[index -`MAX_DOC` hard limit] of 2,147,483,519 (`(2^31)-1`) docs. This figure calulates +`MAX_DOC` hard limit] of 2,147,483,519 (`(2^31)-129`) docs. This figure calulates from the sum of `docs.count` plus `docs.deleted` as reported by <> per shard. Attempting to surpass this limit, e.g. via ingestion's <>, will cause error [source, txt] ---- -Elasticsearch exception [type=illegal_argument_exception, reason=Number of documents in the index can't exceed [2147483519]] +Elasticsearch exception [type=illegal_argument_exception, reason=Number of documents in the index shard cannot exceed [2147483519]] ---- or this error's sister stack trace will appear @@ -566,9 +569,6 @@ or this error's sister stack trace will appear Caused by: java.lang.IllegalArgumentException: number of documents in the index cannot exceed 2147483519 ---- -Noting that in this error message, `index` is referencing a Lucene index which correlates to an -<>. - TIP: This calculation works off all docs within the shard, including nested docs. Therefore, it's results are not guaranteed to line up to the <> which masks nested docs. From deabb2da585108bf4f543903e50be3660b835585 Mon Sep 17 00:00:00 2001 From: Stef Nestor <26751266+stefnestor@users.noreply.github.com> Date: Thu, 11 Jul 2024 08:34:19 -0600 Subject: [PATCH 3/9] feedback Co-authored-by: David Turner --- docs/reference/how-to/size-your-shards.asciidoc | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/docs/reference/how-to/size-your-shards.asciidoc b/docs/reference/how-to/size-your-shards.asciidoc index 1fd48d6c85903..d317f8c47dd7a 100644 --- a/docs/reference/how-to/size-your-shards.asciidoc +++ b/docs/reference/how-to/size-your-shards.asciidoc @@ -548,8 +548,6 @@ PUT _cluster/settings [discrete] ==== Number of documents in the index shard cannot exceed [2147483519] -NOTE: Historically, this error reported `Number of documents in the index shard cannot exceed [2147483519]` where -`index` was referencing a Lucene index which correlates to an <>. Elasticsearch shards reflect Lucene's underlying https://github.com/apache/lucene/issues/5176[index `MAX_DOC` hard limit] of 2,147,483,519 (`(2^31)-129`) docs. This figure calulates @@ -559,7 +557,7 @@ will cause error [source, txt] ---- -Elasticsearch exception [type=illegal_argument_exception, reason=Number of documents in the index shard cannot exceed [2147483519]] +Elasticsearch exception [type=illegal_argument_exception, reason=Number of documents in the shard cannot exceed [2147483519]] ---- or this error's sister stack trace will appear From 532fd8e7ffcd0a925fbe425f013122e017eb0726 Mon Sep 17 00:00:00 2001 From: Stef Nestor <26751266+stefnestor@users.noreply.github.com> Date: Thu, 11 Jul 2024 09:57:28 -0600 Subject: [PATCH 4/9] feedback Co-authored-by: David Turner Co-authored-by: Liam Thompson <32779855+leemthompo@users.noreply.github.com> --- .../how-to/size-your-shards.asciidoc | 20 ++++++++----------- 1 file changed, 8 insertions(+), 12 deletions(-) diff --git a/docs/reference/how-to/size-your-shards.asciidoc b/docs/reference/how-to/size-your-shards.asciidoc index d317f8c47dd7a..1159a965084d5 100644 --- a/docs/reference/how-to/size-your-shards.asciidoc +++ b/docs/reference/how-to/size-your-shards.asciidoc @@ -546,14 +546,13 @@ PUT _cluster/settings ---- [discrete] -==== Number of documents in the index shard cannot exceed [2147483519] +==== Number of documents in the shard cannot exceed [2147483519] Elasticsearch shards reflect Lucene's underlying https://github.com/apache/lucene/issues/5176[index -`MAX_DOC` hard limit] of 2,147,483,519 (`(2^31)-129`) docs. This figure calulates -from the sum of `docs.count` plus `docs.deleted` as reported by <> -per shard. Attempting to surpass this limit, e.g. via ingestion's <>, -will cause error +`MAX_DOC` hard limit] of 2,147,483,519 (`(2^31)-129`) docs. This figure is +the sum of `docs.count` plus `docs.deleted` as reported by the <> +per shard. Exceeding this limit will result in errors like the following: [source, txt] ---- @@ -567,17 +566,14 @@ or this error's sister stack trace will appear Caused by: java.lang.IllegalArgumentException: number of documents in the index cannot exceed 2147483519 ---- -TIP: This calculation works off all docs within the shard, including nested docs. +TIP: This calculation includes nested documents. +This may differ from the results of the <>. Therefore, it's results are not guaranteed to line up to the <> which masks nested docs. -Encountering this error usually suggests an issue with sharding strategy being -out of compliance with the above recommendations on this page, specifically at least -<>. -Frequently, the `docs.deleted` for the index will be elevated that -<> can work as a short-term stop gap to regain -ability to admin the index, e.g. for `my-index-000001` this command would appear +Try using the <> to free up space and recover the +ability to admin the index. For example: [source,console] ---- From 1081f31d6978608bfe4f518acccb411bcbc94303 Mon Sep 17 00:00:00 2001 From: Stef Nestor <26751266+stefnestor@users.noreply.github.com> Date: Thu, 11 Jul 2024 09:57:52 -0600 Subject: [PATCH 5/9] feedback --- docs/reference/how-to/size-your-shards.asciidoc | 7 ------- 1 file changed, 7 deletions(-) diff --git a/docs/reference/how-to/size-your-shards.asciidoc b/docs/reference/how-to/size-your-shards.asciidoc index 1159a965084d5..86d2950a7b425 100644 --- a/docs/reference/how-to/size-your-shards.asciidoc +++ b/docs/reference/how-to/size-your-shards.asciidoc @@ -559,13 +559,6 @@ per shard. Exceeding this limit will result in errors like the following: Elasticsearch exception [type=illegal_argument_exception, reason=Number of documents in the shard cannot exceed [2147483519]] ---- -or this error's sister stack trace will appear - -[source, txt] ----- -Caused by: java.lang.IllegalArgumentException: number of documents in the index cannot exceed 2147483519 ----- - TIP: This calculation includes nested documents. This may differ from the results of the <>. Therefore, it's results are not guaranteed to line up to the <> From bad68971eff78b459cafe39f95099d3fd47aa233 Mon Sep 17 00:00:00 2001 From: Stef Nestor <26751266+stefnestor@users.noreply.github.com> Date: Thu, 11 Jul 2024 17:11:37 -0600 Subject: [PATCH 6/9] feedback --- .../reference/how-to/size-your-shards.asciidoc | 18 ++++++++---------- 1 file changed, 8 insertions(+), 10 deletions(-) diff --git a/docs/reference/how-to/size-your-shards.asciidoc b/docs/reference/how-to/size-your-shards.asciidoc index 86d2950a7b425..9ca46a1d09fe1 100644 --- a/docs/reference/how-to/size-your-shards.asciidoc +++ b/docs/reference/how-to/size-your-shards.asciidoc @@ -559,22 +559,20 @@ per shard. Exceeding this limit will result in errors like the following: Elasticsearch exception [type=illegal_argument_exception, reason=Number of documents in the shard cannot exceed [2147483519]] ---- -TIP: This calculation includes nested documents. -This may differ from the results of the <>. -Therefore, it's results are not guaranteed to line up to the <> -which masks nested docs. +TIP: This calculation may differ from the <> calculation, because the Count API does not include nested documents. -Try using the <> to free up space and recover the -ability to admin the index. For example: +Try using the <> to clear deleted docs. For example: [source,console] ---- POST my-index-000001/_forcemerge?only_expunge_deletes=true ---- -This will kick off an asynchronous task which can be monitored via the <>. +This will launch an asynchronous task which can be monitored via the <>. -Longer-term, this error can usually be resolved by <> -from the index or by <> or <> the index with -corrected shard strategy settings. +For a long-term solution try: + +* <> +* aligning the index to recommendations on this page by either +<> or <> the index From abcac663905b42c09ad20c2fe4384974ccfc0c0d Mon Sep 17 00:00:00 2001 From: Liam Thompson <32779855+leemthompo@users.noreply.github.com> Date: Tue, 16 Jul 2024 10:32:20 +0200 Subject: [PATCH 7/9] Test change to address docs check failure --- docs/reference/how-to/size-your-shards.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/reference/how-to/size-your-shards.asciidoc b/docs/reference/how-to/size-your-shards.asciidoc index 9ca46a1d09fe1..4c7c47df8ff08 100644 --- a/docs/reference/how-to/size-your-shards.asciidoc +++ b/docs/reference/how-to/size-your-shards.asciidoc @@ -554,7 +554,7 @@ Elasticsearch shards reflect Lucene's underlying https://github.com/apache/lucen the sum of `docs.count` plus `docs.deleted` as reported by the <> per shard. Exceeding this limit will result in errors like the following: -[source, txt] +[source] ---- Elasticsearch exception [type=illegal_argument_exception, reason=Number of documents in the shard cannot exceed [2147483519]] ---- From 5d85b9651748239981b5782750305fd72d8fcceb Mon Sep 17 00:00:00 2001 From: Liam Thompson <32779855+leemthompo@users.noreply.github.com> Date: Tue, 16 Jul 2024 10:52:22 +0200 Subject: [PATCH 8/9] Revert test change --- docs/reference/how-to/size-your-shards.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/reference/how-to/size-your-shards.asciidoc b/docs/reference/how-to/size-your-shards.asciidoc index 4c7c47df8ff08..69a3225d9d80b 100644 --- a/docs/reference/how-to/size-your-shards.asciidoc +++ b/docs/reference/how-to/size-your-shards.asciidoc @@ -554,7 +554,7 @@ Elasticsearch shards reflect Lucene's underlying https://github.com/apache/lucen the sum of `docs.count` plus `docs.deleted` as reported by the <> per shard. Exceeding this limit will result in errors like the following: -[source] +[source,txt] ---- Elasticsearch exception [type=illegal_argument_exception, reason=Number of documents in the shard cannot exceed [2147483519]] ---- From 8974e55035768e2d40079bd038967b3e2572d50d Mon Sep 17 00:00:00 2001 From: Liam Thompson <32779855+leemthompo@users.noreply.github.com> Date: Tue, 16 Jul 2024 10:58:29 +0200 Subject: [PATCH 9/9] Test docs check --- docs/reference/how-to/size-your-shards.asciidoc | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/reference/how-to/size-your-shards.asciidoc b/docs/reference/how-to/size-your-shards.asciidoc index 69a3225d9d80b..53f47fc88cdb2 100644 --- a/docs/reference/how-to/size-your-shards.asciidoc +++ b/docs/reference/how-to/size-your-shards.asciidoc @@ -568,6 +568,7 @@ Try using the <> to clear deleted docs. For ---- POST my-index-000001/_forcemerge?only_expunge_deletes=true ---- +// TEST[setup:my_index] This will launch an asynchronous task which can be monitored via the <>.