OOM when performing inference on an extremely large document with a semantic_text field

[maxhniebergall]

Elasticsearch Version

8.15.2, 8.16, main

Installed Plugins

No response

Java Version

bundled

OS Version

linux

Problem Description

The addition of automatic chunking combined with very large documents has caused some ingest processes to crash, with OOM or quiet restarts of the ML node.

We are currently working on long-term solutions to this problem which will be released in an upcoming patch.

Steps to Reproduce

Use semantic_text field to ingest documents over 100k characters in length.

Logs (if relevant)

No response

[elasticsearchmachine]

Pinging @elastic/ml-core (Team:ML)

[maxhniebergall]

this PR may fix close this issue #115477

[wwang500]

this PR may fix close this issue #115477

it got improved in 8.17. not OOM any more, now it throws the error:

{
  "completed": true,
  "task": {
    "node": "t1aieHDxT5yWCKf4TuL8zQ",
    "id": 4685082,
    "type": "transport",
    "action": "indices:data/write/reindex",
    "status": {
      "total": 1,
      "updated": 0,
      "created": 0,
      "deleted": 0,
      "batches": 1,
      "version_conflicts": 0,
      "noops": 0,
      "retries": {
        "bulk": 0,
        "search": 0
      },
      "throttled_millis": 0,
      "requests_per_second": -1,
      "throttled_until_millis": 0
    },
    "description": "reindex from [test] to [reindex_test_4]",
    "start_time_in_millis": 1731775430660,
    "running_time_in_nanos": 31643758184,
    "cancellable": true,
    "cancelled": false,
    "headers": {
      "trace.id": "68fb137d346234295b9a4ed75f87b25b"
    }
  },
  "response": {
    "took": 31643,
    "timed_out": false,
    "total": 1,
    "updated": 0,
    "created": 0,
    "deleted": 0,
    "batches": 1,
    "version_conflicts": 0,
    "noops": 0,
    "retries": {
      "bulk": 0,
      "search": 0
    },
    "throttled": "0s",
    "throttled_millis": 0,
    "requests_per_second": -1,
    "throttled_until": "0s",
    "throttled_until_millis": 0,
    "failures": [
      {
        "index": "reindex_test_4",
        "id": "1",
        "cause": {
          "type": "document_parsing_exception",
          "reason": "[1:175392483] failed to parse field [content] of type [semantic_text] in document with id '1'. Preview of field's value: 'null'",
          "caused_by": {
            "type": "document_parsing_exception",
            "reason": "[1:175392483] The number of nested documents has exceeded the allowed limit of [10000]. This limit can be set by changing the [index.mapping.nested_objects.limit] index level setting."
          }
        },
        "status": 400
      }
    ]
  }
}

[davidkyle]

The number of nested documents has exceeded the allowed limit of [10000]

This error is usually seen with the ELSER tokens where the field is not mapped as a sparse_vector

[wwang500]

Hmm, it complains index.mapping.nested_objects.limit] and this is the mapping before reindex:

PUT reindex_test_4
{
  "mappings": {
    "properties": {
      "content": { 
        "type": "semantic_text", 
        "inference_id": "elser-endpoint" 
      }
    }
  }
}

[maxhniebergall]

@wwang500, do we know how big the document was that caused this?

[wwang500]

@wwang500, do we know how big the document was that caused this?

The test data I used is 60million chars in one document.

[maxhniebergall]

Ok, so 10k chunks at 200 words each is 2M words, at 5 char per word, that would be 10M chars, so by this math it should be impossible to ingest a document with 60M chars with the default settings, and you would need to roughly 6x-10x the default nested documents limit to be able to ingest a document that large. Certainly, I would expect 10xing the default to cause some of the OOM errors that are warned about in the documentation.

I think this means that the limit for the the maximum document size with chunking is roughly 10MB/10M chars.

From here, we should probably document this limitation and suggest breaking up documents into roughly 1MB sections before ingesting.

[sphilipse]

Seeing this cause OOMs again when performing inference with large documents and multiple semantic text fields. Ingesting the rally wikipedia dataset with both an E5 and ELSER dataset has been causing ooms on my project.