[ML] Apply windowing and chunking to long documents

[davidkyle]

Adds a chunkedInfer() method to the InferenceService interface which automatically splits long text before sending the inputs to the model. Chunking is done via a sliding window of length window_size with an overlap of span.

By default the window size is equal to the model's max sequence length and span is 50% of that (after accounting for special tokens). One reason to choose a smaller window size is that processing time is exponential on the number of input tokens, reducing the window size results in some lost context (fewer tokens per input) but may be the fastest strategy for ingesting long text.

This change only applies to the ELSER model and Text Embedding models deployed locally in the cluster

Response Structure

(Field names subject to change)

{
  "sparse_embedding_chunk": [
       {
          "text": "first text chunk",
          "inference": { sparse embedding tokens...}
        },
        {
          "text": "second text chunk",
          "inference": { sparse embedding tokens...}
        }, ...
    ]
}

[elasticsearchmachine]

Pinging @elastic/ml-core (Team:ML)

[elasticsearchmachine]

Hi @davidkyle, I've created a changelog YAML for you.

[carlosdelest]

Overall changes LGTM. 👍

How will chunking options work by default? Will models be deployed with default chunking options?

[jonathan-buttner]

Left a few questions and comments

[davidkyle]

How will chunking options work by default? Will models be deployed with default chunking options?

Currently I'm not sure.

There are 2 options to consider: span and windowSize. Every model has a max sequence length which maps to windowSize if windowSize is not set.

But the model config doesn't have a default value of span. This is a problem as the span parameter is optional (Integer rather than int) but in practice is must be set as there is no default.

The default span could be a function of max sequence length

[jonathan-buttner]

Looks good, just a few questions around switching to readOptional*() and I think we need a few return statements after onFailure() calls.