[ML] Add configurable settings for rerank chunking

[dan-rubinstein]

Description

As part of #121567 we are working to implement chunking for rerank inference calls (at least for the elastic reranker). The initial design of this will be to always chunk documents sent for rerank and return a relevance score for each document that corresponds with the highest score for any of it's chunks. The chunking strategy will be selected for the user based on the token limit. The purpose of this issue is to look into any configurable settings the user may want to be able to control this process. We will need to identify both what values the user might want to configure and where these settings may be stored.

[elasticsearchmachine]

Pinging @elastic/ml-core (Team:ML)

[dan-rubinstein]

Ideas for which values the user would want to configure

It's not yet clear what the motivation behind wanting to configure the values might be. Some potential motivations:

1. Wanting to minimize inference costs.
   • For the elastic reranker which combines the query with the document before sending it to rerank there will be "extra tokens" sent due to chunking as the query will be duplicated as part of the document sent along with each chunk. For other models, the number of overall tokens sent will be the same but the number of values to rerank sent will increase. I'm not sure how this affects costs.
2. Wanting to improve rerank results.
   • For this motivation, the user likely does not have enough information to accomplish this goal. Given that the chunks are not stored or presented to the user in any way, it'll be hard for them to configure any settings with confidence that it will create better results.

All the above being said, some potential configurable values could be:

1. The chunking strategy
   • As stated above, the user does not see the chunks so configuring the strategy may be confusing and open the door for users to generate worse rerank results without knowing why.
2. The maximum number of chunks (across all docs or just relative to each document) to generate per model
   • For this option we'll have to consider what to do when the maximum number is too small to fit the full document.
3. The maximum number of tokens/chunks (across all documents or just relative to each document) to send to the model
   • For this option we'll have to consider what the user gains from having this level of control.

Ideas for where the configurations might be stored

Some places where these values could be stored:

1. As part of the rerank endpoint
2. As part of a rerank retriever
3. As part of the inference call

[dan-rubinstein]

Configurable settings were included as part of the PR to add the RerankRequestChunker #130485