split docs instead of truncating on tokenizer max_input_tokens

[K-Schubert]

Description

Currently, a call to an embedding model will truncate the input text to embed according to the embedding model max_input_tokens if the number of tokens of the input text exceeds this limit.

It would be better to split the text and embed each chunk separately (and keep track of chunk IDs and relationship to one another) to avoid losing data.