NvidiaDocumentEmbedder Failed to query embedding endpoint: Error - "message":"Input length 616 exceeds maximum allowed token size 512"

[smmhsnn]

when I run NvidiaDocumentEmbedder , it gives me the following error even though I use a Document splitter to split my text inputs to only 200 words and also truncate them to make sure they never exceed 512 tokens. Anybody faced the same issue?

ValueError: Failed to query embedding endpoint: Error - {"object":"error","message":"Input length 616 exceeds maximum allowed token size 512","detail":{},"type":"invalid_request_error"}

here is my code:

def truncate_text(text, max_tokens=512):
    tokens = text.split()
    return " ".join(tokens[:max_tokens]) if len(tokens) > max_tokens else text 

def truncate_document_content(document):
     text = document.content
     document.content = truncate_text(text)
     return document

with open("parsed_pdfs.json", 'r') as file:
    dataset = json.load(file)

document_store = InMemoryDocumentStore()
docs = [Document(id=doc["id"], content=doc["content"], 
                 meta=doc["meta"]) for doc in dataset]

cleaner = DocumentCleaner(remove_empty_lines = True,
                        remove_extra_whitespaces = True,
                        remove_repeated_substrings = True)
clean_docs = cleaner.run(docs)

splitter = DocumentSplitter(split_by="word", split_length=200, split_overlap=0)
splitter.warm_up()
splitted_docs = splitter.run(clean_docs["documents"])

docs = splitted_docs["documents"]
documents = [truncate_document_content(doc) for doc in docs]

doc_embedder = NvidiaDocumentEmbedder(model=<name>,
                                    api_url=<url>, api_key=None)
doc_embedder.warm_up()
embedded_docs = doc_embedder.run(documents)

[julian-risch]

Hello @smmhsnn thanks for sharing the code you used. From the initialization of the DocumentSplitter, I can see that you are splitting by word with a limit of 200. That means texts longer than 200 words will be split up into multiple documents. However, even a text with less than 200 words, for example 150 words, can result in more than 200 tokens as some words consist of multiple tokens.
What you could try is set the split_length to an even lower number in your DocumentSplitter or try to identify, which particular document is resulting in so many tokens.
We have an open issue for a preprocessor component that takes the number of tokens into account: #4392

[smmhsnn]

Hello @julian-risch thanks for your reply, yes you are right, I could solve the issue by cleaning the texts to filter out some special charactrs, and setting the spliiter length to only 50 words.