Error in parameters.max_new_tokens: ensure this value is less than or equal to 250' in rag_pipeline.run(..)

[Sandeep-madhyastha]

Wish: Can rag_pipeline.run(..) have another input parameter "max_new_tokens" where in caller can restrict the number of tokens returned.

Scenario : Currently when called with some TGI from hugging face following error is occurred()

Great work :-)Thanks

[anakin87]

Hello!

Please provide a reproducible example of the code that raised that error.

[Sandeep-madhyastha]

Hi Thanks for reply.

The Code: its a copy from Git with just HTML source changed

---

from haystack.document_stores import InMemoryDocumentStore
from haystack.pipeline_utils import build_rag_pipeline, build_indexing_pipeline
from haystack.pipeline_utils.indexing import download_files

#based on https://github.com/deepset-ai/haystack/blame/main/examples/getting_started/rag_custom_data.py

We are model agnostic :) In this getting started you can choose any OpenAI or Huggingface TGI generation model

generation_model = "open_llama_3b_v2" # EleutherAI/gpt-neo-2.7B"
API_KEY = "API-KEY" # ADD YOUR KEY HERE

We support many different databases. Here, we load a simple and lightweight in-memory database.

document_store = InMemoryDocumentStore()

Download example files from web

files = download_files(sources=["https://en.wikipedia.org/wiki/Andre_Pool"]) #["http://www.paulgraham.com/superlinear.html"])

Pipelines are our main abstratcion.

Here we create a pipeline that can index TXT and HTML. You can also use your own private files.

indexing_pipeline = build_indexing_pipeline(
document_store=document_store,
embedding_model="intfloat/e5-base-v2",
supported_mime_types=["text/plain", "text/html"], # "application/pdf"
)
indexing_pipeline.run(files=files) # you can also supply files=[path_to_directory], which is searched recursively

RAG pipeline with vector-based retriever + LLM

rag_pipeline = build_rag_pipeline(
document_store=document_store,
embedding_model="intfloat/e5-base-v2",
generation_model=generation_model,
llm_api_key=API_KEY,
)

For details, like which documents were used to generate the answer, look into the result object

result = rag_pipeline.run(query="When was Andre Pool born?")
print({result.data})

[anakin87]

The problem seems that in this example we are hardcoding the max_new_tokens parameter to 1024

haystack/haystack/pipeline_utils/rag.py

Line 201 in 552f0e3

return HuggingFaceTGIGenerator(model=model_key, token=api_key, generation_kwargs={"max_new_tokens": 1024})

and TGI complains. But in this example, there is no way to change this value. We should improve the example.

For the proper way to build pipelines that are model-agnostic, I suggest reading these:

• Get started docs and recipe
• HuggingFaceTGIGenerator docs