Option to increase the number of input tokens for Falcon 40B Model

[gabriead]

Feature request

I would like to increase the number of input tokens from currently 1024 to it's maximum of 2000 tokens.

Motivation

In order to answer questions given a specific context I want to use as many input tokens as possible.

Your contribution

I tried to extend the number of tokens in the openapi.json but that didn't help

[OlivierDehaene]

You can use text-generation-launcher --help to see all the options available to you. The ones you might be interested in are --max-input-length and --max-total-tokens.

[gabriead]

Hi @OlivierDehaene, thank's for your reply. I am using the generate() method from the client class. That does not provide any of the flags you mentioned. Where am I going wrong here? Could you please specify how I can use the text-generation launcher for inference?

[OlivierDehaene]

Oh as a client you don't have the ability to modify these parameters. Are you using the Huggingface inference API?

[gabriead]

I am not sure, what you mean by 'as a client'. I am using it like so:
from text_generation import Client client = Client("..") print(client.generate("...", max_new_tokens=...).generated_text)
And now I need to extend the number of input tokens to the maximum. Is there a way to integrate that into the framework?