Document Request

[oroojlooy]

Feature request

In Text-Generation-Inference (TGI), I see that there is a parameter of --max-batch-total-tokens, indicating that there is a batch request capability available via TGI. But, when I see the API guide, I cannot find anything related to that. For example, for /generate, the input is

{
  "inputs": "My name is Olivier and I",
  "parameters": {
    "best_of": 1,
    "decoder_input_details": false,
    "details": true,
    "do_sample": true,
    "frequency_penalty": 0.1,
    "grammar": null,
    "max_new_tokens": 20,
    "repetition_penalty": 1.03,
    "return_full_text": false,
    "seed": null,
    "stop": [
      "photographer"
    ],
    "temperature": 0.5,
    "top_k": 10,
    "top_n_tokens": 5,
    "top_p": 0.95,
    "truncate": null,
    "typical_p": 0.95,
    "watermark": true
  }
}

which cannot handle batch requests. I was wondering if there is a batch-request support, and if so, it would be great to add some API guideline/documentation for that.

Motivation

N/A

Your contribution

N/A

[ktrapeznikov]

you can just send multiple simultaneous requests and the TGI will automatically batch them up

[oroojlooy]

you can just send multiple simultaneous requests and the TGI will automatically batch them up

Oh OK. That makes sense.