How to get more than one Inference results with one request?

[activezhao]

In vLLM, the parameter of "N" is about how many Inference results we can get.

For example, when the "N" is 2, we can get the results as flowing:

"Choices": [
      {
        "FinishReason": "stop",
        "Index": 0,
        "Logprobs": {
          "TextOffset": [],
          "TokenLogprobs": [],
          "Tokens": []
        },
        "Text": "(arr, left, right):"
      },
      {
        "FinishReason": "stop",
        "Index": 1,
        "Logprobs": {
          "TextOffset": [],
          "TokenLogprobs": [],
          "Tokens": []
        },
        "Text": "(arr, low, high):"
      }
    ]

In Triton, how to set parameters to achieve the same effect as above?

Thanks.

[byshiue]

Currently, TensorRT-LLM does not support such API directly. You can use same inputs with different random seed to sample different result now.

[activezhao]

Currently, TensorRT-LLM does not support such API directly. You can use same inputs with different random seed to sample different result now.

@byshiue Thanks for your reply, could u please give me a example?

[byshiue]

Like an inputs with batch size 2:

input_ids = [[1, 2], [1, 2]]
random_seed = [1, 2]
topk = [4, 4]

[activezhao]

Like an inputs with batch size 2:

input_ids = [[1, 2], [1, 2]]
random_seed = [1, 2]
topk = [4, 4]

@byshiue Thanks, I understand what you mean, that means I need to make multiple identical requests at once, right?

But, how much overhead will this add? Will this cause the inference to be done twice completely independently, or can some tokens be reused, which can reduce some overhead?

[byshiue]

Like an inputs with batch size 2:

input_ids = [[1, 2], [1, 2]]
random_seed = [1, 2]
topk = [4, 4]

@byshiue Thanks, I understand what you mean, that means I need to make multiple identical requests at once, right?

But, how much overhead will this add? Will this cause the inference to be done twice completely independently, or can some tokens be reused, which can reduce some overhead?

It will introduce some additional cost in context phase now.

[activezhao]

Like an inputs with batch size 2:

input_ids = [[1, 2], [1, 2]]
random_seed = [1, 2]
topk = [4, 4]

@byshiue Thanks, I understand what you mean, that means I need to make multiple identical requests at once, right?
But, how much overhead will this add? Will this cause the inference to be done twice completely independently, or can some tokens be reused, which can reduce some overhead?

It will introduce some additional cost in context phase now.

@byshiue OK, got it. One more thing, can Triton's generate endpoint with a curl command be implemented?
Or I have to use grpc with tensor to achieve it？

curl -X POST localhost:8000/v2/models/ensemble/generate -d '{"text_input": "What is machine learning?", "max_tokens": 20, "bad_words": "", "stop_words": ""}'

[byshiue]

generate endpoint is supported in 23.10 tritonserver, you should be able to run generate endpoint by using 23.10 docker image to build tensorrt llm backend.

[activezhao]

generate endpoint is supported in 23.10 tritonserver, you should be able to run generate endpoint by using 23.10 docker image to build tensorrt llm backend.

@byshiue Yes, you r right. In fact, I' m using 23.10 docker image now, so I have this question?

[byshiue]

It should work. Do you encounter any issue?

[sadreamer]

Hello @byshiue
Any plans to support the shared_context feature (or something similar) in trtllm backend? https://github.com/NVIDIA/FasterTransformer/blob/main/docs/gpt_guide.md#generate-different-sentences-and-enable-shared-context
Thanks

[byshiue]

We don't have such plan now. If you are interested, you could create a feature request to track.