[Question] How can I use ensemble model to get output token one at a time before it's sent to the client ?

[ZihanLiao]

Suffering from repeated output, I'm trying to add some rules to the postprocessing model to avoid duplication. I'm trying to early stop the service when duplication occurs, so i think this should be a wrapper above the streaming mode.
So the question is how to get the token one at a time in the ensemble model to make me do early stopping?

[uppalutkarsh]

@ZihanLiao, if I understand correctly you want to postprocess the output from the tensorrtllm and also want to impose certain stopping logic.

You can use stop_words which gets passed to tensorrt_llm in model repository as stop_words_list. If you explicitly want to add more logic then you can add in postprocessor model.py here.

[ZihanLiao]

@ZihanLiao, if I understand correctly you want to postprocess the output from the tensorrtllm and also want to impose certain stopping logic.

You can use stop_words which gets passed to tensorrt_llm in model repository as stop_words_list. If you explicitly want to add more logic then you can add in postprocessor model.py here.

Thanks for the quick reply.
Indeed, my wish is to add further stopping logic, but I don't have specific stop_words. The implementation here seems like it got the whole sequence generated by tensorrtllm, but I wish to stop the generation when repetition occurred by using the postprocessing logic which might took less time per request.

[byshiue]

You could use decouple mode (streaming) to get the output one by one.

But if you only want to prevent repeated result, you could also try using larger repeat penalties.

[ZihanLiao]

You could use decouple mode (streaming) to get the output one by one.

But if you only want to prevent repeated result, you could also try using larger repeat penalties.

Thanks, I'm now trying the tensorrt_llm_bls model and I met the problem that in streaming mode, "-" is generated iteratively which token_id is 1177 in my scenario.

I also tested inflight_batcher_llm_client.py to solely request the tensorrt_llm and everything is fine. I checked the model input between these two procedures and found them identical. May be something wrong happened in here