Update on the development branch

[kaiyux]

Hi,

The TensorRT-LLM team is pleased to announce that we are pushing an update to the development branch (and the Triton backend) this February 21, 2024.

This update includes:

• Features
  • Enable different rewind tokens per sequence for Medusa
  • OOTB functionality support
    • T5
    • Mixtral 8x7B
  • Experimental: Weightless engine support (see examples/weightless_engine/README.md)
• API
  • Add high-level C++ API for inflight batching
  • Migrate Mixtral to high level API and unified builder workflow
• Bug fixes
  • Fix a weight-only quant bug for Whisper to make sure that the encoder_input_len_range should not be 0, thanks to the contribution from @Eddie-Wang1120 in Fix enc_dec bug and Make several improvements to whisper #992
  • Fix the issue that log probabilities in python runtime are not returned Question: Return log probabilites  #983
• Benchmark/Performance
  • Optimize gptDecoderBatch to support batched sampling
  • Enable FMHA for models in BART, Whisper and NMT family
  • Add emulated static batching in gptManagerBenchmark
• Documentation
  • Blog: Speed up inference with SOTA quantization techniques in TRT-LLM (see docs/source/blogs/quantization-in-TRT-LLM.md)

Thanks,
The TensorRT-LLM Engineering Team

[akhoroshev]

What does it mean Add high-level C++ API for inflight batching ?

[akhoroshev]

I don't see any useful differences between GptManager and Executor. For what reason was Executor introduced?

[pcastonguay]

We have started the development of a higher level C++ API. See https://github.com/NVIDIA/TensorRT-LLM/blob/main/cpp/include/tensorrt_llm/executor/executor.h. Note that currently it only supports single-GPU models. This API is still under active development and not as thoroughly tested/validated as the GptManager interface. It is also subject to change in the future.

[akhoroshev]

If possible add more flexibility to this new API. For example if I want to introduce a new sampling parameter, this is not possible right now with GptManager

[matichon-vultureprime]

Could you share some use cases of the weightless engine, please?

[ttyio]

Hi @matichon-vultureprime , Weightless enables refit only for constant weights that do not impact the builder’s ability to optimize and produce an engine with the same runtime performance as a non-refittable engine. Those weights are then omitted from the serialized engine, resulting in a small plan file that can be refitted at runtime using the weights from the checkpoint. One use case you may be using TensorRT-LLM on multiple GPUs, and wish to avoid an additional copy of weights in the TensorRT plan. Note this is experimental feature, and the full support would after next TensorRT major release.

[ethnzhng]

@ttyio Hi, any updates in regards to weightless/refittable engine support? I see that the weightless example has been removed from the dev branch.

[ttyio]

There is dependency on TRT 10.0, we will release it later after TRT 10.0 GA.

[ethnzhng]

@ttyio I see, and is this expected to come with TRT-LLM 0.9.0?

[ttyio]

TRT 10.0 GA release is scheduled after TRT-LLM 0.9.0, so this is expected to come in the TRT-LLM weekly release after 0.9.0.

[mlmonk]

Could you share if there has been any update on the 800 issue regarding triton backend for enc_dec models?

[symphonylyh]

replied under #800 and let's continue the discussion there, thanks