Understanding of DeepSpeeChat “ Inference”

[gosth]

I just started learning gpt field, and read the article about DeepSpeeChat
There are some doubts about the understanding of “ Inference”.

In article, "Inference" is a sub-process of RLHF training， like this：

some questions：
1、 "Inference" in DeepSpeeChat, is a sub-process of RLHF training. So it is a part of training（an offline process）？
2、 The DeepSpeedChat can improve efficiency.Is it means DeepSpeedChat can only imporve the efficiency of offline process(traing), not online process(online serving)?
3、 “The Hybrid Engine” in article， can only imporve the efficiency of offline process(traing), not online process(online serving)?

[lekurile]

Hi @gosth,

Thanks for the questions.

1. Yes, the Actor model will enter a generation phase where inference execution will be performed.
2. DeepSpeed-Chat improves efficiency during the offline process and the DeepSpeed team has an Inference Engine that can dramatically improve the online process (please see this tutorial here). We also provide examples for the Inference Engine across a variety of tasks (text-generation, fill-mask, translation, etc).
3. One of the things the Hybrid Engine enables in the RLHF training framework is to take advantage of the optimized inference features DeepSpeed provides. For online processing, i.e. serving, the Inference Engine can be used to take advantage of these same features such as optimized kernels and efficient memory management (see 2. above).

Please let us know if you have any additional questions.

Thanks,
Lev

[gosth]

@lekurile Thanks for your reply.It helps me a lot.