Finetune

[liuxinxin123]

Can we support instruction finetune?

[StellaAthena]

Nothing special is required to do instruction-finetuning. It's the same as normal finetuning, just with different data.

if you're interested in RLHF, that's not something we plan on supporting, however there are libraries such as trl and trlX that support this. The later library has a WIP integration with GPT-NeoX

[iNeil77]

Hi @StellaAthena, I think you mean (If I'm not mistaken) to line up the instruct data as an append of instruction and response and continue with regular LM training. However, for most instruction tuning, it is standard to mask the gradient updates for the instruction and only backdrop them for the response. Does the framework make this possible?

[StellaAthena]

Hi @StellaAthena, I think you mean (If I'm not mistaken) to line up the instruct data as an append of instruction and response and continue with regular LM training. However, for most instruction tuning, it is standard to mask the gradient updates for the instruction and only backdrop them for the response. Does the framework make this possible?

There's an open PR adding support for this here. We also have started implementing RLHF features since my comment, e.g., here and here.

[iNeil77]

Thanks for the reply! Will check it out