How to fine tune on massive dataset (10k + dialog lines)

[atl333]

So i have a few 100k+ datasets, and i made some 10k line ones trying to finetune several models but so far this is the only high level programming project where I have seen full documentation. Is it possible to fine tune my bot to, for example, reply on topics about politics with 10k+ lines of politics conversations, just for fun of course, I don't expect it to come up with anything substantial xD

I have an RTX 2060 and CUDA 11.3, pythorch, transformers, cudnn 8, etc

[brightening-eyes]

yes.
write a logic adapter, in it, get the text, preprocess it, feed it to your neural network and return the output.

[MrChadMWood]

So i have a few 100k+ datasets

Any chance you'd be willing to post these?

[Yaso2Go]

yes. write a logic adapter, in it, get the text, preprocess it, feed it to your neural network and return the output.

Can you please show us an example to do so