Train Custom data set PDF

[diouck]

Hello, Super interesting. I would like to train the model on my own data in pdf format.
How to adapt the code. instead of using # The instruction dataset to use
dataset_name = "mlabonne/guanaco-llama2-1k" I would like to replace it with
dataset= doc.pdf but it sucks.
Do you have an idea? THANKS

[mlabonne]

Hi, in your case you want to create a dataset based on your PDFs. The code you're talking about is designed for instruction tuning, which means it requires pairs of instructions and answers. You could generate those using ChatGPT for example (see this: https://github.com/teknium1/GPTeacher).

Another option is to change the training objective from instruction tuning to next-token prediction (the pre-training task), but you can't use this code. I'd recommend checking the Axolotl library in this case: https://github.com/OpenAccess-AI-Collective/axolotl