Training PET on data which is too large to fit in RAM

[ghost]

I am training a pet model on 500gb of text. I have properly processed the data, but I can't load all my data into a variable since I don't have nearly enough RAM to do that.

[chris-aeviator]

Random side note: I believe other projects have been solving this with the deepspeed/ deeperspeed libraries - might need loads of rework codewise before you can use it

[ghost]

Oh. That's sad, because I can't any code rework on my own :(

Random side note: I believe other projects have been solving this with the deepspeed/ deeperspeed libraries - might need loads of rework codewise before you can use it

So is there no simple way to do it? Could you help me?

[ghost]

In what ways did they use ms deepspeed for this?

[timoschick]

Hi @BleepLogger, the focus of PET is few-shot learning from 0-1000 examples. I'm not sure if this is really the right library for you if you've got 500GB of data to train on. We currently don't plan any modifications to PET that would support such large training datasets, so if you really want to use PET, you'll probably have to make them yourself. However, if you just want to use the 500GB of data for pretraining, a better approach would be to first use another library for pretraining and then use the resulting model with PET

[ghost]

Okay, I'm willing to make those modifications on my own. How do I make them?

[ghost]

Also, I have data in 80 parts.
What if instead of fine-tuning PET on all of my data at once, I first fine-tune it on part 1, then fine-tune part 1's resulting model on part 2, and so on. Could this work?

[ghost]

@timoschick

[timoschick]

Okay, I'm willing to make those modifications on my own. How do I make them?

Sorry if I haven't been clear. What I was trying to say is that I don't know what the best way to train on such large datasets would be, so you don't just have to implement any modifications yourself, you'll also have to figure out which modifications are required on your own.