How to load data every epoch

[sadbb]

hi,
because of my task, i must load new train_data every epoch. But in this package, data can only be loaded once at the beginning of training. How can i load data every epoch?
thanks.

[williamFalcon]

could you explain more? do you have pseudocode?

[sadbb]

Thanks for help,

As you can see in your picture, data setup called before training loop. But in my task , like few-shot learning , data must be called between "for epoch" and "for batch" in this picture every new epoch. What can i do for this?
Apologize for my poor English.

[sadbb]

I made some adjustments. Can you help me check it?
In decorators.py, i delete 'setattr(self, attr_name, value)' in line 25 like this:

And in trainer.py, i call data again in function '_train' in line 803 like this:

Am I doing this right? Or he has a bad influence elsewhere？
Thank you for help.

[neggert]

What happens if you just don't use the @dataloader decorator? That should prevent the DataLoader from being cached, and you can just recompute it every time the class method is called (which should be once per epoch). Not sure what other effects that would have, though.

[williamFalcon]

@neggert that's the way to do it. I'll add this to the docs.

[williamFalcon]

@sadbb actually just submitted a PR to enable this.
Once we merge into master, just remove the decorators in the dataloaders. Warning though, your loader will be called every epoch. if any of the loaders are slow to init your script will be very slow.

[yassersouri]

I think it is pretty standard to create a dataloader at the beginning of each epoch. I think it should be the default.