How to do Shuffling in HDF5 input data to prevent training failure

[bearpaw]

I am currently training my multi-label regression model with HDF5 data layer. However, I notice that my training loss going up and down periodically (green curve). Since I merge two dataset together to train the model, and my train_list.txt are something like following,

datasetA_1.h5
datasetA_2.h5
datasetA_3.h5
...
datasetB_1.h5
datasetB_2.h5
datasetB_3.h5

I find that the train loss raises at about 0.5 when training on dataset A h5 files, and jumps to about 0.2 on dataset B. After all h5 files have been trained, it goes back to the first h5. This is the reason why the loss acts like a "square wave", although I shuffled the data in each h5 file.

I think maybe shuffle would solve this problem. Unfortunately, there's no SHUFFLE support in HDF5 data layer (unlike leveldb layer).

Can we solve this problem in an alternative way?

[PatWie]

Maybe:
#1205
helps (Works here). You can use the same idea to shuffle multiple hdf5, too. But training will be slow of you shuffle each sample that will feed through the network.

[bearpaw]

Thanks @PatWie . I've tried to use your code, and it seems correct in practice.

I notice that you shuffle the data in each h5 file, and the h5 file is still read one by one by the traininglist file. Do you think its meaningful to random the order of h5 file either?

[PatWie]

I depends. At least I have to look at code again to figure out why travis-ci shows errors.
Shuffling the order of reading h5 files should be easy to implement, too. And maybe helps. Using a CNN is more art than science.

But IF each of your h5 files only contains samples for one class then nothing helps. I prefer to shuffle the data before creating h5 files.