Keras model takes more epochs to learn when using custom data generator?

[davidvct]

I am training a model using custom data generator, which load batch data from disk on every step. However, it seems the model is taking more epochs to learn, when compared to loading whole datasets into the memory.

The training is for semantic segmentation and is in progress. But I have the prediction print out after every epoch. Now is like epoch 50+, but the prediction image is just blank (detected nothing at all). For comparison, it take less than 30 epochs to get OK prediction if training by loading whole datasets to memory. Total number of classes is 20.

My question, is this normal (taking more epochs to learn)?

I also verified the custom generator by using smaller set of datasets (around 300 images, with only 2 classes). It took 15 epochs to see the prediction.

I am using segmentation library, FPN model, backbone resnet34.

[sachinprasadhs]

Could you please provide the sample reproducible code for the tests you have performed to investigate further on the issue.

[davidvct]

Hi, I already fixed the issue. Root cause was the datasets was the images and masks did not paired (happened intermittently) and image scaling was done incorrect (I used Scikit tranform.fit on batch images instead of partial.fit). So it was not Keras issue.

[sachinprasadhs]

Thanks for confirming, could you please close the issue as well. Thanks!

[google-ml-butler]

Are you satisfied with the resolution of your issue?
Yes
No