Dynamic Batch Sizes

[twerkmeister]

During some of my trainings I noticed, that the memory consumption differs greatly as the sentence size increases within an epoch. For short sentences in the beginning, TTS only consumed about 3GB, but then later for the long example sentences over 7,4GB. I guess a fixed batch size comes from tasks that deal with fixed size tensors, like in image classification. Given that batch size seems important for learning attention, it might be worth experimenting with dynamic batch sizes. That can probably double the batch size for medium sized sentences.

Any thoughts?

[erogol]

@twerkmeister makes sense. My only concern is the learning rate. If the initial batch size vs the last batch size in an epoch is too different, it might turbulence the training. Anyhow, it is worth to try.

[twerkmeister]

Yeah let's keep it in mind as an interesting experiment

[erogol]

It is partially implemented. Not considering the sequence length and active memory usage but I don't see any use for now.