Hi @usimarit , we probably need to increase the shuffle buffer size to 1000 for example, in ASRDatasets here: https://github.com/TensorSpeech/TensorFlowASR/blob/main/tensorflow_asr/datasets/asr_dataset.py#L108 https://github.com/TensorSpeech/TensorFlowASR/blob/main/tensorflow_asr/datasets/asr_dataset.py#L340 because `TFRECORD_SHARDS = 16` is too small. This would help to feed better random data to the model (Reference: https://www.tensorflow.org/api_docs/python/tf/data/Dataset#shuffle)