Handwriting Recognition Training

[ArsalanYounus007]

Hi,

I am trying to train a Handwriting model using doctr - master OR visttr (base).
I created a dataset from IAM, NIST and some custom in-house dataset. Totalling over 300,000 training samples and 100,000 validation samples.
When I started training Master, I got good partial and exact match score. But each epoch is going down in terms of score and going up in terms of Training loss.

Now I will be trying weight decay and lower learning rates.
But I was wondering if there is a better way to handle this.

[felixdittrich92]

Hi @ArsalanYounus007 👋,
For handwritten i think you should adjust the augmentations a bit.
Do you start from scratch or finetuning the pretrained one ?

[ArsalanYounus007]

Hi,
I am using master pretrained model from you guys.
By adjusting the learning rate I am getting 90 exact and 92 partial match on validation set.
However the model is overfitting at this point (0.002 training loss, 0.15 val loss). I am trying weight decay 0.001 now.
What augmentations would you suggest?
I think there is some augmentation in the script already for the training dataset loader.

[felixdittrich92]

Hey :)

I would try the following:

1. Train without the pretrained model and exclude all augmentations keep only the required Resize to see how it performs on "default"
2. Compare the same run by fine tuning on the pretrained run (don't miss to seed everything for the experiments before)
3. Decide and do a full run with augmentations -> compare again

About the augmentations i think i would exclude:

doctr/references/recognition/train_pytorch.py

Line 289 in 1cea7d8

T.RandomApply(GaussianBlur(3), 0.3),

for handwritten because these will make it even harder for the model or reduce the probability a bit to 0.05 for example

Have you also tried parseq already ?

[ArsalanYounus007]

I started with Vistr_base . Got 84% exact and 87% partial match
Then I tried master. Got much better 90% exact and 92% partial match
After I am done with this, I will definitely try parseq again (previous try did not work)

[ArsalanYounus007]

Even with the new model, I am getting a lot of mistakes in unseen data

'Footh Poun, Submandibnlors swellip COMLEIN Aerstartof INf.W. dentallvisst ROVIRE this WK WAL ILX Cindamucin 300mp TIDX , 10cltlan 800. mc Can cont , Norto 22W momiton for immovement '