Specify Tajik VOCAB for training

[sultanovazamat]

Hi, and thanks for the fantastic job!

I am planning to add support for the Tajik language, which has 90% intersection with the Cyrillic alphabet. I have a couple of questions. Could you please answer them?

1. Do I have to update VOCABS for training, or is it used only on inference
2. Is it possible to use a pre-trained model that supports the Cyrillic alphabet and fine-tune it to the Tajik alphabet if the alphabets are almost identical (and how to choose a specific pre-trained model for FT during training)?
3. What dataset sizes would you recommend for training and fine-tuning for good results?

Thanks in advance!

[felixdittrich92]

Hi @sultanovazamat 👋,

1. Yes for both training and inference would be great if you could open a PR to add the Tajik vocab 👍 We have a ticket already to extend the current predefined vocabs [vocabs] Extend the list of predefined vocabularies #1883
2. Currently the --pretrained arg loads always our pretrained model. A way would be to use --resume instead and modify this step slightly:

doctr/references/recognition/train_pytorch.py

Line 255 in 3e21320

# Resume weights

# Resume weights
if isinstance(args.resume, str):
    pbar.write(f"Resuming {args.resume}")
    # This is model specific
    # crnn: ignore_keys=["linear.weight", "linear.bias"]
    # sar_resnet31: ignore_keys=["decoder.embed.weight", "decoder.embed_tgt.weight", "decoder.output_dense.weight", "decoder.output_dense.bias"]
    # master: ignore_keys=["decoder.embed.weight", "linear.weight", "linear.bias"]
    # vitstr: ignore_keys=["head.weight", "head.bias"]
    # parseq: ignore_keys=["embed.embedding.weight", "head.weight", "head.bias"]
    ignore_keys = []
    state_dict = torch.load(args.resume, map_location="cpu")
    for key in ignore_keys:
        state_dict.pop(key)
    missing_keys, unexpected_keys = model.load_state_dict(state_dict, strict=False)
    if set(missing_keys) != set(ignore_keys) or len(unexpected_keys) > 0:
        raise ValueError("unable to load state_dict, due to non-matching keys.")

This resets the layers for fine tuning which would otherwise fail because of a shape mismatch (embedding if exist and last linear (head))

3. Mh hard to answer I have used a 1M synth dataset to fine tune the current models for multilingual support (latin extended - german, spanish, italian, and so on) - In general: real dataset > synth train + real val > full synth

[sultanovazamat]

@felixdittrich92 Thanks for the reply!

1. I will share the Tajik language VOCAB for sure
2. I have tried to FT the pre-trained model on a custom vocab with the following params: --pretrained --freeze-backbone and both training and fine-tuning worked without a shape mismatch error, was it supposed to work or only your suggestion with --resume option is valid?
3. Currently, trying to use 1M synth DS for FT.

The initial version with 250K synth DS already gave 95% exact, but not generalizing well, now increased number of token and added some diversity in augmentation.

[felixdittrich92]

@sultanovazamat

2. No that's fine if you fine-tune on our pretrained checkpoint but it wouldn't work with a different fine tuned model as described in your point 2 an model which was already fine tuned on cyrillic for example in this case my provided snippet would work

Which tool have you used to create the synth dataset ?

[sultanovazamat]

@felixdittrich92 Oh, I got you! If the results from FT are not satisfying, I will try the resume from the Cyrillic model.

I am using https://github.com/Belval/TextRecognitionDataGenerator. Would appreciate if there are any better options that you use.

[felixdittrich92]

@sultanovazamat

I expected that .. and I had the same issue with this tool ..training was fine but models does not generalize on this data ..

What I did:

I modified SynthTiger to my own needs: https://github.com/felixdittrich92/synthtiger/tree/doctr-modified (branch: doctr-modified)

And created following dataset with it:
https://drive.google.com/file/d/1TNQN8uBMiGjzf2GM41BWDICefLubau5q/view?usp=drive_link

To train this model: https://huggingface.co/Felix92/doctr-torch-parseq-multilingual-v1

[sultanovazamat]

@felixdittrich92 This one looks superior to the https://github.com/Belval/TextRecognitionDataGenerator. I should try it for sure.

Btw, could you please suggest a strategy to choose fonts for the synthetic data generation to train/FT model to OCR on national IDs? Currently, I've just visually chosen some similarly looking fonts and ignored the background color of the ID, assuming that doctr will binarize data anyways.

Here are my current options for data augmentation using TextRecognitionDataGenerator:

blur=2,
random_blur=True,
random_skew=True,
skewing_angle=2,

Would you suggest the augmented data to be as diverse as possible or instead be close to the specifics of IDs (font, background, mid-character distance, etc.)?

Thanks in advance!

[felixdittrich92]

I would suggest to avoid any pre-augmentation while creating the dataset I limited this also to a minimum because we apply already augmentations:

doctr/references/recognition/train_pytorch.py

Line 301 in 3e21320

img_transforms=Compose([

Better to modify these to your needs because these are applied while training on the fly dynamic and not static

I would suggest to keep it as close as possible our augmentations will do the rest :)

[felixdittrich92]

Additional this makes it easier to value the evaluation results - if your data is close to original

[sultanovazamat]

@felixdittrich92 Thank you very much for your suggestions! I will get back to share the results soon.

[felixdittrich92]

@felixdittrich92 Thank you very much for your suggestions! I will get back to share the results soon.

And don't miss the PR to add Tajik vocab :D

[felixdittrich92]

Added to #1883