trOCR run example

[vozdemir]

Many thanks for trOCR!! I couldn't run trOCR on text. How to run it, can you give an exaample pls.

Model I am using trOCR.

[wolfshow]

@vozdemir Can you provide more details about your question? You may just download the dataset provided and do the inference with the base/large models. Meanwhile, please refer to #448 for more details.

[NielsRogge]

@wolfshow would be great if you can fix my notebook to run inference with TrOCR on one particular image: https://colab.research.google.com/drive/1BHkOBUGHr1xlVQ9pLVLZPA0VvAxF4Zo_?usp=sharing

[wolfshow]

@wolfshow would be great if you can fix my notebook to run inference with TrOCR on one particular image: https://colab.research.google.com/drive/1BHkOBUGHr1xlVQ9pLVLZPA0VvAxF4Zo_?usp=sharing

@Dod-o will help providing an inference example on that.

[Dod-o]

@NielsRogge The inference example has been uploaded, please see details in pic_inference.py

[SeifeddineGharbi]

@Dod-o when running pic_inference.py, I get the following error: urllib.error.HTTPError: HTTP Error 404: Not Found
it occurs when the script tries to download: Downloading: "https://github.com/pytorch/fairseq/archive/master.zip"

Any help please, thank you!!

[NielsRogge]

@SeifeddineGharbi I figured it out (I had to create a custom fork of Fairseq in order to make it work).

Expect TrOCR to be added to Transformers soon ;)

[SeifeddineGharbi]

@NielsRogge Thank you so much for replying but can you explain a bit further, please?

[wolfshow]

@SeifeddineGharbi I figured it out (I had to create a custom fork of Fairseq in order to make it work).

Expect TrOCR to be added to Transformers soon ;)

We found the fairseq model cannot be easily converted into the hf format. So we need to take more time to re-train the models with the hf library.

[nithinreddyy]

@wolfshow would be great if you can fix my notebook to run inference with TrOCR on one particular image: https://colab.research.google.com/drive/1BHkOBUGHr1xlVQ9pLVLZPA0VvAxF4Zo_?usp=sharing

Hugging face has uploaded the trocr model in their models. You can look into it.

https://huggingface.co/transformers/model_doc/trocr.html

[NielsRogge]

@nithinreddyy haha I wrote that page 😅

[nithinreddyy]

@nithinreddyy haha I wrote that page 😅

Yaaa 😬😬. I gone through the 3 notebooks, but you haven't written code for testing the image and extracting the text (For fine tuning model with IAM dataset). In the 2nd notebook you trained the model with IAM dataset, in 3rd notebook you just checked the test evaluation scores. But how to take one image from test data and extract the text?

[NielsRogge]

My inference notebook does exactly what you want.

[nithinreddyy]

My inference notebook does exactly what you want.

But you are directly loading the model from hugging face. What if we have our own dataset and want to train the model with data? In 2nd notebook You've trained the custom model with IAM dataset and you haven't written code how to extract text from one of the test images. I'm looking for that.

[NielsRogge]

Sorry, I thought you were talking about recognizing text, but you mean extracting text.

The IAM dataset only contains single-line text images, hence one doesn't need to perform any text extraction anymore. However, if you want to apply TrOCR on an entire PDF document, then you first need a text extraction algorithm.

You can for example take a look at this one: https://github.com/qurator-spk/eynollah