【PaddlePaddle Hackathon 4】No.205 add a meter reader notebook

[Liyulingyue]

According to #998, I add a notebook about meter reader. The meter reader recognizes text based on the given layout information and outputs it in a structured format.

Summary

• Reuse some code in 405-paddle-ocr-webcam. Encapsulate the code of 405 and provide an API for OCR that can be called with one click.
• Established a complete structured output process.

[review-notebook-app]

Check out this pull request on 

See visual diffs & provide feedback on Jupyter Notebooks.

---

Powered by ReviewNB

[OpenVINO-dev-contest]

Hi @Liyulingyue thanks for your contribution, and I have put some rough comments on this PR. Could you take a look and fix the code format issue on CI ？

[Liyulingyue]

This comment is only used to upload two imgs

[Liyulingyue]

This comment is only used to upload example imgs

[Liyulingyue]

This comment is only used to upload framework imgs

[OpenVINO-dev-contest]

1. Please use "crop" instead of "cut" in this notebook.
2. Shall we use "Segment code LCD" or "segment code LCD"?

[OpenVINO-dev-contest]

Could you use same words in workflow diagram, or add these numbers in diagram which will make the pipeline more clear.

[OpenVINO-dev-contest]

Since most of codes of 2 main function can be reusable except some special condition, I will highly recommend you to combine them into one, and put it in section 2.

[Liyulingyue]

[OpenVINO-dev-contest]

hi @Liyulingyue Please also help to update the README file.

[OpenVINO-dev-contest]

hi @Liyulingyue Please also help to update the README file, and fix the code format issue on CI

[zhuo-yoyowz]

Suggested changes:

This notebook shows how to create a meter reader with OpenVINO Runtime for industrial digital meters. We use the pre-trained PP-OCR to recognize the required text and numbers on industrial digital meters, with the following inference task pipeline:

1. Configure the screen area of the digital meter reader.
2. Preprocess the input image (affine transformations)
3. Configure the layout information of the digital meter reader, and crop the input image into small pieces with areas required to be recognized.
4. (Optional) Preprocess the cropped images for meters with LCD screen.
5. Perform OCR recognition with OpenVINO.
6. Structure output information.
7. (Optional) Postprocess the output information when necessary.

This notebook will demonstrate how to read digital meters with different types of screens. In case 1 there is a field strength meter with general screen, which means that we can read the pure text directly from the screen with OCR. In case 2, there is a household electricity meter with Segment code LCD screen, which means we can not only see the required text, but also the outline of the redundant text in the background. The background outline could also be misread by the model as the content that needs to be recognized, so we need to add some preprocessing module to remove the redundant background information.

Most of the process for the two types of digital meters is similar, we only make some modifications in the preprocessing for the two different meters. The following figure shows the flowchart of two types of meter readers.

The tutorial consists of the following steps:

1. Prepare the PaddleOCR model.
2. Define configuration and helper functions

• Prepare configuration (take general screen meters as an example)
• Define a helper function to apply affine transformation
• Define helper functions for the preprocessing of text recognition
• Define helper functions for the postprocessing of text recognition

1. Meter reading for general screen digital meters

• Download Image
• Apply the affine transformation
• Define the areas to be recognized
• Recognize the text with OpenVINO on preprocessed images
• Postprocessing to fix the errors in recognition

1. Meter reading for Segment code LCD screen digital meters

• Prepare the Image, and apply configurations and special preprocessing for Segment code LCD screen
• Recognize the text with OpenVINO on preprocessed images (containing affine transformation)
• Postprocessing to fix the errors in recognition

Reply via ReviewNB

[zhuo-yoyowz]

Suggested changes:

note: The concepts of 'DESIGN_SHAPE' and 'DESIGN_LAYOUT' refer to the fact that the layout information of a display screen is determined during design, and these information is independent of the image. Therefore, you can obtain 'DESIGN_LAYOUT' information from your device manual. However, a simple way to run the program is to treat the coordinates of some points in figure as 'DESIGN_LAYOUT'. You can use Paint, a windows application, to obtain the coordinates. You only need to move the mouse to a position on the image, and the tool will display the corresponding coordinates.

Can you provide a link to the "Paint" here, so that other developers could directly link to this app and obtain the DESIGN_LAYOUT information？

For "DESIGN_LAYOUT", when you mention "the layout information of a display screen is determined during design, and these information is independent of the image", so it means the area of the LCD screen on the digital meter (the coordinates of the four vertices of the rectangle area), right?

Reply via ReviewNB

[Liyulingyue]

Sorry for replying late.

For "DESIGN_LAYOUT", when you mention "the layout information of a display screen is determined during design, and these information is independent of the image", so it means the area of the LCD screen on the digital meter (the coordinates of the four vertices of the rectangle area), right?

about the above question, the answer is right.

[Liyulingyue]

update two source img

[OpenVINO-dev-contest]

could you help to fix the layout of these pictures, it looks twisted.

[Liyulingyue]

update image

[HicariHuang]

Line #96.    text_decoder = RecLabelDecode(character_dict_path="../data/text/ppocr_keys_v1.txt",

Missing file: ppocr_keys_v1.txt

Reply via ReviewNB

[Liyulingyue]

The most thing about paddleocr will be very similar to https://github.com/openvinotoolkit/openvino_notebooks/blob/2023.2/notebooks/405-paddle-ocr-webcam/405-paddle-ocr-webcam.ipynb, and the ppocr_keys_v1.txt file can be found in the ipynb.

The most related codes are:

# Download font and a character dictionary for printing OCR results.
font_path = utils.download_file(
    url='https://raw.githubusercontent.com/Halfish/lstm-ctc-ocr/master/fonts/simfang.ttf',
    directory='fonts'
)
character_dictionary_path = utils.download_file(
    url='https://raw.githubusercontent.com/WenmuZhou/PytorchOCR/master/torchocr/datasets/alphabets/ppocr_keys_v1.txt',
    directory='fonts'
)

And you can download the file from that url :)