HWT Table OCR

Description

This repository contains the code and model for a mobile app that scans handwritten tables and converts them into a CSV file. The app utilizes deep learning techniques to recognize and extract table data from handwritten documents, providing a convenient and efficient way to digitize handwritten tables.

Features

• Scan handwritten tables: The app allows users to capture images of handwritten tables using their mobile devices.
• Table recognition: Utilizing a pre-trained machine learning model, the app recognizes and extracts table data from the captured images.
• Data extraction: The app converts the recognized table data into a structured CSV format.
• User-friendly interface: The app provides a user-friendly interface that guides users through the scanning and conversion process.

Installation

Clone the repository to your local machine. Ensure that you have the required dependencies installed. (Please refer to the requirments.txt in the HTR Recognition folder)

Mobile App

• The app was created using Flutter.
• To avoid complexities I uploaded only the important files to use and run the mobile app.

• To use the mobile app, download the main.dart and the workder.dart from the Mobile_app folder and place them in the lib folder of your flutter app
• To look for dependencies, please refer to the pubsec.yaml file in the Mobile_app folder.

Working (APP)

The front end was developed using Flutter and Flask was used for the back end support. During the debugging phase, the flutter app was used to upload the image of the server. Then on the Server end, I had Flask script running, waiting for the image to be received. Once the Flask server receives the Image of the HandWritten Table, it performs further processing and generates a CSV file.

Working (Backend)

Once the Flask script recieves the image, it breaks down the image into little images, each of these images representing a cell of the table. This process was done using the Preprocessing script. Then each of these images is processed using the Handwritten Text recognition model and a resultant word is generated for each cells. At the end, each of these words are combined in a Pandas DataFrame and a CSV file is generated.

Working (Model)

The model training files were taken from the SimpleHTR repository. This Repository allows you to train files using 3 different algorithms. I trained my models on each one of them and have uploaded the model files generated through all 3 algorithm. (you can find them in the HRT Recognition folder). I did a detailed compariso. Please find the comparison graphs at the end of the README.md file.

Research Report

Please find the Research Report at the following link: https://www.overleaf.com/read/qvxgnbqccdsj

Comparison

Please find the comparison below:

Word Accuracy Comparison

In this case, Word Beam Search decoder works the best , you can find this here

Average Train Loss Comparison

Here, Word Beam Search decoder works the best and the rest are almost the same.

Character Error Rate Comparison

Word Beam Search decoder is proven to be the best in this case too and the rest are almost the same.