Add CRAFT character detection node and OCR node. #2650

iory · 2021-12-23T17:16:52Z

What is this?

Add CRAFT: Character-Region Awareness For Text detection. node which detects text region from image and
add Optical Character Reader (OCR) node from image and text area.

This PR is developed on top of this PR.
#2648

Quick Start

Please do rosdep install and catkin build jsk_perception at first.
In order to use this feature, you need to install pytorch <https://pytorch.org/get-started/locally/> (pytorch >= 1.9.0 is recommended).
However, python2 user is not supposed to be able to install that version of torch.
For python2 users, download the appropriate wheel file in your environment from https://download.pytorch.org/whl/cu90/torch_stable.html (melodic) and install with pip as follow:

pip install --user torch-1.1.0-cp27-cp27mu-linux_x86_64.whl
pip install --user torchvision-0.3.0-cp27-cp27mu-manylinux1_x86_64.whl

For python2 users, please do pip install pytesseract==0.3.1 manually because the latest version of pytesseract is not installed automatically.

After that, you can run the following command.

roslaunch jsk_perception sample_craft_node.launch

Japanese OCR recognition

If you want to use a language other than English,
please install the appropriate language data and change the language argument.

For example, if you want to use Japanese,
please install tesseract-ocr-jpn (apt install tesseract-ocr-jpn in Ubuntu) and pass jpn as the language argument.

708yamaguchi

Sorry, my reviews are WIP.
I will add more reviews later.

708yamaguchi · 2021-12-24T07:16:49Z

doc/jsk_perception/nodes/craft_node.rst

+craft_node.py
+=============
+
+What is this?


How about writing the Quick Start in the comments of this PR in the README as well?

For example, the following information seems very important to me.

For python2 users, please do pip install pytesseract==0.3.1 manually because the latest version of pytesseract is not installed automatically.

In addition, I was confused when installing pytorch with python2.
The followint web page show only for python3 user.
https://pytorch.org/get-started/locally/
Could you please add pytorch installation process for python 2 users?

Related to the these python version problem, how about using catkin_virtualenv to use the latest pip programs with Python3.x.
I think this makes the installation process easier.

Related to the these python version problem, how about using catkin_virtualenv to use the latest pip programs with Python3.x. I think this makes the installation process easier.

I'm not familiar with catkin_virtualenv. I have a question.
catkin_install_python needs to call catkin_install_python, but the script files are already written as a target in CMakelists.txt in jsk_perception. https://github.com/jsk-ros-pkg/jsk_recognition/blob/master/jsk_perception/CMakeLists.txt#L430-L435
Does this mean that the script files also need to be changed to work with python3?
If that is true, I think we need to address it with another PR.

How about writing the Quick Start in the comments of this PR in the README as well?

Could you please add pytorch installation process for python 2 users?

Thanks. I added them.
4a49d1d
a51b8ca

I'm not familiar with catkin_virtualenv. I have a question.
catkin_install_python needs to call catkin_install_python, but the script files are already written as a target in CMakelists.txt in jsk_perception. master/jsk_perception/CMakeLists.txt#L430-L435
Does this mean that the script files also need to be changed to work with python3?

I am very sorry, but you are right. We should not use catkin_virtualenv.

This process will also override the standard catkin_install_python macro to wrap a virtualenv loader around the specified python scripts.
https://github.com/locusrobotics/catkin_virtualenv

In addition to that, using catkin_virtualenv adds new dependency to jsk_perception. So we should not use catkin_virtualenv

OK. I understand it.

708yamaguchi · 2021-12-24T12:41:15Z

Thank you very much for your help.
I was able to successfully run the sample!
This PR seems very useful.

Is it difficult to add a test to this?
In my understanding, what we need to do is

Add .test file using sample_craft_node.launch (gpu:=-1 is needed?)
Update CMakeLists.txt
Update BEFORE_SCRIPT in .github/workflow

708yamaguchi · 2021-12-24T14:08:51Z

doc/jsk_perception/nodes/craft_node.rst

+
+.. code-block:: bash
+
+   pip install --user torch-1.1.0-cp27-cp27mu-linux_x86_64.whl


I have a question.

You say (pytorch >= 1.4.0 is recommended), but the installed torch version seems 1.1.0

Does this mean "Python3 users should use torch>=1.4.0, but Python2 users should use torch=1.1.0" ?

OK. I modified the description.
a4beeae
Please check it

Thank you!
Very easy to understand.

708yamaguchi · 2021-12-24T14:40:11Z

When no character is found, I got the error.

Could you check this PR?
iory#14

iory · 2021-12-24T14:42:25Z

When no character is found, I got the error. Could you check this PR?

Merged. Thanks!

708yamaguchi · 2021-12-24T15:11:54Z

I think this PR works well. After tests pass, it will be OK to merge.
Great work!

I have a simple question. (Maybe future work)

When the characters are parallel to the image, the recognition accuracy is high.

However, when it is not, the recognition accuracy seems to be low.

How can we avoid this?
Parameter tunings?
Or if we can know the rotation angle of the strings, we can rotate the image?

tkmtnt7000 · 2021-12-24T16:56:06Z

Sorry to interrupt.
I think it is a nice and useful PR. Overall, the ability to recognize Japanese words is also reasonably high!

OCR	Rect


	(Image source: Amazon.com)

tkmtnt7000 · 2021-12-24T17:08:19Z

doc/jsk_perception/nodes/craft_node.rst

+* ``~gpu`` (Int, default: ``0``)
+
+  GPU id.


If a version that runs on the CPU is supported, it might be easier for users to understand when the value to set in the case of the CPU is written like https://github.com/jsk-ros-pkg/jsk_recognition/blame/master/doc/jsk_perception/nodes/hand_pose_estimation_2d.rst#L39-L41

85a4d68
Thanks. I added the information.

708yamaguchi · 2021-12-24T18:07:42Z

kinetic, melodic, noetic and noetic-full tests do not start.

For example,

Invalid workflow file : .github/workflows/noetic.yml#L35
The workflow is not valid. .github/workflows/noetic.yml (Line: 35, Col: 13): A sequence was not expected

https://github.com/jsk-ros-pkg/jsk_recognition/actions/runs/1619482923

This may be solved with:

SamKirkland/FTP-Deploy-Action#202 (comment)
SamKirkland/FTP-Deploy-Action#202 (comment)
SamKirkland/FTP-Deploy-Action#202 (comment)

iory · 2021-12-24T18:40:31Z

How can we avoid this?
Parameter tunings?
Or if we can know the rotation angle of the strings, we can rotate the image?

I modified the crop code.
2f70e56
However, this is not perfect.
Depending on the position of the four points of the input polygon, the image to be cropped may be distorted.
The solution to this is to crop the text so that it is no longer distorted, or to improve the performance of OCR.
These solutions are future direction.

708yamaguchi · 2021-12-25T01:40:06Z

The recognition accuracy seems to have improved when the characters are rotated.
Thank you!

708yamaguchi · 2021-12-25T04:52:16Z

melodic test fails because pytesseract installation fails

2021-12-24T20:41:06.0722380Z   pip: command [sudo -H pip2 install -U -q pytesseract] failed
2021-12-24T20:41:06.0723230Z   pip: Failed to detect successful installation of [pytesseract]

2021-12-24T20:41:06.9390096Z Traceback (most recent call last):
2021-12-24T20:41:06.9390434Z 
2021-12-24T20:41:06.9390786Z                                                                                 
2021-12-24T20:41:06.9391609Z   File "/workspace/ros/ws_jsk_recognition/src/jsk_recognition/jsk_perception/node_scripts/ocr_node.py", line 25, in <module>
2021-12-24T20:41:06.9392271Z 
2021-12-24T20:41:06.9392625Z                                                                                 
2021-12-24T20:41:06.9393080Z     import pytesseract
2021-12-24T20:41:06.9393385Z 
2021-12-24T20:41:06.9393732Z                                                                                 
2021-12-24T20:41:06.9394356Z ImportError: No module named pytesseract

https://github.com/jsk-ros-pkg/jsk_recognition/runs/4628721970?check_suite_focus=true

We may need to add

pip install pytesseract==0.3.1

to BEFORE_SCRIPT

708yamaguchi · 2021-12-25T04:57:10Z

kinetic, noetic and noetic-full tests fail maybe because format of BEFORE_SCRIPT is not correct (as iory checked with ce1bf09)

mqcmd196 · 2021-12-27T04:33:56Z

LGTM, I could execute the node successfully.

k-okada · 2021-12-30T01:38:49Z

wait for catkin_virutalenv
please add information on how to safely install pytorch like https://github.com/jsk-ros-pkg/jsk_recognition/blob/master/doc/install_chainer_gpu.rst

iory · 2022-06-22T05:36:55Z

I resolved conflicts. I'm waiting for test results.

iory · 2022-06-26T06:02:55Z

@k-okada
I modified the GA config to install pytorch. 0224bd0
Test passed.

k-okada · 2022-06-27T04:51:58Z

@iory please update jsk_travis too, https://github.com/jsk-ros-pkg/jsk_travis/blob/master/docker/Dockerfile.ros-ubuntu:14.04-pcl#L57

pazeshun · 2022-06-27T04:55:54Z

jsk_perception/package.xml

@@ -92,6 +92,7 @@
  <exec_depend>leveldb</exec_depend>
  <exec_depend>python-fcn-pip</exec_depend>  <!-- pip -->
  <!-- }} install fcn -->
+  <exec_depend version_lt="0.3.7">python-pytesseract-pip</exec_depend>


~~@iory Isn't this 0.3.6? 0.3.7 causes syntax error.~~
Sorry, my mistake.
I overlooked version_lt, means "less than"

OK. Thanks.

iory · 2022-06-27T05:21:06Z

please update jsk_travis too, https://github.com/jsk-ros-pkg/jsk_travis/blob/master/docker/Dockerfile.ros-ubuntu:14.04-pcl#L57

OK. I created a PR. jsk-ros-pkg/jsk_travis#451

iory added enhancement document pkg/jsk_perception pkg/jsk_recognition_utils labels Dec 23, 2021

iory requested review from 708yamaguchi, mqcmd196, Kanazawanaoaki and tkmtnt7000 December 23, 2021 17:16

708yamaguchi reviewed Dec 24, 2021

View reviewed changes

iory force-pushed the craft-ocr branch from 57d234b to a4beeae Compare December 24, 2021 14:16

tkmtnt7000 reviewed Dec 24, 2021

View reviewed changes

iory force-pushed the craft-ocr branch from 31972b1 to ce1bf09 Compare December 24, 2021 19:51

iory force-pushed the craft-ocr branch 3 times, most recently from 5f1d22e to 13c8ba2 Compare December 26, 2021 14:44

mqcmd196 approved these changes Dec 27, 2021

View reviewed changes

k-okada added the PR/PleaseFix label Dec 30, 2021

iory added 3 commits June 22, 2022 14:35

Enable cpu inference

70b484c

Add craft node test

6dc0827

Add ocr node test

f7416c1

iory force-pushed the craft-ocr branch from f66015f to bfdef11 Compare June 22, 2022 05:35

iory force-pushed the craft-ocr branch 7 times, most recently from 185e519 to 11cbc31 Compare June 23, 2022 19:54

iory and others added 7 commits June 26, 2022 05:35

Fixed pytorch version and documents for python2 users

591d4ea

Avoid error when length of poly is 0

7927443

Rotate croppped image

b369c1c

Add cpu support information for ocr node documents

909c608

[jsk_perception/craft] Modified permission to 775

a4fbe45

Add torch install for github actions

0224bd0

[jsk_perception/craft] Add version info for pytesseract

5a93d8a

iory force-pushed the craft-ocr branch from 11cbc31 to 93e17d3 Compare June 25, 2022 20:35

[jsk_perception/craft_ocr] Test on greater than indigo

7087b3e

iory force-pushed the craft-ocr branch from 93e17d3 to 7087b3e Compare June 26, 2022 04:08

k-okada merged commit 39e37cb into jsk-ros-pkg:master Jun 27, 2022

iory deleted the craft-ocr branch June 27, 2022 04:54

pazeshun reviewed Jun 27, 2022

View reviewed changes

iory mentioned this pull request Jun 27, 2022

[docker] Install pytesseract and torch jsk-ros-pkg/jsk_travis#451

Open

sawada10 mentioned this pull request May 12, 2023

sample_craft_node.launchが立ち上がらない #2776

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add CRAFT character detection node and OCR node. #2650

Add CRAFT character detection node and OCR node. #2650

iory commented Dec 23, 2021 •

edited

708yamaguchi left a comment

708yamaguchi Dec 24, 2021

708yamaguchi Dec 24, 2021

iory Dec 24, 2021 •

edited

iory Dec 24, 2021

708yamaguchi Dec 24, 2021

iory Dec 24, 2021

708yamaguchi commented Dec 24, 2021

708yamaguchi Dec 24, 2021

iory Dec 24, 2021

708yamaguchi Dec 24, 2021

708yamaguchi commented Dec 24, 2021

iory commented Dec 24, 2021

708yamaguchi commented Dec 24, 2021 •

edited

tkmtnt7000 commented Dec 24, 2021

tkmtnt7000 Dec 24, 2021

iory Dec 24, 2021

tkmtnt7000 Dec 25, 2021

708yamaguchi commented Dec 24, 2021 •

edited

iory commented Dec 24, 2021

708yamaguchi commented Dec 25, 2021

708yamaguchi commented Dec 25, 2021

708yamaguchi commented Dec 25, 2021

mqcmd196 commented Dec 27, 2021 •

edited

k-okada commented Dec 30, 2021

iory commented Jun 22, 2022

iory commented Jun 26, 2022

k-okada commented Jun 27, 2022

pazeshun Jun 27, 2022 •

edited

iory Jun 27, 2022

iory commented Jun 27, 2022


		.. code-block:: bash

		pip install --user torch-1.1.0-cp27-cp27mu-linux_x86_64.whl

Add CRAFT character detection node and OCR node. #2650

Add CRAFT character detection node and OCR node. #2650

Conversation

iory commented Dec 23, 2021 • edited

What is this?

Quick Start

Japanese OCR recognition

708yamaguchi left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

iory Dec 24, 2021 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

708yamaguchi commented Dec 24, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

708yamaguchi commented Dec 24, 2021

iory commented Dec 24, 2021

708yamaguchi commented Dec 24, 2021 • edited

tkmtnt7000 commented Dec 24, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

708yamaguchi commented Dec 24, 2021 • edited

iory commented Dec 24, 2021

708yamaguchi commented Dec 25, 2021

708yamaguchi commented Dec 25, 2021

708yamaguchi commented Dec 25, 2021

mqcmd196 commented Dec 27, 2021 • edited

k-okada commented Dec 30, 2021

iory commented Jun 22, 2022

iory commented Jun 26, 2022

k-okada commented Jun 27, 2022

pazeshun Jun 27, 2022 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

iory commented Jun 27, 2022

iory commented Dec 23, 2021 •

edited

iory Dec 24, 2021 •

edited

708yamaguchi commented Dec 24, 2021 •

edited

708yamaguchi commented Dec 24, 2021 •

edited

mqcmd196 commented Dec 27, 2021 •

edited

pazeshun Jun 27, 2022 •

edited