Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Create a low cost custom scan box #73

Open
tshrinivasan opened this issue Aug 30, 2019 · 19 comments

Comments

@tshrinivasan
Copy link
Contributor

commented Aug 30, 2019

We need to create a low cost custom scan box, so that we can scan books easily.

the existing scanners are costly.

SV600 - 47,000 INR
https://www.amazon.in/Fujitsu-PA03641-B301-ScanSnap-SV600-Scanner/dp/B01AJI0426

CZUR ET 16 Plus Smart Book Scanner - 56,000 INR
https://www.amazon.in/CZUR-Plus-Smart-Scanner-Black/dp/B0758VY4G7

Instead of this, make a scanbox like
https://www.kickstarter.com/projects/limemouse/scanbox-turn-your-smartphone-into-a-portable-scann

https://www.amazon.com/Scanner-Bin-Document-Scanning-Solution/dp/B00XM7LKZM

@tshrinivasan

This comment has been minimized.

Copy link
Contributor Author

commented Aug 30, 2019

photo6285311525753301252

Created the above box.

Size = 1.5 x 1.5 x 1.5 feet

Expenses so far - plywood, carpenter charge - 2100

Pending items

  1. Making a 2.5 inch hole on the top
  2. Painting inside and outside
  3. Adding 4 LED lamps on inner side of TOP
  4. Getting a thick non reflective glass to press the big books

Expected cost - 3000 Rs

With this, anyone can scan a book with a normal digital camera or mobile phone.
This is portable.

Once we did this, we can add more improvements for the next scan boxes.

@gnuanwar

This comment has been minimized.

Copy link

commented Aug 30, 2019

Srini check Kalyan recommendations
[30/08, 7:42 AM] kalyan greenbooks: Design to be improved
[30/08, 7:45 AM] kalyan greenbooks: Lots of glare will be there in this design. Lights to be away from camera area. If you want I give you physical drawing. I think I have it. You may try with that.
[30/08, 9:08 AM] Gnuanwar: Ok give me the drawing we try tell me when I need to collect

@tshrinivasan

This comment has been minimized.

Copy link
Contributor Author

commented Aug 30, 2019

http://techforelders.blogspot.com/2012/12/blog-post.html
Found here a good alternate.

Have to fit the lights on the side walls and fit white sheets in between.

@tshrinivasan

This comment has been minimized.

Copy link
Contributor Author

commented Sep 4, 2019

Added a hole on the top for 2.5 inches diameter.

photo6294053888734636331

photo6294053888734636332

Added two lights on both inner sides.

photo6294053888734636333

Added a 8mm thickness glass of 15 x 17 inch size, to press the big books.

photo6291738312951572656

photo6292026384998050130

Adding a glass, adds more reflections. Exploring on how to avoid this.

photo6291738312951572658

Keep the box away of kids. This seems a great fun box for them :-)

photo6294043739726915845

@tshrinivasan

This comment has been minimized.

Copy link
Contributor Author

commented Sep 4, 2019

Exploring on adding a semi transparent sheets between the lights and the glass to emit limited light.

Trying with sheets used in Drums. This seems thick, easily fixable with screws. emits lights lmitedly. No reflection over glass.

drums

@tshrinivasan

This comment has been minimized.

Copy link
Contributor Author

commented Sep 6, 2019

photo6307632247452313978
photo6309903662381639852
photo6307632247452313979

@tshrinivasan

This comment has been minimized.

Copy link
Contributor Author

commented Sep 6, 2019

Finally, added the thick mica sheet used on the drums to minimize the light reflection.

The box is ready to use now.

@tshrinivasan

This comment has been minimized.

Copy link
Contributor Author

commented Sep 6, 2019

தாமோதரம்.pdf

Here is sample scan from android phone with cam scanner.

Did post processing the scanned images using scantailor.
here is the result from scantailor
thamodaram.pdf

Found that tesseract 4 is good for Tamil OCR.
Here is the result.
thamodaram.txt

@tshrinivasan

This comment has been minimized.

Copy link
Contributor Author

commented Sep 6, 2019

Install Scantailor and tesseract

sudo apt-get install tesseract-ocr yagf tesseract-ocr-tam tesseract-ocr-script-taml tesseract-ocr python3-pyocr ocrodjvu ocrmypdf lios gimagereader

sudo apt-get install scantailor

  1. scan a book
  2. save as images
  3. split, improve using scantailor

split a pdf to multiple images using ghostscript

gs -dNOPAUSE -dBATCH -sDEVICE=png16m -sOutputFile="Pic-%02d.png" output.pdf

Do ocr using tesseract

ls -1Nv *.png > filelist.txt

tesseract -l eng+tam filelist.txt article txt

@tshrinivasan

This comment has been minimized.

Copy link
Contributor Author

commented Sep 6, 2019

if the output from scanning is only double sided PDF, we can split it using mutool.

mutool poster -x 2 input.pdf output.pdf

To crop the PDF, we can use pdfshuffler

image

@tshrinivasan

This comment has been minimized.

Copy link
Contributor Author

commented Sep 6, 2019

Here is a video on how to use scantailor

https://vimeo.com/12524529

@venkatarangan

This comment has been minimized.

Copy link

commented Sep 7, 2019

@tshrinivasan Thanks for the post, the scanner box is super impressive.

I am interested in the software process. One problem which has been plaguing scanning of old Tamil books to PDF has been, the inability to select, copy n paste intelligible Tamil text (Unicode) from the PDF pages to say MS Word or Notepad. For this to work, I guessed we need to have an OCR to generate Tamil text somewhere in the pipeline and then embed the text back to the PDF.

In this context, recently I found all the books in Singapore's NLB Tamil collection, supporting seamless (extremely low error) copy n paste of Tamil text out from PDF. I have been researching how they are doing - I experimented with a python script to use Google Vision OCR to get something working after I saved all pages in a PDF as images. That's where I got stuck, unable to proceed further in the workflow.

Now, seeing the tweet today, I was pleasantly surprised, you have solved the problem. I have two queries:

  1. In the thamodharan.pdf you generated from CamScanner->ScanTailor, the copy n paste of Tamil works. But you are mentioning you doing OCR with tesseract later. Am I missing something? If not, how is copy n paste working before OCR?

  2. Once OCR is done, how to get the Tamil Unicode text back to the PDF.

Lastly, can you write a detailed blog post on the steps outside this GitHub thread, as it will benefit a wider set of people (especially non-programmers) who wish to scan and support Copy N Paste for old Tamil books.

Once again, many thanks for your effort and sharing them.

@tshrinivasan

This comment has been minimized.

Copy link
Contributor Author

commented Sep 8, 2019

@venkatarangan The Tesseract OCR can give output as text file and PDF file.

PDF file is searchable. It adds a text layer ontop of the original image when producing the PDF output.

Will write detailed blog soon.

@tshrinivasan

This comment has been minimized.

Copy link
Contributor Author

commented Sep 8, 2019

Initially, I thought that we do all the scan, fix images, improve images, convert to PDF using mobile itself.

Adobe scan, cam scanner kind of apps do this.

But, the results are not impressive.

Did a test with ScanTailor. It is the magician and King in this field.

Raw scan output

Screenshot from 2019-09-08 20-31-07

Result from Adobe scan mobile app

Screenshot from 2019-09-08 20-39-32

See the difference of contrast across the page due to light reflection over the glass. They are not good for printing and OCRing.

But Found scantailor improves the image quality to a very high level.
It does all the magics to bring the super resulted images.

Screenshot from 2019-09-08 20-34-06

This is perfect for print or OCR.

@tshrinivasan

This comment has been minimized.

Copy link
Contributor Author

commented Sep 8, 2019

Few inputs from bharat varma in twitter
https://twitter.com/BharatVarma3/status/1170241584031391744

Hinge glass at the far edge to ease book placement.

Cut panels on the side & put white acrylic sheets for light diffusion. Keep lights outside.

If using a camera, use a CPL filter to cut glare.

Use a long lens and an inclined base, and you can probably shoot two books at once.


you can add a CPL filter to a Point and shoot camera..
Point and shoots have three possibilities for attaching a CPL filter -

A filter thread on the lens itself, an add on accessory tube that attaches to the body with a threaded receptacle at the end & a similar small tube that can be stuck to the front of the tube housing the lens.


image

Does this section work? :)

The inclined lines are the base & the book.

"O" is camera with a long focal length (less distortion, edge to edge sharpness).

o o is direction.

The camera needs to be parallel to the book. Keeping it at a distance with a longer focal length makes it more forgiving of alignment issues and the minimal distortion ensures better quality images.

@nithyadurai87

This comment has been minimized.

Copy link

commented Sep 17, 2019

To convert all the tif output of scantailor to PDF

ls *.tif | parallel convert {} {.}.pdf

pdfunite *.pdf bookname.pdf

@tshrinivasan

This comment has been minimized.

Copy link
Contributor Author

commented Sep 17, 2019

Here is the first books scanned with this scanbox

https://archive.org/details/thamodharam

https://ia601401.us.archive.org/23/items/thamodharam/thamodharam.pdf

Camera used : Android Phone Honor 9N
Camera Software : Adobe Scan (Saved the original images in gallery, the processed the images via scantailor)

@venkatarangan

This comment has been minimized.

Copy link

commented Sep 18, 2019

@tshrinivasan Considering, the low-cost and OSS setup, the quality of the scan for this PDF is great. Kudos. If you are going to release this book (PDF) as such, I will request you add the tesseract OCR support as well so that the text becomes searchable. Thanks.

@tshrinivasan

This comment has been minimized.

Copy link
Contributor Author

commented Sep 18, 2019

@venkatarangan Sure sir, Dreaming of a website like FreeTamilEbooks.com but only with the scanned PDF files.
Will launch the site soon.

Is it OK to go with the tesseract as there is a lot of room for improvement for it with Tamil ?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
4 participants
You can’t perform that action at this time.