Skip to content
This repository was archived by the owner on Feb 19, 2021. It is now read-only.

Add support to extract text from PDFs that already contain a text layer (i.e. having been previously OCRed)#290

Closed
BastianPoe wants to merge 3 commits into
the-paperless-project:masterfrom
BastianPoe:feature/autodetect_if_ocr_is_required
Closed

Add support to extract text from PDFs that already contain a text layer (i.e. having been previously OCRed)#290
BastianPoe wants to merge 3 commits into
the-paperless-project:masterfrom
BastianPoe:feature/autodetect_if_ocr_is_required

Conversation

@BastianPoe

Copy link
Copy Markdown
Contributor

This change will try to detect if a PDF document already contains text (i.e. has been previously OCRed) and then skips tesseract.

Override via PAPERLESS_OCR_ALWAYS=YES is possible.

@BastianPoe
BastianPoe force-pushed the feature/autodetect_if_ocr_is_required branch from dd28811 to b997eb1 Compare January 28, 2018 18:18
@BastianPoe

Copy link
Copy Markdown
Contributor Author

CI fails, since debian package libpoppler-cpp-dev needs to be installed

@danielquinn

Copy link
Copy Markdown
Collaborator

This is also great work, thanks. As you've already noted, the new system-level dependency of libpoppler-cpp-dev is gumming things up, and would introduce a problem for new users. This isn't to say that we shouldn't do this, but I want to find a way to make the transition less painful for users first.

Merging this branch will require updating the requirements section of the documentation as well as including a note about breaking changes in the release notes. Travis will have to be tinkered with a bit as well to add the dependency somehow. I've been meaning to make some changes to the .travis.yml file lately anyway, so this is as good an excuse as any :-)

So TL;DR: I love this PR and will merge it, but one of us needs to make a few adaptations first. If you've got the time, feel free to try your hand at any of the above, but if not, I'll try to get to these this week.

Thanks again! This is great.

@BastianPoe
BastianPoe force-pushed the feature/autodetect_if_ocr_is_required branch from 2c41d1b to deb2926 Compare January 29, 2018 20:24
@BastianPoe
BastianPoe force-pushed the feature/autodetect_if_ocr_is_required branch from deb2926 to 4050846 Compare January 29, 2018 20:33
@BastianPoe

Copy link
Copy Markdown
Contributor Author

Did the changes (including the release notes) - please have a look.

I also checked again whether there is any pure-python library for reading text from PDFs. The only library I could find is PyPDF2 but the documentation says this (see http://pythonhosted.org/PyPDF2/PageObject.html#PyPDF2.pdf.PageObject.extractText):

Locate all text drawing commands, in the order they are provided in the content stream, and extract the text. This works well for some PDF files, but poorly for others, depending on the generator used. This will be refined in the future. Do not rely on the order of text coming out of this function, as it will change if this function is made more sophisticated.

This is unfortunately not what we need.

@danielquinn

Copy link
Copy Markdown
Collaborator

Ok! I had to take a little time to work out the kinks between this PR and the big Docker one we merged earlier today, but I think I've got it. With the conflicts, I had to do a manual merge, but all of your changes are in there, including credit for your work in the Changelog. Thanks again for this, and I'm sorry that the conflicts mucked things up. So many people are sending me stuff these past few days.

@ddddavidmartin

Copy link
Copy Markdown
Contributor

Nice work on this one @BastianPoe! Just tried it out and it makes a huge difference for generated documents.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants