Advanced text extraction #464

asabramo · 2018-10-16T07:31:44Z

The work on this branch takes text extraction to much more advanced levels.
It mainly includes support for:

Unicode CMAPs (/ToUnicode and /Differences)
Correct line extraction
Extract text widths
Properly handle nested objects
Specific support for the Hebrew language encoding is added. Testing was done on both Hebrew and English documents

These are many changes that I've worked on for a while, I did my best to clean up and order them in the limited time I have to do it.
Note - this is my first time contributing to projects on Github so please let me know if I need to do anything differently.

Add VSCode and GitEye ignores

Higher numbers represent deeper levels of debug. dbg(0, 'text') should always print it dbg(10, 'text') should print only if debugLevel is set to 10 or more

The optional lineCallback argument is called for each line in the extracted text. The callback receives a list of line elements, each containing text, x and y.

Use the BT and Td/TD tags to figure out the real position of a text element in a document. TD are relative, but BT resets the position. This corrects the line detection algorithm to work on documents like BezeqInt

This works for the BezeqInt document. Probably need to support other forms of CMAP. Definitely need to support other forms of encoding - e.g. "differences", as used in the Amex document. This code is based on: https://github.com/mstamy2/PyPDF2/pull/201/commits

This fixes the "Fenix" documents

Only supporting the case where Tm is used to do the same as TD This fixes the "Fenix"

Slightly generalize the text handling to support the TJ element as well. This should work when there are no specific glyph positioning. This solves the case of the Supergaz bill.

CMAP - add support for char based mapping TJ - fix translation This enables the Bezeq Phone bills

TODO - this may not support a combination of bfrange and bfchar in the same cmap... However, this does enable the Pelephone bill format :-)

The electricity bill has some garbage (HTML page) after the EOF marker... PyPDF allows up to 1K of such garbage so the file is rejected. Increased it to make it work. TODO: need a more robust solution, to accept files with more garbage. One way to do it is to cut that garbage before uploading.

The names of the glyphs seem to be "known" with their unicode mappings. This commit brings such name mappings from two different github projects to create a common list. Then the /Differences array is read and parsed into a CMAP compatible structure so that the translation works. This removes the hard coded encoding used for Amex and Mastercard Hebrew documents.

Added extractFonts and extractImages to the PDF reader class. This allows extraction of the fonts and images form the whole PDF in a single command, in a standard way.

Seems that the DIFFERENCES array handling was specific to the way it appeared in the Amex docs. Changed it to handle the Supergaz case properly too. Supergaz fonts are TrueType Also introduced mergine of cmaps from tounicode and differnces in order to make the unittest pass.

This way one can iterate over the response without checking for None. This makes the Hot Mobile bill work

Look for \nend at the end, which could actually be the endobject marker. This is because in the CellcomTV bill there's just no endstream marker for some of the streams (CMAPs)

Seems like Tm should be treated as an absolute location (actually offset from the last location, replacing the previous matrix) This fixes the CellcomTV bill to read the lines correctly

This fixes the PazGaz parsing. Still, it seems that Pazgas don't provide the right name for the newsheqel symbol, and it's identified as {

This is a hack for now, until the line size would be determined properly So for now, the caller needs to tell the pdf extractor what Y difference should be considered a line break. This fixes the BezeqPhone bill

This prevents crashing when parsing the IRS names file.

The T*, " and ' elements were not converted to using the new handleTextElement function. This came up when trying to open the IRS NAMES file.

Faking a newline operator when the text is over so that the new line handler is called also for the last line in the text.

Support ranges of characters in the form: <0000><0002>[<0032><0033><0034>] This appears in the GetTaxi receipt

This fixes the Pelephone bill parsing

TODO: how should we decide how many bytes represent each character in the string? Fixes GetTaxi after the change to support the Pelephone single byte...

Keep all the lines in a dictionary by Y coordinate and run the line callback only at the end (of each page) This fixes the Pelephone date signature

Tom-Evers · 2018-11-08T08:46:50Z

So... I upgraded the extractText() function in #397
Does this conflict with those changes?

asabramo · 2018-11-08T11:00:58Z

Hi, I haven't tried to merge these branches so I can't say for sure what level of conflict exists, but it does look like the changes address similar areas of the code and similar issues. Your changes are far simpler and I expect my changes are more generic. I'm not sure if your change is needed if my code is present, but it's very much possible that your code does address different corner cases. If you have time to try my branch on your example PDFs, that would be the simplest way to find out. Sorry for not having the time to check these things deeper. Assi

…

On Thu, Nov 8, 2018, 10:46 Tom-Evers ***@***.*** wrote: So... I upgraded the extractText() function in #397 <#397> Does this conflict with those changes? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#464 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/Aa_8VNYA4SRszFsKFlmBCeXJS95Xac7Bks5us-9_gaJpZM4Xdvlg> .

MartinThoma · 2022-04-06T10:32:42Z

I like this a lot! I need more time to get into the library / improve tests coverage, but I'm definitely in favor of improving the text extraction!

MartinThoma · 2022-04-09T13:52:29Z

Is it ok if I resolve the issues?

asabramo · 2022-04-09T15:02:58Z

Hi Martin, I'm happy to see you found this work useful. I'm not sure if by "resolve " you mean mark as resolved, or actually push fixes for them... I'm not working on this project at all nowadays, so if you find anything you can improve, you are more than welcome. Personally I won't have the time to review or test, so I trust your judgement. Cheers, Assi

…

On Sat, Apr 9, 2022, 16:52 Martin Thoma ***@***.***> wrote: Is it ok if I resolve the issues? — Reply to this email directly, view it on GitHub <#464 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AGX7YVASMV22P7YRETRWGMDVEGDSTANCNFSM4F3W7FQA> . You are receiving this because you authored the thread.Message ID: ***@***.***>

MartinThoma · 2022-04-09T15:17:34Z

Cool, I understand that. Thank you for the swift answer and the pr! I'll take care of the rest :-)

MartinThoma · 2022-06-05T12:28:22Z

@pubpub-zz The file glyphNamesToUnicode.py seems to be pretty similar to _adobe_glyphs.py. The differences are:

_adobe_glyphs.adobe_glyphs vs glyphNamesToUnicode.glyphNamesToUnicode
_adobe_glyphs creates the dictionary in one run, whereas glyphNamesToUnicode creates an empty dictionary first and adds the items. I think creating it in one run might be better:
"/parenleft": "\u0028", vs glyphNamesToUnicode['parenleft'] = 0x0028 -- note the missing leading backslash and the hex representation vs the unicode symbol representation. I think it's better what we have right now as we can use it directly.
⚠️ _adobe_glyphs has roughly 4,320 symbols whereas glyphNamesToUnicode has roughly 14,680 symbols. I wonder if we need the additional 10,000 symbols. For example, circledot and star are in there which could improve our results for https://arxiv.org/pdf/2201.00151.pdf

Run print({"/" + key: fr"\u{hex(value).upper()[2:]:0>4}" for key, value in sorted(adobe_glyphs.items())}) over the glyphlist from #464 This made use of Glyph Name Formatted Unicode List - GNFUL GlyphNameFormatter version 0.28 - git commit: 406 Unicode version: Unicode 10.0.0 Source code: https://github.com/LettError/glyphNameFormatter/tree/c225719

Run print({"/" + key: fr"\u{hex(value).upper()[2:]:0>4}" for key, value in sorted(adobe_glyphs.items())}) over the glyphlist from #464 This made use of Glyph Name Formatted Unicode List - GNFUL GlyphNameFormatter version 0.28 - git commit: 406 Unicode version: Unicode 10.0.0 Source code: https://github.com/LettError/glyphNameFormatter/tree/c225719 Add circledot and star manually

The highlight of the 2.1.0 release is the most massive improvement to the text extraction capabilities of PyPDF2 since 2016 🥳🎊 A very big thank you goes to [pubpub-zz](https://github.com/pubpub-zz) who took a lot of time and knowledge about the PDF format to finally get those improvements into PyPDF2. Thank you 🤗💚 In case the new function causes any issues, you can use `_extract_text_old` for the old functionality. Please also open a bug ticket in that case. There were several people who have attempted to bring similar improvements to PyPDF2. All of those were valuable. The main reason why they didn't get merged is the big amount of open PRs / issues. pubpub-zz was the most comprehensive PR which also incorporated the latest changes of PyPDF2 2.0.0. Thank you to [VictorCarlquist](https://github.com/VictorCarlquist) for #858 and [asabramo](https://github.com/asabramo) for #464 🤗 New Features (ENH): - Massive text extraction improvement (#924). Closed many open issues: - Exceptions / missing spaces in extract_text() method (#17) 🕺 - Whitespace issues in extract_text() (#42) 💃 - pypdf2 reads the hifenated words in a new line (#246) - PyPDF2 failing to read unicode character (#37) - Unable to read bullets (#230) - ExtractText yields nothing for apparently good PDF (#168) 🎉 - Encoding issue in extract_text() (#235) - extractText() doesn't work on Chinese PDF (#252) - encoding error (#260) - Trouble with apostophes in names in text "O'Doul" (#384) - extract_text works for some PDF files, but not the others (#437) - Euro sign not being recognized by extractText (#443) - Failed extracting text from French texts (#524) - extract_text doesn't extract ligatures correctly (#598) - reading spanish text - mark convert issue (#635) - Read PDF changed from text to random symbols (#654) - .extractText() reads / as 1. (#789) - Update glyphlist (#947) - inspired by #464 - Allow adding PageRange objects (#948) Bug Fixes (BUG): - Delete .python-version file (#944) - Compare StreamObject.decoded_self with None (#931) Robustness (ROB): - Fix some conversion errors on non conform PDF (#932) Documentation (DOC): - Elaborate on PDF text extraction difficulties (#939) - Add logo (#942) - rotate vs Transformation().rotate (#937) - Example how to use PyPDF2 with AWS S3 (#938) - How to deprecate (#930) - Fix typos on robustness page (#935) - Remove scripts (pdfcat) from docs (#934) Developer Experience (DEV): - Ignore .python-version file - Mark deprecated code with no-cover (#943) - Automatically create Github releases from tags (#870) Testing (TST): - Text extraction for non-latin alphabets (#954) - Ignore PdfReadWarning in benchmark (#949) - writer.remove_text (#946) - Add test for Tree and _security (#945) Code Style (STY): - black, isort, Flake8, splitting buildCharMap (#950) Full Changelog: 2.0.0...2.1.0

MartinThoma · 2022-06-19T11:43:10Z

@asabramo We (well, @pubpub-zz deserves all of the credit 😄 ) worked a lot on improving the text extraction capabilities of PyPDF2. I think this PR is no longer necessary.

I'm sorry it took so long. Thank you very much for putting the effort in writing this PR 🤗

Assi.Abramovitz@gmail.com added 30 commits October 16, 2018 10:14

Update gitIgnore

fcd7b19

Add VSCode and GitEye ignores

introduce the dbg function

08ba779

Higher numbers represent deeper levels of debug. dbg(0, 'text') should always print it dbg(10, 'text') should print only if debugLevel is set to 10 or more

introduce lineCallback

46a073b

The optional lineCallback argument is called for each line in the extracted text. The callback receives a list of line elements, each containing text, x and y.

Treat TD like Td

915f40d

use absolute positions for text

232ca33

Use the BT and Td/TD tags to figure out the real position of a text element in a document. TD are relative, but BT resets the position. This corrects the line detection algorithm to work on documents like BezeqInt

Basic support for CMAP

28d57f8

This works for the BezeqInt document. Probably need to support other forms of CMAP. Definitely need to support other forms of encoding - e.g. "differences", as used in the Amex document. This code is based on: https://github.com/mstamy2/PyPDF2/pull/201/commits

Fix CMAP handling to support two byte chars

7737642

This fixes the "Fenix" documents

Basic support for the Tm operands

354a24e

Only supporting the case where Tm is used to do the same as TD This fixes the "Fenix"

Handle TJ text elements

3444f81

Slightly generalize the text handling to support the TJ element as well. This should work when there are no specific glyph positioning. This solves the case of the Supergaz bill.

improve CMAP and TJ handling

ac4e98b

CMAP - add support for char based mapping TJ - fix translation This enables the Bezeq Phone bills

Fix CMAP to support multiple ranges

fd6858d

TODO - this may not support a combination of bfrange and bfchar in the same cmap... However, this does enable the Pelephone bill format :-)

Extract Fonts and Images

f420505

Added extractFonts and extractImages to the PDF reader class. This allows extraction of the fonts and images form the whole PDF in a single command, in a standard way.

fix cmap handling for non unicode

1619d0a

fix getDocumentInfo return value

6561bc8

This way one can iterate over the response without checking for None. This makes the Hot Mobile bill work

accept streams with no endstream

c0a0b74

Look for \nend at the end, which could actually be the endobject marker. This is because in the CellcomTV bill there's just no endstream marker for some of the streams (CMAPs)

Fix Tm handling

b4603fc

Seems like Tm should be treated as an absolute location (actually offset from the last location, replacing the previous matrix) This fixes the CellcomTV bill to read the lines correctly

PyPdf: improve glyph name handling

bbb81f8

This fixes the PazGaz parsing. Still, it seems that Pazgas don't provide the right name for the newsheqel symbol, and it's identified as {

Add linemargin

aa2d46d

This is a hack for now, until the line size would be determined properly So for now, the caller needs to tell the pdf extractor what Y difference should be considered a line break. This fixes the BezeqPhone bill

add the .notdef glyph name

c6dbbb0

handle strange char numbers in cmaps

efd1eec

This prevents crashing when parsing the IRS names file.

Fix handling of rare text elements

1cbce88

The T*, " and ' elements were not converted to using the new handleTextElement function. This came up when trying to open the IRS NAMES file.

don't miss the last line

2394f39

Faking a newline operator when the text is over so that the new line handler is called also for the last line in the text.

fix BFRANGE handling

49bd3fe

Support ranges of characters in the form: <0000><0002>[<0032><0033><0034>] This appears in the GetTaxi receipt

CMAP parsing improvements

ee824d0

This fixes the Pelephone bill parsing

Debugging enhancement

6898737

hack for singlebyte CMAP

22ac7ad

TODO: how should we decide how many bytes represent each character in the string? Fixes GetTaxi after the change to support the Pelephone single byte...

Real lines sorting

2ce9465

Keep all the lines in a dictionary by Y coordinate and run the line callback only at the end (of each page) This fixes the Pelephone date signature

MartinThoma added Large PdfReader The PdfReader component is affected labels Apr 6, 2022

MartinThoma added text-extraction workflow-text-extraction From a users perspective, text extraction is the affected feature/workflow and removed text-extraction labels Apr 16, 2022

This was referenced Apr 16, 2022

extract arabic text from pdf #591

Closed

Initial support for CMap character translation #201

Closed

Advanced text extraction by asabramo #767

Closed

MartinThoma mentioned this pull request Jun 4, 2022

ExtractText2 #929

Merged

MartinThoma mentioned this pull request Jun 5, 2022

ENH: Update glyphlist #947

Merged

MartinThoma closed this Jun 19, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Advanced text extraction #464

Advanced text extraction #464

asabramo commented Oct 16, 2018

Tom-Evers commented Nov 8, 2018

asabramo commented Nov 8, 2018 via email

MartinThoma commented Apr 6, 2022

MartinThoma commented Apr 9, 2022

asabramo commented Apr 9, 2022 via email

MartinThoma commented Apr 9, 2022

MartinThoma commented Jun 5, 2022 •

edited

MartinThoma commented Jun 19, 2022

Advanced text extraction #464

Advanced text extraction #464

Conversation

asabramo commented Oct 16, 2018

Tom-Evers commented Nov 8, 2018

asabramo commented Nov 8, 2018 via email

MartinThoma commented Apr 6, 2022

MartinThoma commented Apr 9, 2022

asabramo commented Apr 9, 2022 via email

MartinThoma commented Apr 9, 2022

MartinThoma commented Jun 5, 2022 • edited

MartinThoma commented Jun 19, 2022

MartinThoma commented Jun 5, 2022 •

edited