Pubpub zz extract text #924
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
fix at least #880 line feeds are applied as stated in the ref TODO : for segmented text, horizontal translation should be analysed to apply some space or delete some characters (to be analysed in crazyones.pdf) TODO : font conversion (to be analysed in crazyones.pdf)
fix wrong characters in crazyones fix space wrongly introduced
various fixes +add pytest with pypdftest
also includes a little of rewrite
Co-authored-by: Martin Thoma <info@martin-thoma.de>
Co-authored-by: Martin Thoma <info@martin-thoma.de>
proposal not fixing change also includes snake_style rewriting
Codecov Report
@@ Coverage Diff @@
## main #924 +/- ##
==========================================
- Coverage 84.22% 82.82% -1.41%
==========================================
Files 16 17 +1
Lines 3842 4104 +262
Branches 808 882 +74
==========================================
+ Hits 3236 3399 +163
- Misses 413 515 +102
+ Partials 193 190 -3
Continue to review full report at Codecov.
|
Merged
MartinThoma
added
the
workflow-text-extraction
* New proposal for spacing/lining with deeper analysis of font data and text positionning Includes : * XObject Processing, * choice between encoding and tounicode fields * partial compliance with Identify-H/V encoding (missing processing on 2-bytes) *legacy conversion reintroduced as old for comparison increase test and refactory depreciation warning ignore in test * take into account Tm scale and improve space calculation
|
PdfWithXForm.pdf was the example from the PR. I agree that There could be some issue with the address that may be a good one... I would propose you to replace it with 2201.00029.pdf that also includes XForms (This is why it was 0% before). Note that in this latest file some XForm images are missing inducing some warnings |
This was referenced Jun 5, 2022
Closed
Closed
Closed
Closed
Closed
MartinThoma
added a commit
that referenced
this pull request
Jun 6, 2022
The highlight of the 2.1.0 release is the most massive improvement to the text extraction capabilities of PyPDF2 since 2016 🥳🎊 A very big thank you goes to [pubpub-zz](https://github.com/pubpub-zz) who took a lot of time and knowledge about the PDF format to finally get those improvements into PyPDF2. Thank you 🤗💚 In case the new function causes any issues, you can use `_extract_text_old` for the old functionality. Please also open a bug ticket in that case. There were several people who have attempted to bring similar improvements to PyPDF2. All of those were valuable. The main reason why they didn't get merged is the big amount of open PRs / issues. pubpub-zz was the most comprehensive PR which also incorporated the latest changes of PyPDF2 2.0.0. Thank you to [VictorCarlquist](https://github.com/VictorCarlquist) for #858 and [asabramo](https://github.com/asabramo) for #464 🤗 New Features (ENH): - Massive text extraction improvement (#924). Closed many open issues: - Exceptions / missing spaces in extract_text() method (#17) 🕺 - Whitespace issues in extract_text() (#42) 💃 - pypdf2 reads the hifenated words in a new line (#246) - PyPDF2 failing to read unicode character (#37) - Unable to read bullets (#230) - ExtractText yields nothing for apparently good PDF (#168) 🎉 - Encoding issue in extract_text() (#235) - extractText() doesn't work on Chinese PDF (#252) - encoding error (#260) - Trouble with apostophes in names in text "O'Doul" (#384) - extract_text works for some PDF files, but not the others (#437) - Euro sign not being recognized by extractText (#443) - Failed extracting text from French texts (#524) - extract_text doesn't extract ligatures correctly (#598) - reading spanish text - mark convert issue (#635) - Read PDF changed from text to random symbols (#654) - .extractText() reads / as 1. (#789) - Update glyphlist (#947) - inspired by #464 - Allow adding PageRange objects (#948) Bug Fixes (BUG): - Delete .python-version file (#944) - Compare StreamObject.decoded_self with None (#931) Robustness (ROB): - Fix some conversion errors on non conform PDF (#932) Documentation (DOC): - Elaborate on PDF text extraction difficulties (#939) - Add logo (#942) - rotate vs Transformation().rotate (#937) - Example how to use PyPDF2 with AWS S3 (#938) - How to deprecate (#930) - Fix typos on robustness page (#935) - Remove scripts (pdfcat) from docs (#934) Developer Experience (DEV): - Ignore .python-version file - Mark deprecated code with no-cover (#943) - Automatically create Github releases from tags (#870) Testing (TST): - Text extraction for non-latin alphabets (#954) - Ignore PdfReadWarning in benchmark (#949) - writer.remove_text (#946) - Add test for Tree and _security (#945) Code Style (STY): - black, isort, Flake8, splitting buildCharMap (#950) Full Changelog: 2.0.0...2.1.0
This was referenced Jun 6, 2022
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The main PR is #881 - my (Martin Thoma) contributions here are neglectable; full credit to @pubpub-zz . The reason for opening this PR was only to show a stylistic code change.