Read PDF changed from text to random symbols

[mathieuboudreau]

Hi there,

I've been using this script regularly on PDF for work,

import PyPDF2
from pathlib import Path
import re
import os

search_terms = ["DATA AVAILABILITY STATEMENT",
                "open source",
                "open-source",
                "opensource",
                "open science",
                "github",
                " git "
                "osf",
                "jupyter",
                "notebook",
                "octave",
                "available online",
                "released",
                "shared",
                " code ",
                "numerical phantom",
                "bitbucket",
                "sourceforge",
                "xnat",
                "reproducible research",
                "julia",
                "image set",
                "image sets",
                "raw k-space data",
                "SHA-1",
                "gitlab",
                "Docker",
                "container",
                "MyBinder",
                "Binder",
                "mrhub",
                "MR-Hub",
                "codeocean",
                "Code Ocean"]

folder_path = '.'

for filename in sorted(os.listdir(Path(folder_path))):
    if filename.endswith(".pdf"):
        # open the pdf file
        object = PyPDF2.PdfFileReader(filename)

        # get number of pages
        num_pages = object.getNumPages()
        
        found_keywords = []
        # search through keywords
        for keyword in search_terms:
            # extract text and do the search
            for page_index in range(0, num_pages):
                page_obj = object.getPage(page_index)
                page_text = page_obj.extractText() 

                search_result = re.search(keyword, page_text)

                if search_result is not None:
                    found_keywords.append(keyword)
                    break
        
        if found_keywords:
            print(filename + " contains " + str(found_keywords))

and for all PDFs I used before a few months ago, the page text was correctly being read as text (e.g. the PDF downloadable here: https://onlinelibrary.wiley.com/doi/10.1002/mrm.28965).

However, now recent PDFs (like this one: https://onlinelibrary.wiley.com/doi/10.1002/mrm.29078) are reading the pages as random symbols, like this (generated by adding print(page_text)):

ƒ
˙−
ˇˇ

ƒ
ˇ˘

−

ˇ˙
ˇ˝−
˜˚
˜ˇ

˜˜


˜˘−

“

“–‡”
“‹
“‹
⁄”


©



−
©

“‹
ƒ
−
ƒ

“‹

−



“‹

ƒ
⁄‡‡”


‹

⁄‡
−
“‹



Žƒ

So clearly, my keyword detection isn't working anymore.

I can't seem to find a difference in the PDF files (they are both Adobe InDesign 15.1 (Windows), Adobe PDF Library 15.0; modified using iText 4.2.0 by 1T3XT). Any clue on how to resolve this for the newer PDFs I'm using?

[johns1c]

The difference is in the Font objects.

The new one has a large number of embedded font objects which are all custom encoded. PyPDF2 is not very good at anything other than the standard encodings.

The old one has 5 fonts and although some are custom WinAnsi encoding is used

in the old one

[johns1c]

is this a Bug or an Limitation?

Interestingly although my viewer (where I have patched PyPDF2 to handle a wider range of encoding) does handle the new one it crashes handing the old one which has indirect content streams which then include forms objects.

[MartinThoma]

is this a Bug or an Limitation?

It is a limitation which is perceived by the user as a bug :-) Although we might not have the capacity to fix this right now, I would say in general we want to fix this. I see it from the users perspective (hence it's a bug, not a new feature). But that's only terminology^^

[johns1c]

Fix available see github johns1c/pypdf2

The files below show the results of running my amended copy (based on 1.26.0) with running the latest version installed yesterday 2022-04-09.

test654_page0_1_26_0.txt

test654_page0_1_27_2.txt

I coded the amendment some time ago and believe it handles custom encoding as well as to_unicode tables. My version of wx.lib.pdfviewer uses this code and seems to handle the vast majority of PDFs.

Martin - do you think that my code could be incorporated into the "production" version?

1 I will check that I have the latest code on github

1. it needs code review - both the approach and the detail code
2. it needs a test suite

I am willing to do the work but I might need some hand holding.

Chris Johnson

[MartinThoma]

do you think that my code could be incorporated into the "production" version?

Yes, for sure!

I am willing to do the work but I might need some hand holding.

That's what we (me and the PyPDF2 community) are here for :-) Just make a PR, we will walk through it :-)

[MartinThoma]

I see that you haven't forked from PyPDF2: https://github.com/johns1c/PyPDF2

Could you maybe fork from PyPDF2: https://docs.github.com/en/get-started/quickstart/fork-a-repo

then make the changes that fixed the issue + create a PR?

[johns1c]

Will take a few days as am away from home Chris (from my iThingy

…

On 16 Apr 2022, at 12:21, Martin Thoma ***@***.***> wrote:  I see that you haven't forked from PyPDF2: https://github.com/johns1c/PyPDF2 Could you maybe fork from PyPDF2: https://docs.github.com/en/get-started/quickstart/fork-a-repo then make the changes that fixed the issue + create a PR? — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.

[johns1c]

Dear Martin, At present my PyPDF2 only contains the Python code that would be installed in site-packages (the PyPDF2 subdirectory of the original). I will need to fork the PyPDF/PyPDF2 code, merge the two versions (mainly incorporating the recent changes to literals) and test. I have a few questions regarding this 1. Does my fork have to be called PyPDF2? 2. Can I run Github actions without breaking anything (or releasing it)? 3. Are existing regression tests sufficient (my own tests tend to be read-only) 4. Is there a way of installing from a fork (say using test PyPi) that would allow me to test without affecting anyone else. 5. Are you currently making changes which I would need to track particularly widespread changes such as the recent literal change which I am now trying to incorporate. Chris Sent from Mail for Windows From: Martin Thoma Sent: 16 April 2022 12:21 To: py-pdf/PyPDF2 Cc: C Johnson; Comment Subject: Re: [py-pdf/PyPDF2] Read PDF changed from text to random symbols(Issue #654) I see that you haven't forked from PyPDF2: https://github.com/johns1c/PyPDF2 Could you maybe fork from PyPDF2: https://docs.github.com/en/get-started/quickstart/fork-a-repo then make the changes that fixed the issue + create a PR? — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.Message ID: ***@***.***>

[MartinThoma]

Good morning!

1. Just click on the Fork-button on the upper-right in Github. The fork can have a different name, but Github will take care of it. The important part is that the git history is the same
2. You don't have access to anything. Forking is making a copy. You don't have access to any credentials. Don't worry :-)
3. That's impossible to answer, but I hope that by now the tests cover most typical use-cases.
4. Yes! You can install from a local copy by pip install -e . in that folder. The -e stands for "editable". It means that your installation points to that directory. So you don't have to re-install for changes - it will automatically use the files you're editing.
5. Don't worry about that. Yes, there will be a lot of changes in the next months, but it might be nothing that affects your PR. We will cross that bridge if we get there.

[MartinThoma]

This issue has been fixed with #924

Example

from PyPDF2 import PdfReader

reader = PdfReader("magnetic.pdf")   # https://onlinelibrary.wiley.com/doi/10.1002/mrm.29078
text = reader.pages[0].extract_text()

text then is:

138 |     Magn Reson Med. 2022;87:138–149.

wileyonlinelibrary.com/journal/mrm
Received: 28 April 2021 | Revised: 30 June 2021 | Accepted: 23 July 2021DOI: 10.1002/mrm.28965  

RESEARCH ARTICLE
Clinical translation of hyperpolarized  13C pyruvate and urea MRI for simultaneous metabolic and perfusion imaging
Hecong/uni00A0Qin1,2 |   Shuyu/uni00A0Tang 1 |   Andrew M./uni00A0Riselli 1 |   Robert A./uni00A0Bok1 |   Romelyn/uni00A0Delos Santos
1 |   Mark/uni00A0van Criekinge 1 |   Jeremy W./uni00A0Gordon 1 |   Rahul/uni00A0Aggarwal
3 |   Rui/uni00A0Chen4 |   Gregory/uni00A0Goddard5 |   Chunxin Tracy/uni00A0Zhang 5 |    Albert/uni00A0Chen
4 |   Galen/uni00A0Reed4 |   Daniel M./uni00A0Ruscitto4 |   James/uni00A0Slater1 |   Renuka/uni00A0Sriram1 |   Peder E. Z./uni00A0Larson
1,2 |   Daniel B./uni00A0Vigneron 1,2 |   John/uni00A0Kurhanewicz 1,2

1Department of Radiology and Biomedical Imaging, University of California, San Francisco, San Francisco, California, USA
2Graduate Program in Bioengineering, University of California, Berkeley and San Francisco, San Francisco, California, USA
3Department of Medicine, University of California, San Francisco, San Francisco, California, USA
4General Electric Healthcare, Milwaukee, Wisconsin, USA
5General Electric Research, Niskayuna, New York, USA
This is an open access article under the terms of the Creat ive Commo ns Attri butio n- NonCo mmerc ial- NoDerivs License, which permits use and distribution in any medium, provided the original work is properly cited, the use is non- commercial and no modifications or adaptations are made.© 2021 The Authors. Magnetic Resonance in Medicine published by Wiley Periodicals LLC on behalf of International Society for Magnetic Resonance in Medicine. CorrespondenceJohn Kurhanewicz, Professor of Radiology and Biomedical Imaging, Pharmaceutical Chemistry, and Urology, University of California, San Francisco (UCSF), 1700 4th St., Suite 203, Box 2520, San Francisco, CA 94143, USA.Email: John.Kurhanewicz@ucsf.eduPresent addressShuyu Tang, HeartVista Inc., Los Altos, California, USAFunding informationNational Institute of Health, Grant/Award Number: P41EB013598 and R01CA214554
 Purpose: The combined hyperpolarized (HP)  13C pyruvate and urea MRI has pro-vided a simultaneous assessment of glycolytic metabolism and tissue perfusion for improved cancer diagnosis and therapeutic evaluation in preclinical studies. This work aims to translate this dual- probe HP imaging technique to clinical research.Methods: A co- polarization system was developed where [1- 
13C]pyruvic acid (PA) and [
13C, 15N2]urea in water solution were homogeneously mixed and polarized on a 5T SPINlab system. Physical and chemical characterizations and toxicology stud-ies of the combined probe were performed. Simultaneous metabolic and perfusion imaging was performed on a 3T clinical MR scanner by alternatively applying a multi- slice 2D spiral sequence for [1- 
13C]pyruvate and its downstream metabolites and a 3D balanced steady- state free precession (bSSFP) sequence for [
13C, 15N2]urea.Results: The combined PA/urea probe has a glass- formation ability similar to neat PA and can generate nearly 40% liquid- state 
13C polarization for both pyruvate and urea in 3- 4 h. A standard operating procedure for routine on- site production was developed and validated to produce 40 mL injection product of approximately 150 mM pyruvate and 35 mM urea. The toxicology study demonstrated the safety profile of the combined probe. Dynamic metabolite- specific imaging of [1- 
13C]pyruvate,   [1- 
13C]lactate, [1- 13C]alanine, and [13C, 15N2]urea was achieved with adequate 

[mathieuboudreau]

Thanks a lot everyone! Really appreciate the work you put into this fix/feature, and overall package!

[MartinThoma]

by now #924 is not relased. I will relase PyPDF2==2.1.0 today.