Excessive spacing when extracting text from PDF

[mush42]

Hello,

Thanks for this awesome work. I'm using PyMuPDF to provide blind people with accessible eBook reading experience with Bookworm.

Describe the bug (mandatory)

PyMuPDf v1.16.0 and up introduced a weired bug with certain PDF documents. When using text extraction functions., excessive spaces are inserted between characters. The extracted text is just a blob of characters. I have to admit that this problem is rare, it happens with some kind of PDF documents. In v1.14X versions the text is extracted correctly.

To Reproduce (mandatory)

Here is the text extraction function straight from the FitzDocument backend:

def _text_from_page(page):
    bloks = page.getTextBlocks()
    text = [blk[4].replace("\n", " ") for blk in bloks]
    return "\r\n".join(text)

Expected behavior (optional)

The text is extracted preserving its basic structure (words and paragraphs).

Your configuration (mandatory)

• Operating system: Windows 10 Pro, 64-bit
• Python: Python 3.7.4 64-bit/32-bit
• PyMuPDF version: v1.16.0, installed from PyPI

Additional context (optional)

The problem still happens with the latest version of PyMuPDF (v1.16.2). Playing with text extraction flags didn't help, it gives several variations none of which solves the issue.

This problem happens with the Dart specs document, and several others. As an example, the Dart language specs is attached.
DartLangSpec-v2.2.pdf

[JorjMcKie]

Hi,
any particular page where this happens, so I don't need to search within 206 of them?

[JorjMcKie]

I have tried page 6 and I believe I understand what you mean.

If printing with this command (page=5 in zero-based count):

>>> for b in page.getText("blocks", flags=1+2):  # flags=3 corresponds to default in earlier versions
	print(b[4])

We get this

D ar t P r ogr am m i n g Lan gu age S p e c i fi c at i on 6
1 Sco pe
e c m a Sc o pe
T h i s E c m a s t an d ar d s p e c i fi e s t h e s y n t ax an d s e m an t i c s of t h e D ar t p r ogr am -
m i n g l an gu age . I t d oe s n ot s p e c i f y t h e AP I s of t h e D ar t l i b r ar i e s e x c e p t w h e r e
t h os e l i b r ar y e l e m e n t s ar e e s s e n t i al t o t h e c or r e c t f u n c t i on i n g of t h e l an gu age
i t s e l f ( e . g. , t h e e x i s t e n c e of c l as s Object w i t h m e t h od s s u c h as noSuchMethod,
runtimeType) .
2 C o nf o r m a nce
e c m a C o nf o r m a nc e
< excess data omitted >

[JorjMcKie]

When printing with:

>>> for b in page.getText("blocks", flags=1+2+8):
	print(b[4])

We get

DartProgram m ingLanguageSpecification 6
1 Scope
ecm aScope
ThisEcm astandard specifiesthesyntaxand sem anticsoftheDartprogram -
m inglanguage.Itdoesnotspecify theAPIsoftheDartlibrariesexceptwhere
those library elem entsare essentialtothe correctfunctioningofthe language
itself(e.g.,the existence ofclassObjectwith m ethodssuch asnoSuchMethod,
runtimeType).
2 C onform ance
ecm aC onform ance
Aconform ingim plem entation oftheDartprogram m inglanguagem ustpro-
videandsupportalltheAPIs(libraries,types,functions,getters,setters,whether
top-level,static,instanceorlocal)m andated in thisspecification.
...

Which is less than satisfying, but better ...
(Oh, btw. I have been using my new patch version 1.16.3, which directly supports getTextBlocks in that way - do not try this with your version).

[JorjMcKie]

As a crosscheck I extracted that page with "mutool draw -o page6.html DartLangSpec-v2.2.pdf". Just to trace who is doing the crime.
This results in a html page looking like this:

And indeed: versions based on MuPDF v1.14 do it "correctly" ... annoying.

Let me check if one of my utilities can be a help ... "I'll be back" (quotation).

[JorjMcKie]

I'm afraid I have to give up. There seems to be no way of recovering a behaviour similar to MuPDF v1.14.
We need to submit an issue to MuPDF (Artifex), which I did just now.
If you want to refer to this, go to this web site: https://bugs.ghostscript.com/show_bug.cgi?id=701602.

If you want, you can also emphasize your interest in resolving it on that site - give it a try ...

[mush42]

Hi @JorjMcKie

Thanks. It is really frustrating when something used to work correctly suddenly breaks.

When I've tried to rollback the version from 1.16 to 1.14, pip could not find a suitable distribution for that version. Could you please provide away to install older versions of PyMuPDF. It is really, really important to have an archive of previous versions for some critical applications.

Kind Regards,

[JorjMcKie]

The unavailability of 1.14 versions on PyPI is my fault. Here are 2 recreated from source. Please let me know if you require other versions ...
You need to rename them from *.zip to *.whl (Github restriction).
PyMuPDF-1.14.20-cp37-cp37m-win32.zip

PyMuPDF-1.14.20-cp37-cp37m-win_amd64.zip

[JorjMcKie]

@mush42 - did this work for you?

[JorjMcKie]

I have just uploaded an official intermediate version 1.14.21 to PyPI.
I hope you can use this while we are waiting for a fix from upstream.