Buffer size mismatch when calling get_text_range

[elonzh]

Issue Template

Checklist

• I confirm this is not a question or feature request. Otherwise, use the Discussions page.
• I confirm this is not an issue encountered with an installed build of pypdfium2, but about some other aspect of the project (specify below). Otherwise, use one of the package templates (PyPA/conda), even if you believe this is not a package-specific issue.
• I confirm this is not about an unofficial build of pypdfium2. We do not support third-party builds, and they are not eligible for a bug report.
• I have read and acknowledged the response policy.

Reason for Generic issue (keyword/topic)

text extraction

Description

> poetry show pypdfium2
Using python3 (3.10.8)
 name         : pypdfium2                 
 version      : 4.27.0                    

88d1ebc0-75ea-4fac-b398-202848e59d80.pdf

Script to reproduce issue

import pypdfium2

path = "88d1ebc0-75ea-4fac-b398-202848e59d80.pdf"

doc = pypdfium2.PdfDocument(path, autoclose=True)
try:
    for page_number, page in enumerate(doc):
        print(f"Reading Page {page_number + 1}")
        text_page = page.get_textpage()
        content = text_page.get_text_range()
        text_page.close()
        page.close()
finally:
    doc.close()

Output:

Reading Page 1
Reading Page 2
Reading Page 3
Reading Page 4
Reading Page 5
Reading Page 6
Reading Page 7
Reading Page 8
Reading Page 9
Traceback (most recent call last):
  File "/home/elonzh/workspace/ivysci/unob/poc.py", line 12, in <module>
    content = text_page.get_text_range()
  File "/home/elonzh/.cache/pypoetry/virtualenvs/unob-hX3-r8Vo-py3.10/lib/python3.10/site-packages/pypdfium2/_helpers/textpage.py", line 89, in get_text_range
    assert in_size == out_size, f"Buffer size mismatch: {in_size} vs {out_size}"
AssertionError: Buffer size mismatch: 3208 vs 3210

[mara004]

Oh, that's not good. Thank you for the bug report; I will investigate.
The assert which caught here was added during refactoring in the course of this issue:
#261 (comment)
#261 (comment)

[mara004]

I think this must be due to a pdfium change.
I reverted to a pdfium version at the time of #261 via ./run emplace auto:6056 (assuming an editable install).

With this, the following test code passes just fine:

from pypdfium2 import *
from pypdfium2.raw import *
from pathlib import Path

in_path = Path("~/Downloads/88d1ebc0-75ea-4fac-b398-202848e59d80.pdf").expanduser()
pdf = PdfDocument(in_path)
tp_a = pdf[8].get_textpage()
tp_b = pdf[9].get_textpage()

print( tp_a.get_text_range() )
print( tp_b.get_text_range() )

But it breaks with latest pdfium (./run emplace auto:6309), triggering the above assert.

So I guess we'll need to find out what has changed, why, and which commit(s) are responsible.

[mara004]

I have consulted pdfium via this ticket: https://bugs.chromium.org/p/pdfium/issues/detail?id=2133

[mara004]

In the meantime, please use get_text_bounded(), unless you specifically need the ability to define a character range.

I've also written a patch that would work around the get_text_range() issue by doubling the allocation, but I'd prefer not to have this. Let's first wait some time for upstream.

[mara004]

Upstream is tough and bureaucratic as usual.
They've classified the issue as WontFix and suggest to generally allocate 4 bytes (which I'd consider bad practice). Otherwise they seem to request a new bug report and new evidence why the API should tell the exact buffer size. Duh...

[elonzh]

Seems we have no choice.

[mara004]

Yeah... I suppose I should merge #301, and if pdfium improves, we can still update the code.
In the meantime, we can add a warning to prefer get_text_bounded() where possible, and maybe even make get_text_range() translate to get_text_bounded() if called without arguments.

[mara004]

Workaround implemented as per previous comment.
GH auto-closed the issue on merge – or would you say we should keep it open?

[elonzh]

The current implementation is already very good. Thank you for your hard work.

[mara004]

It seems like pdfium team actually got back to the issue and reverted FPDFText_GetText() to 2 byte characters only. (Seeing as it was not only that we couldn't tell the exact buffer size anymore, but also that API stability expectations were broken.)

This is speculative, but I suspect they might have received a separate security bug that led to revisiting of the previously raised concerns, as the commit-linked issue is non-public.