Segmentation fault (core dumped)

[gvnramu]

Describe the bug (mandatory)

When applying page.apply_redactions() memory error occured

To Reproduce (mandatory)

>>>import fitz
>>> doc=fitz.open('fail.pdf')
>>> imglist = doc.getPageImageList(0, True)
>>> page = doc[0]
>>> rect = page.getImageBbox(imglist[0])
>>> page.addRedactAnnot(rect)
'Redact' annotation on page 0 of fail.pdf
>>> page.apply_redactions()

Expected behavior (optional)

Apply redactions successfully

Screenshots (optional)

Your configuration (mandatory)

Ubuntu 18.04 LTS - Architecture: x86-64

Python 3.7.4 (default, Aug 13 2019, 20:35:49)
[GCC 7.3.0]
linux

PyMuPDF 1.17.0
Installed via .whl file

[JorjMcKie]

My compliments for your talent to dig out problematic cases 😎!

1. The page does not contain images - although it does show some. The shown images are all contained in so-called Form XObjects, and there is no way for page.getImageBbox() to correctly calculate those, so should not even try to. I forgot to check this condition and raise an exception.

2. Independent from this, I tried to add and then apply a redaction for the full page ... and what happened is the same seg fault as you received it! The error is happening in MuPDF code (not PyMuPDF). So I need to find out what is going wrong there.

[gvnramu]

Hi @JorjMcKie
I got some doubts please explain to them.

1. The page does not contain images? ----- Please explain
2. As of my knowledge images in pdf will save like XObjects only. Then what's the problem?
3. If no images on the page then why doc.getPageImageList(0, True) is returning list of objects
   like this
   

[JorjMcKie]

If you look at this

>>> import fitz
>>> doc = fitz.open("fail.pdf")
>>> for item in doc.getPageImageList(0, True): print(item)

(8, 0, 850, 1100, 8, 'DeviceRGB', '', 'Obj5', 'FlateDecode', 1)
(9, 0, 856, 55, 1, '', '', 'Obj6', 'CCITTFaxDecode', 1)
(10, 0, 32, 26, 1, '', '', 'Obj7', 'CCITTFaxDecode', 1)
(11, 0, 16, 18, 1, '', '', 'Obj8', 'CCITTFaxDecode', 1)
(12, 0, 128, 83, 1, '', '', 'Obj9', 'CCITTFaxDecode', 1)
(13, 0, 296, 44, 1, '', '', 'Obj10', 'CCITTFaxDecode', 1)
(14, 0, 296, 44, 1, '', '', 'Obj11', 'CCITTFaxDecode', 1)
(15, 0, 184, 20, 1, '', '', 'Obj12', 'CCITTFaxDecode', 1)
>>> 

You will notice the "1" as last entry in each item. This is the xref of the Form XObject containing the image. The page invokes that Form XObject, which in turn invokes its embedded images or yet other Form XObjects. What we see here are 8 images which all are invoked by some Form XObject located at xref 1.
Images indeed are also XObjects, but no Form XObjects.
A "0" there would indicate, that the page directly invokes that image - and only in that case page.getImageBbox() will work correctly. I am failing to correctly raise an exception if that's not the case.

As I wrote:
Independently from the above, you shouldn't get a seg fault ...

[JorjMcKie]

I have spotted the code, where the seg fault is caused in MuPDF. I would like to send them a bug report, for which I need an example PDF . Do you agree to take the one you provided here ("fail.pdf")?

[gvnramu]

Yes

[JorjMcKie]

Yes

Thank you! I have submitted a bug report to MuPDF today. I also have modified their code such, that the problem is circumvented -- so far apparently without affecting other functionality.

If you under pressure, I can send you a wheel based on these modifications. Just tell me your system parameters ...

[JorjMcKie]

In the meantime, I already received a response from MuPDF:

What	Removed	Added
Status	UNCONFIRMED	CONFIRMED
Ever confirmed		1
CC		sebastian.rasmussen@artifex.com

< omitted data >

SUMMARY: AddressSanitizer: heap-buffer-overflow source/pdf/pdf-clean.c:534 in
pdf_redact_image_imp

So this is a confirmed problem, that has yet to be looked into.

[gvnramu]

Thanks for your quick response and support

If you under pressure, I can send you a wheel based on these modifications. Just tell me your system parameters ...

Yes... My system parameters

OS: Ubuntu 18.04.4 LTS
Release: 18.04
Codename: bionic
Architecture: x86-64

Python 3.7.4 (default, Aug 13 2019, 20:35:49)
[GCC 7.3.0]
linux

[gvnramu]

In the meantime, I already received a response from MuPDF:

What Removed Added
Status UNCONFIRMED CONFIRMED
Ever confirmed   1
CC   sebastian.rasmussen@artifex.com

< omitted data >

SUMMARY: AddressSanitizer: heap-buffer-overflow source/pdf/pdf-clean.c:534 in
pdf_redact_image_imp

So this is a confirmed problem, that has yet to be looked into.

I've read this comment just now in mupdf. Sebastian Rasmussens said the issue is fixed. Please confirm and remove the example pdf file you attached int the bug report

[JorjMcKie]

@gvnramu - no worries about the example file: they already have removed the data I submitted.

As far as problem fixing is concerned:
They have fixed the problem in their development repositoy on GitHub. That version is not generally / officially available yet however.
In the meantime, I have tried to make a wheel for you which circumvents the problematic MuPDF code. However I learned, that this only worksunder Windows - Linux admonishes a "memory corruption".
What I will try next is getting the unofficial GitHub Mupdf source and go from there ...

[JorjMcKie]

Looks good! Here is the wheel based on that new MuPDF version.
Please remember: it is preliminary. Created for your purposes - in the hope that you will give me feedback about anything unusual / dubious.
I have tested your fail.pdf and it works.
It works even better now: you can also get correct image bboxes for those images, that are contained in "Form XObjects"! I.e. those not containing a "0" as the last entry.
If you apply redactions which surround one of those images, they will also be removed - like the ones inserted directly on the page.
Impressive job done by those MuPDF guys, indeed. They also reacted at a speed which I haven't experienced so far.
PyMuPDF-1.17.1-cp37-cp37m-linux_x86_64.whl.zip

To install, please first unzip, and then do sudo python3.7 -m pip install -U PyMuPDF-1.17.1-cp37-cp37m-linux_x86_64.whl.
As you see, there is a verison update with an increased patch number.

[gvnramu]

Great!!! Working like a charm. Thank you so much @JorjMcKie
Waiting for this version release in PyPI and closing the issue