doc.xref_stream(xref).decode().splitlines() does NOT split the line

[ragebear00]

Description of the bug

after upgrade to PyMuPDF 1.24.2, the following code will print one paragraph for each page, not split lines.

Downgrade to PyMuPDF 1.22.5, doc.xref_stream(xref).decode().splitlines() works again, and it will print "line", not "paragraph" for one page.

How to reproduce the bug

    for page in doc:
        page.clean_contents()  # cleanup page painting commands
        xref = page.get_contents()[0]  # get xref of the resulting source
        cont0 = doc.xref_stream(xref).decode("utf8").splitlines()  # and read it as lines of strings
  
        for line in cont0:
            print(line)

PyMuPDF version

1.24.2

Operating system

Windows

Python version

3.10

[JorjMcKie]

This is not a bug.
MuPDF has changed to no longer generate line breaks between PDF commands, but spaces.

[JorjMcKie]

@huahua74 There still is a way to get a page's contents with line breaks. It is with less comfort - on the other hand, MuPDF team had good reasons for their change.

Line breaks between PDF commands are generated as part of saving the file with the option pretty=True. So if you use this little function with the page as parameter, it will return you the contents as desired:

def pretty_contents(page):
    """Invoke with some PDF page."""
    doc = page.parent  # owing document
    new = fitz.open()  # make temp PDF, copy page into it
    new.insert_pdf(doc, from_page=page.number, to_page=page.number)

    # save temp pdf with pretty option
    pdfdata = new.tobytes(pretty=True, clean=True)
    new.close()  # close and re-open
    new = fitz.open("pdf", pdfdata)

    # read and return /Contents ... now with linebreaks
    contents = new[0].read_contents()
    new.close()
    return contents

contents_lines = pretty_contents(page).splitlines()

[ragebear00]

many thanks! it is extremely helpful! I also find your reply earlier without "splitline()". Runing the script below under PyMuPDF 1.22.5 and 1.24 will end with different results. The following work smooth with 1.24 but not 1.22.5.

  page.clean_contents()
  xref = page.get_contents()[0]  # get xref of resulting /Contents object
  cont = bytearray(page.read_contents())  # read the contents source as a (modifyable) bytearray
  if cont.find(b"/Subtype/Watermark") > 0:  # this will confirm a marked-content watermark is present
      print("marked-content watermark present")
  while True:
      i1 = cont.find(b"/Artifact")  # start of definition
      if i1 < 0: break  # none more left: done
      i2 = cont.find(b"EMC", i1)  # end of definition
      cont[i1-2 : i2+3] = b""  # remove the full definition source "q ... EMC"
  doc.update_stream(xref, cont)  # replace the original source
  doc.ez_save("x.pdf")  # save to new file

[firezym]

I comment out the page.clean_contents(), then the splitlines works ok now.

The code iss revised a little bit from previous code in this discussion #1855. It works ok on current pymupdf release.

pip install PyMuPDF

import pymupdf

def process_page(page : pymupdf.Page):
    """Process one page."""
    # doc = page.parent  # the page's owning document
    # page.clean_contents()  # clean page painting syntax
    xref = page.get_contents()[0]  # get xref of resulting /Contents
    changed = 0  # this will be returned
    # read sanitized contents, splitted by line breaks
    cont_lines = page.read_contents().splitlines()
    print(len(cont_lines))
    # print(cont_lines)
    for i in range(len(cont_lines)):  # iterate over the lines
        line = cont_lines[i]
        # print(line)
        if not (line.startswith(b"/Artifact") and b"/Watermark" in line):
            continue  # this was not for us
        # line number i starts the definition, j ends it:
        print(line)
        j = cont_lines.index(b"EMC", i)
        for k in range(i, j):
            # look for image / xobject invocations in this line range
            do_line = cont_lines[k]
            if do_line.endswith(b"Do"):  # this invokes an image / xobject
                cont_lines[k] = b""  # remove / empty this line
                changed += 1
    if changed > 0:  # if we did anything, write back modified /Contents
        doc.update_stream(xref, b"\n".join(cont_lines))
    return changed

fpath = 'your_pdf_file_path/file_name.pdf'
doc = pymupdf.open(fpath)
changed = 0  # indicates successful removals
for page in doc:
    changed += process_page(page)  # increase number of changes
if changed > 0:
    x = "s" if doc.page_count > 1 else ""
    print(f"{changed} watermarks have been removed on {doc.page_count} page{x}.")
    doc.ez_save(doc.name.replace(".pdf", "-nowm.pdf"))
else:
    print("Nothing to change")

[JorjMcKie]

There have been changes to the underlying MuPDF code for clean_contents():
It does no longer generate line breaks between PDF contents commands, but instead 1 space.
This has happened for an overall improved compressibility of content streams.

Here is a little trick to get back these line breaks:

import pymupdf


def pretty_contents(page):
    doc = page.parent
    new = pymupdf.open()
    new.insert_pdf(doc, from_page=page.number, to_page=page.number)
    pdfdata = new.tobytes(pretty=True, clean=True)
    new.close()
    new = pymupdf.open("pdf", pdfdata)
    xrefs = new[0].get_contents()
    if not xrefs:
        return b""
    contents = new.xref_stream(xrefs[0])
    new.close()
    return contents


doc = pymupdf.open("input.pdf")
page = doc[0]
cont = pretty_contents(page)
# "cont" has line breaks again