Applying Matrix zoom flips antialiasing of saved pixmap

[mlissner]

Describe the bug (mandatory)

I'm dabbling with pixmaps for the first time, and I've found something a bit baffling in the following file:

rectangles_yes.pdf

If you do the following:

import fitz
from fitz import Document, Matrix, Rect
pdf = Document(file)
page = pdf[0]
pixmap = page.get_pixmap(clip=Rect(412.54998779296875, 480.6099853515625, 437.8699951171875, 494.39996337890625))
pixmap.save("some_file.png")

You get a file that looks like this:

On a black background at least (if you use Github's dark theme), you can see there are some white/gray lines along the left, top, and bottom, that form anti-aliasing for the rectangle.

Now, if you switch the code above to do a zoom transformation:

pixmap = page.get_pixmap(
    matrix=Matrix(2, 2), 
    clip=Rect(412.54998779296875, 480.6099853515625, 437.8699951171875, 494.39996337890625),
)
pixmap.save("zoomed_file.png")

You get the zoomed rectangle, but if you look closely, you'll see that it has an left-right flip on the anti-aliasing (it could have a top-bottom flip too, but we wouldn't notice since those are the same):

I was surprised there was any anti-aliasing, let alone that it flipped.

To Reproduce (mandatory)

Use the code and files above.

Expected behavior (optional)

I didn't expect anti-aliasing on this at all, and I definitely didn't expect the zoom to flip it.

Additional context (optional)

This shows up in pixmap.samples and pixmap.samples_mv too.

I plan to work around this by just eliminating a few pixels before making the pixmal, but it's all pretty surprising.

Your configuration (mandatory)

• Operating system, potentially version and bitness

Linux, Ubuntu, 64 bit

• Python version, bitness

3.8, 64 bit

• PyMuPDF version, installation method (wheel or generated from source).

1.19.0

[mlissner]

Oooh, just found part of the answer. I don't know why the aliasing flips on zoom, but this looks like it could disable anti-aliasing completely, which is exciting: #467 (comment)

Maybe a call out in the get_pixmap() call would help folks find this?

[JorjMcKie]

I don't know why the aliasing flips on zoom

It does not do that. The real problem is that the rectangle is simply not that precisely defined as to avoid every single non-black pixel at its borders "shining" into the area.
Choosing a slightly smaller (sub-) rectangle will not exhibit the effect: fitz.Rect(413, 481, 437, 493).

When making the pixmap, you should choose colorpace=fitz.csGRAY. This makes a much smaller pixmap.
Then you can check, whether it only contains one color: pixmap.is_monochrome. This should be much faster, than walking through the pixels yourself (the code behind is written in C).
It currently return true if it is a GRAY pixmap containing only black or only white pixels.
But I think I will make an extended version that supports RGB and CMYK pixmaps too, and check if there is only one color - something like is_monocolor.

If you do this check with that somewhat smaller rect above, you would see these results:

dpi 96 is monochrome: False
dpi 100 is monochrome: True
dpi 200 is monochrome: True
dpi 300 is monochrome: True
dpi 500 is monochrome: True

Where dpi is the short form of the Matrix fitz.Matrix(dpi/72, dpi/72).
Next version 1.19.2 will also provide the dpi parameter directly.

[JorjMcKie]

BTW the different pixmap.samples* object all refer to the same thing: it is a memory area deep inside MuPDF.
pix.samples is a bytes copy of it. The other two, samples_mv and samples_ptr are addressing that area only in two different ways available in Python.
The latter is just a pointer containing the start address. It seems to be usable only by Qt version 6 currently. But in this case it saves two in-memory copies of that potentially large area. This may become notable.

[mlissner]

My current approach is to use the standard deviation of the colors, but it's a bit of a pain since I need to average the RGB tuples. The std deviation lets me set a threshold for things that might have a weird pixel here or there. I already limit to a random sample of the first 1,000 pixels, since it can get far too slow on really big images.

If there were an is_monocolor attribtue on pixmaps, I'd definitely use it instead.

[JorjMcKie]

Confirming:
Next v1.19.2 will have two new attributes for pixmaps: is_monochrome and is_unicolor.

• is_monochrome is true for pixmaps with colorspace gray which only contain black and white pixels.
• is_unicolor is true if there exists only one (arbitrary) color in the pixmap (works for any colorspace).

[JorjMcKie]

Both the above are implemented as C functions and should be fast enough. This checks a 100 x 100 = 10,000 pixel (white) pixmap:

In [6]: pix
Out[6]: Pixmap(DeviceRGB, IRect(100, 100, 200, 200), 0)
In [7]: pix.size
Out[7]: 30088
In [8]: len(pix.samples_mv)
Out[8]: 30000
In [9]: pix.w * pix.h
Out[9]: 10000
In [10]: pix.is_unicolor
Out[10]: True
In [11]: %timeit pix.is_unicolor
32 µs ± 332 ns per loop (mean ± std. dev. of 7 runs, 10000 loops each)
In [12]:

[mlissner]

Yes, that is definitely fast enough. Amazing!

[JorjMcKie]

Resolved by version 1.19.2.

[mlissner]

Just a small follow up here. I switched out my existing approach to this:

# Get a memoryview of the pixels. This is a 1-D array of grayscale
# values.
pixels = pixmap.samples_mv
if len(pixels) > 1000:
    # Big pixmap. Use random sampling to select fewer pixels
    pixels = random.sample(pixels, 1000)
std_dev = statistics.stdev(pixels)
if std_dev > 0:
    # There's some degree of variation in the grayscale values of the
    # pixels. ∴ it's not a uniform box and it's not a bad redaction.
    continue

And swapped in the new is_unicolor attribute:

if not pixmap.is_unicolor:
    # There's some degree of variation in the colors of the pixels.
    # ∴ it's not a uniform box and it's not a bad redaction.
    continue

Some pros/cons:

• Pro: Speed is same as Python version
• Pro: Much simpler code.
• Pro: No need to do random sampling of larger boxes to make it faster
• Pro: Able to use an RGB pixmap instead of a grayscale (speed seems to be identical with both color spaces). This makes it more accurate.
• Con: I had my std dev function set to zero (only allow completely unicolor boxes), but I liked having the ability in the future to allow some amount of slop in the color of the boxes. I lose that flexibility.

Anyway, as hoped, it's a nice improvement overall. Thank you!

[JorjMcKie]

There also is a Pixmap.color_count() method now which counts (or returns) the number of unique colors.
Maybe that one could be extended to also return the number of occurrences of each color. So one could determine that the most frequent color occurs in > 95% of all pixels.

A (probably bad) approximation of this might also be the compressibility ratio len(gzip.compress(pix.samples_mv))/len(pix.samples_mv).

[JorjMcKie]

I am experimenting with the said new method.
I hope you can deal with Jupyter notebooks - attached one here. VS Code is excellent for this.
redact-pixmap.zip

Otherwise please let me know and I will make a PDF or something.

[JorjMcKie]

Here is a (new) PDF version:
redact-pixmap.pdf

There are the following news / changes:

• Pixmap.color_count(True) now delivers a list of dictionaries {pixel: count} for each color, instead of just the color pixels.
• There is a new method Pixmap.color_topusage() which returns a float between 0 and 1. Use it a replacement for Pixmap.is_unicolor when you need a fuzzy answer: if the method returns e.g. 0.95, then you know that the most frequent color is shown by 95% of all pixels.

This should be a good replacement for your prior use of random and statistics.
Its performance is even (much) better than that of color_count() - provided we are dealing with images that are "almost unicolor".

Contrary to color_count(), color_topusage() makes no use of Python internally. It uses the C builtin function qsort (quicksort) on the pixels, which profits a lot from many pixels being equal.

[JorjMcKie]

New wheels with the new method Pixmap.color_topusage() are here as always.

Note: I removed the previously announced color_dist() again.

[mlissner]

Wow, this all sounds great. Thank you as always, @JorjMcKie!

[JorjMcKie]

New version 1.19.3 is being uploaded to PyPI.

You may want to look at the example code snippet for method Rect.torect():
It shows how to detect color equality between text and its environment on a page ... using only one full-page pixmap.
This is possible using the new clip parameter available for color_topusage(clip=rect) - and similarly for color_count().