How to use pyAV plugin to export as GIF, and avoid color loss?

[BootsManOut]

Hello,
imageio uses PIL's quantization method to create the color palette when saving as GIF.
However that quantization method can create color loss in multiple instances.

For example importing this GIF with imageio:

And then saving it with imageio creates this result:

I work with PIL, modify the images with PIL, and afterwards I would like to use imageio and the pyAV plugin, to save the animation as GIF using the FFMPEG quantization method, since it does not have the same issues with color loss.
Something as described in this post:
#220

My knowledge doesn't reach nearly deep enough to understand how to use the plugin with imageio and save it.
How can I code that?

Let's say I have this as my base code, but I would like to save the images using pyAV instead?

from PIL import Image as Img
import numpy as np
import imageio.v3 as iio

PILimages = []
for i in range(0,56):
    digit = str(i)
    PILimages.append(np.asarray(Img.open("frame "+digit+".png")))

iio.imwrite("new gif.gif", PILimages)#instead of saving with imageio, I would like to use pyAV

[FirefoxMetzger]

Interesting find @BootsManOut . Which version of pillow and ImageIO do you have installed? I just tried it in my dev environment and I can't reproduce your artifacts:

import imageio.v3 as iio

frames = iio.imread("https://user-images.githubusercontent.com/63544082/240636834-4b2ed1f7-0730-433f-b18e-1705f9f4d62d.gif")
iio.imwrite("foo.gif", frames, loop=0, duration=20)

Could you share your exact code so I can see what might be going sideways?

---

To answer your original question, you would use FFMPEG's filters to compute an optimized color palette and create the GIF using the result:

    iio.imwrite(
        "new_gif.gif",
        PILimages,
        plugin="pyav",
        codec="gif",
        fps=50,
        out_pixel_format="pal8",
        filter_graph=(
            {  # Nodes
                "split": ("split", ""),
                "palettegen": ("palettegen", ""),
                "paletteuse": ("paletteuse", ""),
            },
            [  # Edges
                ("video_in", "split", 0, 0),
                ("split", "palettegen", 0, 0),
                ("split", "paletteuse", 1, 0),
                ("palettegen", "paletteuse", 0, 1),
                ("paletteuse", "video_out", 0, 0),
            ],
        ),
    )

I never explicitly advertized this in the examples, however, because it triggers a log message in either pyAV or FFMPEG somewhere saying that frames with changing size may be unsupported by some filters. This is a bit annoying (though harmless as far as I know). I have not yet managed to locate where this message comes from, how to disable it, or to calm FFMPEG/pyAV and stop the message. Hence, me not having made an official example out of it (yet).

[BootsManOut]

Hello @FirefoxMetzger ,

Thank you for the sample code. Does it not work for conversion of RGBA images? If I don't convert the images to RGB before using the save function you described, I get an error message because it doesn't expect a 4 dimensional array that includes the alpha value (ValueError: could not broadcast input array from shape (518,518,4) into shape (518,518,3)).
I would need to be able to convert RGBA images, that's where the issue with the PIL quantization lies to begin with.

Also when I convert to RGB first and then save it, it changes the dimensions. How could I set the dimensions manually, please?
This was the result:

You are not reproducing the artifact, because you are not converting anything. You import the GIF in Palette mode and then save it in palette mode. The quantization and color loss happens when you convert RGBA images to P images.
You can recreate it by using the function in my original post, and by importing the animation from a png image sequence.
If you want, here are the frames of the gif animation as RGBA frames:
ex sequence.zip
So you can simply change the import line in my sample code:

PILimages.append(np.asarray(Img.open("ex sequence/ex."+digit+".png")))

[Ai-Himmel]

Hello @FirefoxMetzger ,

Thank you for the sample code. Does it not work for conversion of RGBA images? If I don't convert the images to RGB before using the save function you described, I get an error message because it doesn't expect a 4 dimensional array that includes the alpha value (ValueError: could not broadcast input array from shape (518,518,4) into shape (518,518,3)). I would need to be able to convert RGBA images, that's where the issue with the PIL quantization lies to begin with.

Also when I convert to RGB first and then save it, it changes the dimensions. How could I set the dimensions manually, please? This was the result:

You are not reproducing the artifact, because you are not converting anything. You import the GIF in Palette mode and then save it in palette mode. The quantization and color loss happens when you convert RGBA images to P images. You can recreate it by using the function in my original post, and by importing the animation from a png image sequence. If you want, here are the frames of the gif animation as RGBA frames: ex sequence.zip So you can simply change the import line in my sample code:

PILimages.append(np.asarray(Img.open("ex sequence/ex."+digit+".png")))

For the RGBA part, you can add an extra in_pixel_format="rgba" parameter to solve this problem. The later one may be the probelem as FirefoxMetzger said there is something wrong with pyAV or FFmpeg

[FirefoxMetzger]

or the RGBA part, you can add an extra in_pixel_format="rgba" parameter to solve this problem.

Thanks, @Ai-Himmel 🚀 spot on as usual.

For future reference, pyAV understands all pixel formats supported by FFMPEG. A full list is available here in the docs of libAV. You would pass these to in_pixel_format without the AV_PIX_FMT_ prefix, e.g. in_pixel_format="rgb24" for 8-bit per channel RGB images (which is our default).

Note that some of the formats in the list of libAV are not useful in a numpy context, because they can not be represented as a strided array (it is impossible for numpy to represent these pixel formats). However, most of them should work.

You are not reproducing the artifact, because you are not converting anything. You import the GIF in Palette mode and then save it in palette mode.

You are confusing ImageIO for Pillow @BootsManOut :) We (ImageIO) do rely on Pillow for decoding the GIF, but we do things slightly differently before returning data to the user. In particular, we will un-palette/de-pallete images before returning them and you will get a valid RGB(A) frame, not an image in palette mode.

Similarly, when writing we take a RGB(A) image, ask pillow to convert it to palette mode and then write it into the GIF.

If you don't just want to take my word for this, you can add an intermediate step of writing each frame to PNG and then reading from PNG to build the GIF:

>>> import imageio.v3 as iio
>>> raw_frames = iio.imread("https://user-images.githubusercontent.com/63544082/240636834-4b2ed1f7-0730-433f-b18e-1705f9f4d62d.gif") 
>>> for idx, frame in enumerate(raw_frames):                                                                                         
...     iio.imwrite(f"_foo/{idx:03d}.png", frame)                                                                                    
... 
>>> frames = [iio.imread(f"_foo/{idx:03d}.png") for idx in range(56)]
>>> iio.imwrite("foo.gif", frames, duration=20, loop=0)

The quantization and color loss happens when you convert RGBA images to P images.

This is a different story, and I can indeed reproduce this on master:

>>> raw_frames = iio.imread("https://user-images.githubusercontent.com/63544082/240636834-4b2ed1f7-0730-433f-b18e-1705f9f4d62d.gif", mode="RGBA") 
>>> for idx, frame in enumerate(raw_frames):                                                                                                      
...     iio.imwrite(f"_foo/{idx:03d}.png", frame)
... 
>>> iio.imwrite("foo.gif", frames, duration=20, loop=0)

I can't say for certain why the quantization differs in this case. My guess is that RGBA takes a different path inside pillow. I will try to investigate on the weekend, but can't make any promises on when I will have an answer.

[BootsManOut]

For the RGBA part, you can add an extra in_pixel_format="rgba" parameter to solve this problem. The later one may be the probelem as FirefoxMetzger said there is something wrong with pyAV or FFmpeg

Thank you very much! That works!

You are confusing ImageIO for Pillow @BootsManOut :) We (ImageIO) do rely on Pillow for decoding the GIF, but we do things slightly differently before returning data to the user. In particular, we will un-palette/de-pallete images before returning them and you will get a valid RGB(A) frame, not an image in palette mode.

Sure in that case I have to correct it and say, that the artifact is not reproduced because you converted 'RGB' to 'P', and not 'RBGA' to 'P'.

I can't say for certain why the quantization differs in this case. My guess is that RGBA takes a different path inside pillow. I will try to investigate on the weekend, but can't make any promises on when I will have an answer.

Unless you have time to rewrite the quantization process and rebuild PIL from source, this is not going to be fixed any time soon I'm afraid. It's been known for a while that the library 'libimagequant' uses too few colors for quantization in certain cases.

The only purpose of my post is to find out if it's possible to do it via pyav and ffmpeg.
The function you provided works great! Only is there a way for me to fix the size distortion that happens?

Is there anything I can change in the code to correct it?

from PIL import Image as Img
import numpy as np
import imageio.v3 as iio

PILimages = []
for i in range(0,56):
    digit = str(i)
    if len(digit) == 1:
        digit = "0" + digit
    PILimages.append(np.asarray(Img.open("ex sequence/ex."+digit+".png")))

# iio.imwrite("new gif.gif", PILimages)
iio.imwrite(
        "new_gif.gif",
        PILimages,
        plugin="pyav",
        codec="gif",
        fps=50,
        in_pixel_format="rgba",
        out_pixel_format="pal8",
        filter_graph=(
            {  # Nodes
                "split": ("split", ""),
                "palettegen": ("palettegen", ""),
                "paletteuse": ("paletteuse", ""),
            },
            [  # Edges
                ("video_in", "split", 0, 0),
                ("split", "palettegen", 0, 0),
                ("split", "paletteuse", 1, 0),
                ("palettegen", "paletteuse", 0, 1),
                ("paletteuse", "video_out", 0, 0),
            ],
        ),
    )

This is the output:

[Ai-Himmel]

Is there anything I can change in the code to correct it?

See #1003

[FirefoxMetzger]

@BootsManOut Alrighty, if the RGBA -> P quantization issue is already known in PIL and the other issue (size change) is tracked by the issue created by @Ai-Himmel , shall we close this issue, or is there anything left that we have to discuss / resolve here?

[BootsManOut]

@FirefoxMetzger Before closing the thread, I would only like to know if I have to include an FFMPEG.exe file to the project or if necessary dll files are already included with pyav. I'm aware that for example when using the FFMPY library you need to include the FFMPEG.exe file and add this line to the export parameters to function properly: executable='FFMPEG\\bin\\ffmpeg.exe'
Is there something similar I need to pay attention to when using pyav?
I will create an executable from the code and distribute the application to computers who may not have FFMPEG installed. Is importing the pyav library enough for it to work?

[FirefoxMetzger]

pyav does not need a dedicated FFMPEG.exe, no. It directly binds into the libav version it was compiled with and you "only" have to ship all the necessary pyd and pxd files that make up the project.

That said, the project contains several extension modules, which makes it platform-specific. Be sure to account for this when you choose how to package the project, e.g., don't expect an app packaged on Windows (with pre-compiled extension modules for Windows) to work on Linux.

Also, be mindful of the licenses that apply to the various video codecs within ffmpeg/pyav if you choose to distribute it - especially when doing so commercially.

[BootsManOut]

Thank you for the heads-up.
Indeed, I see that FFMPEG has quite a limited license when it comes to distribution, so this is something I will have to consider.
Thank you very much for your help!
You can close the thread from my side.
Thank you.