How to test videos?

[leotrs]

In the future, we will need to check the videos generated by manim for correctness.

One way to go about it, is to write tests in such a way that they output a single frame in PNG format. And then we can load that PNG in python and check everything, even pixel by pixel if needed. For example, it would be fairly easy to write a test that checks for "this PNG/frame contains a red square of such and such size over a black background".

[yoshiask]

it would be fairly easy to write a test that checks for "this PNG/frame contains a red square of such and such size over a black background"

Until you take into account compression and stuff, at which point something this simple becomes pretty ridiculous

[leotrs]

@yoshiask you raise a good point. Can manim currently output bitmap formats? If it were able to output a bitmap (which is basically an array), then we don't need to worry about compression. This could be an option used only during testing, but that's OK because if the bitmap version is good, then we don't really need to test the compression step.

[huguesdevimeux]

Actually there is get_frame (l. 157, scene.py) that returns an array of the pixels.

EDIT : There is also -s to get the last frame of the video

[leotrs]

Excellent! It shouldn't be too hard to put together a proof-of-concept example test using this idea of testing individual frames.

[huguesdevimeux]

Ok, I will try to do that. We could even use that method of testing instead of the one which have been described in #22.

[leotrs]

Yes, #22 was supposed to be a quick temporary patch.

I still think we should keep those sample scenes tho!

[huguesdevimeux]

from manimlib.imports import *

class Test(Scene): 
    def construct(self): 
        square = Circle()
        self.play(ShowCreation(square))

a = Test()
b = a.get_frame()
# np.save('test.npy', b)
d = np.load('test.npy')
print((b == d).all())

Here is an example of a test. Basically it compares a frame with another saved previously. We could have like a folder with some .npy files of frames of a bunch of scenes, and run some tests comparing with them.

What do you think ?

[eulertour]

If the --save_pngs flag did anything we could just craft the necessary config to make a scene save its frames as pngs, run it with a low frame rate, and compare the pngs against a pre-rendered set of them.

But the --save_pngs flag is broken. https://github.com/ManimCommunity/manim/blob/master/manimlib/scene/scene_file_writer.py#L40

[leotrs]

We could have like a folder with some .npy files of frames of a bunch of scenes, and run some tests comparing with them.

Just to clarify, your example test didn't really do this, right? It is testing a newly-rendered scene with itself, after loading it using numpy. This is more like testing numpy's save/load functions.

If we compare to a previously-rendered frame, then I'm all for this test!

[huguesdevimeux]

@leotrs No, my example compares a previously-rendered frame. This line # np.save('test.npy', b) was intended to show how I would save the NumPy array but won't be used in the test scripts (that's why I commented it).

Sorry, that was unclear

[huguesdevimeux]

If the --save_pngs flag did anything we could just craft the necessary config to make a scene save its frames as pngs, run it with a low frame rate, and compare the pngs against a pre-rendered set of them.

But the --save_pngs flag is broken. https://github.com/ManimCommunity/manim/blob/master/manimlib/scene/scene_file_writer.py#L40

As @yoshiask isn't the "compression and stuff" an issue if we use that ?

[leotrs]

@leotrs No, my example compares a previously-rendered frame. This line # np.save('test.npy', b) was intended to show how I would save the NumPy array

Oh I missed the commented line. This looks good to me. Are you volunteering a PR? :)

[yoshiask]

If the --save_pngs flag did anything we could just craft the necessary config to make a scene save its frames as pngs, run it with a low frame rate, and compare the pngs against a pre-rendered set of them.

But the --save_pngs flag is broken. https://github.com/ManimCommunity/manim/blob/master/manimlib/scene/scene_file_writer.py#L40

As @yoshiask isn't the "compression and stuff" an issue if we use that ?

Yes, it is. We'd have to compare the raw output (byte arrays) to what we'd expect the raw output to be. I doubt we can reliably use any image format.

[huguesdevimeux]

Yup. I will do that asap. I will do that in /tests and will create a new folder like /tests/test_data to store the previously-rendered scenes.
Tell me if you see any problem with doing that

[eulertour]

Yes, it is. We'd have to compare the raw output (byte arrays) to what we'd expect the raw output to be. I doubt we can reliably use any image format.

I found some articles on the topic that make it seem pretty doable.
https://www.pyimagesearch.com/2014/09/15/python-compare-two-images/
https://www.pyimagesearch.com/2017/06/19/image-difference-with-opencv-and-python/