Optimize jxl/avif motion pictures

[DeepTulip]

Open a 3.4MB gif image and open it with almost no delay.
Convert this gif to a 700KB jxl image and open it will have a 2-3 second delay.
Convert this gif to a 270KB avif image and oculante cannot open it, it is always in the "loading" state. Of course this avif image will play normally in chrome.

test-img.zip

win-10 64

[woelper]

Thanks for reporting and attaching images!
Oculante has a quite inefficient design for loading animations - all frames of an animation are being decoded first, then sent off to be played. This amplifies the problem if you open very long and high-res image sequences. 2-3 seconds sounds far too long though. Since in your case you are using the same image sequence, it is quite likely that the decoder for jpeg xl is less optimized than the gif decoder. Also the source gif was probably in a palette 256 color format and the jpeg xl one is 8 bit per channel with alpha channel, making frames significantly larger in memory and thus slower to decode. I can't promise I can do anything in this case, as I am relying on the jxl-oxide library to decode the images. I'll try to see if I can do anything to speed this up though.

Avif animations are not yet supported, only jxl and gif have animation support.

[DeepTulip]

Thanks for the reply. Maybe you can refer to the scheme used by JPEGView. I have tried almost all image viewers recently and only JPEGView supports jxl/avif perfectly and has excellent performance, displaying jxl/avif with almost 0 latency.

[woelper]

I actually just switched from the jpegxl library very recently. Although I don't have any data to support this, I guess that the previous library was faster. I was willing to take a performance hit as I want to minimize the amount of non-rust libraries used. This is not a "religious" decision, but rather makes building the application for many different operating systems and without dependencies much easier for me. jxl-oxide is still very young, but I believe that decoding speed will become better over time.

I've already looked at decoding frames in parallel, but had no luck so far. Maybe there are other tricks which make the experience better. I am happy to work on avif animations in the future.

[woelper]

Good news, I rewrote the animation handling and now send the first frame as soon as it's loaded. This was a bit tricky as I need to compensate frame delays with the time it actually took to decode the frame.
Now animation loading happens without any perceivable delay and is likely to get even faster if the decoding library improves.

This touches all other loading code, so I will have to test this a bit before releasing.

[DeepTulip]

Good. I'll add some other feedback on use.

1, the window size seems to only revert to the size of the last time the window was opened, the window size can be adjusted with the mouse, but it does not affect the image display. This does not make sense.
There are usually three modes of window display, 1, maximize the window, display as many pictures as possible, exclusive to the screen. 2, display pictures according to their size, or shrink them if they are larger than the screen. 3, limit the window size, but display the window according to the picture ratio
I prefer mode-2

2, Cleaner UI
I just view the picture in 99% of the scenes and don't need to edit it. So I would like to add a mode that does not show the editor controls.
By the same token, I don't want to see the title bar either. Dragging an image is dragging a window.
So when I want to view an image quickly, I want it to be displayed quickly and not show anything other than the image. When the mouse is on the left side of the screen, the image is displayed on the right side of the screen against the edge, and vice versa. This minimizes the distraction of viewing the image from the current operation.

I have three scenarios for using the image viewer. 1, quick view. 2, comic mode. 3 professional image viewer with editing capabilities.
In 1, we need a fast, clean UI that doesn't interfere with subsequent operations.
In 2, we need images to be displayed intelligently, zoomed, switched display mode, pre-reading
In 3, what we need is more functions and powerful
Invisible 4, switch between these three scenarios, such as switching to comic mode when in quick view mode.

3, it takes up too much memory.
On my computer, mpv uses only 50MB to play a large video. while oculante uses 350MB to display a small picture. this indirectly reflects the performance of a software.

4, can we use asynchronous solution like rayon library when decoding multi-frame pictures?

5. Activate the window after it is displayed. Many times oculante opens the image but does not show it because it is behind other windows.

[woelper]

1 the window size seems to only revert to the size of the last time the window was opened, the window size can be adjusted with the mouse, but it does not affect the image display. This does not make sense.

This is dependent on the use case. I've had feedback that people don't want their image to be scaled (if they are picking a color for example) when the window is resized. This could be an option of course.

In general, Oculante is more geared towards power users/image professionals, but I am happy to do changes that fit a different audience.

In 1, we need a fast, clean UI that doesn't interfere with subsequent operations.
In 2, we need images to be displayed intelligently, zoomed, switched display mode, pre-reading
In 3, what we need is more functions and powerful
Invisible 4, switch between these three scenarios, such as switching to comic mode when in quick view mode.

This is great feedback.

3, it takes up too much memory.
On my computer, mpv uses only 50MB to play a large video. while oculante uses 350MB to display a small picture. this indirectly reflects the performance of a software.

I am not sure how memory consumption reflects performance. I'd need to dig deeper into concrete examples/benchmarks. I am sure mpv is much more optimized for video playback, where Oculante is not. It was originally made to persist frame data for non-destructive editing, and also caches images, which mpv does not. This can increase memory consumption quite a bit, but it is comparing apples with oranges.

[DeepTulip]

I tested opening a 30KB jpg image with oculante and it used 567MB of memory. No matter what method is used to parse this jpg, it does not use this much memory. And the folder where the jpg is located only has this one image, so this is not a pre-reading or caching usage either.
If a software reads a 1-byte file and uses 20G of memory, I think the software is low performance because the operation that consumes that memory is a low performance.

[EugeneVert]

... opening a 30KB jpg image with oculante and it used 567MB of memory.

Just tried oculate from the last release on linux

While viewing the same 300Kb jpeg image, pmap reports:

name	virtual memory (Kb)	RSS (Kb)
mpv	1'424'096	112'688
oculante	1'370'216	165'332

I don't get such a big difference in memory consumption.
My understanding is that since oculante links statically to most libraries rather than linking to them, it uses less shared memory, but it also uses less virtual memory.

[woelper]

I could also not reproduce this, at least not on Mac & Linux. There are some things embedded in Oculante such as a font with wide language coverage and icons which bring the base memory to about 30MB, but opening a small jpeg should just use a couple of MB - basically the size in pixels times 4 channels times 8bpp. This is loaded, plus cached, plus put into a texture - so you would roughly see 30MB app memory plus 3 times the decoded image. And yes, there could be optimisations done!

On a note, the size of a jpeg on disk, like your example of 30kB image, is rather meaningless. You can easily create a 85x85px jpeg with 30kB size, and a 2300x2300 jpeg that is also 30kB - both are "small" but have drastically different memory usage - the first uses ~30kB mem, the latter ~20000kB:

If you believe you found something unusual, I'd kindly ask you to upload a sample picture, preferably in a separate issue, and describe oculante version and operating system so I can try to reproduce that.

[DeepTulip]

I have already provided my system information before - WIN 10 64, if you can't in testing this platform, uploading images is pointless. I forget where that jpg is saved, but I'm sure it's about 500*500 in size.

[woelper]

I just tested on Windows 10 64 with the 0.6.64 build from the release page, and opening a 500x500 jpeg uses 58.7MB memory. While this can of course be optimized I can not reproduce your case and thus can't do anything about that unless you can provide a better test scenario and sample data.

[DeepTulip]

Try these two images.

This is a screenshot of Process Explorer::

This is a screenshot of Task Manager:

Interestingly, if only one image is opened, they all only take up ~350MB of memory.
But if you open two at the same time, one will stay ~350MB and one will spike to 800+MB.

test-img.zip

---

And when I call GDI+ with AutoHotkey to open images, both images only take up 4MB of memory, most of which is consumed by AutoHotkey itself.

Another thing I noticed is that when oculante is in the background with no operations, it still continues to consume 1-2% of the CPU, why is that?

At some point, when I tested again, the memory usage of 1.jpg/2.jpg was less than 100MB. However, opening both at the same time, there was still an image that used 800+MB.

[woelper]

1.jpg has 64.2 MB memory usage, 2.JPEG has 68.8 MB memory usage on my system. I still can't reproduce the issue you are seeing with these images.

[woelper]

Another thing I noticed is that when oculante is in the background with no operations, it still continues to consume 1-2% of the CPU, why is that?

Oculante uses a library that is similar to a game engine to draw part of its UI. It has an update loop which runs all the time. On all systems except Windows, there is a lazy_loop() parameter enabled to update only when certain events are received, reducing CPU usage. As far as I recall, there was an issue when enabling it on Windows, so Windows uses slightly more CPU than Linux, MacOS and NetBSD.

oculante/src/main.rs

Line 70 in 4001c87

window_config = window_config.vsync(true).high_dpi(true);

[DeepTulip]

I can't provide any more debugging information. But I'm sure that other software opens both images fine. And, no matter how many times they are opened, the memory used by these software does not change.

AutoHotkey => ~4MB
qimgv => ~25MB
JPEGView => ~18MB
ffplay.exe => ~30MB

[woelper]

I've closed this as the original request was done. Avif animation support has a new issue.
I suspect that the memory usage you are seeing might be related to caching - in the next release, try setting the images to cache to 0 in the settings and no caching whatsoever should occur, only keeping the current image in memory.