Pictures app very slow with hig number of pictures

[patchlog]

The Pictures app is very slow when you have thousands of pictures.

It downloads a list of all images right when you enter the Pictures with a call to getimages.php. Mine was 15 mb of data and made me wait about 1 minute just to get to the top screen with only 4 albums.

Why do we need all of this data from the beginning?

[joe-hallenbeck]

Confirmed. I have all my pics since 2001 (about 18.000) on Owncloud. The Gallery-Startpage takes very long to load. On every reload "getimages.php" transferres about 10mb of data.

Debian Sid, Owncloud 5.0.0, Lighttpd 1.4.31, PHP 5.4.4, MariaDB 5.5, Chromium 25.0.1364.160

Screenshot below is on the local 1Gbit network:

[joe-hallenbeck]

If this amount of data is neccessary, it would be good to display at least a progress-bar - so users know, that something's going on. On slow connections like german DSL-Light (384kbit/s) or GSM transferring 10MB takes a while :)

This applies to the thumbnails as well - I changed imagepng(compression 9) and imagejpeg(quality 30) in core/image.php to reduce thumbnail quality and size.
I also changed the amount of pics shown in album-thumbs from 10 to 1 - it might be a nice gimmick to have a mouse-over-effect on album-thumbs, but these .PNGs are really big (about 600kB).

Thumbnail quality should be a setting in the adminitrators panel.

[joe-hallenbeck]

Feature Request: If the browser reports "mobile" (Android/iphone/etc) automatically reduce thumbnail quality and other data. Even if someone has a mobile-flatrate, most often bandwith is reduced after a certain amount of data.

[butonic]

currently getImages() is an initial call to fetch all available images. @icewind1991 I think the best idea would be to limit the result to a single album?

[joe-hallenbeck]

If I understand the code correctly getimages() sends an array of pathnames to all pictures (and some other information). This is text only and maybe cloud be compressed. Another approach would be to use relative pathnames instead of absolute ones for each picture.

[joe-hallenbeck]

Temporary Fix: Activate zlib.output_compression in php.ini - This reduces the amount of data to 1.4MB.
But with compression Ampache is completly broken. No Albums/Artists/etc are shown in Amarok, JustPlayer shows I/O-Stream-Error on connect.

[patchlog]

getimages should fetch only information about the current view. Even with compression it's likely that gathering the data for 10k images by scanning the fs will take a quite a few seconds and it's just unacceptable. I shouldn't have to wait more then 3 seconds to look at any page or else I'll just not use it.

[joe-hallenbeck]

As shown in the screenshot above gathering the data for my 18k images takes about 2.5 seconds on low-end hardware (Old Sempron64 1,8Ghz, 2.5Gb RAM). I think this is acceptable. But it makes a big difference (on slower networks) of sending 10Mb or 1.4Mb. Nevertheless I agree with you, that gathering/sending only the data of the current view is the most effective solution.

[bwoester]

This issue is by design. You simply can't load all data (whatever it is - filepathes in this case) from server and expect that to scale. Also, the suggestions given here won't fix the problem.

1. Compression: Only reduces the impact of the problem (by factor seven, using the numbers provided by @joe-hallenbeck as reference). But even 1 MB of data is way to much for displaying a page as simple as the gallery start page. And still, the more the gallery grows, the worse will be the performance.
2. Limiting the results to a single album as suggested by @butonic: Only moves the problem. Currently, performance degrades when using the gallery with many images. With this suggestion, the overall gallery performance might be better, but as soon as someone creates one huge album, performance for that album will degrade, because of the very same problem - loading too much data.

I think @mihaisecasiu point makes the most sense. Only load what is needed for displaying the current view - at least roughly. Basically, whenever a web service returns a list of resources, like getimages.php does, it should do so by the means of paged result lists. For example, the results of getimages.php could look like this:

{
    "list": [
        {
            "albumId": 1,
            "albumName": "Album 1",
            "images": {
                "list": [
                    "album_1/image1.jpeg",
                    "album_1/image2.jpeg",
                    "album_1/image3.jpeg"
                ],
                "prev": null,
                "next": "getimages.php?album=1&page=2"
            }
        },
        {
            "albumId": 2,
            "albumName": "Album 2",
            "images": {
                "list": [
                    "album_2/image1.jpeg",
                    "album_2/image2.jpeg",
                    "album_2/image3.jpeg"
                ],
                "prev": null,
                "next": "getimages.php?album=2&page=2"
            }
        }
    ],
    "prev": null,
    "next": "getimages.php?page=2"
}

The default's list size could be whatever seems reasonable. Let's say 50, only to have a number at hand. This would be good enough to render the gallery's start page and a bit more. Now, when the user starts scrolling down, or clicks a "show more" link, the gallery could issue a request to the URL provided by the "next" attribute.

Same for album view: We have enough data at hand to initially display one page and maybe a bit more. Depending on user's actions, the gallery could fetch that data on demand, instead of loading everything beforehand.

With this design, you don't have to worry about how much data is available on the server, because you always only transfer a fixed amount of data, thus keeping the app responsive. Of course, scalability restrictions caused by the the filesystem remain, but that is a whole other story. 😉

Downsides are, that the whole app becomes somewhat more complex. The client side portion needs to handle on demand loading when appropriate. Also, the backend now must be capable of sorting the data, because without a fixed ordering, paging isn't possible.

[ftree]

and if you have too many images you get an "Allowed Memory size" Error.
see #1121

[ftree]

I have rewritten the gallery app.
You can download it here: https://github.com/ftree/gallery2
This version loads only the actual folder. So performance depends not on the amount of files. I have testet it with about >70000 files
Also the images are not shown in the original size. Instead you can configure it. At the moment it is set to 1024px

[animalillo]

is this issue even taken into account?, this leaked into OC 6

[ftree]

I took a look into the code, and I have seen that there where some changes regarding the encrypted files.
But the problem that ALL Folders are loaded still exist.
This issue is labeled for OC5.0. So I hope that this issue wil lbe fixed sometimes. That should not be that much effort. It took me one day to reprogamm the application (Gallery2) for loading only the folder, the user want to see.

[pgrm]

Hi, I have a similar problem with all the pictures from my family which will be several 10-100 of thousands. The problem isn't the size of the file for me (26MB - 2.3 MB after Apache's GZip Compression) but rather the fact that it needs to collect all those pictures from the Database or maybe even the hard-drive itself. I have Owncloud running on a Core i5 Server with 16GB of memory and I don't see any CPU Usage-spikes when loading the image gallery but it still takes almost 5 min, which makes me believe that it gets all the info from the hard-disc directly. I'm using Owncloud with maria-db so that one should be also rather quick returning few 100.000 of records.

@ftree I'll have a look at your app, any other ideas for improvement?

[baracoder]

will reorganizing a flat gallery into nested folders help?

[hatl]

Why not just add a thumbnail cache?

[butonic]

@hatl the problem is not loading the images themselves (they can be cached by the browser), but the information which images exist. #790 (comment) correctly identified the getimages call as the problem because it will return a list of ALL images. My first comment would limit the problem to single folders with a huge number of pictures. That could be mitigated by using paging and/or load only the images currently visible. Volunteers welcome!

[butonic]

@craigpg @karlitschek maybe we should add this to 7.0.1? It is annoying.

[ftree]

Did you take a look to my app https://github.com/ftree/gallery2
There I have rewritten the original app, so that only one folder is loaded and not ALL folders. On my system I have >1000 Folder, and about 200000 images its now running without any problems. Also the size for the images can be set. So that not the original size is loaded to the browser. For example, the image size can be set to 1024 (or what else, ofcourse also the original size) . So also the showing of the images is fast.

The App is running only on ownCloud5 at the moment, because I had no time to change it for owncoud6.
But take a look there, how it is done. I did'nt know anything about the original Gallery App. And with anaylse of the code an rewritting the app, it took me three evenings to create the new app. So it should not be the big effort, to rewrite the orignal app.

[karlitschek]

I was under the impression that this is improved in 7 already. @icewind1991 What do you think?

[tavinus]

In OC7 the Pictures app seem to be handling better in general. But there is still a very long wait when first clicking the app. We get a loading message (here for 1 or 2 minutes) and then my folders show and navigation goes fast. I am guessing the app is populating the database when it is loaded, but it does that every time it is loaded.

I tried the Gallery2 app mentioned above and got a database creation error. Tried a few things then just dropped the tables oc_gallery_sharing and oc_pictures_images_cache and the app installed. But then I got no images shown there. Then I disabled g2 and enabled stock gallery again, no problems. Then disabled stock and tried gallery2 and no problems on enabling but no images.

It would be better to just create collections based on folders. I don't really want it to seek all my TB storage for images. I just want some stuff. You can always just browse on Files app and open the image you want. But then you need the Pictures app enabled to be able to view thumbnails (otherwise it will download the image).

Many apps on owncloud 7 let you choose the default folder to scan. It would be a simple solution and it would allow me to NOT show stuff I don't want to as well.

I would prefer an app with a menu to add a Collection and then ask you to point a folder on your storage and you're basically done. Setting names, etc is a plus. Then you can always create a collection on your root folder if you want. See my point? Too bad gallery2 didn't run here. I would like to try it.

[oparoz]

As discussed here: owncloud/gallery#17 , loading only images for the current folder also has its downsides.

[oparoz]

First step, stop loading all images.
Available in this branch: https://github.com/interfasys/galleryplus/tree/performance-fixes

# git clone -b performance-fixes https://github.com/interfasys/galleryplus.git

[leandro-lucarella-sociomantic]

will reorganizing a flat gallery into nested folders help?

No, I have thousands of images all organized in a hierarchical tree, with very few pictures in each actual leaf folder. My gallery never ends to load.

[leandro-lucarella-sociomantic]

Gallery+ works nicely for me, thanks!

[oparoz]

@leandro-lucarella-sociomantic - Ah, cool. You might run into issues as you reach your leaf folders, but that should be fixed in V2.1. I'm rewriting a lot of the Javascript to make it ID based vs path based and also to try and make the view less fragile.

[leandro-lucarella-sociomantic]

On Thu, Feb 26, 2015 at 07:59:44AM -0800, Olivier Paroz wrote:

@leandro-lucarella-sociomantic - Ah, cool. You might run into issues
as you reach your leaf folders, but that should be fixed in V2.1. I
just had to rewrite a lot of the Javascript to make it ID based vs
Path based and also to try and make the view less fragile.

It's working fine so far, thanks again!

[GreatEmerald]

So are there plans to include the updated app into the base ownCloud?

[oparoz]

@GreatEmerald - Yes indeed. It will be part of 8.2. You can use it now for 8.1 and 8.0 if you install Gallery from the Multimedia section.