Limit media files count in media manager

[mxbossard]

Hi,

We currently have a lot of images in the default namespace. We choose to work like this for multiple reasons, but the loading is now really slow.

The media manager lack pagination, so I checked the source code. Every files in the namespace directory are scanned and media meta informations are gathered in a PHP array and then sent to the client.

I didn't think much of how to paginate results of the search function (https://github.com/splitbrain/dokuwiki/blob/b058d418d940f5ab7eb2cd126767f4bcd6fdd289/inc/search.php#L26). But I think this function would at least deserve a size limit guard which build and return a sized limited array.

By default this size limit guard would be infinite (-1) to not break current implementation. And I could definitely use a limit in media manager plugin to display a max number of media file. The guard could be set to a reasonable value like 1000, because I cannot imagine à scenario where 1000 files need to be searched, but I may be wrong.

Dokuwiki would benefit of this guard by consuming less resources when every files are not necessary, but only first ones.

I can submit a PR to limit the result count of the search function. And another to add a media size result limit configuration in media manager.

[phy25]

Releted to #2619

[mxbossard]

The implementation of the #2619 does not modify the search function. So for each browsed page, the search will be called and process every files in the directory.
Imo, the pagination should be implemented at search level.

[b01xy]

+1001 to modify the media manager behavior :)
don't know if I understand everything correctly but an ideal behavior could be :

• to have the ability to search on all images
• to only display a limited amount of images, with different possibility to sort this short list (last uploaded, by name, ...) to speed-up the loading of the media manager

[nderambure]

Really need a fix for this media manager, but pagination seems a bit overloaded. We need AJAX loading of small groups of images, and thus cut down the limit of first search return.

[phy25]

Due to how search() works now, it will be a lot of work to add pagination to it. PR welcome.

@nderambure I believe "AJAX loading of small groups of images" is the same as "pagination" here.

[nderambure]

I think pagination as mxbossard describes is a PHP pagination with reloading of media manager each time a click is made. But I may be wrong.

Anyway, I hope a fix will be found as this is a real major problem when a wiki has more than 100 images in root namespace (and it still is too much too load).

[Klap-in]

The clicking in namespace tree of the media managers uses already ajax.php for retrieving the images of a namespace. So I understand that you like instead of this entire list, to retrieve the list of media in smaller segments by ajax.

However, I'm wondering: what is causing the slow user experience? Is it due showing the list in the browser? Or server side, the assembling of the html response? or is it the search action (by search() )?.

[mxbossard]

I think the problem is the slow build of the page which delay js loading of media manager a lot.

I wrote a patch to limit the size of the data array manipulated in the search function. If it works I'll submit the patch. For now, only the media manager will use this limit. The purpose is to always display X first media in a namespace or after a search.

[nderambure]

@Klap-in : there is 2 distincts problems that could be solved independantly or together.

As @mxbossard said, the initial PHP build of the images array is a huge work for the server.
FOr example, in our server, we have 1700+ images in root level, resulting on 2,5 seconds of loading, just for the PHP (the server is a Xeon octocore with 16Go Ram from 2014).
Then, the loading of the hundreds of thumbnails in the browser takes 65 seconds and 23Mo each time we need to insert an image in wysiwyg (see below).

We could retrieve small segments of images as you said, and in top of this, lazy-loading images would be great for bandwith.

[sokolgeo]

I am in same boat: over 3'000 files in root /media folder of DokuWiki made Media Manager simply unusable because of slow load time. Ironically, the only function I actually need from MediaManager is to delete the current media file, which I open from a wiki page. But slow / long load time for MediaManager effectively prevents this only useful scenario.

Long story short I ended up blocking the generation of file list in MediaManager totally:

media.php -> function media_tab_files($ns,$auth=null,$jump='') -> commented line
/* media_filelist($ns,$auth,$jump,true,_media_get_sort_type()); */

Now I can at least use MediaManager to delete the current media file from a wiki page, without suffering slow load time of the MediaManager. Ugly hack, let's see how finding proper solution goes ahead.

[Klap-in]

Maybe it is also interesting to test the development version. The development version has some improvements of the Media manager such as lazy loading. Could be that will help your case as well.

(https://www.dokuwiki.org/changes, or more detailed https://www.dokuwiki.org/devel:releases )

[Raggaer]

In my case we have like 2000 images, resulting in 2000 requests to fetch everytime we open the popup or the fullscreen media manager, making it very slow.

I decided to remove the image previews on the page (inc/media.php) to solve this for now (simply commenting the element).

[Klap-in]

@Raggaer this is the experience for the last release? The development version has implemented lazy loading, which should improve your use case.

[Raggaer]

@Klap-in The problem was with the latest stable release. I will try that one locally to see how it improves, maybe an option to hide image thumbs on the media manager could be added too.