Huge performance problems (OOM + Execution time) when listing large folders

[LukasReschke]

While debugging #12785 I had to create a lot of files in the user's data folder with the following script, to be precise I created 22559 files.

When now going to the file overview as the user where I created the files the Apache log gets sometimes filled with the following fatal errors such as:

[Sat Jan 17 21:59:24.553293 2015] [:error] [pid 20417] [client XXXXX:55157] PHP Fatal error:  Out of memory (allocated 168296448) (tried to allocate 78 bytes) in /var/www/html/owncloud/3rdparty/doctrine/dbal/lib/Doctrine/DBAL/Statement.php on line 231, referer: http://XXXXX/owncloud/index.php/apps/files/
[Sat Jan 17 22:00:19.838615 2015] [:error] [pid 20419] [client XXXXX:54882] PHP Fatal error:  Out of memory (allocated 340262912) (tried to allocate 72 bytes) in /var/www/html/owncloud/lib/private/json.php on line 132, referer: http://XXXXX/owncloud/index.php/apps/files/
[Sat Jan 17 22:00:20.680055 2015] [:error] [pid 20368] [client XXXXX:54873] PHP Fatal error:  Out of memory (allocated 306184192) (tried to allocate 99 bytes) in /var/www/html/owncloud/lib/private/connector/sabre/filesplugin.php on line 73, referer: http://XXXXX/owncloud/index.php/settings/admin

Thus it is effectively not possible to use ownCloud at all… Sometimes it works but then the AJAX request gathering the file list is very slow (apps/files/ajax/list.php requires more than 50 seconds) and requires way too much memory (more than 150MB for just getting the file list!) than it actually should, just take a look at the following profiling overview:

https://blackfire.io/profiles/e2ddd96d-08c1-4afc-8a2e-414b5a4cb5b4/graph

Memory

1. We waste 24MB memory for OC_Helper::mimetypeIcon
2. We waste 58.6MB memory for OC_JSON::encode
3. We waste 62.9MB memory for PDOStatement::fetchAll

CPU

1. We waste more than 30 seconds on sorting and comparing the file names
2. We waste more than 10 seconds on a method that should just format a date…

This is on current master on my local development machine (SQLite) but behaves identically as the 7.0.3 Ubuntu machine (MySQL) where I debug the config.php problem (big fun)

@DeepDiver1975 @karlitschek FYI – major performance problem.

[LukasReschke]

Ewww, Punic is actually really really slow… Let me see if I can find some quick wins in their code base…

[LukasReschke]

Got a simple patch that gives us on average 20% performance win on time-manipulation related actions by using a simple array cache for the timezone. This gives us a 2 second performance gain here in this simple example and a lot of other gains in other cases. PR is upstream: punic/punic#30

Still not perfectly happy with the duration of all Punic related stuff … mhm…

[LukasReschke]

Some patches for the mimetype icon problem: #13438

[PVince81]

Is that formatting the date for the web UI ? I think we can probably drop that completely in favor of timestamps.
But @DeepDiver1975 seemed to think we should rather use the ISO format here: #12639 (comment) so it might not be the right approach anyway.

[LukasReschke]

Is that formatting the date for the web UI ? I think we can probably drop that completely in favor of timestamps.

It is. But those 10 seconds on dates are negligible when considering the 30 seconds on sorting stuff :( – Also using more than 50MB here just for JSON encoding is pretty … much … :)

[PVince81]

Some ideas to optimize the web UI workflow:

1. Call json_encode() only on every file, then string concat into the final array
2. After converting a file with json_encode(), remove it from the original array, with shift() (if that doesn't cause additional perf issues for PHP...)
3. Rely on DB sort (but might cause trouble with UTF-8 chars)

Ideally we should resort to pagination, but that will cause trouble with WebDAV...

[LukasReschke]

Another 4 seconds are saved by optimizing NaturalSort, thanks @Xenopathic: #13511

[Xenopathic]

@LukasReschke Closed? Is this no longer a significant issue?

[karlitschek]

@Xenopathic the assumption is that the issue is mostly fixed by the two referenced pull requests

[LukasReschke]

(and also the referenced issue at #13473 is quite good)