High memory usage

[Zegorax]

I'm trying to generate previews for 3.3TB of data. I ran ./occ preview:generate-all -vvv and this process is using a lot of RAM, and eventually exits by its own.

NextCloud is running in a container using the official Docker image.

What should I do to complete the process and generate all previews ?

[Zegorax]

I have applied these settings :

  'preview_max_x'   => 1024,
  'preview_max_y'   => 1024,
  'preview_max_scale_factor'   => 10,
  'preview_max_filesize_image' => 50,

and it seems to be working. For an unknown reason, the commands on the readme did not set the settings to config.php and I think occ was not using them as a result. I'll confirm later on when all of the data have been processed.

[fate8383]

I have the same problem, high memory usage, even corrupted few php files trying to kill the process, had to reinstall php. I had many segmentation fault errors, after reinstalling php i am fine with that. But still limiting memory to 512 or leaving it -1 on cli/php.ini nothing helps. Had many tries, maybe 100s, still either process is killed for high memory usage or halts until max execution time. Using nextcloud 19, but i don't think it is related to it, it simply using too much memory to handle such process. Any idea how can I use this? it is only for 50G photos, i have done this before on version 18.3 nextcloud. I have to redo it now because new hard rive etc...but i am not able to. Please advice. Thanks

[Zegorax]

@fate8383 Try the settings I've posted above, they work. It's been 2 days and it is processing fine for now, no sign of high memory usage.

But this issue should not be closed like this, because the problem is present when not modifying the default settings.

[fate8383]

i have already made those changes, still no help. now have installed zram increased all swap and zram to be able to finish it. it is using all of the allocated rams in swap and zram. there is definitely some memory leak or something.

[basti1982]

I think that I'm running into the same issue. ./occ preview:generate-all get's killed after a couple of images.
I applied the configuration from here https://ownyourbits.com/2019/06/29/understanding-and-improving-nextcloud-previews/ and also tried the settings mentioned in this thread. Process still get's killed.

php config:
memory_limit => -1 => -1
max_execution_time => 0 => 0

[fate8383]

There is no solution, I split into folders my images and I go folder by folder to create previews. ...generate-all --path=‘path/to/file’
Added swap and ZRAM 5g together. Killed all other apps/daemons/cronjobs which are using ram, changed
cli/php.ini
memory -1
maximum execution 3600
I am getting 10 min of it working and at least it kills softly. I restart again, it is mind blowingly annoying to do it this way but I will finish it and put back all my settings again. Next time only run it for specific folder when I have new photos.

[JoeH82]

Hi,

same here. I'm facing this issue after upgrading from NC18 to NC19. Not sure if i got an update of preview generator with that, cause im using the docker image of NC and this switched from18 to 19 including the installed apps.
Not able to generate previews for bunch of high resolutio images :(

[fate8383]

yeah definitely something with v18 to v19 update caused this. app is not updated for v19, it just uses the same settings for v18...I hope and wish the owner looks at it and updates if possible.

[basti1982]

Here's a part of the output of dmesg -T | grep -i "killed process"

[Fr Jun 19 14:30:01 2020] OOM killed process 25825 (php) total-vm:2896928kB, anon-rss:2581984kB, file-rss:5708kB [Fr Jun 19 15:29:53 2020] OOM killed process 28708 (php) total-vm:2553020kB, anon-rss:2229412kB, file-rss:5700kB [Fr Jun 19 16:29:44 2020] OOM killed process 31529 (php) total-vm:2573892kB, anon-rss:2215016kB, file-rss:5808kB

It really looks like that the php process is using too much memory.

[fate8383]

here is my workaround
add this to Generate.php after use commands:
ini_set('max_execution_time', '90');
then add crontab every 6 min to run the command without verbose and with --path='path/to/your/photos/folder/'
it gets you 500-1000 photos done autonomously, after that many photos 90s fails during scanning already generated previews but you can avoid this by splitting photos to folders

[rullzer]

This is nothing the app can fix. The app just asks the server to generate the previews while it iterates over the folders.
If some file is consuming a ton of memory there is not much the app can do about that.

[Zegorax]

@rullzer The problem is still existing. I'm trying now on a brand new installation and the same problem happens.

[kfirfer]

Problem exists for me aswell

[lrissman-github]

Fresh install using docker -- Issue still exists for me.

when running this command: php -d memory_limit=-1 occ preview:generate-all it simply runs until all of my memory is overwhelmed and the system crashes.

[lrissman-github]

I have something that worked for me, and am sharing it here -- it's not a fix.

I run a SBC (Rockpro64) with 4G of ram, so I always need to be careful with memory lest I use more than physical and then start swapping = system halt due to waiting on storage = bad. :)

I did these things:

1. I updated to 19.0.1 RC 1 and had the same issue with php ./occ perview:generate-all where it would exit after a few files.
2. stopped all my other containers except nextcloud and mariadb
3. logged into my nextcloud container as my www user (abc in my case) and ran this command php -d memory_limit=2G ./occ preview:generate-all -vvv

My experience just by watching htop and the progress is this:

• It starts out well using around 300M of ram and does that for awhile flying through files at about 10-15 per second..
• Then 'something' happens and it jumps to using close to 1G and sometimes way over.
• Then eventually dies as expected when memory exceeds my PHP memory limit settings: PHP Fatal error: Allowed memory size of 2147483648 bytes exhausted (tried to allocate 14508792 bytes) in /config/www/nextcloud/lib/private/legacy/OC_Image.php on line 649

It is almost like it's not clearing out memory fast enough when processing images and builds up until it overruns the system...

So here's my final -- a never-ending loop because the script will not regenerate for files that already have images.. I also lowered the available memory so that I could run my other containers:

while true; do php -d memory_limit=1G ./occ preview:generate-all -vvv ; sleep 5 ; done

This will run until it dies, and then start again in 5 seconds, forever.. so every few hours I will check on it until it finally finishes all the thumnails and then never run it again..

I tend to get between 1-3 minutes per process needing to be restarted and it has no problems with whatever file it died at the last time in the next run.. so, looking good for me for the last 2 hours.

[rullzer]

I'm not saying it can't happen. I'm saying it is not this app that is causing it.
The issue is most likely in the nextcloud server somewhere some memory is leaking. But that is nothing this app can fix