With MDC enabled NC becomes excruciatingly slow

[YouveGotMeowxy]

Describe the bug
I've tested this a few times now; with MDC enabled NC becomes so slow to load anything, and most times when it finishes loading, most of the "stuff" isn't there (no folder images, icons, etc.)

After disabling MDC, NC becomes responsive again.

Oddly, but this seems to possibly only happen upon first start of NC with MDC already enabled. If I disabled and then re-enable MDC once, then it seems NC is back to normal responsiveness, though I haven't really put that to the test yet to confirm that NC continues to stay responsive.

To Reproduce
Steps to reproduce the behavior:

1. Enable MDC
2. Try to use Nextcloud
3. That's it

Expected behavior
No negative speed affect of NC at all. :)

Desktop (please complete the following information):

• OS Docker (LSIO container) NC
• Nextcloud version latest 25
• Database configuration mariadb latest version
• MediaDC version latest version

Additional context
Not sure what other info might be needed, just let me know and I will post it.

Update:

After doing the r-enable thing, I tried loading the MDC page, and I just get the loading indicator and a blank tab perpetually. Looking in the NC log I see this:

01/20/2023 10:45:02 PM
ArgumentCountError: Too few arguments to function OCA\MediaDC\BackgroundJob\CollectorCleanupJob::__construct(), 0 passed in /config/www/nextcloud/lib/private/BackgroundJob/JobList.php on line 304 and exactly 2 expected in /config/www/nextcloud/apps/mediadc/lib/BackgroundJob/CollectorCleanupJob.php:41
01/20/2023 10:45:02 PM
Stack trace:
01/20/2023 10:45:02 PM
#0 /config/www/nextcloud/lib/private/BackgroundJob/JobList.php(304): OCA\MediaDC\BackgroundJob\CollectorCleanupJob->__construct()
01/20/2023 10:45:02 PM
#1 /config/www/nextcloud/lib/private/BackgroundJob/JobList.php(243): OC\BackgroundJob\JobList->buildJob()
01/20/2023 10:45:02 PM
#2 /config/www/nextcloud/cron.php(145): OC\BackgroundJob\JobList->getNext()
01/20/2023 10:45:02 PM
#3 {main}

[andrey18106]

@YouveGotMeowxy Hi! Thanks for reporting this. It needs more investigation to understand what's the reason, as I didn't notice such symptoms.

About log entry - mentioned commit should fix this, background cleanup job was broken.

[YouveGotMeowxy]

@andrey18106 ty, how long before it's available to update via the apps in NC?

[andrey18106]

@andrey18106 ty, how long before it's available to update via the apps in NC?

In a few days, I think, after we transfer this repository to cloud-py-api org.

Also performance issues could probably be because of unused and being replaced Thrift third party dependency in cloud_py_api, it'll be updated soon too.

[YouveGotMeowxy]

Sounds good!

[andrey18106]

@YouveGotMeowxy Could you verify please performance changes for your instance with the latest updates of cloud_py_api and mediadc? If it's ok, we can close this issue.

[YouveGotMeowxy]

@YouveGotMeowxy Could you verify please performance changes for your instance with the latest updates of cloud_py_api and mediadc? If it's ok, we can close this issue.

Yessir, I will as soon as I get my PC that was running docker back up again. The root hard drive died on it and I'm in the process of getting it all back together again.

[ksihota]

I just installed both cloud_py_api and mediadc on my system and ran a test using two directories with a lot of duplicates. It pushed my CPU usage up to 100% Basically NextCloud froze up. I ended up having to restart my PHP-FPM server to use nextcloud.
Is there a way to throttle the resources used to perform the task? I would prefer to do that and let the app run in the background for a few hours rather than having it bring down my server.

System Information

Operating system: Ubuntu Linux 20.04.5
Firewall version: ConfigServer Security & Firewall 14.17
Time on system: Wednesday, March 1, 2023 2:12 PM
Kernel and CPU: Linux 5.4.0-139-generic on x86_64
Processor information: Intel(R) Xeon(R) CPU X5550 @ 2.67GHz, 8 cores
System uptime: 6 days, 0 hours, 21 minutes
Running processes: 248
CPU load averages: 4.62 (1 min) 4.55 (5 mins) 4.54 (15 mins)
Real memory: 2.13 GiB used / 19.04 GiB cached / 23.51 GiB total
Virtual memory: 379.75 MiB used / 7.99 GiB total
Local disk space: 777.59 GiB used / 1.06 TiB free / 1.82 TiB total
Package updates: All installed packages are up to date

[bigcat88]

@ksihota does encryption enabled on your instance?

[ksihota]

@bigcat88 Just checked to make sure and No, encryption is not enabled.

I'm pretty sure its because there were GBs of duplicates. Doesn't happen with few matches

[bigcat88]

There is only a single threaded code for duplicates. As I understand there was a multi-threaded load on 50% of logical cores?
And it was caused by php process(so reloading php-fpm helps?)

[ksihota]

It is obviously a resources issue with PHP. I have no idea what is causing it. I assumed it was just the number of duplicates it was finding. The specific folder I targeted had been a copy of the original so my guess was that all images were duplicates. My aim in running the task was to see if the app would be good at cleaning up directory duplicates or partial duplicates.
NextCloud became unresponsive so I restarted my PHP server and I was able to get back into Nextcloud and MDC

[ksihota]

I am getting a lot of these:
Resource: Virtual Memory Size
Exceeded: 931 > 512 (MB)
Executable: /tmp/onefile_1432088_1677697958_628787/main
Command Line: /home/user/nextcloud_data/appdata_ocoi9tgt1m1j/mediadc/binaries/main -t 1

I could push my PHP memory up to 1024 but I figured 512 should be enough to cover nearly anything.

[bigcat88]

We'll post a new version in a few days and see how it goes.
And we will try to resolve this issue tomorrow

[ksihota]

Thanks, but I didn't mean to push this in, I just thought it might be something you would like to be aware of. I only just started using the app so I am just learning some of its ins and outs.
So far it looks like its working quite nicely. I'm still looking for an easier way to delete a lot of files all at once. I have made a few other comments is some of the other issues.
Thanks for this useful project.
PS is there actually a forum for this or is everything here?