Set default limit for concurrent archivers #18375
Currently when someone sets up a core:archive crontab, the recommendation is to just simply set it without any additional options: https://matomo.org/docs/setup-auto-archiving/
In a lot of cases this works without any issues. However, if for some reason a particular run of the core:archive takes longer than usual (For example lots of tracking data all of a sudden) or for instances that track a lot data already this may lead to many concurrent archivers running at the same time.
It would be great if we could set a default limit to the number of concurrently running archivers (Maybe some like a default max of 3 archivers).
One example of how this impacted 2 customers recently: One customer ended up with over 60 core:archive crontabs starting over a period of one or two days. This caused the CPU of the Reader database to hit 100% consistently and none of the core:archivers were able to finish because the database server was already way overloaded with requests.
The other customer did not have a Reader database setup and they ended up with around 25 concurrent archivers running which caused CPU to hit 100% consistently. This prevented them from being able to use Matomo (They were not even able to load the UI)
The text was updated successfully, but these errors were encountered:
BTW another solution, without possibly breaking it could be to simply adjust the guide. There will be potentially always few people where it would break something, just the same way as it would be fixing something for few potentially. We could maybe then change the default in Matomo 5 or so.
Sounds good to maybe include this sort of change in a bigger version release. Changing the guide will help people who set it up in the future, but I think by setting a default like this could potentially help quite a few people who already have it setup and maybe don't even realise that sometimes their server is slow for this reason.
@tsteur Did a simple PR for that one, trying to complete those two. But looks really simple, am I on the right path?
@peterhashair in this case it be more about the archive command https://github.com/matomo-org/matomo/blob/4.12.3/plugins/CoreConsole/Commands/CoreArchiver.php#L114 which people set up as part of https://matomo.org/docs/setup-auto-archiving/
The value set in the PR would be otherwise overwritten in https://github.com/matomo-org/matomo/blob/4.12.3/plugins/CoreConsole/Commands/CoreArchiver.php#L47 I believe.