Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Set default limit for concurrent archivers #18375

Closed
Starker3 opened this issue Nov 24, 2021 · 12 comments
Closed

Set default limit for concurrent archivers #18375

Starker3 opened this issue Nov 24, 2021 · 12 comments
Labels
Enhancement For new feature suggestions that for example enhance Matomo's cabapilities..
Milestone

Comments

@Starker3
Copy link
Contributor

Currently when someone sets up a core:archive crontab, the recommendation is to just simply set it without any additional options: https://matomo.org/docs/setup-auto-archiving/

In a lot of cases this works without any issues. However, if for some reason a particular run of the core:archive takes longer than usual (For example lots of tracking data all of a sudden) or for instances that track a lot data already this may lead to many concurrent archivers running at the same time.
This in turn results in even slower archiving performance and becomes a negative feedback loop eventually resulting in the database server hitting 100% CPU usage.

It would be great if we could set a default limit to the number of concurrently running archivers (Maybe some like a default max of 3 archivers).
I don't think this would have any impact on people not already setting the number of concurrent archivers. The one negative would be that if someone wanted to run a manual core:archive command and there are already core:archive commands running, then this would cause that archiver to exit.
In this case maybe an additional option to disable the check for maximum archivers might work for example. (We could also hint this in the output when the archiver exits for this reason).

One example of how this impacted 2 customers recently: One customer ended up with over 60 core:archive crontabs starting over a period of one or two days. This caused the CPU of the Reader database to hit 100% consistently and none of the core:archivers were able to finish because the database server was already way overloaded with requests.

The other customer did not have a Reader database setup and they ended up with around 25 concurrent archivers running which caused CPU to hit 100% consistently. This prevented them from being able to use Matomo (They were not even able to load the UI)

@Starker3 Starker3 added the Enhancement For new feature suggestions that for example enhance Matomo's cabapilities.. label Nov 24, 2021
@tsteur
Copy link
Member

tsteur commented Nov 24, 2021

BTW another solution, without possibly breaking it could be to simply adjust the guide. There will be potentially always few people where it would break something, just the same way as it would be fixing something for few potentially. We could maybe then change the default in Matomo 5 or so.

@Starker3
Copy link
Contributor Author

Sounds good to maybe include this sort of change in a bigger version release. Changing the guide will help people who set it up in the future, but I think by setting a default like this could potentially help quite a few people who already have it setup and maybe don't even realise that sometimes their server is slow for this reason.

@Starker3
Copy link
Contributor Author

This could actually be something that is easily setup if we work on #17719
We would have it as an option during the crontab installation

@tsteur tsteur added this to the 5.0.0 milestone Nov 24, 2021
@jane-twizel
Copy link

Will do now as we aren't likely to work on #17719 for a while.

@sgiehl
Copy link
Member

sgiehl commented Aug 12, 2022

So what should we change as part of this issue? Only change the default limit of concurrent archivers to 3. Or also an additional option to force running an archiving even if the limit is already hit?

@justinvelluppillai
Copy link
Contributor

@sgiehl it would be good to do both. @tsteur can you confirm your expectation/preference here also?

@tsteur
Copy link
Member

tsteur commented Aug 14, 2022

We would

  • Update the guide if that's not done yet
  • Change the default limit of concurrent archivers to 3.
  • There would indeed also need to be a way to not have concurrent archivers restricted. Eg using concurrent-archivers=-1 (and/or 0 could be also treated as unlimited) and concurrent-archivers= or something similar (or a different option).

@peterhashair
Copy link
Contributor

@tsteur Did a simple PR for that one, trying to complete those two. But looks really simple, am I on the right path?

  • Change the default limit of concurrent archivers to 3.
  • There would indeed also need to be a way to not have concurrent archivers restricted. Eg using concurrent-archivers=-1 (and/or 0 could be also treated as unlimited) and concurrent-archivers= or something similar (or a different option).

@tsteur
Copy link
Member

tsteur commented Nov 7, 2022

@peterhashair
Copy link
Contributor

we need to update this https://developer.matomo.org/guides/archiving-behavior-specification once Matomo 5.x document is ready.

@justinvelluppillai
Copy link
Contributor

This should be ready to action, if you haven't already @peterhashair ?

@peterhashair
Copy link
Contributor

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Enhancement For new feature suggestions that for example enhance Matomo's cabapilities..
Projects
None yet
Development

No branches or pull requests

6 participants