Join GitHub today
GitHub is home to over 40 million developers working together to host and review code, manage projects, and build software together.Sign up
Implemented pausing mechanisms into cli/finder_indexer.php. #13502
Summary of Changes
This patch implements a pausing mechanism that pauses for a defined or dynamically adjusted amount of time between batches, therefore giving the server a little time to catch up.
Note: You might or might not notice (on huge number of articles with a lot of different text) that the machine(server) might get unresponsive.
Even if you don't notice that, you will notice that the apache/php- and mysql-processes are each constantly fully consuming a CPU core. Which depending on the cores available (and the settings of mysql in core usage) might translate to 50% on a dual core, 25% on a quad core or 12,5% on an octa-core processor. While this alone might not be an issue, it does add up the the problem at hand.
You also might or might not notice (depending on the storage subsystem [hdd, ssd or hybrid]) that there is a lot of IO/WAIT during the task. It gets worse, if it is a very long running task. I have seen this resulting in an unresponsive site, during indexing. Where unresponsive can mean that the response could take from 5-10 seconds to a lot more, or a complete timeout.
You should also notice that the site/admin interface is now a lot more responsive, as the machine has less IO/WAITS and less constantly fully consuming of CPU cores.
Documentation Changes Required
Yes. The new arguments should be included in the documentation here: https://docs.joomla.org/Setting_up_automatic_Smart_Search_indexing
The reason for that is, that with thousands of articles or K2 items, the server would often become unresponsive, because of a lot of processing and IO/WAiT on the mysql server. This patch implements a pausing mechanism that pauses for a defined or dynamically adjusted amount of time between batches, therefore giving the server a little time to catch up.
Thank you for this feature. I do consider it a new feature and not just a bugfix, and thus I would recommend to add this to 4.0 instead of 3.x. I would actually be interested if this issue persists with the new DB structure in 4.0.
And yes, some things take a VERY long time. sigh
@Hackwar Yes, it can be seen both ways. But it fixes a problem that is happening on the 3.x version. And since people will be on that version for some time to come, we might as well provide a better experience.
As for 4.0, I have not tested, and unfortunately I didn't have any time to check out all the great stuff you people have done with the new version. But if we have this on 3.x it shouldn't be too much of an effort, if needed, to port it to 4.0.