[Maintenance] Optimize tables with full text indexes periodically

[BlackbitDevs]

Full text indexes get slower over time because of gaps caused by deleting entries and because word counts get fragmented. In a concrete case we had the problem of ~600K entries in search_backend_data and the query

SELECT id FROM search_backend_data WHERE MATCH (`data`,`properties`) AGAINST ('123456' IN BOOLEAN MODE);

took 28 seconds.
This gets used in

pimcore/bundles/AdminBundle/Helper/GridHelperService.php

Line 617 in f415841

$conditionFilters[] = 'oo_id IN (SELECT id FROM search_backend_data WHERE maintype = "object" AND MATCH (`data`,`properties`) AGAINST (' . $list->quote($query) . ' IN BOOLEAN MODE))';

and also in other places like object search.

After running
OPTIMIZE TABLE search_backend_data
this same query takes 0.1 seconds.
It is even documented in the MySQL docs: https://dev.mysql.com/doc/refman/8.0/en/fulltext-fine-tuning.html#fulltext-optimize

This PR adds a maintenance task which optimizes the tables with fulltext indexes once every 7 days - currently those are search_backend_data and email_log. I also thought about fetching all tables which have fulltext indexes and optimize those but this would pull control of third-party tables like from plugins or application-specific implementation away from the owners.

[brusch]

LGTM, but tasks need to be registered as services, see:

pimcore/bundles/CoreBundle/Resources/config/maintenance.yaml

Line 59 in f611fab

Pimcore\Maintenance\Tasks\VersionsCleanupTask:

Can you please update your PR accordingly?
Thanks in advance!

[BlackbitDevs]

Sorry, forgot the service. Is fixed now.

[brusch]

👍 Thanks!

[BlackbitDevs]

Just as a note: I tried to migrate this maintenance task to a Pimcore 6.9 system (with PdoStore as lock backend) and stumbled upon the MySQL error

General error 2014 Cannot execute queries while other unbuffered queries are active. Consider using PDOStatement::fetchAll().  Alternatively, if your code is only ever going to run against mysql, you may enable query buffering by setting the PDO::MYSQL_ATTR_USE_BUFFERED_QUERY attribute.

There I changed the calls in

pimcore/lib/Maintenance/Tasks/FullTextIndexOptimizeTask.php

Lines 41 to 42 in ffe18e9

	Db::get()->exec('OPTIMIZE TABLE search_backend_data');
	Db::get()->exec('OPTIMIZE TABLE email_log');

to use fetchAll() instead of exec() and this works. Do not know what caused the error but if this ever occurs also for Pimcore 10, this may become useful...

[DrLuke]

to use fetchAll() instead of exec() and this works. Do not know what caused the error but if this ever occurs also for Pimcore 10, this may become useful...

We just upgraded to Pimcore 10 and had this issue.

Can be solved by overriding the class:
overrides/Pimcore/Maintenance/Tasks/FullTextIndexOptimizeTask.php

<?php

namespace Pimcore\Maintenance\Tasks;

use Pimcore\Db;
use Pimcore\Maintenance\TaskInterface;
use Symfony\Component\Lock\LockFactory;

/**
 * @internal
 */
class FullTextIndexOptimizeTask implements TaskInterface
{
    /** @var \Symfony\Component\Lock\LockInterface */
    private $lock;

    public function __construct(LockFactory $lockFactory)
    {
        $this->lock = $lockFactory->createLock(self::class, 86400 * 7, false);
    }

    /**
     * {@inheritdoc}
     */
    public function execute()
    {
        // Override to make it work
        // See: https://github.com/pimcore/pimcore/pull/11817#issuecomment-1105026544
        if ($this->lock->acquire(false)) {
            Db::get()->fetchAll('OPTIMIZE TABLE search_backend_data');
            Db::get()->fetchAll('OPTIMIZE TABLE email_log');
        }
    }
}

And then in your composer.json:

"psr-4": {
  "App\\": "src/",
  "Pimcore\\Maintenance\\Tasks\\": "overrides/Pimcore/Maintenance/Tasks",
}