[Bug]: Class rebuild fails during pimcore install script

[wwidergoldbsc]

Pimcore version

11.1, 11.0.2

Steps to reproduce

Example tested on pimcore demo project:

1. Follow setup steps in Pimcore Demo to initialize the demo project until step 4.
2. make sure DB is empty:
   run bin/console doctrine:database:drop --force --no-interaction
   run bin/console doctrine:database:create --if-not-exists
3. install pimcore without the demo-dumps:
   run php vendor/bin/pimcore-install --mysql-host-socket=db --mysql-username=pimcore --mysql-password=pimcore --mysql-database=pimcore --skip-database-data-dump

Actual Behavior

The installer will output

12:04:14 EMERGENCY [pimcore] PDOException: SQLSTATE[42S02]: Base table or view not found: 1146 Table 'pimcore.targeting_target_groups' doesn't exist in /var/www/html/vendor/doctrine/dbal/src/Driver/PDO/Connection.php:71
Stack trace:
#0 /var/www/html/vendor/doctrine/dbal/src/Driver/PDO/Connection.php(71): PDO->query('SELECT id FROM ...')
pimcore/pimcore#1 /var/www/html/vendor/doctrine/dbal/src/Driver/Middleware/AbstractConnectionMiddleware.php(33): Doctrine\DBAL\Driver\PDO\Connection->query('SELECT id FROM ...')
pimcore/pimcore#2 /var/www/html/vendor/doctrine/dbal/src/Logging/Connection.php(43): Doctrine\DBAL\Driver\Middleware\AbstractConnectionMiddleware->query('SELECT id FROM ...')
pimcore/pimcore#3 /var/www/html/vendor/doctrine/dbal/src/Driver/Middleware/AbstractConnectionMiddleware.php(33): Doctrine\DBAL\Logging\Connection->query('SELECT id FROM ...')
pimcore/pimcore#4 /var/www/html/vendor/symfony/doctrine-bridge/Middleware/Debug/DBAL3/Connection.php(58): Doctrine\DBAL\Driver\Middleware\AbstractConnectionMiddleware->query('SELECT id FROM ...')
pimcore/pimcore#5 /var/www/html/vendor/doctrine/dbal/src/Connection.php(1100): Symfony\Bridge\Doctrine\Middleware\Debug\DBAL3\Connection->query('SELECT id FROM ...')
pimcore/pimcore#6 /var/www/html/vendor/doctrine/dbal/src/Connection.php(952): Doctrine\DBAL\Connection->executeQuery('SELECT id FROM ...', Array, Array)
pimcore/pimcore#7 /var/www/html/vendor/pimcore/personalization-bundle/src/Model/Tool/Targeting/TargetGroup/Listing/Dao.php(33): Doctrine\DBAL\Connection->fetchFirstColumn('SELECT id FROM ...', Array)
pimcore/pimcore#8 [internal function]: Pimcore\Bundle\PersonalizationBundle\Model\Tool\Targeting\TargetGroup\Listing\Dao->load()
pimcore/pimcore#9 /var/www/html/vendor/pimcore/pimcore/lib/Model/AbstractModel.php(220): call_user_func_array(Array, Array)
pimcore/pimcore#10 /var/www/html/vendor/pimcore/personalization-bundle/src/Pimcore/Model/DataObject/ClassDefinition/Data/TargetGroup.php(80): Pimcore\Model\AbstractModel->__call('load', Array)
pimcore/pimcore#11 /var/www/html/vendor/pimcore/personalization-bundle/src/Pimcore/Model/DataObject/ClassDefinition/Data/TargetGroup.php(116): Pimcore\Model\DataObject\ClassDefinition\Data\TargetGroup->configureOptions()
pimcore/pimcore#12 /var/www/html/var/classes/definition_CustomerSegment.php(246): Pimcore\Model\DataObject\ClassDefinition\Data\TargetGroup::__set_state(Array)
pimcore/pimcore#13 /var/www/html/vendor/pimcore/pimcore/lib/Model/DataObject/ClassDefinition/ClassDefinitionManager.php(74): include('/var/www/html/v...')
pimcore/pimcore#14 /var/www/html/vendor/pimcore/pimcore/bundles/CoreBundle/src/Command/ClassesRebuildCommand.php(98): Pimcore\Model\DataObject\ClassDefinition\ClassDefinitionManager->createOrUpdateClassDefinitions()
pimcore/pimcore#15 /var/www/html/vendor/symfony/console/Command/Command.php(326): Pimcore\Bundle\CoreBundle\Command\ClassesRebuildCommand->execute(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
pimcore/pimcore#16 /var/www/html/vendor/symfony/console/Application.php(1096): Symfony\Component\Console\Command\Command->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
pimcore/pimcore#17 /var/www/html/vendor/symfony/framework-bundle/Console/Application.php(126): Symfony\Component\Console\Application->doRunCommand(Object(Pimcore\Bundle\CoreBundle\Command\ClassesRebuildCommand), Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
pimcore/pimcore#18 /var/www/html/vendor/symfony/console/Application.php(324): Symfony\Bundle\FrameworkBundle\Console\Application->doRunCommand(Object(Pimcore\Bundle\CoreBundle\Command\ClassesRebuildCommand), Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
pimcore/pimcore#19 /var/www/html/vendor/symfony/framework-bundle/Console/Application.php(80): Symfony\Component\Console\Application->doRun(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
pimcore/pimcore#20 /var/www/html/vendor/symfony/console/Application.php(175): Symfony\Bundle\FrameworkBundle\Console\Application->doRun(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
pimcore/pimcore#21 /var/www/html/vendor/symfony/runtime/Runner/Symfony/ConsoleApplicationRunner.php(49): Symfony\Component\Console\Application->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
pimcore/pimcore#22 /var/www/html/vendor/autoload_runtime.php(29): Symfony\Component\Runtime\Runner\Symfony\ConsoleApplicationRunner->run()
pimcore/pimcore#23 /var/www/html/bin/console(20): require_once('/var/www/html/v...')
pimcore/pimcore#24 {main}

Next Doctrine\DBAL\Driver\PDO\Exception: SQLSTATE[42S02]: Base table or view not found: 1146 Table 'pimcore.targeting_target_groups' doesn't exist in /var/www/html/vendor/doctrine/dbal/src/Driver/PDO/Exception.php:28
Stack trace:
#0 /var/www/html/vendor/doctrine/dbal/src/Driver/PDO/Connection.php(76): Doctrine\DBAL\Driver\PDO\Exception::new(Object(PDOException))
pimcore/pimcore#1 /var/www/html/vendor/doctrine/dbal/src/Driver/Middleware/AbstractConnectionMiddleware.php(33): Doctrine\DBAL\Driver\PDO\Connection->query('SELECT id FROM ...')
pimcore/pimcore#2 /var/www/html/vendor/doctrine/dbal/src/Logging/Connection.php(43): Doctrine\DBAL\Driver\Middleware\AbstractConnectionMiddleware->query('SELECT id FROM ...')
pimcore/pimcore#3 /var/www/html/vendor/doctrine/dbal/src/Driver/Middleware/AbstractConnectionMiddleware.php(33): Doctrine\DBAL\Logging\Connection->query('SELECT id FROM ...')
pimcore/pimcore#4 /var/www/html/vendor/symfony/doctrine-bridge/Middleware/Debug/DBAL3/Connection.php(58): Doctrine\DBAL\Driver\Middleware\AbstractConnectionMiddleware->query('SELECT id FROM ...')
pimcore/pimcore#5 /var/www/html/vendor/doctrine/dbal/src/Connection.php(1100): Symfony\Bridge\Doctrine\Middleware\Debug\DBAL3\Connection->query('SELECT id FROM ...')
pimcore/pimcore#6 /var/www/html/vendor/doctrine/dbal/src/Connection.php(952): Doctrine\DBAL\Connection->executeQuery('SELECT id FROM ...', Array, Array)
pimcore/pimcore#7 /var/www/html/vendor/pimcore/personalization-bundle/src/Model/Tool/Targeting/TargetGroup/Listing/Dao.php(33): Doctrine\DBAL\Connection->fetchFirstColumn('SELECT id FROM ...', Array)
pimcore/pimcore#8 [internal function]: Pimcore\Bundle\PersonalizationBundle\Model\Tool\Targeting\TargetGroup\Listing\Dao->load()
pimcore/pimcore#9 /var/www/html/vendor/pimcore/pimcore/lib/Model/AbstractModel.php(220): call_user_func_array(Array, Array)
pimcore/pimcore#10 /var/www/html/vendor/pimcore/personalization-bundle/src/Pimcore/Model/DataObject/ClassDefinition/Data/TargetGroup.php(80): Pimcore\Model\AbstractModel->__call('load', Array)
pimcore/pimcore#11 /var/www/html/vendor/pimcore/personalization-bundle/src/Pimcore/Model/DataObject/ClassDefinition/Data/TargetGroup.php(116): Pimcore\Model\DataObject\ClassDefinition\Data\TargetGroup->configureOptions()
pimcore/pimcore#12 /var/www/html/var/classes/definition_CustomerSegment.php(246): Pimcore\Model\DataObject\ClassDefinition\Data\TargetGroup::__set_state(Array)
pimcore/pimcore#13 /var/www/html/vendor/pimcore/pimcore/lib/Model/DataObject/ClassDefinition/ClassDefinitionManager.php(74): include('/var/www/html/v...')
pimcore/pimcore#14 /var/www/html/vendor/pimcore/pimcore/bundles/CoreBundle/src/Command/ClassesRebuildCommand.php(98): Pimcore\Model\DataObject\ClassDefinition\ClassDefinitionManager->createOrUpdateClassDefinitions()
pimcore/pimcore#15 /var/www/html/vendor/symfony/console/Command/Command.php(326): Pimcore\Bundle\CoreBundle\Command\ClassesRebuildCommand->execute(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
pimcore/pimcore#16 /var/www/html/vendor/symfony/console/Application.php(1096): Symfony\Component\Console\Command\Command->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
pimcore/pimcore#17 /var/www/html/vendor/symfony/framework-bundle/Console/Application.php(126): Symfony\Component\Console\Application->doRunCommand(Object(Pimcore\Bundle\CoreBundle\Command\ClassesRebuildCommand), Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
pimcore/pimcore#18 /var/www/html/vendor/symfony/console/Application.php(324): Symfony\Bundle\FrameworkBundle\Console\Application->doRunCommand(Object(Pimcore\Bundle\CoreBundle\Command\ClassesRebuildCommand), Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
pimcore/pimcore#19 /var/www/html/vendor/symfony/framework-bundle/Console/Application.php(80): Symfony\Component\Console\Application->doRun(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
pimcore/pimcore#20 /var/www/html/vendor/symfony/console/Application.php(175): Symfony\Bundle\FrameworkBundle\Console\Application->doRun(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
pimcore/pimcore#21 /var/www/html/vendor/symfony/runtime/Runner/Symfony/ConsoleApplicationRunner.php(49): Symfony\Component\Console\Application->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
pimcore/pimcore#22 /var/www/html/vendor/autoload_runtime.php(29): Symfony\Component\Runtime\Runner\Symfony\ConsoleApplicationRunner->run()
pimcore/pimcore#23 /var/www/html/bin/console(20): require_once('/var/www/html/v...')
pimcore/pimcore#24 {main}

Next Doctrine\DBAL\Exception\TableNotFoundException: An exception occurred while executing a query: SQLSTATE[42S02]: Base table or view not found: 1146 Table 'pimcore.targeting_target_groups' doesn't exist in /var/www/html/vendor/doctrine/dbal/src/Driver/API/MySQL/ExceptionConverter.php:49
Stack trace:
#0 /var/www/html/vendor/doctrine/dbal/src/Connection.php(1938): Doctrine\DBAL\Driver\API\MySQL\ExceptionConverter->convert(Object(Doctrine\DBAL\Driver\PDO\Exception), Object(Doctrine\DBAL\Query))
pimcore/pimcore#1 /var/www/html/vendor/doctrine/dbal/src/Connection.php(1880): Doctrine\DBAL\Connection->handleDriverException(Object(Doctrine\DBAL\Driver\PDO\Exception), Object(Doctrine\DBAL\Query))
pimcore/pimcore#2 /var/www/html/vendor/doctrine/dbal/src/Connection.php(1105): Doctrine\DBAL\Connection->convertExceptionDuringQuery(Object(Doctrine\DBAL\Driver\PDO\Exception), 'SELECT id FROM ...', Array, Array)
pimcore/pimcore#3 /var/www/html/vendor/doctrine/dbal/src/Connection.php(952): Doctrine\DBAL\Connection->executeQuery('SELECT id FROM ...', Array, Array)
pimcore/pimcore#4 /var/www/html/vendor/pimcore/personalization-bundle/src/Model/Tool/Targeting/TargetGroup/Listing/Dao.php(33): Doctrine\DBAL\Connection->fetchFirstColumn('SELECT id FROM ...', Array)
pimcore/pimcore#5 [internal function]: Pimcore\Bundle\PersonalizationBundle\Model\Tool\Targeting\TargetGroup\Listing\Dao->load()
pimcore/pimcore#6 /var/www/html/vendor/pimcore/pimcore/lib/Model/AbstractModel.php(220): call_user_func_array(Array, Array)
pimcore/pimcore#7 /var/www/html/vendor/pimcore/personalization-bundle/src/Pimcore/Model/DataObject/ClassDefinition/Data/TargetGroup.php(80): Pimcore\Model\AbstractModel->__call('load', Array)
pimcore/pimcore#8 /var/www/html/vendor/pimcore/personalization-bundle/src/Pimcore/Model/DataObject/ClassDefinition/Data/TargetGroup.php(116): Pimcore\Model\DataObject\ClassDefinition\Data\TargetGroup->configureOptions()
pimcore/pimcore#9 /var/www/html/var/classes/definition_CustomerSegment.php(246): Pimcore\Model\DataObject\ClassDefinition\Data\TargetGroup::__set_state(Array)
pimcore/pimcore#10 /var/www/html/vendor/pimcore/pimcore/lib/Model/DataObject/ClassDefinition/ClassDefinitionManager.php(74): include('/var/www/html/v...')
pimcore/pimcore#11 /var/www/html/vendor/pimcore/pimcore/bundles/CoreBundle/src/Command/ClassesRebuildCommand.php(98): Pimcore\Model\DataObject\ClassDefinition\ClassDefinitionManager->createOrUpdateClassDefinitions()
pimcore/pimcore#12 /var/www/html/vendor/symfony/console/Command/Command.php(326): Pimcore\Bundle\CoreBundle\Command\ClassesRebuildCommand->execute(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
pimcore/pimcore#13 /var/www/html/vendor/symfony/console/Application.php(1096): Symfony\Component\Console\Command\Command->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
pimcore/pimcore#14 /var/www/html/vendor/symfony/framework-bundle/Console/Application.php(126): Symfony\Component\Console\Application->doRunCommand(Object(Pimcore\Bundle\CoreBundle\Command\ClassesRebuildCommand), Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
pimcore/pimcore#15 /var/www/html/vendor/symfony/console/Application.php(324): Symfony\Bundle\FrameworkBundle\Console\Application->doRunCommand(Object(Pimcore\Bundle\CoreBundle\Command\ClassesRebuildCommand), Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
pimcore/pimcore#16 /var/www/html/vendor/symfony/framework-bundle/Console/Application.php(80): Symfony\Component\Console\Application->doRun(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
pimcore/pimcore#17 /var/www/html/vendor/symfony/console/Application.php(175): Symfony\Bundle\FrameworkBundle\Console\Application->doRun(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
pimcore/pimcore#18 /var/www/html/vendor/symfony/runtime/Runner/Symfony/ConsoleApplicationRunner.php(49): Symfony\Component\Console\Application->run(Object(Symfony\Component\Console\Input\ArgvInput), Object(Symfony\Component\Console\Output\ConsoleOutput))
pimcore/pimcore#19 /var/www/html/vendor/autoload_runtime.php(29): Symfony\Component\Runtime\Runner\Symfony\ConsoleApplicationRunner->run()
pimcore/pimcore#20 /var/www/html/bin/console(20): require_once('/var/www/html/v...')
pimcore/pimcore#21 {main}

In ExceptionConverter.php line 49:

  An exception occurred while executing a query: SQLSTATE[42S02]: Base table
  or view not found: 1146 Table 'pimcore.targeting_target_groups' doesn't exi
  st


In Exception.php line 28:

  SQLSTATE[42S02]: Base table or view not found: 1146 Table 'pimcore.targetin
  g_target_groups' doesn't exist


In Connection.php line 71:

  SQLSTATE[42S02]: Base table or view not found: 1146 Table 'pimcore.targetin
  g_target_groups' doesn't exist

Even though the Pimcore install itself is successful, trying to install bundles manually after this will create and overwrite any class definitions related to the installed bundles, essentially discarding any changes made to bundle classes.

Expected Behavior

1. Pimcore installer runs without errors
2. Existing class definitions are detected during bundle installation and not overwritten

[fashxp]

Are you sure, you are using the latest version of the demo?

missing table should be created in bootstrap, see https://github.com/pimcore/demo/blob/11.x/dump/data-0-bootstrap.sql#L3183

[wwidergoldbsc]

I'm explicitly running the installer with the --skip-database-data-dump flag to reproduce the scenario in another project, where the setup relies on the pimcore installation working on a clean database.
The bug is not specific to the demo project, I just used it as an example.

[kingjia90]

Confirming the issue and it's surprisingly more related to pimcore/pimcore#16157 rather than the demo install itself.
Potentially we just need to add a flag to the command to allow skipping the class rebuild too on https://github.com/pimcore/pimcore/blob/9f9336f0b82e07b5c07d58370d464b9f70e7c5f1/bundles/InstallBundle/src/Installer.php#L446

Internally is a sort of "known issue", see https://github.com/pimcore/ee-pimcore/pull/19#issuecomment-1845337937

[wwidergoldbsc]

We were able to work around this issue by:

1. installing Pimcore without the included pimcore:deployment:classes-rebuild -c by removing the part you mentioned
2. running bin/console pimcore:bundle:install PimcorePersonalizationBundle --no-interaction to install the PimcorePersonalizationBundle and avoid the exception above regarding missing tables
3. running bin/console pimcore:deployment:classes-rebuild --create-classes --delete-classes -- without errors this time
4. installing all remaining bundles

[kingjia90]

Thank you for the feedback.

I've found a potential solution to this, by swapping the order like in pimcore/pimcore#16729 in combination with install-bundles option or (the step event install_classes), should work, at least by avoiding to manually do the steps you mentioned

/vendor/bin/pimcore-install \
  --admin-username=admin --admin-password=admin \
  --mysql-database=pimcore \
  --no-interaction
  --skip-database-data-dump
  --install-bundles=PimcorePersonalizationBundle

[kingjia90]

Ah wait, it's not intended to work like that, it's for the internal bundles (/bundles) only
https://github.com/pimcore/pimcore/pull/14194/files#diff-4eee5188dd424c74c2215a7595b21d4bde4443e3732f5b86d71484393076c498R62-R72
but i think we can tweak a little there to work for Personalization bundle too

[kingjia90]

@mattamon Do you see any contraindications on doing as pimcore/pimcore#16729?
Would be possible to extend the installable bundle to what the ones that are already enabled, but maybe the installer isn't yet run or so?
eg. by checking if is installable

[mattamon]

@kingjia90 no objections. I guess I was not fully aware of this.

If you want to install an external bundle, you can always add it via the EventSubscriber like we do it here.
https://github.com/pimcore/skeleton/blob/11.x/src/EventSubscriber/BundleSetupSubscriber.php

Not sure about checking installable though.

[kingjia90]

@wwidergoldbsc So i've merged pimcore/pimcore#16729 and with the suggestion to use the InstallEvents::EVENT_BUNDLE_SETUP to add the bundles that should be installed before rebuilding the class, it should work around the problem.

This issue is now considered as done, therefore closing for the moment, feel free to re-open (or even better a new one, but maybe not in the demo repository 😄 ) if the problem still occurs.

Thank you again for reporting and testing

[matthiashamacher]

@kingjia90
We've tested it and generally it works. One thing we noticed is that the installBundles() function is not executed if you only have the bundles in the Kernel.php. This is because of these lines of code: InstallCommand.php#L214-L226

It only executes the even subscriber if the --install-bundles option is set in the command. Our workaround is now, to also have the bundle set in that option, then it works. Maybe we should discuss to move the event subscriber execution out of that if? In that case either of those would trigger the bundle installation.

I'm happy to provide a PR for that change, just wanted to get your input first.

[kingjia90]

Thank you for the feedback, glad to hear that it works now.

Could be fine to move the event of that, but the current documentation is quite defined with the current implementations and could be tricky to refactor without BC breaks, what about a new event or so?

[matthiashamacher]

Yeah ok. I had a closer look into the function and I noticed that you don't have to give an option there. Previously we had --install-bundles=PimcorePersonalizationBundle and the bundle again as required bundle in the subscriber. Because of that duplicate notation I thought it might be a good idea to move that. But I now noticed, that you can just use --install-bundles which then triggers the subscriber and also installs the bundles. So I guess its ok then to have it only defined in one place. Maybe there is a use case where you don't want to install the bundles so this would be needed as it is.