Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DoctrineMetadataCacheWarmer must load metadata first, check priority of you warmers when updated from 2.2.4 -> 2.3.0 #1310

Closed
ivanbogomoloff opened this issue Mar 24, 2021 · 30 comments

Comments

@ivanbogomoloff
Copy link

Hello all.
I got this message when run bin/console cache:clear or bin/console cache:warmup.

Conditions:

I have symfony 5.2.4 project out box
Then i run composer update doctrine-bundle updated to 2.2.4 -> 2.3.0.

Then i got error.

DoctrineMetadataCacheWarmer must load metadata first, check priority of you warmers

@dmaicher
Copy link
Contributor

I cannot reproduce this on any of my projects. I also tested it on symfony-demo and it works fine.

Can you provide a reproducer @ivanbogomoloff ?

@ivanbogomoloff
Copy link
Author

I cannot reproduce this on any of my projects. I also tested it on symfony-demo and it works fine.

Can you provide a reproducer @ivanbogomoloff ?

No, i can't. It's magically gone.
But now It occurs only in project with a lot of code, but when i copy composer.json and try to create raw new project (i mean out box is composer command create project) it works fine.
Little notice, that error occurs only after rm -rf var/cache/prod and when APP_ENV=prod

@alexrozz
Copy link

I have the same issue on cache:clear, doctrine:migrations:migrate and any other.
Could anybody explain the reason of this exception? What warmers must run before?
In DoctrineMetadataCacheWarmer:47 there is a check if metadata exists, then crush. What the purpose, anybody knows?

@dmaicher
Copy link
Contributor

Could anybody explain the reason of this exception? What warmers must run before?
In DoctrineMetadataCacheWarmer:47 there is a check if metadata exists, then crush. What the purpose, anybody knows?

@alexrozz

With DoctrineBundle 2.3.0 we introduced a new and faster caching mechanism for the Doctrine meta data. For this to work properly we have to ensure that DoctrineMetadataCacheWarmer is the first to actually load all meta data so the cache can be properly warmed up.

Are you able to create a reproducer? Happy to take a look then

@jkabat
Copy link

jkabat commented Mar 30, 2021

I'm facing the issue as well. In my project it has relation to "msgphp/msgphp" package. In my understanding problematic is the change of warmers priority introduced in #1260 - other packages may be bitten as well if they depend on specific priority value.

@dmaicher
Copy link
Contributor

dmaicher commented Mar 30, 2021

In my understanding problematic is the change of warmers priority introduced in #1260

we did not change any priority of cache warmers 😉 We introduced a new DoctrineMetadataCacheWarmer with a high priority of 1000.

Again: if someone can provide a reproducer I can re-open this and take a look.

@jkabat
Copy link

jkabat commented Mar 30, 2021

I have created a reproducer, but I think it is related with the library Im using (it processes/loads entity metadata and should probably be run before doctrine warmup step which is not true anymore as new warmer has priority 1000, while msgphp one only 100).

https://github.com/jkabat/doctrine-bundle-test

bin/console cache:clear --env=prod --no-debug # http://prntscr.com/10zwd8u

In my application setup I had better insight with cache:warmup command: http://prntscr.com/10zwfkn. Unfortunately in reproducer I do not see this kind of error.

@ostrolucky
Copy link
Member

Reproducer doesn't work for me

❯ /usr/local/Cellar/php@7.4/7.4.16/bin/php bin/console cache:clear --env=prod --no-debug -vvv

 // Clearing the cache for the prod environment with debug false

 // Clearing outdated warmup directory...

 // Warming up cache...

[info] User Deprecated: The "metadata_cache_driver" configuration key is deprecated. PHP Array cache is now automatically registered when %kernel.debug% is false.

[info] User Deprecated: The "metadata_cache_driver" configuration key is deprecated. PHP Array cache is now automatically registered when %kernel.debug% is false.


In MappingException.php line 65:

  [Doctrine\Persistence\Mapping\MappingException]
  No mapping file found named 'EmailPassword.orm.xml' for class 'MsgPhp\User\Credential\EmailPassword'.


Exception trace:
  at /Users/gabriel.ostrolucky/Documents/doctrine-bundle-test/vendor/doctrine/persistence/lib/Doctrine/Persistence/Mapping/MappingException.php:65
 Doctrine\Persistence\Mapping\MappingException::mappingFileNotFound() at /Users/gabriel.ostrolucky/Documents/doctrine-bundle-test/vendor/doctrine/persistence/lib/Doctrine/Persistence/Mapping/Driver/SymfonyFileLocator.php:230
 Doctrine\Persistence\Mapping\Driver\SymfonyFileLocator->findMappingFile() at /Users/gabriel.ostrolucky/Documents/doctrine-bundle-test/vendor/doctrine/persistence/lib/Doctrine/Persistence/Mapping/Driver/FileDriver.php:92
 Doctrine\Persistence\Mapping\Driver\FileDriver->getElement() at /Users/gabriel.ostrolucky/Documents/doctrine-bundle-test/vendor/doctrine/orm/lib/Doctrine/ORM/Mapping/Driver/XmlDriver.php:67
 Doctrine\ORM\Mapping\Driver\XmlDriver->loadMetadataForClass() at /Users/gabriel.ostrolucky/Documents/doctrine-bundle-test/vendor/doctrine/persistence/lib/Doctrine/Persistence/Mapping/Driver/MappingDriverChain.php:79
 Doctrine\Persistence\Mapping\Driver\MappingDriverChain->loadMetadataForClass() at /Users/gabriel.ostrolucky/Documents/doctrine-bundle-test/vendor/doctrine/orm/lib/Doctrine/ORM/Mapping/ClassMetadataFactory.php:149
 Doctrine\ORM\Mapping\ClassMetadataFactory->doLoadMetadata() at /Users/gabriel.ostrolucky/Documents/doctrine-bundle-test/vendor/doctrine/persistence/lib/Doctrine/Persistence/Mapping/AbstractClassMetadataFactory.php:306
 Doctrine\Persistence\Mapping\AbstractClassMetadataFactory->loadMetadata() at /Users/gabriel.ostrolucky/Documents/doctrine-bundle-test/vendor/doctrine/orm/lib/Doctrine/ORM/Mapping/ClassMetadataFactory.php:79
 Doctrine\ORM\Mapping\ClassMetadataFactory->loadMetadata() at /Users/gabriel.ostrolucky/Documents/doctrine-bundle-test/vendor/doctrine/persistence/lib/Doctrine/Persistence/Mapping/AbstractClassMetadataFactory.php:184
 Doctrine\Persistence\Mapping\AbstractClassMetadataFactory->getMetadataFor() at /Users/gabriel.ostrolucky/Documents/doctrine-bundle-test/vendor/msgphp/domain/Infrastructure/Doctrine/Event/ObjectMappingListener.php:163
 MsgPhp\Domain\Infrastructure\Doctrine\Event\ObjectMappingListener->getMetadata() at /Users/gabriel.ostrolucky/Documents/doctrine-bundle-test/vendor/msgphp/domain/Infrastructure/Doctrine/Event/ObjectMappingListener.php:100
 MsgPhp\Domain\Infrastructure\Doctrine\Event\ObjectMappingListener->processFieldMapping() at /Users/gabriel.ostrolucky/Documents/doctrine-bundle-test/vendor/msgphp/domain/Infrastructure/Doctrine/Event/ObjectMappingListener.php:77
 MsgPhp\Domain\Infrastructure\Doctrine\Event\ObjectMappingListener->processClassFields() at /Users/gabriel.ostrolucky/Documents/doctrine-bundle-test/vendor/msgphp/domain/Infrastructure/Doctrine/Event/ObjectMappingListener.php:81
 MsgPhp\Domain\Infrastructure\Doctrine\Event\ObjectMappingListener->processClassFields() at /Users/gabriel.ostrolucky/Documents/doctrine-bundle-test/vendor/msgphp/domain/Infrastructure/Doctrine/Event/ObjectMappingListener.php:62
 MsgPhp\Domain\Infrastructure\Doctrine\Event\ObjectMappingListener->loadClassMetadata() at /Users/gabriel.ostrolucky/Documents/doctrine-bundle-test/vendor/symfony/doctrine-bridge/ContainerAwareEventManager.php:64
 Symfony\Bridge\Doctrine\ContainerAwareEventManager->dispatchEvent() at /Users/gabriel.ostrolucky/Documents/doctrine-bundle-test/vendor/doctrine/orm/lib/Doctrine/ORM/Mapping/ClassMetadataFactory.php:239
 Doctrine\ORM\Mapping\ClassMetadataFactory->doLoadMetadata() at /Users/gabriel.ostrolucky/Documents/doctrine-bundle-test/vendor/doctrine/persistence/lib/Doctrine/Persistence/Mapping/AbstractClassMetadataFactory.php:306
 Doctrine\Persistence\Mapping\AbstractClassMetadataFactory->loadMetadata() at /Users/gabriel.ostrolucky/Documents/doctrine-bundle-test/vendor/doctrine/orm/lib/Doctrine/ORM/Mapping/ClassMetadataFactory.php:79
 Doctrine\ORM\Mapping\ClassMetadataFactory->loadMetadata() at /Users/gabriel.ostrolucky/Documents/doctrine-bundle-test/vendor/doctrine/persistence/lib/Doctrine/Persistence/Mapping/AbstractClassMetadataFactory.php:184
 Doctrine\Persistence\Mapping\AbstractClassMetadataFactory->getMetadataFor() at /Users/gabriel.ostrolucky/Documents/doctrine-bundle-test/vendor/doctrine/persistence/lib/Doctrine/Persistence/Mapping/AbstractClassMetadataFactory.php:90
 Doctrine\Persistence\Mapping\AbstractClassMetadataFactory->getAllMetadata() at /Users/gabriel.ostrolucky/Documents/doctrine-bundle-test/vendor/doctrine/doctrine-bundle/CacheWarmer/DoctrineMetadataCacheWarmer.php:51
 Doctrine\Bundle\DoctrineBundle\CacheWarmer\DoctrineMetadataCacheWarmer->doWarmUp() at /Users/gabriel.ostrolucky/Documents/doctrine-bundle-test/vendor/symfony/framework-bundle/CacheWarmer/AbstractPhpFileCacheWarmer.php:49
 Symfony\Bundle\FrameworkBundle\CacheWarmer\AbstractPhpFileCacheWarmer->warmUp() at /Users/gabriel.ostrolucky/Documents/doctrine-bundle-test/vendor/symfony/http-kernel/CacheWarmer/CacheWarmerAggregate.php:97
 Symfony\Component\HttpKernel\CacheWarmer\CacheWarmerAggregate->warmUp() at /Users/gabriel.ostrolucky/Documents/doctrine-bundle-test/vendor/symfony/http-kernel/Kernel.php:633
 Symfony\Component\HttpKernel\Kernel->initializeContainer() at /Users/gabriel.ostrolucky/Documents/doctrine-bundle-test/vendor/symfony/http-kernel/Kernel.php:136
 Symfony\Component\HttpKernel\Kernel->boot() at /Users/gabriel.ostrolucky/Documents/doctrine-bundle-test/vendor/symfony/http-kernel/Kernel.php:153
 Symfony\Component\HttpKernel\Kernel->reboot() at /Users/gabriel.ostrolucky/Documents/doctrine-bundle-test/vendor/symfony/framework-bundle/Command/CacheClearCommand.php:189
 Symfony\Bundle\FrameworkBundle\Command\CacheClearCommand->warmup() at /Users/gabriel.ostrolucky/Documents/doctrine-bundle-test/vendor/symfony/framework-bundle/Command/CacheClearCommand.php:129
 Symfony\Bundle\FrameworkBundle\Command\CacheClearCommand->execute() at /Users/gabriel.ostrolucky/Documents/doctrine-bundle-test/vendor/symfony/console/Command/Command.php:255
 Symfony\Component\Console\Command\Command->run() at /Users/gabriel.ostrolucky/Documents/doctrine-bundle-test/vendor/symfony/console/Application.php:1027
 Symfony\Component\Console\Application->doRunCommand() at /Users/gabriel.ostrolucky/Documents/doctrine-bundle-test/vendor/symfony/framework-bundle/Console/Application.php:97
 Symfony\Bundle\FrameworkBundle\Console\Application->doRunCommand() at /Users/gabriel.ostrolucky/Documents/doctrine-bundle-test/vendor/symfony/console/Application.php:273
 Symfony\Component\Console\Application->doRun() at /Users/gabriel.ostrolucky/Documents/doctrine-bundle-test/vendor/symfony/framework-bundle/Console/Application.php:83
 Symfony\Bundle\FrameworkBundle\Console\Application->doRun() at /Users/gabriel.ostrolucky/Documents/doctrine-bundle-test/vendor/symfony/console/Application.php:149
 Symfony\Component\Console\Application->run() at /Users/gabriel.ostrolucky/Documents/doctrine-bundle-test/bin/console:42

cache:clear [--no-warmup] [--no-optional-warmers] [-h|--help] [-q|--quiet] [-v|vv|vvv|--verbose] [-V|--version] [--ansi] [--no-ansi] [-n|--no-interaction] [-e|--env ENV] [--no-debug] [--] <command>

❯

@Eydamos
Copy link

Eydamos commented Jun 17, 2021

I have the same issue. Unfortunately the project is private so I can't share it. I also, up to this point, failed to reproduce the behaviour in a new project. But after hours of debugging I can already give some insight why this is happening.

In my project I have several connections/entity manager and therefore multiple cache warmer are registered all with priority 1000.

For simplicity let's say we have the connections conn_a and conn_b which have the entity managers em_a and em_b and their cache warmers are also registered in this order. So we have two iterations of cache warmers first the cache warmer who gets the entity manager em_a and then the cache warmer who gets the entity manager em_b.

Now when I clear the cache the following happens:

Iteration 1 (em_a)

In DoctrineMetadataCacheWarmer in line 59 we have $metadataFactory->getAllMetadata(); which will load all the metadata for all entities of the entity manager em_a.
During this method call the ClassMetadataFactory will call in line 244 if ($this->evm->hasListeners(Events::loadClassMetadata)) {.

And here comes the tricky part I was not able to reproduce yet as I don't know how symfony determines which event manager should get a certain subscriber but the following happens:

The event manager of em_a has a doctrine event subscriber assigned that I have written.
This subscriber has a dependency on a custom repository.
This custom repository will get the entity manager em_b and calls $entityManager->getRepository(SomeEntityFromConnB::class) in it's constructor.
Now the entity_manager em_b will generate the metadata for SomeEntityFromConnB class.

Iteration 2 (em_b)

Now the cache warmer of entity manager em_b will do it's run and will fetch from it the meta data factory in line 47 of the DoctrineMetadataCacheWarmer and as the entity manager has already created metadata for SomeEntityFromConnB even though there are still several entities that do not have meta data generated the if condition in line 48 will throw an exception.

I will continue my quest to provide an example so you can debug it yourself but I hope I have already clearly outlined the cause for the bug.

@dmaicher
Copy link
Contributor

@TimWerdin this sounds indeed like it could be the issue...

But then creating a reproducer should be possible when simply following your comments?

@Eydamos
Copy link

Eydamos commented Jun 18, 2021

I guess it will not take too much time to be able to produce a demo but I'm currently busy with testing and deploying a big project. I will continue my work next Thuesday

@Eydamos
Copy link

Eydamos commented Jun 28, 2021

I was a little bit busy last week but now I could prepare a demo application which shows the cause of the bug. You can find it here: https://github.com/TimWerdin/doctrine-bug-demo

@dmaicher
Copy link
Contributor

dmaicher commented Jun 28, 2021

The reproducer looks good to me. Thanks @TimWerdin

@Eydamos
Copy link

Eydamos commented Sep 10, 2021

Is there any progress on this issue? This is limiting us to use older versions of doctrine to prevent conflicts

@dmaicher
Copy link
Contributor

Is there any progress on this issue? This is limiting us to use older versions of doctrine to prevent conflicts

Did not have much time recently. You could test #1378 and give some feedback there

@DoobleD
Copy link

DoobleD commented Nov 28, 2021

Anybody has a workaround for this? I cannot downgrade to 2.2.4 because of another issue in that version which was solved on 2.3.

@cywbl
Copy link

cywbl commented Feb 7, 2022

Hi all- just adding some info, I am experiencing this error on doctrine-bundle v2.5.5. Does anyone have a workaround?

@dmaicher
Copy link
Contributor

dmaicher commented Feb 7, 2022

A workaround is to define a cache pool for the metadata yourself.

See the older version of the recipe for example.

@pscheit
Copy link

pscheit commented Jun 1, 2022

note to me: make sure that the metadata cache key applies to all managers in the config (to apply the workaround) I had one manager triggering the autoloading of a repo in the other manager as well

@lin-lu
Copy link

lin-lu commented Jun 14, 2022

A workaround is to define a cache pool for the metadata yourself.

See the older version of the recipe for example.

Hi @dmaicher, could you explain a bit more in detail how the workaround works? I have the same config in the doctrine.yaml config file, but still got the same error. Thanks!

@dmaicher
Copy link
Contributor

A workaround is to define a cache pool for the metadata yourself.
See the older version of the recipe for example.

Hi @dmaicher, could you explain a bit more in detail how the workaround works? I have the same config in the doctrine.yaml config file, but still got the same error. Thanks!

which version of DoctrineBundle are you using? latest?

If you configure your own psr6 metadata cache then the DoctrineMetadataCacheWarmer should not be doing anything.

@Eydamos
Copy link

Eydamos commented Jun 17, 2022

which version of DoctrineBundle are you using? latest?

If you configure your own psr6 metadata cache then the DoctrineMetadataCacheWarmer should not be doing anything.

Lin is currently on vacation so I will continue the conversation for him.
We are using doctrine/doctrine-bundle in version 2.6.3
Our config/packages/doctrine.yaml only defines the connections and entity managers

doctrine:
    dbal:
        connections:
            foo:
                driver: pdo_mysql
                host: '%env(HOST_FOO)%'
                user:  '%env(USER_FOO)%'
                password: '%env(PASSWORD_FOO)%'
                dbname: '%env(DB_NAME_FOO)%'
                server_version: '%env(SERVER_VERSION_FOO)%'
                keep_slave: true
            bar:
                driver: pdo_mysql
                host: '%env(HOST_BAR)%'
                user:  '%env(USER_BAR)%'
                password: '%env(PASSWORD_BAR)%'
                dbname: '%env(DB_NAME_BAR)%'
                server_version: '%env(SERVER_VERSION_BAR)%'
                keep_slave: true
    orm:
        entity_managers:
            foo:
                naming_strategy: doctrine.orm.naming_strategy.underscore
                connection: foo
            bar:
                naming_strategy: doctrine.orm.naming_strategy.underscore
                connection: bar

and our config/packages/prod/doctrine.yaml is the default one from the recipe:

doctrine:
    orm:
        auto_generate_proxy_classes: false
        metadata_cache_driver:
            type: pool
            pool: doctrine.system_cache_pool
        query_cache_driver:
            type: pool
            pool: doctrine.system_cache_pool
        result_cache_driver:
            type: pool
            pool: doctrine.result_cache_pool

framework:
    cache:
        pools:
            doctrine.result_cache_pool:
                adapter: cache.app
            doctrine.system_cache_pool:
                adapter: cache.system

Sorry if I still don't get it but what exactly do I have to do?
Do I need to add an entry to framework.cache.pools and replace the value for doctrine.orm.metadata_cache_driver.pool then?

@dmaicher
Copy link
Contributor

@TimWerdin so then you are only configuring the cache in config/packages/prod/doctrine.yaml for the default connection/manager? What about the other one?

@Eydamos
Copy link

Eydamos commented Jun 17, 2022

We are not aware that we need to configure a cache per connection/entity_manager or that this is even possible. At least I can't find anything in the documentation
I was of the impression that the cache is always used globally for all connections/entity_managers.

@barbuslex
Copy link

I have the same error message when i am execute symfony console c:c on production environnement. Have you a workaround to fix it temporarily ? Thanks

@barbuslex
Copy link

barbuslex commented Sep 14, 2022

I have fixed it.

In config/packages/doctrine.yaml :

  • Replace :
when@prod:
    doctrine:
        orm:
            auto_generate_proxy_classes: false
            query_cache_driver:
                type: pool
                pool: doctrine.system_cache_pool
            result_cache_driver:
                type: pool
                pool: doctrine.result_cache_pool
  • With :
when@prod:
    doctrine:
        orm:
            auto_generate_proxy_classes: false
            metadata_cache_driver:
                type: pool
                pool: doctrine.system_cache_pool
            query_cache_driver:
                type: pool
                pool: doctrine.system_cache_pool
            result_cache_driver:
                type: pool
                pool: doctrine.result_cache_pool

Just add :

metadata_cache_driver:
    type: pool
    pool: doctrine.system_cache_pool

@mdevlamynck
Copy link

mdevlamynck commented Apr 7, 2023

I've encountered the same issue in tests:

  • we have multiple connections (different sqlite db in various states)
  • we run with kernel.debug=false to speed up the tests

This results in the cache warmer being added twice during autoconfiguration:

$ bin/console debug:container --env=test --tag=kernel.cache_warmer

Symfony Container Services Tagged with "kernel.cache_warmer" Tag
================================================================

 --------------------------------------------------- ---------- -------------------------------------------------------------------------
  Service ID                                          priority   Class name
 --------------------------------------------------- ---------- -------------------------------------------------------------------------
  doctrine.orm.default_metadata_cache_warmer          1000       Doctrine\Bundle\DoctrineBundle\CacheWarmer\DoctrineMetadataCacheWarmer
  doctrine.orm.empty_database_metadata_cache_warmer   1000       Doctrine\Bundle\DoctrineBundle\CacheWarmer\DoctrineMetadataCacheWarmer
  cache_pool_clearer.cache_warmer                     64         Symfony\Bundle\FrameworkBundle\CacheWarmer\CachePoolClearerCacheWarmer
  ...

The workaround above works because the cache warmer is only added if no metadata_cache_driver is defined.

@hothson
Copy link

hothson commented Jun 14, 2023

Remove APP_DEBUG=0 in .env file

@ivanbogomoloff
Copy link
Author

i don't encountered this issue in version ^2.7 , so i decided to close issue

@justin-oh
Copy link

justin-oh commented Aug 18, 2023

I'm commenting in case anyone encounters this specific error through a Google search. (Sorry in advance if this comment re-opens the issue.)

In my case, the error was caused by me initializing something in a service constructor that queries the database. Example:

<?php

namespace App\Service;

class MyService
{
    private $data;
    private OtherService $otherService;

    public function __construct(OtherService $otherService)
    {
        $this->otherService = $otherService;

        $this->data = $otherService->queryForData();
    }
}

My solution was to initialize $this->data in a separate function:

private $dataInit = false;

private function initData(): void
{
    if ($this->dataInit) {
        return;
    }

    $this->dataInit = true;

    $this->data = $otherService->queryForData();
}

Then I could call $this->initData() before I need to reference $this->data. The query is only performed once and only when absolutely necessary.

In hindsight, I should have took this approach regardless of any error. Especially in a Dependency Injection framework like Symfony.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.