Cache of Doctrine SQL Statement leads to 503 error in Apache #44270

Art4 · 2021-11-25T11:07:00Z

Art4
Nov 25, 2021

Hello 👋

(Please note the Update 1 below)

I'm not sure if this is the right place to ask, but I'm a little lost with this mysterious problem...

The problem occurs only on the prod server and I was not able to reproduce it locally.

Lets say I have a Symfony app (v5.3.10) with a page under http://example.org/admin where the admin dashboard lives. After a new deploy of the app I can only request /admin one time successfully. It shows me the admin dashboard without errors.
But if I request this page a second time the server gives me immediately (not after some seconds or a timeout) a 503 error.

The full error response is:

HTTP/2 503 Service Unavailable
content-length: 299
content-type: text/html; charset=iso-8859-1
date: Thu, 25 Nov 2021 09:09:08 GMT
server: Apache
X-Firefox-Spdy: h2

<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>503 Service Unavailable</title>
</head><body>
<h1>Service Unavailable</h1>
<p>The server is temporarily unable to service your
request due to maintenance downtime or capacity
problems. Please try again later.</p>
</body></html>

Obviously this error comes from Apache, not from Symfony. But interesting is that if I clear the cache and then request the /admin page it will be show successfully one time. On the second request (and every other request) Apache will response with 503 error again. After a cache:clear the page will work again one time.

This problem occurs only on a specific route or controller. So /admin is affected, but not /admin/subpage.
I first thought that this problem has something to do with the AdminController, so I've debugged the controller (yes, on prod server 🙈 ) to simply show some fake data in a simple rendered twig template but the problem is the same. I've also found out that this error occurs somewhere after the controller returns the Response class (by calling $this->render()).

And now I've found 2 other endpoints/controller with the same strange behaviour. It seems that this problem occurs independent from the other controllers. So I can request /admin one time, after then it returns the 503 error. Next I can request /another-route one time, and the next request will return the 503 error and so on.

Another point that I noticed is that this behavior first occurred after I switched from the legacy security system to the new authenticator-based Security system, so maybe there is a connection there too.
see Update 1: This behavior occured first after I switched from doctrine/orm 2.9.5 to 2.10.1.

I've looked into the server logs but there is nothing logged. So for me it seems like a weird bug with the Symfony cache and Symfony Security in combination with Apache.

I'm not sure if someone could help me but I would be glad for every hint what or where I could look next to find the root of this problem.

Thank you.

Update 1 (2021-11-27)

I was able to reproduce the issue on a staging area and looked deeper into the cache problem. I've found out that the problem comes from a specific Doctrine cache. In my case of the /admin endpoint there will be created a cache in var/cache/prod/pools/iWwGDKOazk/R/6/085f6b9716d8e39e78c0de0dcdefffd8. If I remove only this cache file the request will work one time because on the next request the cache file will be there again.

This behavior occured first after I switched from doctrine/orm 2.9.5 to 2.10.1, so maybe there is a connection too.

This is the content of the cache file:

<?php //085f6b9716d8e39e78c0de0dcdefffd8

return [PHP_INT_MAX, static function () { return \Symfony\Component\VarExporter\Internal\Hydrator::hydrate(
    $o = [
        clone (($p = &\Symfony\Component\VarExporter\Internal\Registry::$prototypes)['Doctrine\\ORM\\Query\\ParserResult'] ?? \Symfony\Component\VarExporter\Internal\Registry::p('Doctrine\\ORM\\Query\\ParserResult')),
        clone ($p['Doctrine\\ORM\\Query\\Exec\\SingleSelectExecutor'] ?? \Symfony\Component\VarExporter\Internal\Registry::p('Doctrine\\ORM\\Query\\Exec\\SingleSelectExecutor')),
        clone ($p['Doctrine\\ORM\\Query\\ResultSetMapping'] ?? \Symfony\Component\VarExporter\Internal\Registry::p('Doctrine\\ORM\\Query\\ResultSetMapping')),
    ],
    null,
    [
        'Doctrine\\ORM\\Query\\ParserResult' => [
            '_sqlExecutor' => [
                $o[1],
            ],
            '_resultSetMapping' => [
                $o[2],
            ],
            '_parameterMappings' => [
                [
                    1 => [
                        0,
                    ],
                ],
            ],
        ],
        'Doctrine\\ORM\\Query\\Exec\\AbstractSqlExecutor' => [
            '_sqlStatements' => [
                1 => 'SELECT count(u0_.id) AS sclr_0 FROM users u0_ WHERE u0_.deleted_at IS NULL AND u0_.created_at > ?',
            ],
        ],
        'stdClass' => [
            'scalarMappings' => [
                2 => [
                    'sclr_0' => 1,
                ],
            ],
            'typeMappings' => [
                2 => [
                    'sclr_0' => 'integer',
                ],
            ],
        ],
    ],
    $o[0],
    []
); }];
~

Answered by thomas2411

Aug 19, 2022

After digging deeper I found that as before, the problem is with cache.system I do not have APCu on server so with this information: https://symfony.com/blog/new-in-symfony-3-1-cache-component#symfony-integration we know that cache.system uses APCu by default, if not found fallback to filesystem.

So with this setting:

framework:
    cache:
        app: cache.adapter.redis
        pools:
            doctrine.result_cache_pool:
                adapter: cache.app
            doctrine.system_cache_pool:
                adapter: cache.system

I have 503 error. If I change cache.system to cache.adapter.redis it works. And what is weird if I change cache.system to cache.adapter.filesystem it wor…

View full answer

Art4 · 2021-11-29T15:25:50Z

Art4
Nov 29, 2021
Author

I'm sure that this is a problem with doctrine/orm or doctrine/dbal, so I created a ticket there, see doctrine/orm#9207.

Update: It looks like this problem only occurs with PHP 7.4. After switching to PHP 8, the problem was solved.

0 replies

DigitalTimK · 2022-01-18T10:11:14Z

DigitalTimK
Jan 18, 2022

Thank you for this post. The description sound similar to the problem I am facing. So I don't feel so alone and desperate anymore ;-)

3 replies

Art4 Jan 18, 2022
Author

You're welcome. My "workaround" was an update to PHP 8.

DigitalTimK Jan 18, 2022

I saw this in doctrine/orm#9207, but I cannot change PHP Version on the Production Server.

Art4 Jan 18, 2022
Author

Maybe you could change the framework.cache.pools.doctrine.system_cache_pool config from cache.system to cache.app. This had also worked in my tests, but I don`t know all the effects of this change.

bezin · 2022-07-20T06:58:04Z

bezin
Jul 20, 2022

Hello, I ran into this issue the other day as well with a project and I can confirm that swapping the Cache fixes this issue. My hoster digged into this more deeply and came to the conclusion, that the Ion Cube Loader misinterprets the format of the cache files as its own file format, tries to parse it and fails. This crashes the PHP process and hence Apache returns 503.

I am not gonna dig into this more deeply as it is supposed to be a PHP 7.4 issue only and we will be on PHP 8.0 soon with this project, but maybe this information saves somebody some hours :-)

0 replies

thomas2411 · 2022-08-14T13:44:05Z

thomas2411
Aug 14, 2022

I had the same problem today. Out of nowhere it started 503 Service unavailable but only for some of the Entities in my project, not all of them. Removing query_cache_driver removed the issue. Changing doctrine.system_cache_pool config from cache.system to cache.app fixed the problem too, for now. I have:

PHP 8.1.1
Symfony 6.1.3 and
"doctrine/annotations": "^1.13.3",
"doctrine/doctrine-bundle": "^2.7.0",
"doctrine/doctrine-migrations-bundle": "^3.2.2",
"doctrine/orm": "^2.13.1"

It drives me nuts. I spent with server admin a few hours and what they said it is something from the PHP app. A switched from Redis cache to filesystem and the same issue appears so it is not a Redis problem. In DEV environment, where this cache is not used it works fine. On prod does not.

On logs I cannot see anything. Just when it runs query to problematic Entity it disconnecting and throws 503.

3 replies

Art4 Aug 15, 2022
Author

What's your PHP version?

thomas2411 Aug 15, 2022

8.1.1. This week I will try to update to 8.1.9

thomas2411 Aug 16, 2022

We have update to 8.1.9, the problem still exists. What is interesting is that the same situation is on 3 other servers (but the same hosting company) where there are other apps in Symfony, different version of Symfony, Doctrine etc. And it happened a few days ago. So it might be connected with server software but the hosting company is not willing to tell me what has changed.

thomas2411 · 2022-08-19T08:46:03Z

thomas2411
Aug 19, 2022

After digging deeper I found that as before, the problem is with cache.system I do not have APCu on server so with this information: https://symfony.com/blog/new-in-symfony-3-1-cache-component#symfony-integration we know that cache.system uses APCu by default, if not found fallback to filesystem.

So with this setting:

framework:
    cache:
        app: cache.adapter.redis
        pools:
            doctrine.result_cache_pool:
                adapter: cache.app
            doctrine.system_cache_pool:
                adapter: cache.system

I have 503 error. If I change cache.system to cache.adapter.redis it works. And what is weird if I change cache.system to cache.adapter.filesystem it works too. But if I change cache.system to cache.adapter.apcu I get an error while clearing the cache that apcu is not enabled in php. So what is going on here?

1 reply

Art4 Dec 13, 2022
Author

I just updated to PHP 8.1 and got this error again. After changing the cache from cache.system for cache.app and clearing the cache the 503 error disappears.

thomas2411 · 2022-08-22T10:08:26Z

thomas2411
Aug 22, 2022

As @bezin wrote, the same situation was here. The server admin finally admitted that IonCube might have been updated recently. The remove it from my PHP setup and it works.

5 replies

stof Sep 13, 2022
Collaborator

if IonCube breaks the Symfony filesystem cache adapter, I suggest reporting a bug to IonCube. Their extension should not break working code.

creiner Sep 20, 2022

Did anybody reported the bug to the IonCube helpdesk?
Since it needs registration, I cannot see it.

kevinpapst Oct 22, 2022

Same issue here. Hosting company All-Inkl forces that ioncube stuff on every customer.
I am trying to get their support have a look and raise a ticket, they should be big enough.

justpilot Oct 22, 2022

Same issue here. Hosting company All-Inkl forces that ioncube stuff on every customer. I am trying to get their support have a look and raise a ticket, they should be big enough.

Same here, all-inkl.com customer. I called the support. They are unwilling to change a thing.

obspsr Nov 24, 2022

Same issue which costs me 4 hours ... I did exactly the same as did Art4 at top of discussion and was very close to
revert my code to old fashion without using Response.... BUT !

public function indexAction(MemberRepository
            $membersRepository): Response {
        return $this->render('member/index.html.twig', [
                    'piwamMembers' => $membersRepository->findBy(
                            [],
                            ['lastname' => 'ASC'],)
        ]);
    }

Thanks to your help, I am on hosting24 with CPanel/PHP version minimal control (I use .htaccess in zones ...)
And I found IonCube activated ... I deactivated and all is wright !!
Thanks for coop,
Serge

My 'bin/console about' if it can help ....

-------------------- --------------------------------- 
  Symfony                                               
 -------------------- --------------------------------- 
  Version              5.4.13                           
  Long-Term Support    Yes                              
  End of maintenance   11/2024 (in +737 days)           
  End of life          11/2025 (in +1102 days)          
 -------------------- --------------------------------- 
  Kernel                                                
 -------------------- --------------------------------- 
  Type                 App\Kernel                       
  Environment          dev                              
  Debug                true                             
  Charset              UTF-8                            
  Cache directory      ./var/cache/dev (30.8 MiB)       
  Build directory      ./var/cache/dev (30.8 MiB)       
  Log directory        ./var/log (147.4 MiB)            
 -------------------- --------------------------------- 
  PHP                                                   
 -------------------- --------------------------------- 
  Version              8.0.25                           
  Architecture         64 bits                          
  Intl locale          en_US                            
  Timezone             UTC (2022-11-24T13:48:19+00:00)  
  OPcache              true                             
  APCu                 true                             
  Xdebug               true                             
 -------------------- ---------------------------------

justpilot · 2022-09-13T10:38:54Z

justpilot
Sep 13, 2022

I have the same error from now on, without changing my scripts.
When I clear the cache I can call the API Endpoint once, after that it's status 503.

PHP: 8.1.7

This program makes use of the Zend Scripting Language Engine:
Zend Engine v4.1.7, Copyright (c) Zend Technologies
with the ionCube PHP Loader v12.0.2, Copyright (c) 2002-2022, by ionCube Ltd.
with Zend OPcache v8.1.7, Copyright (c), by Zend Technologies

<title>503 Service Unavailable</title>

<h1>Service Unavailable</h1>
<p>The server is temporarily unable to service your
request due to maintenance downtime or capacity
problems. Please try again later.</p>

Anybody with a solution?

1 reply

justpilot Sep 13, 2022

As suggested above the temporary solution is:

File: doctrine.yaml

cache.system -> cache.adapter.filesystem

[...]

    framework:
        cache:
            pools:
                doctrine.result_cache_pool:
                    adapter: cache.app
                doctrine.system_cache_pool:
                    adapter: cache.adapter.filesystem
                    #adapter: cache.system

justpilot · 2022-10-22T10:49:36Z

justpilot
Oct 22, 2022

Please check if the problem only occurs when dealing with DateTimeImmutable property type.
I changed the type to DateTime Mutable and the problem with the cache is gone.

0 replies

joerndyherrn · 2022-10-24T07:31:03Z

joerndyherrn
Oct 24, 2022

@justpilot seems not to be the problem...

1 reply

justpilot Oct 24, 2022

@justpilot seems not to be the problem...

Thank you, then this was just a coincidence!

e9li · 2022-11-11T11:04:06Z

e9li
Nov 11, 2022

Yesterday this error out of nowhere occurred in a project of mine. It affects only some of the database queries and only on mariaDB (10.3.36) and in production, on mySQL (5.7.34) everything works fine. I do not have the time to dive into this but maybe my dirty quick fix helps someone.

In my case removing "left join" helps to get the queries to work again.

public function getEventsForCalender(DateTime $startDate, DateTime $endDate): array
{
    return $this->createQueryBuilder('event')
//            ->leftJoin('event.tag', 'tag')
//            ->addSelect('tag')
//            ->leftJoin('event.locations', 'locations')
//            ->addSelect('locations')
        ->andWhere('event.user IS NULL')
        ->andWhere('event.startDate >= :startDate and event.startDate <= :endDate')
        ->setParameter('startDate', $startDate)
        ->setParameter('endDate', $endDate)
        ->addOrderBy('event.startDate', 'ASC')
        ->getQuery()
        ->getResult();
}

As soon as I add the joins the 503 occurs, on mariaDB.
Interesting that only some queries are affected by this error.
The following query with the same joins works without any error.

public function getAbsencesForDashboard(User|UserInterface $user): array
{
    return $this->createQueryBuilder('event')
        ->leftJoin('event.user', 'user')
        ->addSelect('user')
        ->leftJoin('event.tag', 'tag')
        ->addSelect('tag')
        ->leftJoin('event.locations', 'locations')
        ->addSelect('locations')
        ->andWhere('event.startDate = CURRENT_DATE()')
        ->andWhere('event.user IS NOT NULL')
        ->getQuery()
        ->getResult();
}

Symfony 6.1
doctrine/annotations 1.13.3
doctrine/doctrine-bundle 2.7.1
doctrine/orm 1.13.3

0 replies

justpilot · 2022-12-20T18:50:08Z

justpilot
Dec 20, 2022

I started getting some problems again.

I'm using API Platform. My API endpoint has the security setting 'is_granted("ROLE_USER") and object.customer == user'.
When I clear the cache (e.g. just by deleting the var/cache directory) the first call to this API endpoint works fine - Status 200.
But when I call the API point the second time, I'm getting a 403 - Forbidden error.
When I clear the cache again, I can call the endpoint again without a problem, but just once.

 -------------------- ------------------------------------------- 
  Symfony                                                         
 -------------------- ------------------------------------------- 
  Version              6.2.2                                      
  Long-Term Support    No                                         
  End of maintenance   07/2023 (in +223 days)                     
  End of life          07/2023 (in +223 days)                     
 -------------------- ------------------------------------------- 
  Kernel                                                          
 -------------------- ------------------------------------------- 
  Type                 App\Kernel                                 
  Environment          prod                                       
  Debug                false                                      
  Charset              UTF-8                                      
  Cache directory      ./var/cache/prod (5.7 MiB)                 
  Build directory      ./var/cache/prod (5.7 MiB)                 
  Log directory        ./var/log (0 B)                            
 -------------------- ------------------------------------------- 
  PHP                                                             
 -------------------- ------------------------------------------- 
  Version              8.1.13                                     
  Architecture         64 bits                                    
  Intl locale          de_DE                                      
  Timezone             Europe/Berlin (2022-12-20T19:40:53+01:00)  
  OPcache              false                                      
  APCu                 false                                      
  Xdebug               false                                      
 -------------------- -------------------------------------------

And checking the PHP info:

This program makes use of the Zend Scripting Language Engine:
Zend Engine v4.1.13, Copyright (c) Zend Technologies
with the ionCube PHP Loader v12.0.2, Copyright (c) 2002-2022, by ionCube Ltd.
with Zend OPcache v8.1.13, Copyright (c), by Zend Technologies

As you see the ionCube is activated on this server.

Here is my doctrine.yaml file:


doctrine:
    dbal:
        #url: '%env(resolve:DATABASE_URL)%'
        url: '%env(DATABASE_URL)%'

        # IMPORTANT: You MUST configure your server version,
        # either here or in the DATABASE_URL env var (see .env file)
        #server_version: '14'
    orm:
        auto_generate_proxy_classes: true
        naming_strategy: doctrine.orm.naming_strategy.underscore_number_aware
        auto_mapping: true
        mappings:
            App:
                is_bundle: false
                dir: '%kernel.project_dir%/src/Entity'
                prefix: 'App\Entity'
                alias: App

when@test:
    doctrine:
        dbal:
            # "TEST_TOKEN" is typically set by ParaTest
            dbname_suffix: '_test%env(default::TEST_TOKEN)%'

when@prod:
    doctrine:
        orm:
            auto_generate_proxy_classes: false
            proxy_dir: '%kernel.build_dir%/doctrine/orm/Proxies'
            query_cache_driver:
                type: pool
                pool: doctrine.system_cache_pool
            result_cache_driver:
                type: pool
                pool: doctrine.result_cache_pool

    framework:
        cache:
            pools:
                doctrine.result_cache_pool:
                    adapter: cache.app
                doctrine.system_cache_pool:
                    adapter: cache.adapter.filesystem
                    #adapter: cache.system

I'm running out of ideas. That kills - all of a sudden - everything on the production server.

I'm happy to here any suggestions.

3 replies

kevinpapst Dec 20, 2022

The problem is that ioncubeLoader is interpreting the cache file as encrypted.
I assume that you could offload the cache into memory (acpu, redis). That way ioncube should not interfer.

justpilot Dec 21, 2022

@kevinpapst Thank You!

I switched to a managed server, where I'm the only customer now. IonCube module was removed. The server has the following configuration now:

This program makes use of the Zend Scripting Language Engine:
Zend Engine v4.1.13, Copyright (c) Zend Technologies
with Zend OPcache v8.1.13, Copyright (c), by Zend Technologies


**Zend OPcache**

Opcode Caching | Up and Running
-- | --
Optimization | Enabled
SHM Cache | Enabled
File Cache | Disabled
JIT | Disabled
Startup | OK
Shared memory model | mmap
Cache hits | 29606
Cache misses | 11891
Used memory | 64216056
Free memory | 61280976
Wasted memory | 8720696
Interned Strings Used memory | 6290984
Interned Strings Free memory | 8
Cached scripts | 7902
Cached keys | 10374
Max keys | 16229
OOM restarts | 0
Hash keys restarts | 0
Manual restarts | 0

  -------------------- ------------------------------------------- 
  Symfony                                                         
 -------------------- ------------------------------------------- 
  Version              6.2.2                                      
  Long-Term Support    No                                         
  End of maintenance   07/2023 (in +222 days)                     
  End of life          07/2023 (in +222 days)                     
 -------------------- ------------------------------------------- 
  Kernel                                                          
 -------------------- ------------------------------------------- 
  Type                 App\Kernel                                 
  Environment          prod                                       
  Debug                false                                      
  Charset              UTF-8                                      
  Cache directory      ./var/cache/prod (5.5 MiB)                 
  Build directory      ./var/cache/prod (5.5 MiB)                 
  Log directory        ./var/log (32 KiB)                         
 -------------------- ------------------------------------------- 
  PHP                                                             
 -------------------- ------------------------------------------- 
  Version              8.1.13                                     
  Architecture         64 bits                                    
  Intl locale          de_DE                                      
  Timezone             Europe/Berlin (2022-12-21T23:40:32+01:00)  
  OPcache              false                                      
  APCu                 false                                      
  Xdebug               false                                      
 -------------------- -------------------------------------------

The problem remains the same.

When I run composer dump-autoload --optimize I get the error 403 immediately.

As I mentioned before, when I change the security setting on the ApiResource from

new Get(
            security: '(is_granted("ROLE_USER") and object.customer == user) or is_granted("ROLE_ADMIN")'
        ),

to

new Get(
            security: 'is_granted("ROLE_USER")'
        ),

it works fine. But the first security configuration is correct, it worked already for a long time.

Switching from prod to dev Mode - everything works fine. Prod Mode on my Linux Dev machine works fine as well.

justpilot Dec 23, 2022

Problem solved!
The PHP CLI version still had ionCube enabled. So the ionCube is the root of the problem! As you see it sometimes causes not only 503 errors.

davefx · 2024-07-29T12:50:16Z

davefx
Jul 29, 2024

I just found this error with PHP 8.1 and ionCube 12. So I'm afraid this problem is back, or maybe it never went away.

0 replies

Uh oh!

Cache of Doctrine SQL Statement leads to 503 error in Apache #44270

Uh oh!

Uh oh!

Update 1 (2021-11-27)

Replies: 12 comments · 17 replies

Uh oh!

Uh oh!

Art4 Nov 29, 2021 Author

Uh oh!

Uh oh!

Art4 Jan 18, 2022 Author

Uh oh!

Uh oh!

Uh oh!

Art4 Jan 18, 2022 Author

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Art4 Aug 15, 2022 Author

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Art4 Dec 13, 2022 Author

Uh oh!

Uh oh!

stof Sep 13, 2022 Collaborator

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Replies: 12 comments 17 replies

Art4
Nov 29, 2021
Author

Art4 Jan 18, 2022
Author

Art4 Jan 18, 2022
Author

Art4 Aug 15, 2022
Author

Art4 Dec 13, 2022
Author

stof Sep 13, 2022
Collaborator