Exception thrown unless Symfony cache directory is also changed to `/tmp/...` #39

nealio82 · 2018-07-09T12:45:46Z

Following the example code for Symfony and deploying, the request fails with the response {"message": "Internal server error"}.

Checking CloudWatch logs shows 2 relevant entries:

Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to allocate 20480 bytes) in /var/task/vendor/monolog/monolog/src/Monolog/Handler/StreamHandler.php on line 171

and

Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to allocate 73728 bytes) in /var/task/vendor/symfony/debug/DebugClassLoader.php on line 145

Altering Kernel.php to include the following makes the problem go away

    public function getCacheDir()
    {
        // When on the lambda only /tmp is writable
        if (getenv('LAMBDA_TASK_ROOT') !== false) {
            return '/tmp/cache/'.$this->environment;
        }

        return $this->getProjectDir().'/var/cache/'.$this->environment;
    }

However...

Presumably setting the cache dir to Lambda's local /tmp kind-of defeats the point pre-warming the cache with the hooks in the .bref.yml file? The object is to avoid launching the application without the cache already in place, right?

.bref.yml:

hooks:
    build:
        - 'APP_ENV=prod php bin/console cache:clear --no-debug --no-warmup'
        - 'APP_ENV=prod php bin/console cache:warmup'

serverless.yml:

service: test

provider:
  name: aws
  runtime: nodejs6.10

package:
  exclude:
    - '*'
    - '**'
  include:
    - bref.php
    - 'src/**'
    - 'vendor/**'
    - composer.json # Symfony uses it to figure out the root directory
    - 'bin/**'
    - 'config/**'
    - 'var/cache/prod/**' # We want to deploy the production caches

functions:
  # By default we create one "main" function
  main:
    handler: handler.handle
    timeout: 20 # Timeout in seconds, the default is 6 seconds
    # The function will match all HTTP URLs
    events:
      - http: 'ANY /'
      - http: 'ANY {proxy+}'
    environment:
      APP_ENV: 'prod'
      APP_DEBUG: '0'

bref.php:

<?php

use App\Kernel;
use Bref\Bridge\Symfony\SymfonyAdapter;
use Symfony\Component\Debug\Debug;
use Symfony\Component\Dotenv\Dotenv;

require __DIR__.'/vendor/autoload.php';

Debug::enable();

// The check is to ensure we don't use .env in production
if (!isset($_SERVER['APP_ENV'])) {
    (new Dotenv)->load(__DIR__.'/.env');
}
if ($_SERVER['APP_DEBUG'] ?? ('prod' !== ($_SERVER['APP_ENV'] ?? 'dev'))) {
    umask(0000);
}
$kernel = new Kernel($_SERVER['APP_ENV'] ?? 'dev', (bool) ($_SERVER['APP_DEBUG'] ?? ('prod' !== ($_SERVER['APP_ENV'] ?? 'dev'))));

$app = new \Bref\Application;
$app->httpHandler(new SymfonyAdapter($kernel));
$app->cliHandler(new \Symfony\Bundle\FrameworkBundle\Console\Application($kernel));
$app->run();

I get the same behaviour with cloning the mnapoli/bref-symfony-demo repo and running bref deploy, and also with adding bref to a symfony/website-skeleton project.

The text was updated successfully, but these errors were encountered:

mnapoli · 2018-07-09T13:12:47Z

Regarding the memory size problem, you could try increasing the memory limit of PHP by customizing the php.ini config, see https://github.com/mnapoli/bref#php-configuration to see how to set php.ini flags. However 128Mb seems already quite much, I'm surprised Symfony would consume that much.

Presumably setting the cache dir to Lambda's local /tmp kind-of defeats the point pre-warming the cache with the hooks in the .bref.yml file? The object is to avoid launching the application without the cache already in place, right?

Agreed, this is not really efficient in production that's why it's better to pre-generate the cache.

I get the same behaviour with cloning the mnapoli/bref-symfony-demo repo and running bref deploy

This is surprising, but since I wrote (and tested) bref-symfony-demo we have added some extensions to PHP (mainly opcache, the other extensions are disabled by default), maybe that could be it? I don't really know much more right now I'll try to dig in.

nealio82 · 2018-07-09T13:18:49Z

The first thing I tried was to increase the memory_limit (I should have mentioned that above, sorry!)

I think the cache is not being pre-warmed during the deploy process, because presumably it should exist and be readable if it has been!?

As mentioned, if you set the cache dir to the writeable /tmp/ dir then the problem disappears and PHP doesn't complain about memory any more

mnapoli · 2018-07-09T13:28:48Z

OK, I'll try to reproduce that tonight and see how it goes. Thanks for opening the issue!

t-geindre · 2018-07-09T14:14:56Z

We use our own kernel here, but since we also use symfony/dependency-injection we had to find a solution to pre-warm the cache, which essentially means to build and dump the container.

The only way we found to do so is to use relative paths.

For instance, we have a parameter %cache_dir%. Its value is always ./var/cache. This allows us to dump the container before uploading it on lambda.

Our kernel always check that the current working directory is $_ENV['LAMBDA_TASK_ROOT'].

This works and we don't build the container on each execution. I think you could do something similar with the Symfony kernel.

That beeing said, if you have a better solution, I would be very interested.

mnapoli · 2018-07-09T17:49:00Z

@nealio82 I've updated (composer update) everything to the latest versions on the symfony-demo project and redeployed, it still works: https://7oaryq3rzl.execute-api.eu-west-3.amazonaws.com/dev I don't understand what breaks for you 🤔

@t-geindre Yes! I had issues with absolute/relative paths when deploying the API platform demo for my benchmarks. In the end I gave up and stored the cache in /tmp because I was not measuring the cold starts, so I just had to warmup the application and that's it. But this is not ideal at all.

nealio82 · 2018-07-09T18:17:31Z

I probably won’t have time tonight, but i’ll create a version which breaks / screengrab reproduction steps

…

On Mon, 9 Jul 2018 at 18:49, Matthieu Napoli ***@***.***> wrote: @nealio82 <https://github.com/nealio82> I've updated (composer update) everything to the latest versions on the symfony-demo project and redeployed, it still works: https://7oaryq3rzl.execute-api.eu-west-3.amazonaws.com/dev I don't understand what breaks for you 🤔 @t-geindre <https://github.com/t-geindre> Yes! I had issues with absolute/relative paths when deploying the API platform demo for my benchmarks. In the end I gave up and stored the cache in /tmp because I was not measuring the cold starts, so I just had to warmup the application and that's it. But this is not ideal at all. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#39 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABCVBs2lAnH5tdsVqgYsKKU_LW_aU-swks5uE5eOgaJpZM4VHpC8> .

nealio82 · 2018-07-10T12:07:41Z

Interestingly, if I clone the mnapoli/bref-symfony-demo repo, I get a different (and nicer) exception message about the cache.

The symfony/website-skeleton still gives me out of memory, but the SF demo now specifically mentions the cache.

I made a screencast here: https://youtu.be/Ar3USl8h8Ug

mnapoli · 2018-07-10T12:15:21Z

Thanks for the screencast! Could you try one last time mnapoli/bref-symfony-demo by running composer update before bref deploy to install the last version of Bref?

nealio82 · 2018-07-10T12:23:28Z

I reverted my changes in Kernel.php and did the following:

➜  bref-symfony-demo git:(master) ✗ composer update
Loading composer repositories with package information
Updating dependencies (including require-dev)

Prefetching 2 packages 🎵
  - Downloading (100%)

Package operations: 2 installs, 20 updates, 0 removals
  - Installing matomo/ini (2.0.0) Loading from cache
  - Updating symfony/process (v4.0.9 => v4.1.1) Loading from cache
  - Updating zendframework/zend-diactoros (1.7.1 => 1.8.1) Loading from cache
  - Updating aws/aws-sdk-php (3.55.9 => 3.62.12) Loading from cache
  - Updating symfony/http-foundation (v4.0.9 => v4.1.1) Loading from cache
  - Installing symfony/polyfill-ctype (v1.8.0) Loading from cache
  - Updating symfony/yaml (v4.0.9 => v4.1.1) Loading from cache
  - Updating symfony/finder (v4.0.9 => v4.1.1) Loading from cache
  - Updating symfony/filesystem (v4.0.9 => v4.1.1) Loading from cache
  - Updating symfony/console (v4.0.9 => v4.1.1) Loading from cache
  - Updating mnapoli/bref (0.2.4 => 0.2.18) Loading from cache
  - Updating symfony/routing (v4.0.9 => v4.1.1) Loading from cache
  - Updating symfony/event-dispatcher (v4.0.9 => v4.1.1) Loading from cache
  - Updating symfony/debug (v4.0.9 => v4.1.1) Loading from cache
  - Updating symfony/http-kernel (v4.0.9 => v4.1.1) Loading from cache
  - Updating symfony/dependency-injection (v4.0.9 => v4.1.1) Loading from cache
  - Updating symfony/config (v4.0.9 => v4.1.1) Loading from cache
  - Updating symfony/cache (v4.0.9 => v4.1.1) Loading from cache
  - Updating symfony/framework-bundle (v4.0.9 => v4.1.1) Loading from cache
  - Updating symfony/twig-bridge (v4.0.9 => v4.1.1) Loading from cache
  - Updating symfony/twig-bundle (v4.0.9 => v4.1.1) Loading from cache
  - Updating symfony/dotenv (v4.0.9 => v4.1.1) Loading from cache
Writing lock file
Generating autoload files

What about running composer global require symfony/thanks && composer thanks now?
This will spread some 💖  by sending a ★  to the GitHub repositories of your fellow package maintainers.

Executing script cache:clear [OK]

➜  bref-symfony-demo git:(master) ✗ php vendor/bin/bref deploy
Serverless: Packaging service...
Serverless: Excluding development dependencies...
Serverless: Uploading CloudFormation file to S3...
Serverless: Uploading artifacts...
Serverless: Uploading service .zip file to S3 (22.52 MB)...
Serverless: Validating template...
Serverless: Updating Stack...
Serverless: Checking Stack update progress...
..
Serverless: Stack update finished...
Service Information
service: bref-demo-symfony
stage: dev
region: eu-west-3
stack: bref-demo-symfony-dev
api keys:
  None
endpoints:
  ANY - https://j1wj4kj13e.execute-api.eu-west-3.amazonaws.com/dev
  ANY - https://j1wj4kj13e.execute-api.eu-west-3.amazonaws.com/dev/{proxy+}
functions:
  main: bref-demo-symfony-dev-main
Deployment success
 8/8 [░░░░░░░░░░░░░░░░░░░░░░░░░░░░]  1 min%

I see the same exception(s) related to the cache again

[STDERR] 2018-07-10T12:19:17+00:00 [critical] Uncaught Exception: Unable to write in the cache directory (/var/task/var/cache/prod/twig/dc).
2018-07-10T12:19:17+00:00 [critical] Uncaught PHP Exception RuntimeException: "Unable to write in the cache directory (/var/task/var/cache/prod/twig/dc)." at /var/task/vendor/twig/twig/lib/Twig/Cache/Filesystem.php line 59

[STDERR] 2018-07-10T12:19:17+00:00 [critical] Exception thrown when handling an exception (RuntimeException: Unable to write in the cache directory (/var/task/var/cache/prod/twig/dc). at /var/task/vendor/twig/twig/lib/Twig/Cache/Filesystem.php line 59)

[STDERR] 2018-07-10T12:19:17+00:00 [critical] Uncaught Exception: Unable to write in the cache directory (/var/task/var/cache/prod/twig/dc).

Fatal error: Uncaught RuntimeException: Unable to create the cache directory (/var/task/var/cache/prod/twig/ba). in /var/task/vendor/twig/twig/lib/Twig/Cache/Filesystem.php:55
Stack trace:
#0 /var/task/vendor/twig/twig/lib/Twig/Environment.php(369): Twig_Cache_Filesystem->write('/var/task/var/c...', '<?php\n\n/* hello...')
#1 /var/task/vendor/twig/twig/lib/Twig/Environment.php(289): Twig_Environment->loadTemplate('hello.html.twig')
#2 /var/task/vendor/symfony/framework-bundle/Controller/ControllerTrait.php(224): Twig_Environment->render('hello.html.twig', Array)
#3 /var/task/src/Controller/HomeController.php(12): Symfony\Bundle\FrameworkBundle\Controller\Controller->render('hello.html.twig')
#4 /var/task/vendor/symfony/http-kernel/HttpKernel.php(149): App\Controller\HomeController->index()
#5 /var/task/vendor/symfony/http-kernel/HttpKernel.php(66): Symfony\Component\HttpKernel\HttpKernel->handleRaw(Object(Symfony\Component\HttpFoundation\Request), 1)
#6 /var/task/vendor/symfony/http-kernel/Kernel.php(188): Symfony\Compone in /var/task/vendor/twig/twig/lib/Twig/Cache/Filesystem.php on line 59

nealio82 · 2018-07-12T09:53:06Z

If I add the cache param to config/packages/twig.yml with the value of either /tmp/... or false, then everything works again.

twig:
    paths: ['%kernel.project_dir%/templates']
    debug: '%kernel.debug%'
    strict_variables: '%kernel.debug%'
    cache: '/tmp/twig'

see https://twig.symfony.com/doc/2.x/api.html#environment-options and http://symfony.com/doc/current/reference/configuration/twig.html#cache

However, I don't like setting /tmp/... in the twig config, and I don't know enough about Twig to tell if disabling the cache is a bad idea.

I also don't get why Twig needs to write to the cache if the templates are pre-compiled!?

mnapoli · 2018-07-12T16:39:25Z

OK I think I remember now where the problem came from (when I tried deploying the API platform demo).

Some things in the Symfony cache are using absolute paths, as @t-geindre mentioned. You need to tweak Symfony and its config to force it to use relative paths (because the paths on your machine and on AWS lambda are different).

That works, except for Twig that IIRC uses realpath() as some point in the code (I think it's here). Because of that call, a relative path is turned into an absolute path and it breaks things. I don't remember the details anymore though.

Also it is important to note that not all the Symfony cache is pregenerated, for example you have to put Doctrine's cache (which is not pregenerated) in /tmp manually.

Here is an example of what I did for Doctrine in config/packages/doctrine.yaml:

parameters:
    writable_cache_dir: '/tmp/cache'

[…]

doctrine:
    orm:
        […]
        metadata_cache_driver:
            cache_provider: metadata_cache
        result_cache_driver:
            cache_provider: result_cache
        query_cache_driver:
            cache_provider: query_cache

[…]

doctrine_cache:
    providers:
        metadata_cache:
            aliases: [doctrine.orm.default_metadata_cache]
            file_system:
                directory: "%writable_cache_dir%/doctrine/metadata"
        result_cache:
            aliases: [doctrine.orm.default_result_cache]
            file_system:
                directory: "%writable_cache_dir%/doctrine/result"
        query_cache:
            aliases: [doctrine.orm.default_query_cache]
            file_system:
                directory: "%writable_cache_dir%/doctrine/query"

I did the same for the user cache:

framework:
    cache:
        directory: "%writable_cache_dir%/app"

nealio82 · 2018-07-12T17:31:08Z

Ok, so I assume that means at some point we have to have a writeable filesystem in order for Twig to work properly?

However, that raises 2 questions:

Should the Kernel.php in the documentation be updated to change the cache dir to /tmp to make sure Twig can always write to it?
Why does your example work for you and not for me? You saw in the screencast that I made no adjustments to the example repo's code!

mnapoli · 2018-07-13T19:25:09Z

Should the Kernel.php in the documentation be updated to change the cache dir to /tmp to make sure Twig can always write to it?

That could be a temporary solution. I still hope to find the correct configuration to apply to solve that (or maybe fix the problem in Twig/Symfony). But yeah it's better to have helpful documentation for now.

Why does your example work for you and not for me? You saw in the screencast that I made no adjustments to the example repo's code!

Yes I don't know how to explain that either :/

nealio82 · 2018-07-13T21:15:08Z

Ok, I can make a PR for this tomorrow :)

…

On Fri, 13 Jul 2018 at 20:25, Matthieu Napoli ***@***.***> wrote: Should the Kernel.php in the documentation be updated to change the cache dir to /tmp to make sure Twig can always write to it? That could be a temporary solution. I still hope to find the correct configuration to apply to solve that (or maybe fix the problem in Twig/Symfony). But yeah it's better to have helpful documentation for now. Why does your example work for you and not for me? You saw in the screencast that I made no adjustments to the example repo's code! Yes I don't know how to explain that either :/ — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#39 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABCVBtB0FkhbTNKAuTqR0t1-Rb5GHQVEks5uGPQVgaJpZM4VHpC8> .

nealio82 · 2018-07-17T09:36:39Z

Workaround documentation added in #42

mnapoli · 2018-08-24T13:07:26Z

Does anyone here has a public repository to reproduce this?

nealio82 · 2018-08-24T20:43:28Z

I can make one early next week (probably Monday)

…

On Fri, 24 Aug 2018 at 14:07, Matthieu Napoli ***@***.***> wrote: Does anyone here has a public repository to reproduce this? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#39 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABCVBovYn0MTlVJHvnXypDufuXWx6MmXks5uT_qPgaJpZM4VHpC8> .

mnapoli · 2018-09-23T20:09:12Z

For those reading this issue, I reproduced it when writing a simple Symfony 4 application, starting with Flex. I just have a base template (the layout) and a few pages that extend from that. I don't have time right now to create a whole repository to reproduce that but that can maybe be a starting point for others.

nealio82 · 2018-10-01T05:37:44Z

Weirdly I tried yesterday; starting with Symfony 4 & extending a base template, then passing in some variables to render. I didn’t see the issue at all. (And i’m pretty sure the first time I saw the issue I didn’t even need to extend a base template)

thibaudlemaire · 2019-05-24T09:16:19Z

Hi!
I personally chose to redirect Twig's cache to config/packages/prod/twig.yaml to be able to warm up other bundles.

# config/packages/prod/twig.yaml
twig:
    cache: '/tmp/cache/twig'

I know it's a temporary solution but I preferer doing this because making /tmp the default Symfony cache directory increases the cold start from 500ms to 2 seconds.
I hope Twig's team will work on this problem to comply with the Symfony4 best practices (var/cache should be read-only and warmable).

mnapoli · 2019-05-24T09:26:44Z

Thanks that's a much better solution! I'll try to update the documentation in the coming weeks (I should have more time than this month), if anyone wants to get on this though feel free.

nealio82 · 2019-05-25T16:12:31Z

@thibaudlemaire by any chance do you know why it adds a 4x overhead? Have you looked into that with Blackfire, for example? The Symfony best practice is to override it in the Kernel class: https://symfony.com/doc/current/configuration/override_dir_structure.html#override-the-cache-directory

thibaudlemaire · 2019-05-25T16:42:29Z

When you override the cache directory, pre-warmed cache cannot be used because either it's warmed up in your local /tmp dir that is not packed by sam package command, or during Symfony execution it tries to retrieve cache from the lambda container's /tmp directory that is empty on first request. I think we cannot apply blindly what's explained in the link you gave.

The 4x overhead is due to the cache warmup when a Lambda container is created. I don't know exactly what is compiled at the first execution but I suppose : Doctrine proxies, annotations, Twig views, Kernel config, services container, etc.

That's why we need to make Symfony use the pre-warmed cache. To do so, I thought about two solutions :

Use /tmp as cache directory and copy (or symlink) pre-warmed cache files on lambda container initialisation. I don't know exactly how to do it. A solution may be to add this feature to Bref's event handler to make it copy (or symlink) cache on first invoke.
Or use the default Symfony cache directory (./var/cache), warm it up before deployment so that sam package pack cache too. This should be possible for all bundles because Symfony recommends not to write in the cache directory after deployment. For bundles that are not compliant with this best practice (like Twig), we still can override the cache directory to make them write in /tmp.

I use the second one because it's easier. But it's not ideal because twig views still need to be compiled on first invoke.

nealio82 · 2019-05-31T14:38:08Z

When you override the cache directory, pre-warmed cache cannot be used because either it's warmed up in your local /tmp dir that is not packed by sam package command, or during Symfony execution it tries to retrieve cache from the lambda container's /tmp directory that is empty on first request. I think we cannot apply blindly what's explained in the link you gave.

Ah, of course! 🤦‍♂

Makefile cpu-specific

mnapoli · 2023-03-28T12:22:04Z

I'll be closing this issue since it was opened in 2019, and since then we have the Symfony bridge that should take care of the cache 🎉

nealio82 changed the title ~~Exception thrown unless cache directory is also changed to /tmp/...~~ Exception thrown unless Symfony cache directory is also changed to /tmp/... Jul 9, 2018

mnapoli mentioned this issue Jul 9, 2018

Exception is thrown unless translations are explicitly defined #40

Closed

mnapoli added the bug label Aug 24, 2018

brefphp-bot pushed a commit to brefphp-bot/bref that referenced this issue Jan 2, 2022

Merge pull request brefphp#39 from deleugpn/runtime-pr

6400b32

Makefile cpu-specific

mnapoli closed this as completed Mar 28, 2023

This issue was closed.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Exception thrown unless Symfony cache directory is also changed to `/tmp/...` #39

Exception thrown unless Symfony cache directory is also changed to `/tmp/...` #39

nealio82 commented Jul 9, 2018 •

edited

Loading

mnapoli commented Jul 9, 2018 •

edited

Loading

nealio82 commented Jul 9, 2018 •

edited

Loading

mnapoli commented Jul 9, 2018

t-geindre commented Jul 9, 2018

mnapoli commented Jul 9, 2018

nealio82 commented Jul 9, 2018 via email

nealio82 commented Jul 10, 2018

mnapoli commented Jul 10, 2018

nealio82 commented Jul 10, 2018

nealio82 commented Jul 12, 2018 •

edited

Loading

mnapoli commented Jul 12, 2018 •

edited

Loading

nealio82 commented Jul 12, 2018

mnapoli commented Jul 13, 2018

nealio82 commented Jul 13, 2018 via email

nealio82 commented Jul 17, 2018

mnapoli commented Aug 24, 2018

nealio82 commented Aug 24, 2018 via email

mnapoli commented Sep 23, 2018

nealio82 commented Oct 1, 2018

thibaudlemaire commented May 24, 2019

mnapoli commented May 24, 2019

nealio82 commented May 25, 2019

thibaudlemaire commented May 25, 2019

nealio82 commented May 31, 2019

mnapoli commented Mar 28, 2023

Exception thrown unless Symfony cache directory is also changed to /tmp/... #39

Exception thrown unless Symfony cache directory is also changed to /tmp/... #39

Comments

nealio82 commented Jul 9, 2018 • edited Loading

mnapoli commented Jul 9, 2018 • edited Loading

nealio82 commented Jul 9, 2018 • edited Loading

mnapoli commented Jul 9, 2018

t-geindre commented Jul 9, 2018

mnapoli commented Jul 9, 2018

nealio82 commented Jul 9, 2018 via email

nealio82 commented Jul 10, 2018

mnapoli commented Jul 10, 2018

nealio82 commented Jul 10, 2018

nealio82 commented Jul 12, 2018 • edited Loading

mnapoli commented Jul 12, 2018 • edited Loading

nealio82 commented Jul 12, 2018

mnapoli commented Jul 13, 2018

nealio82 commented Jul 13, 2018 via email

nealio82 commented Jul 17, 2018

mnapoli commented Aug 24, 2018

nealio82 commented Aug 24, 2018 via email

mnapoli commented Sep 23, 2018

nealio82 commented Oct 1, 2018

thibaudlemaire commented May 24, 2019

mnapoli commented May 24, 2019

nealio82 commented May 25, 2019

thibaudlemaire commented May 25, 2019

nealio82 commented May 31, 2019

mnapoli commented Mar 28, 2023

Exception thrown unless Symfony cache directory is also changed to `/tmp/...` #39

Exception thrown unless Symfony cache directory is also changed to `/tmp/...` #39

nealio82 commented Jul 9, 2018 •

edited

Loading

mnapoli commented Jul 9, 2018 •

edited

Loading

nealio82 commented Jul 9, 2018 •

edited

Loading

nealio82 commented Jul 12, 2018 •

edited

Loading

mnapoli commented Jul 12, 2018 •

edited

Loading