Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

After upgrade 2.1.2 to 2.1.7, Redis cache ends up in bad state intermittently, then all requests return 503 #10002

Closed
apicht opened this Issue Jun 21, 2017 · 22 comments

Comments

Projects
None yet
9 participants
@apicht
Copy link

apicht commented Jun 21, 2017

Preconditions

  1. Magento CE 2.1.7
  2. OS: RHEL 7.2
  3. Apache 2.4.6-45
  4. PHP 7.0.19-1 (IUS)
  5. Redis 3.2.4-1 (IUS)
  6. Percona MySQL 5.7.15-9
  7. 2 identical servers behind an F5 load balancer, plus a separate admin server.
  8. Media is shared between the servers via NFS.

Steps to reproduce

  1. Run production traffic for hours to days until the 503s begin.
  2. We have been unable to reproduce on staging, even replaying captured production traffic. However, capturing an rdb dump from the Redis server while in the error state and transplanting it to a test server has allowed us to at least reproduce the error state.

Expected result

  1. Home page should load without issue

Actual result

  1. User receives error processing request page with error number.
  2. var/report contains file with error number, exception contained therein is one of the following, but during a particular error state the same exception occurs on every request:
a:4:{i:0;s:134:"Warning: array_merge(): Argument #1 is not an array in /data/www/magento2/vendor/magento/framework/Image/AdapterFactory.php on line 37";i:1;s:7726:"#0 [internal function]: Magento\Framework\App\ErrorHandler->handler(2, 'array_merge(): ...', '/data/www/magen...', 37, Array)
#1 /data/www/magento2/vendor/magento/framework/Image/AdapterFactory.php(37): array_merge(NULL, Array)
#2 /data/www/magento2/vendor/magento/framework/ObjectManager/Factory/AbstractFactory.php(93): Magento\Framework\Image\AdapterFactory->__construct(Object(Magento\Framework\ObjectManager\ObjectManager), Object(Magento\Framework\Image\Adapter\Config), Array)
#3 /data/www/magento2/vendor/magento/framework/ObjectManager/Factory/Dynamic/Developer.php(89): Magento\Framework\ObjectManager\Factory\AbstractFactory->createObject('Magento\\Framewo...', Array)
#4 /data/www/magento2/vendor/magento/framework/ObjectManager/ObjectManager.php(71): Magento\Framework\ObjectManager\Factory\Dynamic\Developer->create('Magento\\Framewo...')
#5 /data/www/magento2/vendor/magento/framework/ObjectManager/Factory/AbstractFactory.php(126): Magento\Framework\ObjectManager\ObjectManager->get('Magento\\Framewo...')
#6 /data/www/magento2/vendor/magento/framework/ObjectManager/Factory/Dynamic/Developer.php(53): Magento\Framework\ObjectManager\Factory\AbstractFactory->resolveArgument(Array, 'Magento\\Framewo...', NULL, 'adapterFactory', 'Magento\\Framewo...')
#7 /data/www/magento2/vendor/magento/framework/ObjectManager/Factory/Dynamic/Developer.php(82): Magento\Framework\ObjectManager\Factory\Dynamic\Developer->_resolveArguments('Magento\\Framewo...', Array, Array)
#8 /data/www/magento2/vendor/magento/framework/ObjectManager/ObjectManager.php(71): Magento\Framework\ObjectManager\Factory\Dynamic\Developer->create('Magento\\Framewo...')
#9 /data/www/magento2/vendor/magento/framework/ObjectManager/Factory/AbstractFactory.php(126): Magento\Framework\ObjectManager\ObjectManager->get('Magento\\Framewo...')
#10 /data/www/magento2/vendor/magento/framework/ObjectManager/Factory/Dynamic/Developer.php(53): Magento\Framework\ObjectManager\Factory\AbstractFactory->resolveArgument(Array, 'Magento\\Framewo...', NULL, 'imageFactory', 'MageArray\\Blog\\...')
#11 /data/www/magento2/vendor/magento/framework/ObjectManager/Factory/Dynamic/Developer.php(82): Magento\Framework\ObjectManager\Factory\Dynamic\Developer->_resolveArguments('MageArray\\Blog\\...', Array, Array)
#12 /data/www/magento2/vendor/magento/framework/ObjectManager/ObjectManager.php(71): Magento\Framework\ObjectManager\Factory\Dynamic\Developer->create('MageArray\\Blog\\...')
#13 /data/www/magento2/vendor/magento/framework/ObjectManager/Factory/AbstractFactory.php(126): Magento\Framework\ObjectManager\ObjectManager->get('MageArray\\Blog\\...')
#14 /data/www/magento2/vendor/magento/framework/ObjectManager/Factory/Dynamic/Developer.php(53): Magento\Framework\ObjectManager\Factory\AbstractFactory->resolveArgument(Array, 'MageArray\\Blog\\...', NULL, 'dataHelper', 'MageArray\\Blog\\...')
#15 /data/www/magento2/vendor/magento/framework/ObjectManager/Factory/Dynamic/Developer.php(82): Magento\Framework\ObjectManager\Factory\Dynamic\Developer->_resolveArguments('MageArray\\Blog\\...', Array, Array)
#16 /data/www/magento2/vendor/magento/framework/ObjectManager/ObjectManager.php(57): Magento\Framework\ObjectManager\Factory\Dynamic\Developer->create('MageArray\\Blog\\...', Array)
#17 /data/www/magento2/var/generation/MageArray/Blog/Model/BlogpostFactory.php(43): Magento\Framework\ObjectManager\ObjectManager->create('MageArray\\Blog\\...', Array)
#18 /data/www/magento2/app/code/MageArray/Blog/Controller/Router.php(154): MageArray\Blog\Model\BlogpostFactory->create()
#19 /data/www/magento2/vendor/magento/framework/App/FrontController.php(50): MageArray\Blog\Controller\Router->match(Object(Magento\Framework\App\Request\Http))
#20 /data/www/magento2/vendor/magento/framework/Interception/Interceptor.php(74): Magento\Framework\App\FrontController->dispatch(Object(Magento\Framework\App\Request\Http))
#21 /data/www/magento2/vendor/magento/framework/Interception/Chain/Chain.php(70): Magento\Framework\App\FrontController\Interceptor->___callParent('dispatch', Array)
#22 /data/www/magento2/vendor/magento/framework/Interception/Chain/Chain.php(63): Magento\Framework\Interception\Chain\Chain->invokeNext('Magento\\Framewo...', 'dispatch', Object(Magento\Framework\App\FrontController\Interceptor), Array, 'requestPreproce...')
#23 /data/www/magento2/vendor/magento/module-store/App/FrontController/Plugin/RequestPreprocessor.php(94): Magento\Framework\Interception\Chain\Chain->Magento\Framework\Interception\Chain\{closure}(Object(Magento\Framework\App\Request\Http))
#24 /data/www/magento2/vendor/magento/framework/Interception/Chain/Chain.php(67): Magento\Store\App\FrontController\Plugin\RequestPreprocessor->aroundDispatch(Object(Magento\Framework\App\FrontController\Interceptor), Object(Closure), Object(Magento\Framework\App\Request\Http))
#25 /data/www/magento2/vendor/magento/framework/Interception/Chain/Chain.php(63): Magento\Framework\Interception\Chain\Chain->invokeNext('Magento\\Framewo...', 'dispatch', Object(Magento\Framework\App\FrontController\Interceptor), Array, 'install')
#26 /data/www/magento2/vendor/magento/framework/Module/Plugin/DbStatusValidator.php(69): Magento\Framework\Interception\Chain\Chain->Magento\Framework\Interception\Chain\{closure}(Object(Magento\Framework\App\Request\Http))
#27 /data/www/magento2/vendor/magento/framework/Interception/Chain/Chain.php(67): Magento\Framework\Module\Plugin\DbStatusValidator->aroundDispatch(Object(Magento\Framework\App\FrontController\Interceptor), Object(Closure), Object(Magento\Framework\App\Request\Http))
#28 /data/www/magento2/vendor/magento/framework/Interception/Chain/Chain.php(63): Magento\Framework\Interception\Chain\Chain->invokeNext('Magento\\Framewo...', 'dispatch', Object(Magento\Framework\App\FrontController\Interceptor), Array, 'front-controlle...')
#29 /data/www/magento2/vendor/magento/module-page-cache/Model/App/FrontController/VarnishPlugin.php(55): Magento\Framework\Interception\Chain\Chain->Magento\Framework\Interception\Chain\{closure}(Object(Magento\Framework\App\Request\Http))
#30 /data/www/magento2/vendor/magento/framework/Interception/Chain/Chain.php(67): Magento\PageCache\Model\App\FrontController\VarnishPlugin->aroundDispatch(Object(Magento\Framework\App\FrontController\Interceptor), Object(Closure), Object(Magento\Framework\App\Request\Http))
#31 /data/www/magento2/vendor/magento/framework/Interception/Interceptor.php(138): Magento\Framework\Interception\Chain\Chain->invokeNext('Magento\\Framewo...', 'dispatch', Object(Magento\Framework\App\FrontController\Interceptor), Array, 'front-controlle...')
#32 /data/www/magento2/vendor/magento/module-page-cache/Model/App/FrontController/BuiltinPlugin.php(68): Magento\Framework\App\FrontController\Interceptor->Magento\Framework\Interception\{closure}(Object(Magento\Framework\App\Request\Http))
#33 /data/www/magento2/vendor/magento/framework/Interception/Interceptor.php(142): Magento\PageCache\Model\App\FrontController\BuiltinPlugin->aroundDispatch(Object(Magento\Framework\App\FrontController\Interceptor), Object(Closure), Object(Magento\Framework\App\Request\Http))
#34 /data/www/magento2/var/generation/Magento/Framework/App/FrontController/Interceptor.php(26): Magento\Framework\App\FrontController\Interceptor->___callPlugins('dispatch', Array, Array)
#35 /data/www/magento2/vendor/magento/framework/App/Http.php(135): Magento\Framework\App\FrontController\Interceptor->dispatch(Object(Magento\Framework\App\Request\Http))
#36 /data/www/magento2/vendor/magento/framework/App/Bootstrap.php(258): Magento\Framework\App\Http->launch()
#37 /data/www/magento2/pub/index.php(37): Magento\Framework\App\Bootstrap->run(Object(Magento\Framework\App\Http))
a:4:{i:0;s:60:"Time to live is a mandatory parameter for set public headers";i:1;s:9650:"#0 /data/www/magento2/var/generation/Magento/Framework/App/Response/Http/Interceptor.php(50): Magento\Framework\App\Response\Http->setPublicHeaders(NULL)
#1 /data/www/magento2/vendor/magento/module-page-cache/Model/Layout/LayoutPlugin.php(48): Magento\Framework\App\Response\Http\Interceptor->setPublicHeaders(NULL)
#2 /data/www/magento2/vendor/magento/framework/Interception/Interceptor.php(152): Magento\PageCache\Model\Layout\LayoutPlugin->afterGenerateXml(Object(Magento\Framework\View\Layout\Interceptor), Object(Magento\Framework\View\Layout\Interceptor))
#3 /data/www/magento2/var/generation/Magento/Framework/View/Layout/Interceptor.php(78): Magento\Framework\View\Layout\Interceptor->___callPlugins('generateXml', Array, Array)
#4 /data/www/magento2/vendor/magento/framework/View/Layout/Builder.php(104): Magento\Framework\View\Layout\Interceptor->generateXml()
#5 /data/www/magento2/vendor/magento/framework/View/Layout/Builder.php(64): Magento\Framework\View\Layout\Builder->generateLayoutXml()
#6 /data/www/magento2/vendor/magento/framework/View/Layout.php(244): Magento\Framework\View\Layout\Builder->build()
#7 /data/www/magento2/vendor/magento/framework/View/Layout.php(859): Magento\Framework\View\Layout->build()
#8 /data/www/magento2/var/generation/Magento/Framework/View/Layout/Interceptor.php(414): Magento\Framework\View\Layout->getBlock('page_content_he...')
#9 /data/www/magento2/vendor/magento/module-cms/Helper/Page.php(171): Magento\Framework\View\Layout\Interceptor->getBlock('page_content_he...')
#10 /data/www/magento2/vendor/magento/module-cms/Controller/Index/Index.php(43): Magento\Cms\Helper\Page->prepareResultPage(Object(Magento\Cms\Controller\Index\Index\Interceptor), 'home')
#11 /data/www/magento2/var/generation/Magento/Cms/Controller/Index/Index/Interceptor.php(24): Magento\Cms\Controller\Index\Index->execute(NULL)
#12 /data/www/magento2/vendor/magento/framework/App/Action/Action.php(102): Magento\Cms\Controller\Index\Index\Interceptor->execute()
#13 /data/www/magento2/vendor/magento/framework/Interception/Interceptor.php(74): Magento\Framework\App\Action\Action->dispatch(Object(Magento\Framework\App\Request\Http))
#14 /data/www/magento2/vendor/magento/framework/Interception/Chain/Chain.php(70): Magento\Cms\Controller\Index\Index\Interceptor->___callParent('dispatch', Array)
#15 /data/www/magento2/vendor/magento/framework/Interception/Chain/Chain.php(63): Magento\Framework\Interception\Chain\Chain->invokeNext('Magento\\Cms\\Con...', 'dispatch', Object(Magento\Cms\Controller\Index\Index\Interceptor), Array, 'contextPlugin')
#16 /data/www/magento2/vendor/magento/module-store/App/Action/Plugin/Context.php(106): Magento\Framework\Interception\Chain\Chain->Magento\Framework\Interception\Chain\{closure}(Object(Magento\Framework\App\Request\Http))
#17 /data/www/magento2/vendor/magento/framework/Interception/Chain/Chain.php(67): Magento\Store\App\Action\Plugin\Context->aroundDispatch(Object(Magento\Cms\Controller\Index\Index\Interceptor), Object(Closure), Object(Magento\Framework\App\Request\Http))
#18 /data/www/magento2/vendor/magento/framework/Interception/Chain/Chain.php(63): Magento\Framework\Interception\Chain\Chain->invokeNext('Magento\\Cms\\Con...', 'dispatch', Object(Magento\Cms\Controller\Index\Index\Interceptor), Array, 'customer-app-ac...')
#19 /data/www/magento2/vendor/magento/module-customer/Model/App/Action/ContextPlugin.php(61): Magento\Framework\Interception\Chain\Chain->Magento\Framework\Interception\Chain\{closure}(Object(Magento\Framework\App\Request\Http))
#20 /data/www/magento2/vendor/magento/framework/Interception/Chain/Chain.php(67): Magento\Customer\Model\App\Action\ContextPlugin->aroundDispatch(Object(Magento\Cms\Controller\Index\Index\Interceptor), Object(Closure), Object(Magento\Framework\App\Request\Http))
#21 /data/www/magento2/vendor/magento/framework/Interception/Chain/Chain.php(63): Magento\Framework\Interception\Chain\Chain->invokeNext('Magento\\Cms\\Con...', 'dispatch', Object(Magento\Cms\Controller\Index\Index\Interceptor), Array, 'storeCheck')
#22 /data/www/magento2/vendor/magento/module-store/App/Action/Plugin/StoreCheck.php(44): Magento\Framework\Interception\Chain\Chain->Magento\Framework\Interception\Chain\{closure}(Object(Magento\Framework\App\Request\Http))
#23 /data/www/magento2/vendor/magento/framework/Interception/Chain/Chain.php(67): Magento\Store\App\Action\Plugin\StoreCheck->aroundDispatch(Object(Magento\Cms\Controller\Index\Index\Interceptor), Object(Closure), Object(Magento\Framework\App\Request\Http))
#24 /data/www/magento2/vendor/magento/framework/Interception/Interceptor.php(138): Magento\Framework\Interception\Chain\Chain->invokeNext('Magento\\Cms\\Con...', 'dispatch', Object(Magento\Cms\Controller\Index\Index\Interceptor), Array, 'tax-app-action-...')
#25 /data/www/magento2/vendor/magento/module-tax/Model/App/Action/ContextPlugin.php(91): Magento\Cms\Controller\Index\Index\Interceptor->Magento\Framework\Interception\{closure}(Object(Magento\Framework\App\Request\Http))
#26 /data/www/magento2/vendor/magento/framework/Interception/Interceptor.php(142): Magento\Tax\Model\App\Action\ContextPlugin->aroundDispatch(Object(Magento\Cms\Controller\Index\Index\Interceptor), Object(Closure), Object(Magento\Framework\App\Request\Http))
#27 /data/www/magento2/var/generation/Magento/Cms/Controller/Index/Index/Interceptor.php(39): Magento\Cms\Controller\Index\Index\Interceptor->___callPlugins('dispatch', Array, Array)
#28 /data/www/magento2/vendor/magento/framework/App/FrontController.php(55): Magento\Cms\Controller\Index\Index\Interceptor->dispatch(Object(Magento\Framework\App\Request\Http))
#29 /data/www/magento2/vendor/magento/framework/Interception/Interceptor.php(74): Magento\Framework\App\FrontController->dispatch(Object(Magento\Framework\App\Request\Http))
#30 /data/www/magento2/vendor/magento/framework/Interception/Chain/Chain.php(70): Magento\Framework\App\FrontController\Interceptor->___callParent('dispatch', Array)
#31 /data/www/magento2/vendor/magento/framework/Interception/Chain/Chain.php(63): Magento\Framework\Interception\Chain\Chain->invokeNext('Magento\\Framewo...', 'dispatch', Object(Magento\Framework\App\FrontController\Interceptor), Array, 'requestPreproce...')
#32 /data/www/magento2/vendor/magento/module-store/App/FrontController/Plugin/RequestPreprocessor.php(94): Magento\Framework\Interception\Chain\Chain->Magento\Framework\Interception\Chain\{closure}(Object(Magento\Framework\App\Request\Http))
#33 /data/www/magento2/vendor/magento/framework/Interception/Chain/Chain.php(67): Magento\Store\App\FrontController\Plugin\RequestPreprocessor->aroundDispatch(Object(Magento\Framework\App\FrontController\Interceptor), Object(Closure), Object(Magento\Framework\App\Request\Http))
#34 /data/www/magento2/vendor/magento/framework/Interception/Chain/Chain.php(63): Magento\Framework\Interception\Chain\Chain->invokeNext('Magento\\Framewo...', 'dispatch', Object(Magento\Framework\App\FrontController\Interceptor), Array, 'install')
#35 /data/www/magento2/vendor/magento/framework/Module/Plugin/DbStatusValidator.php(69): Magento\Framework\Interception\Chain\Chain->Magento\Framework\Interception\Chain\{closure}(Object(Magento\Framework\App\Request\Http))
#36 /data/www/magento2/vendor/magento/framework/Interception/Chain/Chain.php(67): Magento\Framework\Module\Plugin\DbStatusValidator->aroundDispatch(Object(Magento\Framework\App\FrontController\Interceptor), Object(Closure), Object(Magento\Framework\App\Request\Http))
#37 /data/www/magento2/vendor/magento/framework/Interception/Chain/Chain.php(63): Magento\Framework\Interception\Chain\Chain->invokeNext('Magento\\Framewo...', 'dispatch', Object(Magento\Framework\App\FrontController\Interceptor), Array, 'front-controlle...')
#38 /data/www/magento2/vendor/magento/module-page-cache/Model/App/FrontController/VarnishPlugin.php(55): Magento\Framework\Interception\Chain\Chain->Magento\Framework\Interception\Chain\{closure}(Object(Magento\Framework\App\Request\Http))
#39 /data/www/magento2/vendor/magento/framework/Interception/Chain/Chain.php(67): Magento\PageCache\Model\App\FrontController\VarnishPlugin->aroundDispatch(Object(Magento\Framework\App\FrontController\Interceptor), Object(Closure), Object(Magento\Framework\App\Request\Http))
#40 /data/www/magento2/vendor/magento/framework/Interception/Interceptor.php(138): Magento\Framework\Interception\Chain\Chain->invokeNext('Magento\\Framewo...', 'dispatch', Object(Magento\Framework\App\FrontController\Interceptor), Array, 'front-controlle...')
#41 /data/www/magento2/vendor/magento/module-page-cache/Model/App/FrontController/BuiltinPlugin.php(68): Magento\Framework\App\FrontController\Interceptor->Magento\Framework\Interception\{closure}(Object(Magento\Framework\App\Request\Http))
#42 /data/www/magento2/vendor/magento/framework/Interception/Interceptor.php(142): Magento\PageCache\Model\App\FrontController\BuiltinPlugin->aroundDispatch(Object(Magento\Framework\App\FrontController\Interceptor), Object(Closure), Object(Magento\Framework\App\Request\Http))
#43 /data/www/magento2/var/generation/Magento/Framework/App/FrontController/Interceptor.php(26): Magento\Framework\App\FrontController\Interceptor->___callPlugins('dispatch', Array, Array)
#44 /data/www/magento2/vendor/magento/framework/App/Http.php(135): Magento\Framework\App\FrontController\Interceptor->dispatch(Object(Magento\Framework\App\Request\Http))
#45 /data/www/magento2/vendor/magento/framework/App/Bootstrap.php(258): Magento\Framework\App\Http->launch()
#46 /data/www/magento2/pub/index.php(37): Magento\Framework\App\Bootstrap->run(Object(Magento\Framework\App\Http))
#47 {main}";s:3:"url";s:1:"/";s:11:"script_name";s:10:"/index.php";}

Additional Information

What seems to happen is something corrupts the Redis cache, and from that point forward all requests start to return an error page with a 503. Deleting the bd5_SYSTEM_CACHE_EXISTS key from Redis, or issuing a FLUSHDB command, clears the error. The corrupted cache state even prevents the bin/magento CLI from functioning correctly.

@southerncomputer

This comment has been minimized.

Copy link
Contributor

southerncomputer commented Jun 21, 2017

How is your env.php setup?

Here's mine - i've replaced GZIP with LZF since it is faster:

'cache' =>
array (
'frontend' =>
array (
'default' =>
array (
'backend' => 'Cm_Cache_Backend_Redis',
'backend_options' =>
array (
'server' => '/run/redis/redis6380.sock',
'port' => '',
'persistent' => '',
'database' => '0',
'force_standalone' => '0',
'connect_retries' => '1',
'read_timeout' => '10',
'automatic_cleaning_factor' => '0',
'compress_data' => '1',
'compress_tags' => '0',
'compress_threshold' => '20480',
'compression_lib' => 'lzf',
),
),
),
),
'session' =>
array (
'save' => 'redis',
'redis' =>
array (
'host' => '/run/redis/redis6379.sock',
'port' => '',
'password' => '',
'timeout' => '2.5',
'persistent_identifier' => '',
'database' => '0',
'compress_data' => 1,
'compress_tags' => 1,
'compression_threshold' => '2048',
'compression_library' => 'lzf',
'log_level' => '1',
'max_concurrency' => '32',
'break_after_frontend' => '5',
'break_after_adminhtml' => '30',
'first_lifetime' => '6000',
'bot_first_lifetime' => '60',
'bot_lifetime' => '7200',
'disable_locking' => '0',
'min_lifetime' => '60',
'max_lifetime' => '2592000',
),
),

@apicht

This comment has been minimized.

Copy link
Author

apicht commented Jun 21, 2017

env.php:

<?php
return array (
  'backend' => 
  array (
    'frontName' => 'redacted',
  ),
  'install' => 
  array (
    'date' => 'Mon, 11 Apr 2016 22:01:25 +0000',
  ),
  'crypt' => 
  array (
    'key' => 'redacted',
  ),
  'session' => 
  array (
    'save' => 'redis',
    'redis' => 
    array (
      'host' => 'redis-master.redacted.com',
      'port' => '6379',
      'password' => 'redacted',
      'timeout' => '2.5',
      'persistent_identifier' => '',
      'database' => '2',
      'compression_threshold' => '2048',
      'compression_library' => 'gzip',
      'log_level' => '1',
      'max_concurrency' => '6',
      'break_after_frontend' => '5',
      'break_after_adminhtml' => '30',
      'first_lifetime' => '600',
      'bot_first_lifetime' => '60',
      'bot_lifetime' => '7200',
      'disable_locking' => '0',
      'min_lifetime' => '60',
      'max_lifetime' => '2592000',
    ),
  ),
  'db' => 
  array (
    'table_prefix' => '',
    'connection' => 
    array (
      'default' => 
      array (
        'host' => 'mysql-master.redacted.com',
        'dbname' => 'magento2',
        'username' => 'prod',
        'password' => 'redacted',
        'active' => '1',
      ),
    ),
  ),
  'resource' => 
  array (
    'default_setup' => 
    array (
      'connection' => 'default',
    ),
  ),
  'x-frame-options' => 'SAMEORIGIN',
  'MAGE_MODE' => 'production',
  'cache_types' => 
  array (
    'config' => 1,
    'layout' => 1,
    'block_html' => 1,
    'collections' => 1,
    'reflection' => 1,
    'db_ddl' => 1,
    'eav' => 1,
    'config_integration' => 1,
    'config_integration_api' => 1,
    'full_page' => 1,
    'translate' => 1,
    'config_webservice' => 1,
    'customer_notification' => 1,
  ),
  'cache' => 
  array (
    'frontend' => 
    array (
      'default' => 
      array (
        'backend' => 'Cm_Cache_Backend_Redis',
        'backend_options' => 
        array (
          'server' => 'redis-master.redacted.com',
          'port' => '6379',
          'password' => 'redacted',
          'database' => '0',
        ),
      ),
      'page_cache' => 
      array (
        'backend' => 'Cm_Cache_Backend_Redis',
        'backend_options' => 
        array (
          'server' => 'redis-master.redacted.com',
          'port' => '6379',
          'password' => 'redacted',
          'database' => '1',
          'compress_data' => '0',
        ),
      ),
    ),
  ),
);

@southerncomputer - Thanks for the tip on lzf, I'll have to try it once this is resolved.

@southerncomputer

This comment has been minimized.

Copy link
Contributor

southerncomputer commented Jun 21, 2017

Wait you have your full page_cache using Redis? I don't think anything but Varnish is supported in production!

If you had varnish installed you'd also have:
'http_cache_hosts' =>
array (
0 =>
array (
'host' => '127.0.0.1',
'port' => '80',
),
),

I would consider installing varnish 4 and removing page_cache section from your setup!

Btw, Since redis is only single-core non-threaded. It is better to run multiple instances of redis if you have spare cores to keep things separated such as back-end cache and session cache! for performance reasons!

@apicht

This comment has been minimized.

Copy link
Author

apicht commented Jun 21, 2017

I can't seem to find anything suggesting that Redis is not supported in production. We've been successfully using Redis on Magento sites since 1.x and this would come as a surprise to me.

The TTL exception above led me to believe this was somehow related to the FPC, but flushing the Redis FPC database doesn't clear the exception.

I'm aware of the single-thread issue with Redis, but this particular site doesn't get enough traffic to tax 1 core (~98% idle).

@southerncomputer

This comment has been minimized.

Copy link
Contributor

southerncomputer commented Jun 21, 2017

http://devdocs.magento.com/guides/v2.1/config-guide/varnish/config-varnish.html Strongly recommends varnish caching for production.. I just assumed everyone took that as meaning you should use varnish or-else. M2 is definitely a bit more complicated to deploy than M1!

maybe you are being affected by this issue:
#8554

I've patched my 2.1.2-2.1.7 against this but i don't use a CMS on the main page! good luck!

@amoskvenk

This comment has been minimized.

Copy link

amoskvenk commented Jul 20, 2017

This is exactly an issue, which we have faced on our project after upgrading to 2.1.6. The root cause of the issue is in the class "\Magento\Config\App\Config\Type\System": there is a code, which checks, are the config records present in Cache storage, by checking the availability of another record ("SYSTEM_CACHE_EXISTS") in Cache storage. :-D
As the result: on Production, which has a lot of concurrent requests, some records may be deleted from Redis by tag, but some other threads may renew some records in cache. As the result: you may get some config records absent in Redis, however SYSTEM_CACHE_EXISTS record may still be present and have positive value. As the result: Magento does not renew the config records from Disk and your application stops working. With our traffic we got such errors approximately every 1-2 hours.
The solution is: the availability of a record in Cache storage can be 100% determined by only reading this record from cache. You cannot 100% be sure that all other records are still present in Cache by reading the value of another record. We have fixed this issue a little bit tricky by overriding the entire class (because a lot of methods in this class are PRIVATE!) "\Magento\Config\App\Config\Type\System" and making changes in several methods. However I believe Magento core team will understand my comment and will make better fix for Core. In 2.1.7 there is the same code in that class as in 2.1.6.

Note: on Production we use Redis for regular cache and Varnish for FullPageCache. However the described issue was applicable for our projects too.

@elzekool

This comment has been minimized.

Copy link
Contributor

elzekool commented Jul 20, 2017

@amoskvenk Is there a possibility that you share your rewritten file in a gist until the Magento team fixes it?

@amoskvenk

This comment has been minimized.

Copy link

amoskvenk commented Jul 20, 2017

Sorry, but the code I wrote belongs to my company and I cannot share it by NDA. I can only give some tips how to implement it: need to make changes only in the method "readFromCache". If a config record is not loaded from cache, then it is: broken or just never exist in a config tree. Therefore after reloading need to check: if your $pathParts[0] and $pathParts[1] keys are present in the loaded config array. If keys are present, then the config was broken in Cache and now it is repaired and renewed. If keys are not present, then such path is never present in a config and need to push serialized empty array into Cache storage. Otherwise your config cache will be reloaded on each request from disk and your server will just stop working.

@apicht

This comment has been minimized.

Copy link
Author

apicht commented Jul 20, 2017

@southerncomputer FWIW, we just finally got Varnish through staging, QA, and onto production. The same thing keeps happening with Varnish in place.

@elzekool

This comment has been minimized.

Copy link
Contributor

elzekool commented Jul 21, 2017

@amoskvenk Thank you for your help. With your pointers I managed to create a patch myself. I can understand the NDA restrictions. If all works correctly I will share my patched file after the weekend

@elzekool

This comment has been minimized.

Copy link
Contributor

elzekool commented Jul 24, 2017

As promised, this is a patched version for System.php that fixes the Redis issue. Use this patched version as a temporary fix, I strongly advise you not to change core Magento files! Patch it in your own extension!

https://gist.github.com/elzekool/b3e77a48ff3010d082c94f98c3e3f0ea

@7ochem

This comment has been minimized.

Copy link
Contributor

7ochem commented Jul 24, 2017

@elzekool could you also provide a unified diff patch file? So I could see the actual changes and apply the patch to my own existing System.php file?

@amoskvenk

This comment has been minimized.

Copy link

amoskvenk commented Jul 25, 2017

@elzekool Nice! Your code is almost like mine. Good job!

@amoskvenk

This comment has been minimized.

Copy link

amoskvenk commented Jul 25, 2017

Aha. Sorry. Have found one issue in your code. You add empty array to the entire config array ($config). Then you save the entire config into Cache. This will lead to full cache reload problem every time, because each config record, which is not present neither in the DB, nor in a default.xml file, will delete empty arrays of other such config records from the entire config object. So on the next request another config record will reload the config again and will just add empty array into the $config array, but other empty arrays will be cleared. Therefore correct solution is: to add this empty array into a Separate cache record into own cache key. Therefore after reloading of the config cache from disk you will not delete empty arrays of other non-existing config records in cache, which will not trigger the cache reloading again.

@mrIntegrator

This comment has been minimized.

Copy link

mrIntegrator commented Jul 28, 2017

This seems to have been resolved by MAGETWO-69840 on the branch 2.1.8-preview. The class has been completely refactored.
https://github.com/magento/magento2/commits/2.1.8-preview/app/code/Magento/Config/App/Config/Type/System.php

@southerncomputer

This comment has been minimized.

Copy link
Contributor

southerncomputer commented Jul 28, 2017

so can you just apply the changed files for 2.1.8 to 2.1.7..2 ?

@mrIntegrator

This comment has been minimized.

Copy link

mrIntegrator commented Jul 28, 2017

I'll let you know after I've tried it!

@mrIntegrator

This comment has been minimized.

Copy link

mrIntegrator commented Jul 28, 2017

The MAGETWO-69840 commits can be applied to 2.1.7. So far, the bug has not reoccured for us. Don't take our word for it though; always test your application before applying the patch to production.

@southerncomputer

This comment has been minimized.

Copy link
Contributor

southerncomputer commented Jul 28, 2017

Alright i've patched it into my dev - patches clean! when is 2.1.8 coming out btw?

@mrIntegrator

This comment has been minimized.

Copy link

mrIntegrator commented Jul 28, 2017

I'm guessing that at the latest it would be in September when they launch 2.2, but I really don't know for sure. I wish they would release at regular intervals (every 6 or 8 weeks) instead. It would be easier to schedule maintenance that way to. At least they make the future releases visible here.

@magento-engcom-team

This comment has been minimized.

Copy link
Contributor

magento-engcom-team commented Nov 24, 2017

@apicht, thank you for your report.
The issue is already fixed in 2.1.10

@conflicker

This comment has been minimized.

Copy link

conflicker commented Mar 26, 2018

@magento-engcom-team is the fix also applicable in Magento EE?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.