Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

Loading…

read error on connection #18

Closed
fr0x opened this Issue · 15 comments

4 participants

@fr0x

I saw you popped into this issue with PHPRedis here:
phpredis/phpredis#70

It might completely reside with them but thought I would post here in case you had found a solution outside of PHPRedis to handle this with Cm_Cache_Backend_Redis.

The only new thing I saw of note in that thread was within the past few weeks:

"I encountered this same error message. Using the tip from @redzarf - I did some research and found that some of the strings we were trying to store are indeed more than 64KB (65,536 Characters).

As a quick solution I started gzip'ing data before storing it in redis. Havn't seen the "read error on connection" error since."

Here are some of my dump(s) when this hits:

Error:
read error on connection

Trace:
#0 /app/code/community/Cm/Cache/Backend/Redis.php(128): Credis_Client->__call('hGet', Array)
#1 /app/code/community/Cm/Cache/Backend/Redis.php(128): Credis_Client->hGet('zc:k:139_TRANSL...', 'd')
#2 /lib/Zend/Cache/Core.php(303): Cm_Cache_Backend_Redis->load('139_TRANSLATE_E...', false)
#3 /app/code/core/Mage/Core/Model/Cache.php(351): Zend_Cache_Core->load('TRANSLATE_EN_US...')
#4 /app/code/core/Mage/Core/Model/App.php(1126): Mage_Core_Model_Cache->load('translate_en_US...')
#5 /app/code/core/Mage/Core/Model/Translate.php(521): Mage_Core_Model_App->loadCache('translate_en_US...')
#6 /app/code/core/Mage/Core/Model/Translate.php(121): Mage_Core_Model_Translate->_loadCache()
#7 /app/code/core/Mage/Core/Model/App/Area.php(146): Mage_Core_Model_Translate->init('frontend')
#8 /app/code/core/Mage/Core/Model/App/Area.php(121): Mage_Core_Model_App_Area->_initTranslate()
#9 /app/code/core/Mage/Core/Model/App/Area.php(93): Mage_Core_Model_App_Area->_loadPart('translate')
#10 /app/code/core/Mage/Core/Model/App.php(768): Mage_Core_Model_App_Area->load()
#11 /app/code/core/Mage/Core/Controller/Varien/Action.php(493): Mage_Core_Model_App->loadArea('frontend')
#12 /app/code/core/Mage/Core/Controller/Front/Action.php(59): Mage_Core_Controller_Varien_Action->preDispatch()
#13 /app/code/core/Mage/Core/Controller/Varien/Action.php(409): Mage_Core_Controller_Front_Action->preDispatch()
#14 /app/code/core/Mage/Core/Controller/Varien/Router/Standard.php(250): Mage_Core_Controller_Varien_Action->dispatch('view')
#15 /app/code/core/Mage/Core/Controller/Varien/Front.php(176): Mage_Core_Controller_Varien_Router_Standard->match(Object(Mage_Core_Controller_Request_Http))
#16 /app/code/core/Mage/Core/Model/App.php(349): Mage_Core_Controller_Varien_Front->dispatch()
#17 /app/Mage.php(640): Mage_Core_Model_App->run(Array)
#18 /index.php(83): Mage::run('', 'store')
#19 {main}

Error:
read error on connection

Trace:
#0 /app/code/community/Cm/Cache/Backend/Redis.php(128): Credis_Client->__call('hGet', Array)
#1 /app/code/community/Cm/Cache/Backend/Redis.php(128): Credis_Client->hGet('zc:k:139_DB_PDO...', 'd')
#2 /lib/Zend/Cache/Core.php(303): Cm_Cache_Backend_Redis->load('139_DB_PDO_MYSQ...', false)
#3 /lib/Varien/Db/Adapter/Pdo/Mysql.php(1442): Zend_Cache_Core->load('DB_PDO_MYSQL_DD...')
#4 /lib/Varien/Db/Adapter/Pdo/Mysql.php(1564): Varien_Db_Adapter_Pdo_Mysql->loadDdlCache('eav_attribute', 1)
#5 /app/code/core/Mage/Catalog/Model/Resource/Product/Attribute/Collection.php(55): Varien_Db_Adapter_Pdo_Mysql->describeTable('eav_attribute')
#6 /app/code/core/Mage/Core/Model/Resource/Db/Collection/Abstract.php(135): Mage_Catalog_Model_Resource_Product_Attribute_Collection->_initSelect()
#7 /app/code/core/Mage/Core/Model/Config.php(1350): Mage_Core_Model_Resource_Db_Collection_Abstract->__construct(Array)
#8 /app/code/core/Mage/Core/Model/Config.php(1386): Mage_Core_Model_Config->getModelInstance('catalog_resourc...', Array)
#9 /app/Mage.php(460): Mage_Core_Model_Config->getResourceModelInstance('catalog/product...', Array)
#10 /app/code/core/Mage/Catalog/Model/Layer.php(226): Mage::getResourceModel('catalog/product...')
#11 /app/code/core/Mage/Catalog/Block/Layer/View.php(163): Mage_Catalog_Model_Layer->getFilterableAttributes()
#12 /app/code/local/Lp/Catalog/Block/Layer/View.php(121): Mage_Catalog_Block_Layer_View->_getFilterableAttributes()
#13 /app/code/core/Mage/Core/Block/Abstract.php(238): Lp_Catalog_Block_Layer_View->_prepareLayout()
#14 /app/code/core/Mage/Core/Model/Layout.php(430): Mage_Core_Block_Abstract->setLayout(Object(Mage_Core_Model_Layout))
#15 /app/code/core/Mage/Core/Model/Layout.php(446): Mage_Core_Model_Layout->createBlock('catalog/layer_v...', 'catalog.leftnav')
#16 /app/code/core/Mage/Core/Model/Layout.php(238): Mage_Core_Model_Layout->addBlock('catalog/layer_v...', 'catalog.leftnav')
#17 /app/code/core/Mage/Core/Model/Layout.php(204): Mage_Core_Model_Layout->_generateBlock(Object(Mage_Core_Model_Layout_Element), Object(Mage_Core_Model_Layout_Element))
#18 /app/code/core/Mage/Core/Model/Layout.php(209): Mage_Core_Model_Layout->generateBlocks(Object(Mage_Core_Model_Layout_Element))
#19 /app/code/core/Mage/Core/Controller/Varien/Action.php(345): Mage_Core_Model_Layout->generateBlocks()
#20 /app/code/core/Mage/Catalog/controllers/CategoryController.php(150): Mage_Core_Controller_Varien_Action->generateLayoutBlocks()
#21 /app/code/core/Mage/Core/Controller/Varien/Action.php(420): Mage_Catalog_CategoryController->viewAction()
#22 /app/code/core/Mage/Core/Controller/Varien/Router/Standard.php(250): Mage_Core_Controller_Varien_Action->dispatch('view')
#23 /app/code/core/Mage/Core/Controller/Varien/Front.php(176): Mage_Core_Controller_Varien_Router_Standard->match(Object(Mage_Core_Controller_Request_Http))
#24 /app/code/core/Mage/Core/Model/App.php(349): Mage_Core_Controller_Varien_Front->dispatch()
#25 /app/Mage.php(640): Mage_Core_Model_App->run(Array)
#26 /index.php(83): Mage::run('', 'store')
#27 {main}

@fr0x

As far as consistency....this seems to happen around once a day (when it does trigger, it tends to trigger 4-7 errors in a row, all read error on connection). Time it does happen is not consistent nor is the load on the server at the time of it happening.

@colinmollenhour

Interesting. If you don't like compression perhaps setting the "compress_threshold" option at 64k would be a good solution then.

You might also check your Redis config regarding disk activity as the cache is pretty low-priority and should be written to disk rarely if at all and the append-only log doesn't need to be enabled IMO.

@fr0x

The over 64KB post was actually from someone else. I do have compression enabled in my Cm_Cache_Backend_Redis

1 <!-- 0-9 for compression level, recommended: 0 or 1 -->
1 <!-- 0-9 for compression level, recommended: 0 or 1 -->
20480 <!-- Strings below this size will not be compressed -->
gzip <!-- Supports gzip, lzf and snappy -->

(think those are all default values)

Just was more curious if you still see the "read error on connection" error popup with Cm_Cache_Backend_Redis and/or the times you ran into the issue if it was data that was > 64KB

@colinmollenhour

I don't recall ever discovering some exact size that triggered errors consistently and I can't remember what the largest size I tested was but it seems like it may have been over 64k..

@jonathanselander

I run into this as well, but only when i flushAll() from PHP, it doesn't seem to happen if i use redis-cli.

@davidyilee

We had the same issue, turning on lzf compression for strings above 40k fix this for us. We did not try pushing it up to 64k to test the actual boundary.

@colinmollenhour

Yeah this issue is definitely a phpredis bug: phpredis/phpredis#70

Redis server's redis.h contains a 64k constant named: REDIS_INLINE_MAX_SIZE. However, I have no idea how to fix the bug but I believe the fault lies with phpredis. I suppose either Credis_Client or Cm_Cache_Backend_Redis could try to split larger string into multiple keys but with expiration and all this would be messy..

@colinmollenhour

I just updated the unit tests for lib/Credis and can't reproduce the issue with 128K length strings.. I was using an old phpredis version and also just built the latest and still couldn't reproduce it.

@jonathanselander
@colinmollenhour

The other stated reason for "read error on connection" issues seemed to be timeouts. Does the flushAll operation take much time to complete? If so it may be that the read timeout is occurring. I can see this may be the case if flushAll causes a wait on disk activity and your disk is slow or highly utilized.. A similar situation could explain the errors on cache loads: the Redis server is busy handling other requests and the read timeout is too short.

@colinmollenhour

I somehow overlooked this before, but phpredis currently uses the connect timeout as the read timeout as well.. This is not ideal, but as a workaround you could set your timeout higher to avoid the read errors. The drawback is if there actually is a connection problem the error will not be thrown as quickly so don't set it too high.

I added a setReadTimeout method to Credis_Client and support for a read_timeout option in Cm_Cache_Backend_Redis which will not work for phpredis until phpredis/phpredis#260 is merged. So, go post a note there kindly asking the maintainer to merge it.. :)

If it is very urgent you could merge that pull request with phpredis yourself and run with your forked version.

If anyone tries this please let me know if it fixes your issues.

@DerekMarcinyshyn DerekMarcinyshyn referenced this issue from a commit in DerekMarcinyshyn/Cm_Cache_Backend_Redis
@colinmollenhour Add read_timeout option (see README). Refs #18 f198b62
@colinmollenhour

FYI, a version of phpredis including the read timeout feature has finally been merged and tagged. Does setting a longer read timeout fix your issue?

@fr0x
@fr0x

After upgrading both phpredis and Cm_Cache_Backend_Redis I haven't had this issue come back up (has been about 2 weeks). Not sure if the combination was required or if upgrading both was required (I dont believe the version of Cm_Cache_Backend_Redis had timeout / retry values implemented yet).

@colinmollenhour

Cool, thanks for the update. Upgrading both would be probably be required in your case. Closing ticket unless there are any objections.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.