Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Models meta data cache fails to save and the app is DOWN!!! #13433

Closed
Shay12tg opened this issue Jul 14, 2018 · 12 comments

Comments

@Shay12tg
Copy link

commented Jul 14, 2018

If the model meta data cache fails to save, the app crashes... WTF?? :(
Over time it happens a lot and for me it's an issue... please help me find a solution for a stable app.

I use memcache to save metadata cache with the provided adapter ( Phalcon\Mvc\Model\Metadata\Memcache ).
I installed a simple local memcached instance.
it works.... until it went to production and it started failing.
So I added a memcached health check (connect and save random data) to the health check file. but no luck... still getting these errors about once a week and the app goes down for a few minutes/hours.

The app should not fail in this case (imo). it should catch the error, throw a warning and fallback to memory or something...

To recreate, use a memcached metadata adapter with no server.. it's about the same..
The app doesn't have to fail if the cache fails...

Error from cloudwatch:

{
    "code": 0,
    "message": "Failed storing data in memcached",
    "file": "phalcon/cache/backend/memcache.zep",
    "line": 241,
    "trace": "
#0 [internal function]: Phalcon\\Cache\\Backend\\Memcache->save('meta-files-File...', Array)
#1 [internal function]: Phalcon\\Mvc\\Model\\MetaData\\Memcache->write('meta-files-File...', Array)
#2 [internal function]: Phalcon\\Mvc\\Model\\MetaData->_initialize(Object(Files), 'files-Files', 'Files', '')
#3 [internal function]: Phalcon\\Mvc\\Model\\MetaData->writeMetaDataIndex(Object(Files), 10, Array)
#4 [internal function]: Phalcon\\Mvc\\Model\\MetaData->setAutomaticCreateAttributes(Object(Files), Array)
#5 [internal function]: Phalcon\\Mvc\\Model->skipAttributesOnCreate(Array)
#6 /var/www/api/app/models/Files.php(46): Phalcon\\Mvc\\Model->skipAttributes(Array)
#7 [internal function]: Files->initialize()
#8 [internal function]: Phalcon\\Mvc\\Model\\Manager->initialize(Object(Files))
#9 [internal function]: Phalcon\\Mvc\\Model->__construct()
#10 [internal function]: Phalcon\\Mvc\\Model::_invokeFinder('findFirstByID', Array)
#11 /var/www/api/app/controllers/FilesController.php(284): Phalcon\\Mvc\\Model::__callStatic('findFirstByID', Array)
#12 [internal function]: FilesController->previewAction('1352')
#13 [internal function]: Phalcon\\Dispatcher->callActionMethod(Object(FilesController), 'previewAction', Array)\n#14 [internal function]: Phalcon\\Dispatcher->dispatch()
#15 /var/www/api/index.php(44): Phalcon\\Mvc\\Application->handle()
#16 {main}
"
}

Details

  • Phalcon version: 3.4.0
  • PHP Version: 7.2.7
  • Operating System: Ubuntu 16.04 x64
  • Installation type: package manager
  • Zephir version (if any): 0.10.10-d1b4cc68d9
  • Server: Apache
  • Other related info (Database, table schema): MySQL..
@sergeyklay

This comment has been minimized.

Copy link
Member

commented Jul 15, 2018

@Shay12tg We'll need a script to reproduce. As small as possible please.

@Shay12tg

This comment has been minimized.

Copy link
Author

commented Jul 15, 2018

@sergeyklay thanks for the quick response.
I get a different error on production (fails to save, not connect) but I can't reproduce it.
However if it fails to connect it still produces a fatal error. warning is enough imo. cache doesn't have to break the app.

<?php

$di = new Phalcon\DI\FactoryDefault();

$di->set('db', function() use ($config) {
	return new Phalcon\Db\Adapter\Pdo\Mysql([
		'host' => 'localhost',
		'dbname' => 'test',
		'username'	=> 'root',
		'password'	=> ''
	]);
});

$di->set('modelsMetadata', function(){
	return new Phalcon\Mvc\Model\Metadata\Memcache([
		'port' => 1337
	]);
});

class Files extends \Phalcon\Mvc\Model
{
	public $id;
	public $path;
	public $code;

	public function initialize()
	{
		$this->skipAttributes(['code']);
	}

	public function getSource()
	{
		return 'Files';
	}
}

echo Files::findFirstById(1) ? 'found' : 'notfound';
@Shay12tg

This comment has been minimized.

Copy link
Author

commented Jul 15, 2018

Without DB:

<?php

$di = new Phalcon\DI\FactoryDefault();

$di->set('modelsMetadata', function(){
	return new Phalcon\Mvc\Model\Metadata\Memcache([
		'port' => 1337
	]);
});
$data = $di->get('modelsMetadata')->read('a');

This case should fail, but the case above should not. the app can continue without the cache or fallback to memory

@Tanami

This comment has been minimized.

Copy link

commented Aug 21, 2018

The app should not fail in this case (imo). it should catch the error, throw a warning and fallback to memory or something...

I think what you mean is "I need to write some code to check if the cache backend is up and if not fall back to a different cache method".

@Jurigag

This comment has been minimized.

Copy link
Member

commented Oct 25, 2018

To be honest in our production application we also have this problem with meta data memcache adapter, sometimes we even have some weird things written to metadata. It happens more often if we have enabled persistent connection. On other hand we use apc/apcu on fpm(don't ask me why) and it works perfectly fine.

@niden niden added the Bug - Medium label Feb 15, 2019

@niden niden added this to To do in 4.0 Release via automation Feb 15, 2019

@niden

This comment has been minimized.

Copy link
Member

commented Feb 15, 2019

Relates to #13439

@niden

This comment has been minimized.

Copy link
Member

commented Apr 28, 2019

The Memcache adapter has been deprecated so this is no longer relevant (at least for this particular use case).

However I have added an exception to the MetaData::write() so that if the data cannot be stored in any of the available adapters for v4, an exception will be thrown to inform the user.

@niden

This comment has been minimized.

Copy link
Member

commented Apr 28, 2019

The relevant commit is here:

niden@7e6be9b

@niden niden closed this Apr 28, 2019

4.0 Release automation moved this from To do to Done Apr 28, 2019

@niden niden reopened this Apr 28, 2019

4.0 Release automation moved this from Done to In progress Apr 28, 2019

@niden

This comment has been minimized.

Copy link
Member

commented Apr 28, 2019

Reopening this after discussion with @CameronHall

If the metadata cache dies it should not kill the whole application.

  • Introduce a method to throw or not throw an exception on write fail something like
$metadata->doNotRaiseExceptionOnWriteFailOtherwiseCameronWillThrowAFit(true);

that will not throw an exception if the write fails but rather issue a warning in the logs for the devops team.

@sergeyklay

This comment has been minimized.

Copy link
Member

commented May 2, 2019

@niden Consider using globals:

cphalcon/config.json

Lines 35 to 112 in 9fae862

"globals": {
"db.escape_identifiers": {
"type": "bool",
"default": true
},
"db.force_casting": {
"type": "bool",
"default": false
},
"orm.parser_cache": {
"type": "hash",
"default": "NULL"
},
"orm.ast_cache": {
"type": "hash",
"default": "NULL"
},
"orm.cache_level": {
"type": "int",
"default": 3
},
"orm.unique_cache_id": {
"type": "int",
"default": 3
},
"orm.events": {
"type": "bool",
"default": true
},
"orm.virtual_foreign_keys": {
"type": "bool",
"default": true
},
"orm.column_renaming": {
"type": "bool",
"default": true
},
"orm.not_null_validations": {
"type": "bool",
"default": true
},
"orm.exception_on_failed_save": {
"type": "bool",
"default": false
},
"orm.enable_literals": {
"type": "bool",
"default": true
},
"orm.late_state_binding": {
"type": "bool",
"default": false
},
"orm.enable_implicit_joins": {
"type": "bool",
"default": true
},
"orm.cast_on_hydrate": {
"type": "bool",
"default": false
},
"orm.ignore_unknown_columns": {
"type": "bool",
"default": false
},
"orm.update_snapshot_on_save": {
"type": "bool",
"default": true
},
"orm.disable_assign_setters": {
"type": "bool",
"default": false
},
"orm.case_insensitive_column_map": {
"type": "bool",
"default": false
}
},

For more see: https://docs.zephir-lang.com/0.11/en/globals

@Jurigag

This comment has been minimized.

Copy link
Member

commented May 7, 2019

I agree, better use global.

Also in some cases - metadata can't be saved because it can be actually too large - The maximum size of a value you can store in memcached is 1 megabyte there are cases that for some kind of huge model it can be bigger.

@niden niden referenced this issue May 21, 2019

Closed

T13433 metadata cache exception #14108

4 of 5 tasks complete

@niden niden removed the Docs needed label May 21, 2019

@niden niden referenced this issue May 21, 2019

Merged

T13433 metadata cache exception 2 #14112

4 of 5 tasks complete
@niden

This comment has been minimized.

Copy link
Member

commented May 26, 2019

Resolved #14112

@niden niden closed this May 26, 2019

4.0 Release automation moved this from In progress to Done May 26, 2019

@niden niden added the 4.0 label Jun 21, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
5 participants
You can’t perform that action at this time.