New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[WIP] 7.0 kernel branch #1875

Merged
merged 34 commits into from Dec 15, 2017
Commits
Jump to file or symbol
Failed to load files and symbols.
+1,541 −7,538
Diff settings

Always

Just for now

Viewing a subset of changes. View all

[7.x] Refactor to use Symfony Cache for performance & better cache cl…

…aring with tags (#1920)

- Removes dependency on Stash
- Improves performance by:
  1. Use Symfony Cache
  2. Use tagging to be able to cache more, and:
  3. allow cache item duplication to avoid extra identifier to id lookups before item lookup
- Reduces code complexity of cache, for instance no cache warming anymore as it was often warming the wrong object
- More reliable, thanks to cache tagging only relevant cache items are cleared on operations, not the whole cache
  • Loading branch information...
andrerom authored and alongosz committed Mar 21, 2017
commit 124a0b719859d156f3ec0f15a60b2f116aa672d4
@@ -11,7 +11,14 @@ fi
# Enable redis
if [ "$CUSTOM_CACHE_POOL" = "singleredis" ] ; then
echo extension = redis.so >> ~/.phpenv/versions/$(phpenv version-name)/etc/php.ini
echo 'extension = redis.so' >> ~/.phpenv/versions/$(phpenv version-name)/etc/php.ini
# Configure redis to work in memory mode and avoid running out of memory
redis-cli config set appendfsync "no"
redis-cli config set maxmemory "60mb"
# commented out to detect if a test uses more then max memory or if clearing is not done correctly between tests
#redis-cli config set maxmemory-policy "allkeys-lru"
redis-cli config set save ""
fi
# Setup DB
View
@@ -21,7 +21,6 @@
"symfony-cmf/routing": "~1.1",
"qafoo/rmf": "1.0.*",
"kriswallsmith/buzz": ">=0.9",
"tedivm/stash-bundle": "~0.6.1",
"sensio/distribution-bundle": "^5.0",
"nelmio/cors-bundle": "^1.3.3",
"hautelook/templated-uri-bundle": "~1.0 | ~2.0",
View
@@ -14,6 +14,17 @@ Changes affecting version compatibility with former or future versions.
Id of Role is added on deletePolicy() to be able to properly clear cache
for the affected role.
* "cache_pool" service is now a Symfony 3 Cache Pool instead of Stash. if you type hinted against PSR-6 you should be
somewhat safe, but be on the lookout for nuances in behaviour. If you used Stash features like cache hierarchy,
you'll need to adapt. Recommendation is to adapt to use Symfony Cache, but you can also setup and use Stash yourself.
* "cache_pool" service is now a Symfony 3 Cache Pool instead of Stash. if you type hinted against PSR-6 you should be
somewhat safe, but be on the lookout for nuances in behaviour. If you used Stash features like cache hierarchy,
you'll need to adapt. Recommendation is to adapt to use Symfony Cache, but you can also setup and use Stash yourself.
* "cache_pool_name" siteaccess setting has been removed & replaced by "cache_service_name" as the semantic is different.
The new setting should contain the full service name of a symfony cache service, by default app_cache.app is used.
## Deprecations
@@ -0,0 +1,91 @@
# Persistence Cache
Persistence cache refers to a decorator implementation of `eZ\Publish\SPI\Persistence` that only caches data to reduce
lockups to underlying persisted storage engine. Implementation can be found in `eZ\Publish\Core\Persistence\Cache`.
As of kernel 7.0 the cache pool used internally is Symfony cache. And for the most part we use it as an implementation
of PSR-6. However as we have more advance needs for cache invalidation we specify that we require implementation of
`Symfony\Component\Cache\Adapter\TagAwareAdapterInterface`, as we take advantage of Cache tagging.
Tags are set when a cache item is saved, and used by any code that needs to invalidate a specif part of the cache.
You can think of it as a additional index for the cache item that you may invalidate it by, allowing you to index
the cache in a ways so exact items are invalidated on specific bulk operations affecting a sub set of the cache pool.
### Tag usage in Persistence Cache
List of tags and their meaning:
- `content-<content-id>` :
_Meta tag used on anything that are affected on changes to content, on content itself as well as location and so on._
- `content-fields-<content-id>` :
_Used on content/user object itself (with fields), for use when operations affects field values but not content meta info._
- `content-fields-type-<content-type-id>` :
_Same as above, but specifically for use when content type changes affecting content fields of it's type._
- `location-<location-id>` :
_Meta tag used on anything that are affected on changes to location, needs to be invalidated if location changes._
- `location-path-<location-id>` :
_Same as above, additional tags for all parents, for operations that changes the tree itself, like move/remove/(..)._
- `location-data-<location-id>` :
_Used on location, and invalidated when operations affect the properties on location only, e.g. swap/update._
- `location-path-data-<location-id>` :
_Same as above, but for operations affecting data in the tree, e.g. hide/unhide._
- `language-<language-id>` :
_Used on languages, and invalidated when operations affect a language._
- `type-<type-id>` :
_Used on types, and invalidated when operations affect a type._
- `type-group-<type-group-id>` :
_Used on type groups, and invalidated when operations affect a type groups._
- `type-map` :
_Used on type map info, like calculated info on searchable fields, invalidated on type changes._
- `state-<type-id>` :
_Used on states, and invalidated when operations affect a state._
- `state-group-<state-group-id>` :
_Used on state groups, and invalidated when operations affect a state groups._
- `section-<section-id>` :
_Used on sections, and invalidated when operations affect a section._
- `urlAlias-location-<location-id>` :
_Used on url alias, invalidated on location operations affecting url alias._
- `urlAlias-notFound` :
_Used for not found lookups for url alias by url as this is hot spot, invalidated on urlAlias creation._
- `role-<role-id>` :
_Used on roles, and invalidated when operations affect a role (role and policy operations)._
- `policy-<policy-id>` :
_Used on policies, and invalidated when operations affect a policy._
- `role-assignment-<role-assignment-id>` :
_Used on role assignment, and invalidated when operations affect a role assignment._
- `role-assignment-group-list-<content-id>` :
_Used for list of role assignment, and invalidated when operations affect it from content side._
- `role-assignment-role-list-<content-id>` :
_Used for list of role assignment, and invalidated when operations affect it from role side._
### Example of internal use
On `LocationHandler->create()` the following invalidation is done:
```php
$this->cache->invalidateTags(['content-' . $locationStruct->contentId, 'role-assignment-group-list-' . $locationStruct->contentId]);
```
This is done since the operation might affect content if main location changes, and as it might affect roles assignments
on the given that are connected to the given content (as in user might get additional roles when new location is added for user).
@@ -9,13 +9,14 @@
namespace eZ\Bundle\EzPublishCoreBundle\ApiLoader;
use eZ\Publish\Core\MVC\ConfigResolverInterface;
use Symfony\Component\Cache\Adapter\TagAwareAdapter;
use Symfony\Component\DependencyInjection\ContainerAwareInterface;
use Symfony\Component\DependencyInjection\ContainerAwareTrait;
/**
* Class CacheFactory.
*
* Service "ezpublish.cache_pool", selects a Stash cache service based on siteaccess[-group] setting "cache_pool_name"
* Service "ezpublish.cache_pool", selects a Symfony cache service based on siteaccess[-group] setting "cache_service_name"
*/
class CacheFactory implements ContainerAwareInterface
{
@@ -24,10 +25,12 @@ class CacheFactory implements ContainerAwareInterface
/**
* @param ConfigResolverInterface $configResolver
*
* @return \Stash\Interfaces\PoolInterface
* @return \Symfony\Component\Cache\Adapter\TagAwareAdapter
*/
public function getCachePool(ConfigResolverInterface $configResolver)
{
return $this->container->get(sprintf('stash.%s_cache', $configResolver->getParameter('cache_pool_name')));
return new TagAwareAdapter(
$this->container->get($configResolver->getParameter('cache_service_name'))
);
}
}
@@ -56,9 +56,10 @@ public function addSemanticConfig(NodeBuilder $nodeBuilder)
->scalarNode('dsn')->info('Full database DSN. Will replace settings above.')->example('mysql://root:root@localhost:3306/ezdemo')->end()
->end()
->end()
->scalarNode('cache_pool_name')
->example('ez_site_x')
->info('The cache pool name to use for a siteaccess / siteaccess-group, *must* be present under stash.caches: yml config. Default value is "default". NB! Setting is Deprecated, will be made redundant in future version.')
->scalarNode('cache_service_name')
->example('cache.app')
->defaultValue('cache.app')
->info('The cache pool service name to use for a siteaccess / siteaccess-group, *must* be present.')
->end()
->scalarNode('var_dir')
->cannotBeEmpty()
@@ -156,8 +157,8 @@ public function mapConfig(array &$scopeSettings, $currentScope, ContextualizerIn
if (isset($scopeSettings['repository'])) {
$contextualizer->setContextualParameter('repository', $currentScope, $scopeSettings['repository']);
}
if (isset($scopeSettings['cache_pool_name'])) {
$contextualizer->setContextualParameter('cache_pool_name', $currentScope, $scopeSettings['cache_pool_name']);
if (isset($scopeSettings['cache_service_name'])) {
$contextualizer->setContextualParameter('cache_service_name', $currentScope, $scopeSettings['cache_service_name']);
}
if (isset($scopeSettings['var_dir'])) {
$contextualizer->setContextualParameter('var_dir', $currentScope, $scopeSettings['var_dir']);
@@ -65,7 +65,7 @@ parameters:
ezsettings.default.languages: []
ezsettings.default.translation_siteaccesses: []
ezsettings.default.related_siteaccesses: []
ezsettings.default.cache_pool_name: "default" # The cache pool name to use for a siteaccess / siteaccess-group
ezsettings.default.cache_service_name: "cache.app" # The cache pool serive name to use for a siteaccess / siteaccess-group
ezsettings.default.var_dir: "var" # The root directory where all log files, cache files and other stored files are created
ezsettings.default.storage_dir: "storage" # Where to place new files for storage, it's relative to var directory
ezsettings.default.binary_dir: "original"
@@ -9,6 +9,10 @@
namespace eZ\Bundle\EzPublishCoreBundle\Tests\ApiLoader;
use eZ\Bundle\EzPublishCoreBundle\ApiLoader\CacheFactory;
use eZ\Publish\Core\MVC\ConfigResolverInterface;
use Symfony\Component\Cache\Adapter\AdapterInterface;
use Symfony\Component\Cache\Adapter\TagAwareAdapter;
use Symfony\Component\DependencyInjection\ContainerInterface;
use PHPUnit\Framework\TestCase;
class CacheFactoryTest extends TestCase
@@ -26,8 +30,8 @@ class CacheFactoryTest extends TestCase
protected function setUp()
{
parent::setUp();
$this->configResolver = $this->createMock('eZ\\Publish\\Core\\MVC\\ConfigResolverInterface');
$this->container = $this->createMock('Symfony\\Component\\DependencyInjection\\ContainerInterface');
$this->configResolver = $this->createMock(ConfigResolverInterface::class);
$this->container = $this->createMock(ContainerInterface::class);
}
/**
@@ -36,9 +40,9 @@ protected function setUp()
public function providerGetService()
{
return array(
array('default', 'stash.default_cache'),
array('ez_site1', 'stash.ez_site1_cache'),
array('xyZ', 'stash.xyZ_cache'),
array('default', 'default'),
array('ez_site1', 'ez_site1'),
array('xyZ', 'xyZ'),
);
}
@@ -50,18 +54,18 @@ public function testGetService($name, $expected)
$this->configResolver
->expects($this->once())
->method('getParameter')
->with('cache_pool_name')
->with('cache_service_name')
->will($this->returnValue($name));
$this->container
->expects($this->once())
->method('get')
->with($expected)
->will($this->returnValue(false));
->will($this->returnValue($this->getMock(AdapterInterface::class)));
$factory = new CacheFactory();
$factory->setContainer($this->container);
$this->assertFalse($factory->getCachePool($this->configResolver));
$this->assertInstanceOf(TagAwareAdapter::class, $factory->getCachePool($this->configResolver));
}
}
@@ -116,7 +116,7 @@ public function testNonExistentSettings()
{
$this->load();
$this->assertConfigResolverParameterValue('url_alias_router', true, 'ezdemo_site');
$this->assertConfigResolverParameterValue('cache_pool_name', 'default', 'ezdemo_site');
$this->assertConfigResolverParameterValue('cache_service_name', 'cache.app', 'ezdemo_site');
$this->assertConfigResolverParameterValue('var_dir', 'var', 'ezdemo_site');
$this->assertConfigResolverParameterValue('storage_dir', 'storage', 'ezdemo_site');
$this->assertConfigResolverParameterValue('binary_dir', 'original', 'ezdemo_site');
@@ -144,7 +144,7 @@ public function testMiscSettings()
array(
'system' => array(
'ezdemo_site' => array(
'cache_pool_name' => $cachePoolName,
'cache_service_name' => $cachePoolName,
'var_dir' => $varDir,
'storage_dir' => $storageDir,
'binary_dir' => $binaryDir,
@@ -159,7 +159,7 @@ public function testMiscSettings()
)
);
$this->assertConfigResolverParameterValue('cache_pool_name', $cachePoolName, 'ezdemo_site');
$this->assertConfigResolverParameterValue('cache_service_name', $cachePoolName, 'ezdemo_site');
$this->assertConfigResolverParameterValue('var_dir', $varDir, 'ezdemo_site');
$this->assertConfigResolverParameterValue('storage_dir', $storageDir, 'ezdemo_site');
$this->assertConfigResolverParameterValue('binary_dir', $binaryDir, 'ezdemo_site');
@@ -1877,6 +1877,11 @@ public function testPublishVersionNotCreatingUnlimitedArchives()
$content = $this->createContentVersion1();
// load first to make sure list gets updated also (cache)
$versionInfoList = $contentService->loadVersions($content->contentInfo);
$this->assertEquals(1, count($versionInfoList));
$this->assertEquals(1, $versionInfoList[0]->versionNo);
// Create a new draft with versionNo = 2
$draftedContentVersion = $contentService->createContentDraft($content->contentInfo);
$contentService->publishVersion($draftedContentVersion->getVersionInfo());
@@ -9,24 +9,32 @@
namespace eZ\Publish\API\Repository\Tests\Regression;
use eZ\Publish\API\Repository\Tests\BaseTest;
use Symfony\Component\Cache\Adapter\TagAwareAdapter;
use Symfony\Component\Cache\Adapter\RedisAdapter;
use Symfony\Component\Cache\Adapter\ArrayAdapter;
/**
* Test case to verify Integration tests are setup with the right instances.
*/
class EnvTest extends BaseTest
{
/**
* Verify Redis is setup if asked for.
* Verify Redis cache is setup if asked for, if not file system.
*/
public function testVerifyStashDriver()
public function testVerifyCacheDriver()
{
/** @var \Stash\Pool $pool */
$pool = $this->getSetupFactory()->getServiceContainer()->get('ezpublish.cache_pool');
$this->assertInstanceOf(TagAwareAdapter::class, $pool);
$reflectionPool = new \ReflectionProperty($pool, 'itemsAdapter');
$reflectionPool->setAccessible(true);
$innerPool = $reflectionPool->getValue($pool);
if (getenv('CUSTOM_CACHE_POOL') === 'singleredis') {
$this->assertInstanceOf('\Stash\Driver\Redis', $pool->getDriver());
$this->assertInstanceOf(RedisAdapter::class, $innerPool);
} else {
$this->assertInstanceOf('\Stash\Driver\Ephemeral', $pool->getDriver());
$this->assertInstanceOf(ArrayAdapter::class, $innerPool);
}
}
}
@@ -259,10 +259,10 @@ protected function clearInternalCaches()
$contentTypeHandler->clearCache();
}
/** @var $decorator \eZ\Publish\Core\Persistence\Cache\Tests\Helpers\IntegrationTestCacheServiceDecorator */
$decorator = $this->getServiceContainer()->get('ezpublish.cache_pool.spi.cache.decorator');
/** @var $cachePool \Psr\Cache\CacheItemPoolInterface */
$cachePool = $this->getServiceContainer()->get('ezpublish.cache_pool');
$decorator->clearAllTestData();
$cachePool->clear();
}
/**
@@ -9,6 +9,7 @@
namespace eZ\Publish\Core\Persistence\Cache;
use eZ\Publish\SPI\Persistence\Handler as PersistenceHandler;
use Symfony\Component\Cache\Adapter\TagAwareAdapterInterface;
/**
* Class AbstractHandler.
@@ -18,7 +19,7 @@
abstract class AbstractHandler
{
/**
* @var \eZ\Publish\Core\Persistence\Cache\CacheServiceDecorator
* @var \Symfony\Component\Cache\Adapter\TagAwareAdapterInterface
*/
protected $cache;
@@ -35,12 +36,12 @@
/**
* Setups current handler with everything needed.
*
* @param \eZ\Publish\Core\Persistence\Cache\CacheServiceDecorator $cache
* @param \Symfony\Component\Cache\Adapter\TagAwareAdapterInterface $cache
* @param \eZ\Publish\SPI\Persistence\Handler $persistenceHandler
* @param \eZ\Publish\Core\Persistence\Cache\PersistenceLogger $logger
*/
public function __construct(
CacheServiceDecorator $cache,
TagAwareAdapterInterface $cache,
PersistenceHandler $persistenceHandler,
PersistenceLogger $logger
) {
Oops, something went wrong.
ProTip! Use n and p to navigate between commits in a pull request.