Permalink
Browse files

feature #27543 [Cache] serialize objects using native arrays when pos…

…sible (nicolas-grekas)

This PR was merged into the 4.2-dev branch.

Discussion
----------

[Cache] serialize objects using native arrays when possible

| Q             | A
| ------------- | ---
| Branch?       | master
| Bug fix?      | no
| New feature?  | no
| BC breaks?    | no
| Deprecations? | no
| Tests pass?   | yes
| Fixed tickets | -
| License       | MIT
| Doc PR        | -

This PR allows leveraging OPCache shared memory when storing objects in `Php*` pool storages (as done by default for all system caches). This improves performance a bit further when loading e.g. annotations, etc. (bench coming);

Instead of using native php serialization, this uses a marshaller that represents objects in plain static arrays. Unmarshalling these arrays is faster than unserializing the corresponding PHP strings (because it works with copy-on-write, while unserialize cannot.)

php-serialization is still a possible format because we have to use it when serializing structures with internal references or with objects implementing `Serializable`. The best serialization format is selected automatically so this is completely seamless.

ping @palex-fpt since you gave me the push to work on this, and are pursuing a similar goal in #27484. I'd be thrilled to get some benchmarks on your scenarios.

Commits
-------

866420e [Cache] serialize objects using native arrays when possible
  • Loading branch information...
fabpot committed Jun 18, 2018
2 parents d075d0c + 866420e commit c0ca2afbb3a395aadf1f9e2bb8f430c14899534f
Showing with 1,390 additions and 88 deletions.
  1. +1 −0 .php_cs.dist
  2. +1 −0 src/Symfony/Bundle/FrameworkBundle/Resources/config/cache.xml
  3. +28 −10 src/Symfony/Component/Cache/Adapter/PhpArrayAdapter.php
  4. +5 −1 src/Symfony/Component/Cache/Adapter/PhpFilesAdapter.php
  5. +184 −0 src/Symfony/Component/Cache/Marshaller/PhpMarshaller.php
  6. +119 −0 src/Symfony/Component/Cache/Marshaller/PhpMarshaller/Configurator.php
  7. +30 −0 src/Symfony/Component/Cache/Marshaller/PhpMarshaller/Reference.php
  8. +58 −0 src/Symfony/Component/Cache/Marshaller/PhpMarshaller/Registry.php
  9. +22 −10 src/Symfony/Component/Cache/Simple/PhpArrayCache.php
  10. +5 −1 src/Symfony/Component/Cache/Simple/PhpFilesCache.php
  11. +25 −5 src/Symfony/Component/Cache/Tests/Adapter/PhpArrayAdapterTest.php
  12. +28 −0 src/Symfony/Component/Cache/Tests/Marshaller/Fixtures/array-iterator.optimized.php
  13. +30 −0 src/Symfony/Component/Cache/Tests/Marshaller/Fixtures/array-iterator.php
  14. +28 −0 src/Symfony/Component/Cache/Tests/Marshaller/Fixtures/array-object-custom.optimized.php
  15. +30 −0 src/Symfony/Component/Cache/Tests/Marshaller/Fixtures/array-object-custom.php
  16. +36 −0 src/Symfony/Component/Cache/Tests/Marshaller/Fixtures/array-object.optimized.php
  17. +42 −0 src/Symfony/Component/Cache/Tests/Marshaller/Fixtures/array-object.php
  18. +1 −0 src/Symfony/Component/Cache/Tests/Marshaller/Fixtures/bool.php
  19. +20 −0 src/Symfony/Component/Cache/Tests/Marshaller/Fixtures/clone.optimized.php
  20. +24 −0 src/Symfony/Component/Cache/Tests/Marshaller/Fixtures/clone.php
  21. +30 −0 src/Symfony/Component/Cache/Tests/Marshaller/Fixtures/datetime.optimized.php
  22. +32 −0 src/Symfony/Component/Cache/Tests/Marshaller/Fixtures/datetime.php
  23. +39 −0 src/Symfony/Component/Cache/Tests/Marshaller/Fixtures/private.optimized.php
  24. +43 −0 src/Symfony/Component/Cache/Tests/Marshaller/Fixtures/private.php
  25. +19 −0 src/Symfony/Component/Cache/Tests/Marshaller/Fixtures/serializable.optimized.php
  26. +23 −0 src/Symfony/Component/Cache/Tests/Marshaller/Fixtures/serializable.php
  27. +7 −0 src/Symfony/Component/Cache/Tests/Marshaller/Fixtures/simple-array.php
  28. +27 −0 src/Symfony/Component/Cache/Tests/Marshaller/Fixtures/spl-object-storage.optimized.php
  29. +31 −0 src/Symfony/Component/Cache/Tests/Marshaller/Fixtures/spl-object-storage.php
  30. +30 −0 src/Symfony/Component/Cache/Tests/Marshaller/Fixtures/wakeup.optimized.php
  31. +34 −0 src/Symfony/Component/Cache/Tests/Marshaller/Fixtures/wakeup.php
  32. +169 −0 src/Symfony/Component/Cache/Tests/Marshaller/PhpMarshallerTest.php
  33. +26 −8 src/Symfony/Component/Cache/Tests/Simple/PhpArrayCacheTest.php
  34. +4 −1 src/Symfony/Component/Cache/Traits/ApcuTrait.php
  35. +44 −20 src/Symfony/Component/Cache/Traits/PhpArrayTrait.php
  36. +113 −31 src/Symfony/Component/Cache/Traits/PhpFilesTrait.php
  37. +2 −1 src/Symfony/Component/Cache/composer.json
View
@@ -20,6 +20,7 @@ return PhpCsFixer\Config::create()
->append(array(__FILE__))
->exclude(array(
// directories containing files with content that is autogenerated by `var_export`, which breaks CS in output code
'Symfony/Component/Cache/Tests/Marshaller/Fixtures',
'Symfony/Component/DependencyInjection/Tests/Fixtures',
'Symfony/Component/Routing/Tests/Fixtures/dumper',
// fixture templates
@@ -41,6 +41,7 @@
<argument /> <!-- namespace -->
<argument>0</argument> <!-- default lifetime -->
<argument>%kernel.cache_dir%/pools</argument>
<argument>true</argument>
<call method="setLogger">
<argument type="service" id="logger" on-invalid="ignore" />
</call>
@@ -87,16 +87,21 @@ public function get(string $key, callable $callback, float $beta = null)
if (null === $this->values) {
$this->initialize();
}
if (null === $value = $this->values[$key] ?? null) {
if (!isset($this->keys[$key])) {
if ($this->pool instanceof CacheInterface) {
return $this->pool->get($key, $callback, $beta);
}
return $this->doGet($this->pool, $key, $callback, $beta ?? 1.0);
}
$value = $this->values[$this->keys[$key]];
if ('N;' === $value) {
return null;
}
if ($value instanceof \Closure) {
return $value();
}
if (\is_string($value) && isset($value[2]) && ':' === $value[1]) {
return unserialize($value);
}
@@ -115,15 +120,22 @@ public function getItem($key)
if (null === $this->values) {
$this->initialize();
}
if (!isset($this->values[$key])) {
if (!isset($this->keys[$key])) {
return $this->pool->getItem($key);
}
$value = $this->values[$key];
$value = $this->values[$this->keys[$key]];
$isHit = true;
if ('N;' === $value) {
$value = null;
} elseif ($value instanceof \Closure) {
try {
$value = $value();
} catch (\Throwable $e) {
$value = null;
$isHit = false;
}
} elseif (\is_string($value) && isset($value[2]) && ':' === $value[1]) {
try {
$value = unserialize($value);
@@ -167,7 +179,7 @@ public function hasItem($key)
$this->initialize();
}
return isset($this->values[$key]) || $this->pool->hasItem($key);
return isset($this->keys[$key]) || $this->pool->hasItem($key);
}
/**
@@ -182,7 +194,7 @@ public function deleteItem($key)
$this->initialize();
}
return !isset($this->values[$key]) && $this->pool->deleteItem($key);
return !isset($this->keys[$key]) && $this->pool->deleteItem($key);
}
/**
@@ -198,7 +210,7 @@ public function deleteItems(array $keys)
throw new InvalidArgumentException(sprintf('Cache key must be string, "%s" given.', is_object($key) ? get_class($key) : gettype($key)));
}
if (isset($this->values[$key])) {
if (isset($this->keys[$key])) {
$deleted = false;
} else {
$fallbackKeys[] = $key;
@@ -224,7 +236,7 @@ public function save(CacheItemInterface $item)
$this->initialize();
}
return !isset($this->values[$item->getKey()]) && $this->pool->save($item);
return !isset($this->keys[$item->getKey()]) && $this->pool->save($item);
}
/**
@@ -236,7 +248,7 @@ public function saveDeferred(CacheItemInterface $item)
$this->initialize();
}
return !isset($this->values[$item->getKey()]) && $this->pool->saveDeferred($item);
return !isset($this->keys[$item->getKey()]) && $this->pool->saveDeferred($item);
}
/**
@@ -253,11 +265,17 @@ private function generateItems(array $keys): \Generator
$fallbackKeys = array();
foreach ($keys as $key) {
if (isset($this->values[$key])) {
$value = $this->values[$key];
if (isset($this->keys[$key])) {
$value = $this->values[$this->keys[$key]];
if ('N;' === $value) {
yield $key => $f($key, null, true);
} elseif ($value instanceof \Closure) {
try {
yield $key => $f($key, $value(), true);
} catch (\Throwable $e) {
yield $key => $f($key, null, false);
}
} elseif (\is_string($value) && isset($value[2]) && ':' === $value[1]) {
try {
yield $key => $f($key, unserialize($value), true);
@@ -20,10 +20,14 @@ class PhpFilesAdapter extends AbstractAdapter implements PruneableInterface
use PhpFilesTrait;
/**
* @param $appendOnly Set to `true` to gain extra performance when the items stored in this pool never expire.
* Doing so is encouraged because it fits perfectly OPcache's memory model.
*
* @throws CacheException if OPcache is not enabled
*/
public function __construct(string $namespace = '', int $defaultLifetime = 0, string $directory = null)
public function __construct(string $namespace = '', int $defaultLifetime = 0, string $directory = null, bool $appendOnly = false)
{
$this->appendOnly = $appendOnly;
self::$startTime = self::$startTime ?? $_SERVER['REQUEST_TIME'] ?? time();
parent::__construct('', $defaultLifetime);
$this->init($namespace, $directory);
@@ -0,0 +1,184 @@
<?php
/*
* This file is part of the Symfony package.
*
* (c) Fabien Potencier <fabien@symfony.com>
*
* For the full copyright and license information, please view the LICENSE
* file that was distributed with this source code.
*/
namespace Symfony\Component\Cache\Marshaller;
use Symfony\Component\Cache\Marshaller\PhpMarshaller\Configurator;
use Symfony\Component\Cache\Marshaller\PhpMarshaller\Reference;
use Symfony\Component\Cache\Marshaller\PhpMarshaller\Registry;
/**
* @author Nicolas Grekas <p@tchwork.com>
*
* PhpMarshaller allows serializing PHP data structures using var_export()
* while preserving all the semantics associated to serialize().
*
* By leveraging OPcache, the generated PHP code is faster than doing the same with unserialize().
*
* @internal
*/
class PhpMarshaller
{
public static function marshall($value, int &$objectsCount)
{
if (!\is_object($value) && !\is_array($value)) {
return $value;
}
$objectsPool = new \SplObjectStorage();
$value = array($value);
$objectsCount = self::doMarshall($value, $objectsPool);
$classes = array();
$values = array();
$wakeups = array();
foreach ($objectsPool as $i => $v) {
list(, $classes[], $values[], $wakeup) = $objectsPool[$v];
if ($wakeup) {
$wakeups[$wakeup] = $i;
}
}
ksort($wakeups);
$properties = array();
foreach ($values as $i => $vars) {
foreach ($vars as $class => $values) {
foreach ($values as $name => $v) {
$properties[$class][$name][$i] = $v;
}
}
}
if (!$classes) {
return $value[0];
}
return new Configurator(new Registry($classes), $properties, $value[0], $wakeups);
}
public static function optimize(string $exportedValue)
{
return preg_replace(sprintf("{%s::__set_state\(array\(\s++'0' => (\d+),\s++\)\)}", preg_quote(Reference::class)), Registry::class.'::$objects[$1]', $exportedValue);
}
private static function doMarshall(array &$array, \SplObjectStorage $objectsPool): int
{
$objectsCount = 0;
foreach ($array as &$value) {
if (\is_array($value) && $value) {
$objectsCount += self::doMarshall($value, $objectsPool);
}
if (!\is_object($value)) {
continue;
}
if (isset($objectsPool[$value])) {
++$objectsCount;
$value = new Reference($objectsPool[$value][0]);
continue;
}
$class = \get_class($value);
$properties = array();
$sleep = null;
$arrayValue = (array) $value;
$proto = (Registry::$reflectors[$class] ?? Registry::getClassReflector($class))->newInstanceWithoutConstructor();
if ($value instanceof \ArrayIterator || $value instanceof \ArrayObject) {
// ArrayIterator and ArrayObject need special care because their "flags"
// option changes the behavior of the (array) casting operator.
$reflector = $value instanceof \ArrayIterator ? 'ArrayIterator' : 'ArrayObject';
$reflector = Registry::$reflectors[$reflector] ?? Registry::getClassReflector($reflector);
$properties = array(
$arrayValue,
$reflector->getMethod('getFlags')->invoke($value),
$value instanceof \ArrayObject ? $reflector->getMethod('getIteratorClass')->invoke($value) : 'ArrayIterator',
);
$reflector = $reflector->getMethod('setFlags');
$reflector->invoke($proto, \ArrayObject::STD_PROP_LIST);
if ($properties[1] & \ArrayObject::STD_PROP_LIST) {
$reflector->invoke($value, 0);
$properties[0] = (array) $value;
} else {
$reflector->invoke($value, \ArrayObject::STD_PROP_LIST);
$arrayValue = (array) $value;
}
$reflector->invoke($value, $properties[1]);
if (array(array(), 0, 'ArrayIterator') === $properties) {
$properties = array();
} else {
if ('ArrayIterator' === $properties[2]) {
unset($properties[2]);
}
$properties = array($reflector->class => array("\0" => $properties));
}
} elseif ($value instanceof \SplObjectStorage) {
foreach (clone $value as $v) {
$properties[] = $v;
$properties[] = $value[$v];
}
$properties = array('SplObjectStorage' => array("\0" => $properties));
} elseif ($value instanceof \Serializable) {
++$objectsCount;
$objectsPool[$value] = array($id = \count($objectsPool), serialize($value), array(), 0);
$value = new Reference($id);
continue;
}
if (\method_exists($class, '__sleep')) {
if (!\is_array($sleep = $value->__sleep())) {
trigger_error('serialize(): __sleep should return an array only containing the names of instance-variables to serialize', E_USER_NOTICE);
$value = null;
continue;
}
$sleep = array_flip($sleep);
}
$proto = (array) $proto;
foreach ($arrayValue as $name => $v) {
$k = (string) $name;
if ('' === $k || "\0" !== $k[0]) {
$c = $class;
} elseif ('*' === $k[1]) {
$c = $class;
$k = substr($k, 3);
} else {
$i = strpos($k, "\0", 2);
$c = substr($k, 1, $i - 1);
$k = substr($k, 1 + $i);
}
if (null === $sleep) {
$properties[$c][$k] = $v;
} elseif (isset($sleep[$k]) && $c === $class) {
$properties[$c][$k] = $v;
unset($sleep[$k]);
}
if (\array_key_exists($name, $proto) && $proto[$name] === $v) {
unset($properties[$c][$k]);
}
}
if ($sleep) {
foreach ($sleep as $k => $v) {
trigger_error(sprintf('serialize(): "%s" returned as member variable from __sleep() but does not exist', $k), E_USER_NOTICE);
}
}
$objectsPool[$value] = array($id = \count($objectsPool));
$objectsCount += 1 + self::doMarshall($properties, $objectsPool);
$objectsPool[$value] = array($id, $class, $properties, \method_exists($class, '__wakeup') ? $objectsCount : 0);
$value = new Reference($id);
}
return $objectsCount;
}
}
Oops, something went wrong.

0 comments on commit c0ca2af

Please sign in to comment.