Use APC cache for autoloading #127

Closed
jcleveley opened this Issue Nov 22, 2011 · 21 comments

Comments

Projects
None yet
8 participants
@jcleveley

Looking at the autoload class it seems its checking directories every page request, perhaps something like ApcUniversalClassLoader might be good to cache the class directories.

John

@Seldaek

This comment has been minimized.

Show comment
Hide comment
@Seldaek

Seldaek Nov 22, 2011

Member

The plan is to add a new command to dump an optimized autoloader later on (probably a class map one though, APC is better but still not the fastest) for production use.

Member

Seldaek commented Nov 22, 2011

The plan is to add a new command to dump an optimized autoloader later on (probably a class map one though, APC is better but still not the fastest) for production use.

@jcleveley

This comment has been minimized.

Show comment
Hide comment
@jcleveley

jcleveley Nov 22, 2011

class map would be even better! Plus its universal for people without APC

class map would be even better! Plus its universal for people without APC

@roadrunner

This comment has been minimized.

Show comment
Hide comment
@roadrunner

roadrunner Jan 5, 2012

@Seldaek, what is your suggestion over APC autoloading in production environment?

@Seldaek, what is your suggestion over APC autoloading in production environment?

@Seldaek

This comment has been minimized.

Show comment
Hide comment
@Seldaek

Seldaek Jan 7, 2012

Member

As said above, a class map autoloader. Basically an autoloader that contains a big array with all known classes and the file they're in. Then autoloading is a simple matter of an isset() call + an include. That's the fastest as far as I know, but it takes some work to generate that class map.

Member

Seldaek commented Jan 7, 2012

As said above, a class map autoloader. Basically an autoloader that contains a big array with all known classes and the file they're in. Then autoloading is a simple matter of an isset() call + an include. That's the fastest as far as I know, but it takes some work to generate that class map.

@roadrunner

This comment has been minimized.

Show comment
Hide comment
@roadrunner

roadrunner Jan 8, 2012

With array map, it seems to be significant performance could be gained when the number of used classes around 1k.
Here is my simple test: https://gist.github.com/1577462
By the way, thanks for the tip.

With array map, it seems to be significant performance could be gained when the number of used classes around 1k.
Here is my simple test: https://gist.github.com/1577462
By the way, thanks for the tip.

@sliver

This comment has been minimized.

Show comment
Hide comment

sliver commented Mar 3, 2012

Is this help ?
symfony/symfony#2471

@Seldaek

This comment has been minimized.

Show comment
Hide comment
@Seldaek

Seldaek Mar 5, 2012

Member

@sliver yes it does, and I'm aware of that code, but haven't had time to look into this yet. Other things have higher priorities.

Member

Seldaek commented Mar 5, 2012

@sliver yes it does, and I'm aware of that code, but haven't had time to look into this yet. Other things have higher priorities.

@krymen

This comment has been minimized.

Show comment
Hide comment
@krymen

krymen Apr 26, 2012

Contributor

@Seldaek Any moves in this topic? I would use it in my project which is based on Symfony 2.0.12.

Contributor

krymen commented Apr 26, 2012

@Seldaek Any moves in this topic? I would use it in my project which is based on Symfony 2.0.12.

@Seldaek

This comment has been minimized.

Show comment
Hide comment
@Seldaek

Seldaek Apr 26, 2012

Member

Nope no news yet, still have other higher prio items to do.

Member

Seldaek commented Apr 26, 2012

Nope no news yet, still have other higher prio items to do.

@krymen

This comment has been minimized.

Show comment
Hide comment
@krymen

krymen Apr 26, 2012

Contributor

@Seldaek If you explain me how it might work I will try to do that.

Contributor

krymen commented Apr 26, 2012

@Seldaek If you explain me how it might work I will try to do that.

@Seldaek

This comment has been minimized.

Show comment
Hide comment
@Seldaek

Seldaek Apr 26, 2012

Member

@krymen The thing we need is a classmap dumper not an APC one, like #541 but a bit different (see my comments there). Then we can add an APC one later if needed, but as far as I know it's not faster.

Member

Seldaek commented Apr 26, 2012

@krymen The thing we need is a classmap dumper not an APC one, like #541 but a bit different (see my comments there). Then we can add an APC one later if needed, but as far as I know it's not faster.

@krymen

This comment has been minimized.

Show comment
Hide comment
@krymen

krymen Apr 26, 2012

Contributor

@Seldaek Thx. Yeah i was talking about classmap dumper. I will check your comments later and try to implement it.

Contributor

krymen commented Apr 26, 2012

@Seldaek Thx. Yeah i was talking about classmap dumper. I will check your comments later and try to implement it.

@aderuwe

This comment has been minimized.

Show comment
Hide comment
@aderuwe

aderuwe Apr 26, 2012

@krymen I started work on the classmap dumper in #541, but personal issues take up all my time since then... Glad to hear you might pick it up.

aderuwe commented Apr 26, 2012

@krymen I started work on the classmap dumper in #541, but personal issues take up all my time since then... Glad to hear you might pick it up.

@stof

This comment has been minimized.

Show comment
Hide comment
@stof

stof Apr 26, 2012

Contributor

@krymen You could copy the new ApcClassLoader available in Symfony 2.1 as it can be used to wrap a Composer autoloader (there is the equivalent using Xcache too)

Contributor

stof commented Apr 26, 2012

@krymen You could copy the new ApcClassLoader available in Symfony 2.1 as it can be used to wrap a Composer autoloader (there is the equivalent using Xcache too)

@krymen

This comment has been minimized.

Show comment
Hide comment
@krymen

krymen Apr 26, 2012

Contributor

@stof You are everywhere man;) Thx for advice.

Contributor

krymen commented Apr 26, 2012

@stof You are everywhere man;) Thx for advice.

@Seldaek Seldaek closed this in 48c46ce Aug 14, 2012

@gharlan gharlan referenced this issue in redaxo/redaxo Mar 17, 2013

Closed

Apc cache für rex-config + classloader #251

cordoval pushed a commit to cordoval/symfony-bootstrap that referenced this issue Nov 6, 2013

merged branch marcw/feat-apc-autoloading (PR #339)
Commits
-------

ac3ce44 Adds (commented by default) APC autoloading to autoload.php file.

Discussion
----------

Adds (commented by default) APC autoloading to autoload.php file.

I propose this patch because composer do not have a performant autoloading system yet (see composer/composer#127)

digitalkaoz pushed a commit to digitalkaoz/composer that referenced this issue Nov 22, 2013

@apinstein

This comment has been minimized.

Show comment
Hide comment
@apinstein

apinstein Jan 10, 2014

We switched from a sandboxed pear to composer a few months back, and there has been a huge hit on our performance. It's still below the "noticeable" threshold, but I have been thinking about looking into it since I have a lot of experience optimizing php require/autoloading esp w/r/t apc/opcode caches.

Graph: in early November we switched, and our "best-case" response time dropped by more than half.
httpresponsetime-year

I wanted to touch base and see if anyone's aware of a particular thread/issue that might be better suited to this topic before diving in.

We switched from a sandboxed pear to composer a few months back, and there has been a huge hit on our performance. It's still below the "noticeable" threshold, but I have been thinking about looking into it since I have a lot of experience optimizing php require/autoloading esp w/r/t apc/opcode caches.

Graph: in early November we switched, and our "best-case" response time dropped by more than half.
httpresponsetime-year

I wanted to touch base and see if anyone's aware of a particular thread/issue that might be better suited to this topic before diving in.

@Seldaek

This comment has been minimized.

Show comment
Hide comment
@Seldaek

Seldaek Jan 10, 2014

Member

@apinstein which autoloading are you using? Did you try the -o flag? I'd be curious to see the impact.

Member

Seldaek commented Jan 10, 2014

@apinstein which autoloading are you using? Did you try the -o flag? I'd be curious to see the impact.

@apinstein

This comment has been minimized.

Show comment
Hide comment
@apinstein

apinstein Jan 10, 2014

I ran with that flag and benchmarked before/after. No difference.

Then I ran a diff to see what actually changed… almost nothing…

Does the optimization flag require anything from the packages themselves to be amenable to optimization?

diff -ru vendor/composer/autoload_classmap.php vendor.optimized/composer/autoload_classmap.php
--- vendor/composer/autoload_classmap.php       2014-01-10 12:11:36.000000000 -0600
+++ vendor.optimized/composer/autoload_classmap.php     2014-01-10 12:09:06.000000000 -0600
@@ -271,6 +271,12 @@
     'ForeignKey' => $vendorDir . '/propel/propel1/generator/lib/model/ForeignKey.php',
     'FormatterElement' => $vendorDir . '/phing/phing/classes/phing/tasks/ext/phpunit/FormatterElement.php',
     'FtpDeployTask' => $vendorDir . '/phing/phing/classes/phing/tasks/ext/FtpDeployTask.php',
+    'FuelPHP\\Event\\Container' => $vendorDir . '/fuelphp/event/src/FuelPHP/Event/Container.php',
+    'FuelPHP\\Event\\Facade\\Base' => $vendorDir . '/fuelphp/event/src/FuelPHP/Event/Facade/Base.php',
+    'FuelPHP\\Event\\Facade\\Event' => $vendorDir . '/fuelphp/event/src/FuelPHP/Event/Facade/Event.php',
+    'FuelPHP\\Event\\Facade\\Queue' => $vendorDir . '/fuelphp/event/src/FuelPHP/Event/Facade/Queue.php',
+    'FuelPHP\\Event\\Listener' => $vendorDir . '/fuelphp/event/src/FuelPHP/Event/Listener.php',
+    'FuelPHP\\Event\\Queue' => $vendorDir . '/fuelphp/event/src/FuelPHP/Event/Queue.php',
     'FunctionParam' => $vendorDir . '/phing/phing/classes/phing/tasks/system/PhpEvalTask.php',
     'GeneratorConfig' => $vendorDir . '/propel/propel1/generator/lib/config/GeneratorConfig.php',
     'GeneratorConfigInterface' => $vendorDir . '/propel/propel1/generator/lib/config/GeneratorConfigInterface.php',
@@ -1048,6 +1054,16 @@
     'SvnSwitchTask' => $vendorDir . '/phing/phing/classes/phing/tasks/ext/svn/SvnSwitchTask.php',
     'SvnUpdateTask' => $vendorDir . '/phing/phing/classes/phing/tasks/ext/svn/SvnUpdateTask.php',
     'SymfonyConsoleTask' => $vendorDir . '/phing/phing/classes/phing/tasks/ext/SymfonyConsole/SymfonyConsoleTask.php',
+    'Symfony\\Component\\Yaml\\Dumper' => $vendorDir . '/symfony/yaml/Symfony/Component/Yaml/Dumper.php',
+    'Symfony\\Component\\Yaml\\Escaper' => $vendorDir . '/symfony/yaml/Symfony/Component/Yaml/Escaper.php',
+    'Symfony\\Component\\Yaml\\Exception\\DumpException' => $vendorDir . '/symfony/yaml/Symfony/Component/Yaml/Exception/DumpException.php',
+    'Symfony\\Component\\Yaml\\Exception\\ExceptionInterface' => $vendorDir . '/symfony/yaml/Symfony/Component/Yaml/Exception/ExceptionInterface.php',
+    'Symfony\\Component\\Yaml\\Exception\\ParseException' => $vendorDir . '/symfony/yaml/Symfony/Component/Yaml/Exception/ParseException.php',
+    'Symfony\\Component\\Yaml\\Exception\\RuntimeException' => $vendorDir . '/symfony/yaml/Symfony/Component/Yaml/Exception/RuntimeException.php',
+    'Symfony\\Component\\Yaml\\Inline' => $vendorDir . '/symfony/yaml/Symfony/Component/Yaml/Inline.php',
+    'Symfony\\Component\\Yaml\\Parser' => $vendorDir . '/symfony/yaml/Symfony/Component/Yaml/Parser.php',
+    'Symfony\\Component\\Yaml\\Unescaper' => $vendorDir . '/symfony/yaml/Symfony/Component/Yaml/Unescaper.php',
+    'Symfony\\Component\\Yaml\\Yaml' => $vendorDir . '/symfony/yaml/Symfony/Component/Yaml/Yaml.php',
     'SymlinkTask' => $vendorDir . '/phing/phing/classes/phing/tasks/ext/SymlinkTask.php',
     'TabToSpaces' => $vendorDir . '/phing/phing/classes/phing/filters/TabToSpaces.php',
     'Table' => $vendorDir . '/propel/propel1/generator/lib/model/Table.php',
diff -ru vendor/composer/autoload_real.php vendor.optimized/composer/autoload_real.php
--- vendor/composer/autoload_real.php   2014-01-10 12:11:36.000000000 -0600
+++ vendor.optimized/composer/autoload_real.php 2014-01-10 12:09:06.000000000 -0600
@@ -2,7 +2,7 @@

 // autoload_real.php @generated by Composer

-class ComposerAutoloaderInit1ba13e1f6158f5dd202dc2c4caf54ba3
+class ComposerAutoloaderInit52bab1cff3cdbd4aba44fa910f880f48
 {
     private static $loader;

@@ -19,9 +19,9 @@
             return self::$loader;
         }

-        spl_autoload_register(array('ComposerAutoloaderInit1ba13e1f6158f5dd202dc2c4caf54ba3', 'loadClassLoader'), true, true);
+        spl_autoload_register(array('ComposerAutoloaderInit52bab1cff3cdbd4aba44fa910f880f48', 'loadClassLoader'), true, true);
         self::$loader = $loader = new \Composer\Autoload\ClassLoader();
-        spl_autoload_unregister(array('ComposerAutoloaderInit1ba13e1f6158f5dd202dc2c4caf54ba3', 'loadClassLoader'));
+        spl_autoload_unregister(array('ComposerAutoloaderInit52bab1cff3cdbd4aba44fa910f880f48', 'loadClassLoader'));

         $vendorDir = dirname(__DIR__);
         $baseDir = dirname($vendorDir);

I ran with that flag and benchmarked before/after. No difference.

Then I ran a diff to see what actually changed… almost nothing…

Does the optimization flag require anything from the packages themselves to be amenable to optimization?

diff -ru vendor/composer/autoload_classmap.php vendor.optimized/composer/autoload_classmap.php
--- vendor/composer/autoload_classmap.php       2014-01-10 12:11:36.000000000 -0600
+++ vendor.optimized/composer/autoload_classmap.php     2014-01-10 12:09:06.000000000 -0600
@@ -271,6 +271,12 @@
     'ForeignKey' => $vendorDir . '/propel/propel1/generator/lib/model/ForeignKey.php',
     'FormatterElement' => $vendorDir . '/phing/phing/classes/phing/tasks/ext/phpunit/FormatterElement.php',
     'FtpDeployTask' => $vendorDir . '/phing/phing/classes/phing/tasks/ext/FtpDeployTask.php',
+    'FuelPHP\\Event\\Container' => $vendorDir . '/fuelphp/event/src/FuelPHP/Event/Container.php',
+    'FuelPHP\\Event\\Facade\\Base' => $vendorDir . '/fuelphp/event/src/FuelPHP/Event/Facade/Base.php',
+    'FuelPHP\\Event\\Facade\\Event' => $vendorDir . '/fuelphp/event/src/FuelPHP/Event/Facade/Event.php',
+    'FuelPHP\\Event\\Facade\\Queue' => $vendorDir . '/fuelphp/event/src/FuelPHP/Event/Facade/Queue.php',
+    'FuelPHP\\Event\\Listener' => $vendorDir . '/fuelphp/event/src/FuelPHP/Event/Listener.php',
+    'FuelPHP\\Event\\Queue' => $vendorDir . '/fuelphp/event/src/FuelPHP/Event/Queue.php',
     'FunctionParam' => $vendorDir . '/phing/phing/classes/phing/tasks/system/PhpEvalTask.php',
     'GeneratorConfig' => $vendorDir . '/propel/propel1/generator/lib/config/GeneratorConfig.php',
     'GeneratorConfigInterface' => $vendorDir . '/propel/propel1/generator/lib/config/GeneratorConfigInterface.php',
@@ -1048,6 +1054,16 @@
     'SvnSwitchTask' => $vendorDir . '/phing/phing/classes/phing/tasks/ext/svn/SvnSwitchTask.php',
     'SvnUpdateTask' => $vendorDir . '/phing/phing/classes/phing/tasks/ext/svn/SvnUpdateTask.php',
     'SymfonyConsoleTask' => $vendorDir . '/phing/phing/classes/phing/tasks/ext/SymfonyConsole/SymfonyConsoleTask.php',
+    'Symfony\\Component\\Yaml\\Dumper' => $vendorDir . '/symfony/yaml/Symfony/Component/Yaml/Dumper.php',
+    'Symfony\\Component\\Yaml\\Escaper' => $vendorDir . '/symfony/yaml/Symfony/Component/Yaml/Escaper.php',
+    'Symfony\\Component\\Yaml\\Exception\\DumpException' => $vendorDir . '/symfony/yaml/Symfony/Component/Yaml/Exception/DumpException.php',
+    'Symfony\\Component\\Yaml\\Exception\\ExceptionInterface' => $vendorDir . '/symfony/yaml/Symfony/Component/Yaml/Exception/ExceptionInterface.php',
+    'Symfony\\Component\\Yaml\\Exception\\ParseException' => $vendorDir . '/symfony/yaml/Symfony/Component/Yaml/Exception/ParseException.php',
+    'Symfony\\Component\\Yaml\\Exception\\RuntimeException' => $vendorDir . '/symfony/yaml/Symfony/Component/Yaml/Exception/RuntimeException.php',
+    'Symfony\\Component\\Yaml\\Inline' => $vendorDir . '/symfony/yaml/Symfony/Component/Yaml/Inline.php',
+    'Symfony\\Component\\Yaml\\Parser' => $vendorDir . '/symfony/yaml/Symfony/Component/Yaml/Parser.php',
+    'Symfony\\Component\\Yaml\\Unescaper' => $vendorDir . '/symfony/yaml/Symfony/Component/Yaml/Unescaper.php',
+    'Symfony\\Component\\Yaml\\Yaml' => $vendorDir . '/symfony/yaml/Symfony/Component/Yaml/Yaml.php',
     'SymlinkTask' => $vendorDir . '/phing/phing/classes/phing/tasks/ext/SymlinkTask.php',
     'TabToSpaces' => $vendorDir . '/phing/phing/classes/phing/filters/TabToSpaces.php',
     'Table' => $vendorDir . '/propel/propel1/generator/lib/model/Table.php',
diff -ru vendor/composer/autoload_real.php vendor.optimized/composer/autoload_real.php
--- vendor/composer/autoload_real.php   2014-01-10 12:11:36.000000000 -0600
+++ vendor.optimized/composer/autoload_real.php 2014-01-10 12:09:06.000000000 -0600
@@ -2,7 +2,7 @@

 // autoload_real.php @generated by Composer

-class ComposerAutoloaderInit1ba13e1f6158f5dd202dc2c4caf54ba3
+class ComposerAutoloaderInit52bab1cff3cdbd4aba44fa910f880f48
 {
     private static $loader;

@@ -19,9 +19,9 @@
             return self::$loader;
         }

-        spl_autoload_register(array('ComposerAutoloaderInit1ba13e1f6158f5dd202dc2c4caf54ba3', 'loadClassLoader'), true, true);
+        spl_autoload_register(array('ComposerAutoloaderInit52bab1cff3cdbd4aba44fa910f880f48', 'loadClassLoader'), true, true);
         self::$loader = $loader = new \Composer\Autoload\ClassLoader();
-        spl_autoload_unregister(array('ComposerAutoloaderInit1ba13e1f6158f5dd202dc2c4caf54ba3', 'loadClassLoader'));
+        spl_autoload_unregister(array('ComposerAutoloaderInit52bab1cff3cdbd4aba44fa910f880f48', 'loadClassLoader'));

         $vendorDir = dirname(__DIR__);
         $baseDir = dirname($vendorDir);
@stof

This comment has been minimized.

Show comment
Hide comment
@stof

stof Jan 10, 2014

Contributor

well, it looks like most of your code uses the classmap autoloading rather than PSR-0, so the optimization of the PSR autoloading will indeed not change much for you

Contributor

stof commented Jan 10, 2014

well, it looks like most of your code uses the classmap autoloading rather than PSR-0, so the optimization of the PSR autoloading will indeed not change much for you

@apinstein

This comment has been minimized.

Show comment
Hide comment
@apinstein

apinstein Jan 10, 2014

Ah ok… well so then my original question probably remains.. if I want to work on addressing this, is there a better issue/thread I should use? Or is this one fine?

On Jan 10, 2014, at 1:25 PM, Christophe Coevoet notifications@github.com wrote:

well, it looks like most of your code uses the classmap autoloading rather than PSR-0, so the optimization of the PSR autoloading will indeed not change much for you


Reply to this email directly or view it on GitHub.

Ah ok… well so then my original question probably remains.. if I want to work on addressing this, is there a better issue/thread I should use? Or is this one fine?

On Jan 10, 2014, at 1:25 PM, Christophe Coevoet notifications@github.com wrote:

well, it looks like most of your code uses the classmap autoloading rather than PSR-0, so the optimization of the PSR autoloading will indeed not change much for you


Reply to this email directly or view it on GitHub.

@stof

This comment has been minimized.

Show comment
Hide comment
@stof

stof Jan 10, 2014

Contributor

create a new issue for it

Contributor

stof commented Jan 10, 2014

create a new issue for it

mbence pushed a commit to mbence/csaladsegito that referenced this issue Feb 3, 2014

merged branch marcw/feat-apc-autoloading (PR #339)
Commits
-------

ac3ce44 Adds (commented by default) APC autoloading to autoload.php file.

Discussion
----------

Adds (commented by default) APC autoloading to autoload.php file.

I propose this patch because composer do not have a performant autoloading system yet (see composer/composer#127)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment