Skip to content

Commit

Permalink
Updated readme and switched benchmarks to prefer CLI
Browse files Browse the repository at this point in the history
  • Loading branch information
@TRPB
TRPB committed Aug 12, 2015
1 parent 59cc201 commit 3999761
Show file tree
Hide file tree
Showing 5 changed files with 124 additions and 94 deletions.
52 changes: 50 additions & 2 deletions README.md
View file
Expand Up @@ -5,10 +5,58 @@ Containers currently tested are:
- Auryn
- Dice
- Laravel


- Leage
- Njasm
- Phalcon
- PHP-DI
- Pimple
- Symfony DI
- Zend\Di
- Zend\Servicemanager


**Nette is currently disabled due to not working as advertised. It fails to load Nette\DI\ContainerLoader which prevents any tests being done**

**Ray.Di does not seem to work without annotations. Requiring the classes to be coupled to the container goes against the spirit of DI and as such is not tested here**

**To use phalcon you must have the phalcon.so installed. You may need to edit php-phalcon.ini and adjust the extension dir/path to phalcon.so**

# Usage

1a) Clone this repo and Run `composer install`
or
1b) Install via composer

3) Run `php test1-5_runner.php` and `php test6_runner.php` (These will take some time!)
4) Open the files `test1-5_results.html` and `test6_results.html`


# Tests

## Test 1

Test 1 tests the total time for the containers to construct a single object repeatedly, including the time it takes for the containers to autoload their files.

## Test 2

Test 2 is the same as test one, only autoload time is not included. If there is a big difference between the results for test 1 and test 2 for a container, the container is loading a lot of files

## Test 3

Creation of a deep object graph. This test measures the time it takes the container to construct a 10 level deep object graph repeatedly. This is the longest test and the truest indicator of raw performance.

## Test 4

This is a test of how quickly the same object (service) can be repeatedly requested from the container

## Test 5

Test 5 repeatedly has the container construct and object and inject a service

## Test 6

*Not all containers are currently set up for this test.*

This test is the most useful for the faster containers, it measures scalability by measuring the entire script execution time for the PHP process to launch, construct/configure the container and thenhave the container construct a specified number of objects. Fast containers with a slow startup time will score worse with fewer objects but improve in the rankings as the number of objects is increased. Slower containers with fast startup times will rank highly with fewer objects but will lose out to faster containers once the number of objects gets high enough.


1 change: 1 addition & 0 deletions test1-5_results.html
View file
@@ -0,0 +1 @@
<style>td,th {padding: 5px; border: 1px solid #aaa; text-align: right;}</style><h2>Test 1 - Create single object (incl autoload time)</h2><table><thead><tr><th>Container</th><th>Time</th><th>Memory</th><th>Files</th></thead><tr><td>aura</td><td>0.0518</td><td>0.4496</td><td>16</td></tr><tr><td>auryn</td><td>0.0584</td><td>0.5103</td><td>15</td></tr><tr><td>dice</td><td>0.0075</td><td>0.4178</td><td>10</td></tr><tr><td>laravel</td><td>0.0713</td><td>0.5142</td><td>11</td></tr><tr><td>league</td><td>0.0388</td><td>0.4533</td><td>20</td></tr><tr><td>njasm</td><td>0.1727</td><td>0.4843</td><td>25</td></tr><tr><td>phalcon</td><td>0.0119</td><td>0.2546</td><td>0</td></tr><tr><td>php-di</td><td>0.1272</td><td>0.569</td><td>41</td></tr><tr><td>pimple</td><td>0.0226</td><td>0.4035</td><td>10</td></tr><tr><td>symfonydi</td><td>0.0232</td><td>0.4612</td><td>13</td></tr><tr><td>zend-di</td><td>0.2058</td><td>0.71</td><td>33</td></tr><tr><td>zend-servicemanager</td><td>0.075</td><td>0.5842</td><td>14</td></tr></table><h2>Test 2 - Create single object (excl autoload time)</h2><table><thead><tr><th>Container</th><th>Time</th><th>Memory</th><th>Files</th></thead><tr><td>aura</td><td>0.0509</td><td>0.4493</td><td>16</td></tr><tr><td>auryn</td><td>0.058</td><td>0.5108</td><td>15</td></tr><tr><td>dice</td><td>0.0077</td><td>0.4178</td><td>10</td></tr><tr><td>laravel</td><td>0.0724</td><td>0.5143</td><td>11</td></tr><tr><td>league</td><td>0.0359</td><td>0.4532</td><td>20</td></tr><tr><td>njasm</td><td>0.1712</td><td>0.4839</td><td>25</td></tr><tr><td>phalcon</td><td>0.0115</td><td>0.2546</td><td>0</td></tr><tr><td>php-di</td><td>0.1245</td><td>0.5676</td><td>41</td></tr><tr><td>pimple</td><td>0.0227</td><td>0.4034</td><td>10</td></tr><tr><td>symfonydi</td><td>0.0228</td><td>0.4614</td><td>13</td></tr><tr><td>zend-di</td><td>0.2054</td><td>0.7098</td><td>33</td></tr><tr><td>zend-servicemanager</td><td>0.0758</td><td>0.5843</td><td>14</td></tr></table><h2>Test 3 - Create deep object graph</h2><table><thead><tr><th>Container</th><th>Time</th><th>Memory</th><th>Files</th></thead><tr><td>aura</td><td>0.5955</td><td>0.4515</td><td>16</td></tr><tr><td>auryn</td><td>1.0879</td><td>0.5108</td><td>15</td></tr><tr><td>dice</td><td>0.1954</td><td>0.4178</td><td>10</td></tr><tr><td>laravel</td><td>1.1432</td><td>0.5143</td><td>11</td></tr><tr><td>league</td><td>0.4666</td><td>0.4532</td><td>20</td></tr><tr><td>njasm</td><td>1.6928</td><td>0.5174</td><td>25</td></tr><tr><td>phalcon</td><td>0.139</td><td>0.2809</td><td>0</td></tr><tr><td>php-di</td><td>2.5765</td><td>0.5884</td><td>46</td></tr><tr><td>pimple</td><td>0.4314</td><td>0.4239</td><td>10</td></tr><tr><td>symfonydi</td><td>0.0535</td><td>0.4649</td><td>13</td></tr><tr><td>zend-di</td><td>8.6317</td><td>0.7105</td><td>34</td></tr><tr><td>zend-servicemanager</td><td>0.7308</td><td>0.5915</td><td>14</td></tr></table><h2>Test 4 - Fetch the same instance from the container repeatedly</h2><table><thead><tr><th>Container</th><th>Time</th><th>Memory</th><th>Files</th></thead><tr><td>aura</td><td>0.0037</td><td>0.4498</td><td>16</td></tr><tr><td>auryn</td><td>0.0169</td><td>0.511</td><td>15</td></tr><tr><td>dice</td><td>0.0037</td><td>0.4178</td><td>10</td></tr><tr><td>laravel</td><td>0.0046</td><td>0.5143</td><td>11</td></tr><tr><td>league</td><td>0.0071</td><td>0.4532</td><td>20</td></tr><tr><td>njasm</td><td>0.1761</td><td>0.4957</td><td>26</td></tr><tr><td>phalcon</td><td>0.0023</td><td>0.2556</td><td>0</td></tr><tr><td>php-di</td><td>0.0045</td><td>0.5387</td><td>36</td></tr><tr><td>pimple</td><td>0.0035</td><td>0.4035</td><td>10</td></tr><tr><td>symfonydi</td><td>0.0131</td><td>0.4614</td><td>13</td></tr><tr><td>zend-di</td><td>0.0595</td><td>0.7102</td><td>33</td></tr><tr><td>zend-servicemanager</td><td>0.0091</td><td>0.5843</td><td>14</td></tr></table><h2>Test 5 - Inject a shared instance into a new object repeatedly</h2><table><thead><tr><th>Container</th><th>Time</th><th>Memory</th><th>Files</th></thead><tr><td>aura</td><td>0.0625</td><td>0.4495</td><td>16</td></tr><tr><td>auryn</td><td>0.1473</td><td>0.511</td><td>15</td></tr><tr><td>dice</td><td>0.028</td><td>0.4178</td><td>10</td></tr><tr><td>laravel</td><td>0.1355</td><td>0.5143</td><td>11</td></tr><tr><td>league</td><td>0.0616</td><td>0.4533</td><td>20</td></tr><tr><td>njasm</td><td>0.3523</td><td>0.5051</td><td>27</td></tr><tr><td>phalcon</td><td>0.0188</td><td>0.2577</td><td>0</td></tr><tr><td>php-di</td><td>0.2479</td><td>0.5848</td><td>46</td></tr><tr><td>pimple</td><td>0.0299</td><td>0.4044</td><td>10</td></tr><tr><td>symfonydi</td><td>0.0334</td><td>0.4623</td><td>13</td></tr><tr><td>zend-di</td><td>0.508</td><td>0.7087</td><td>34</td></tr><tr><td>zend-servicemanager</td><td>0.0929</td><td>0.5843</td><td>14</td></tr></table>
106 changes: 34 additions & 72 deletions test1-5_runner.php
View file
Expand Up @@ -2,12 +2,26 @@
//You shouldn't have max_exectution_time set high enough to run these benchmarks
ini_set('max_execution_time', 90000);
opcache_reset();
$isCli = php_sapi_name() == 'cli';
function cliPrint($text, $newLine = true) {
$isCli = php_sapi_name() == 'cli';
if ($isCli) {
echo $text;
if ($newLine) echo "\n";
}
}

cliPrint('Starting benchmarks');

$html = '';

//Number of times to run each test before taking an average

$runs = 10;
cliPrint('Running each test ' . $runs . ' times');

//Containers to be tested (dir names)
$containers = ['zend-servicemanager', 'dice'];
$containers = ['aura', 'auryn', 'dice', 'laravel', 'league', 'njasm', 'phalcon', 'php-di', 'pimple', 'symfonydi', 'zend-di', 'zend-servicemanager'];


//Default ini file to use for tests
Expand All @@ -21,7 +35,7 @@

//The number of tests
$numTests = 5;

cliPrint('Running tests 1 - ' . $numTests);

function average($array, $dp = 4) {
sort($array, SORT_NUMERIC);
Expand Down Expand Up @@ -51,7 +65,7 @@ function runScript($file, $iniFile, $args = []) {


//Some very basic styling
echo '<style>td,th {padding: 5px; border: 1px solid #aaa; text-align: right;}</style>';
$html .= '<style>td,th {padding: 5px; border: 1px solid #aaa; text-align: right;}</style>';

$testdescriptions = [1 => 'Create single object (incl autoload time)',
2 => 'Create single object (excl autoload time)',
Expand All @@ -61,20 +75,22 @@ function runScript($file, $iniFile, $args = []) {
];

for ($test = 1; $test <= $numTests; $test++) {
echo '<h2>Test ' . $test . ' - ' . $testdescriptions[$test] . '</h2>';
echo '<table>';

echo '<thead><tr><th>Container</th><th>Time</th><th>Memory</th><th>Files</th></thead>';
$html .= '<h2>Test ' . $test . ' - ' . $testdescriptions[$test] . '</h2>';
$html .= '<table>';
cliPrint('Starting test:' . $test);

$html .= '<thead><tr><th>Container</th><th>Time</th><th>Memory</th><th>Files</th></thead>';

foreach ($containers as $container) {
//chdir($cwd);
//chdir('./' . $container);
cliPrint('');
cliPrint('Benchmarking container:' . $container);
$memory = [];
$time = [];
$files = [];
$output = [];

for ($i = 0; $i < $runs; $i++) {
cliPrint($container . ' test' . $test . ' : ' . $i . '/' . $runs);
$output = runScript('./' . $container . '/test' . $test . '.php', $inis[$container]);
$result = json_decode($output[0]);
if (!is_object($result)) echo $container . $test . '<br />';
Expand All @@ -84,73 +100,19 @@ function runScript($file, $iniFile, $args = []) {
}

//average memory and file count is pointless here, but it's included for completenesss
echo '<tr>';
echo '<td>' . $container .'</td>';
echo '<td>' . average($time) . '</td>';
echo '<td>' . average($memory) . '</td>';
echo '<td>' . (average($files)-2) . '</td>';
echo '</tr>';
$html .= '<tr>';
$html .= '<td>' . $container .'</td>';
$html .= '<td>' . average($time) . '</td>';
$html .= '<td>' . average($memory) . '</td>';
$html .= '<td>' . (average($files)-2) . '</td>';
$html .= '</tr>';

flush();
}


echo '</table>';
flush();
}


die;
echo '<h2>Test 6</h2>';
$html .= '</table>';

//Calculate the average overhead of running $test6Iterations php scripts with a specified ini file
$blankScript = 'blank.php';

$overheads = [];
chdir($cwd);
for ($i = 0; $i < $runs; $i++) {
$t1 = microtime(true);

for ($i = 0; $i < $test6Iterations; $i++) {
runScript($blankScript, $defaultIni);
}
$t2 = microtime(true);
$overheads[] = $t2 - $t1;
}


$overhead = average($overheads);

echo 'Overhead time: ' . $overhead;

echo '<table>';

echo '<thead><tr><th>Container</th><th>Time</th><th>Memory</th><th>Files</th></thead>';


foreach ($containers as $container) {
chdir($cwd);
chdir('./' . $container);

$results = [];
for ($i = 0; $i < $runs; $i++) {
$t1 = microtime(true);
for ($i = 0; $i < $test6Iterations; $i++) {
$output = runScript('./test6a.php', $inis[$container], [1]);
}
$t2 = microtime(true);

$result = json_decode($output[0]);
$results[] = $t2 - $t1;
}

$result = average($results);

echo '<tr>';
echo '<td>' . $container .'</td>';
echo '<td>' . ($result - $overhead) . '</td>';
echo '<td>' . average([$result['memory']]) . '</td>';
echo '<td>' . average([$result['files']]) . '</td>';
echo '</tr>';

}
if (!$isCli) echo $html;
else file_put_contents('test1-5_results.html', $html);
4 changes: 4 additions & 0 deletions test6_results.html
View file
@@ -0,0 +1,4 @@
<h2>Test 6 - Scalability</h2><p>This test measures the entire script execution time for the PHP process to launch, construct/configure the container and then
have the container construct a specified number of objects. Fast containers with a slow startup time will score worse with fewer objects but improve in the rankings
as the number of objects is increased. Slower containers with fast startup times will rank highly with fewer objects but will lose out to faster containers once the number of objects gets high enough</p>
Overhead time: 3.495<table><thead><tr><th>Container</th><th>10 objects</th><th>50 objects</th><th>100 objects</th><th>200 objects</th><th>500 objects</th><th>1000 objects</th><th>1500 objects</th></thead><tr><td>pimple</td><td>0.1767</td><td>0.1982</td><td>0.2589</td><td>0.3281</td><td>0.5135</td><td>0.8565</td><td>1.2091</td></tr><tr><td>phalcon</td><td>1.0103</td><td>0.9813</td><td>1.0059</td><td>1.0058</td><td>0.9699</td><td>0.9867</td><td>0.9583</td></tr><tr><td>dice</td><td>0.1571</td><td>0.2241</td><td>0.2817</td><td>0.2907</td><td>0.4508</td><td>0.7915</td><td>1.0386</td></tr><tr><td>symfonydi</td><td>0.4626</td><td>0.4771</td><td>0.4728</td><td>0.5013</td><td>0.5943</td><td>0.6568</td><td>0.7271</td></tr>
55 changes: 35 additions & 20 deletions test6_runner.php
View file
Expand Up @@ -2,24 +2,32 @@
//You shouldn't have max_exectution_time set high enough to run these benchmarks
ini_set('max_execution_time', 90000);
opcache_reset();
$isCli = php_sapi_name() == 'cli';
function cliPrint($text, $newLine = true) {
$isCli = php_sapi_name() == 'cli';
if ($isCli) {
echo $text;
if ($newLine) echo "\n";
}
}

cliPrint('Starting benchmarks');

$html = '';


//Increasing these numbers will inprove accuracy but increase the time this script takes to run.

//Number of times to run each test before taking an average, the higher the better
$runs = 6;
$runs = 10;

//Number of iterations (HTTP requests to mimic) in each test. This will make each test longer
//use 100 or more for a more accurate result
$iterations = 500;
$iterations = 250;


//Containers to be tested (dir names)
//$containers = ['dice','pimple', 'symfonydi'];
//$containers = ['pimple', 'dice', 'symfonydi'];
$containers = ['pimple', 'dice', 'symfonydi'];
//$containers = ['aura', 'dice', 'nette', 'orno', 'phalcon', 'php-di', 'pimple', 'symfonydi'];

$containers = ['pimple', 'phalcon', 'dice', 'symfonydi'];

//Default ini file to use for tests
$defaultIni = getcwd() . DIRECTORY_SEPARATOR . 'php.ini';
Expand Down Expand Up @@ -58,9 +66,9 @@ function runScript($file, $iniFile, $args = []) {
}


echo '<h2>Test 6 - Scalability</h2>';
$html .= '<h2>Test 6 - Scalability</h2>';

echo '<p>This test measures the entire script execution time for the PHP process to launch, construct/configure the container and then
$html .= '<p>This test measures the entire script execution time for the PHP process to launch, construct/configure the container and then
have the container construct a specified number of objects. Fast containers with a slow startup time will score worse with fewer objects but improve in the rankings
as the number of objects is increased. Slower containers with fast startup times will rank highly with fewer objects but will lose out to faster containers once the number of objects gets high enough</p>
';
Expand All @@ -82,46 +90,53 @@ function runScript($file, $iniFile, $args = []) {

$overhead = average($overheads);

echo 'Overhead time: ' . $overhead;
$html .= 'Overhead time: ' . $overhead;

echo '<table>';
$html .= '<table>';

echo '<thead><tr><th>Container</th>';
$html .= '<thead><tr><th>Container</th>';


//The number of J objects on each iteration for the tests. Each J object consists of 10 objects in total
$objects = [1, 5, 10, 20, 50, 100, 150];


foreach ($objects as $object) {
echo '<th>' . $object *10 . ' objects</th>';
$html .= '<th>' . $object *10 . ' objects</th>';
}

echo '</thead>';
$html .= '</thead>';

cliPrint('Starting benchmark');

foreach ($containers as $container) {
cliPrint('Benchmarking container: ' . $container);

echo '<tr>';
echo '<td>' . $container .'</td>';
$html .= '<tr>';
$html .= '<td>' . $container .'</td>';

foreach ($objects as $objectcount) {
$results = [];
for ($i = 0; $i < $runs; $i++) {
cliPrint('Benchmarking container: ' . $container . ' with ' . $objectcount*10 . ' objects run ' . $i . '/' . $runs);
$t1 = microtime(true);
for ($i = 0; $i < $iterations; $i++) {
for ($j = 0; $j < $iterations; $j++) {
$output = runScript($container . '/test6a.php', $inis[$container], [$objectcount]);
}
$t2 = microtime(true);

$test = json_decode($output[0]);
if (!is_object($test)) print_r($output);
$results[] = $t2 - $t1;
}
}
$result = average($results);

echo '<td>' . ($result - $overhead) . '</td>';
$html .= '<td>' . ($result - $overhead) . '</td>';
}

echo '</tr>';
$html .= '</tr>';

}

if (!$isCli) echo $html;
else file_put_contents('test6_results.html', $html);

0 comments on commit 3999761

Please sign in to comment.