Add Pux pure PHP to the benchmark information #17

Closed
rk opened this Issue Jan 14, 2014 · 21 comments

Comments

Projects
None yet
5 participants

rk commented Jan 14, 2014

I hate to say it, but comparing a C extension to a pure PHP implementation is like comparing a racehorse against a donkey. Especially for something like this. So saying the extension is faster than the Symphony router isn't saying much.

Personally, I'd like to know where the pure PHP implementation of Pux stands in relation to the extension, as well as other libraries.

Owner

c9s commented Jan 14, 2014

The benchmark is just to show Pux with C extension can reach the goal.

It's fine if you say "comparing a C extension to a pure PHP implementation is like comparing a racehorse against a donkey" while you are comparing language.

but the benchmark is not comparing programming languages or PHP programming techniques, it's comparing PHP routers.

The dispatching speed in Pux without C extension is already faster than symfony/router. so I think there is no much need to include the pure php benchmark.

and actually while pux extension provides the the same class name, I can't compare them in the same benchmark test script.

rk commented Jan 14, 2014

I do understand the challenge of testing the pure php implementation against the extension. What I'm concerned is that someone is just going to point out that it's an unfair comparison (extension vs native PHP). Plus, I want to know if Pux w/o the compiled extension can beat the other router implementations.

And no, the benchmark is comparing a C extension against a native PHP library. There's a night and day difference. The C extension starts as compiled machine code without the overhead of being interpreted, and the PHP library has to be interpreted. It's inaccurate until you compare Pux against Symfony on the same terms.

Owner

c9s commented Jan 14, 2014

Maybe I can make a benchmark for pure php implementations in https://github.com/c9s/router-benchmark maybe this weekend.

Owner

c9s commented Jan 14, 2014

alright I just tested the pure PHP benchmark it's here, this uses the same test code in the https://github.com/c9s/router-benchmark benchmark test suite. all implementations are in pure php.

php dispatch.php
n=10000
Runing php array - . 58366.171782266/s
Runing pux - . 34159.074953945/s
Runing klein - . 741.86689636814/s
Runing ham - . 10384.390179169/s
Runing aura - . 21274.46843045/s
Runing symfony/routing - . 900.48034589679/s
                     Rate   Mem php array pux aura ham symfony/routing klein
      php array  58.37K/s    0B        ---58% -36%-17%             -1%   -1%
            pux  34.16K/s    0B      170%  -- -62%-30%             -2%   -2%
           aura  21.27K/s    0B      274%160%   ---48%             -4%   -3%
            ham  10.38K/s    0B      562%328% 204%  --             -8%   -7%
symfony/routing  900.48/s  786K     6481%3793%2362%1153%              --  -82%
          klein  741.87/s  262K     7867%4604%2867%1399%            121%    --
================================== Bar Chart ==================================
        php array  58.37K/s | ████████████████████████████████████████████████████████████  |
              pux  34.16K/s | ███████████████████████████████████                           |
             aura  21.27K/s | █████████████████████                                         |
              ham  10.38K/s | ██████████                                                    |
  symfony/routing  900.48/s |                                                               |
            klein  741.87/s |                                                               |
============================== System Information ==============================
PHP Version: 5.5.6
CPU Brand String: Intel(R) Core(TM) i5-3427U CPU @ 1.80GHz

rk commented Jan 14, 2014

Excellent! Thanks Pedro, I was genuinely curious if Pux could beat Aura's implementation. A 2x improvement with the extension is actually smaller than I'd expected. Good work.

On Jan 14, 2014, at 10:46 AM, Pedro notifications@github.com wrote:

alright I just tested the pure PHP benchmark it's here, this uses the same test code in the https://github.com/c9s/router-benchmark benchmark test suite.

php dispatch.php
n=10000
Runing php array - . 58366.171782266/s
Runing pux - . 34159.074953945/s
Runing klein - . 741.86689636814/s
Runing ham - . 10384.390179169/s
Runing aura - . 21274.46843045/s
Runing symfony/routing - . 900.48034589679/s

                 Rate   Mem php array pux aura ham symfony/routing klein
  php array  58.37K/s    0B        ---58% -36%-17%             -1%   -1%
        pux  34.16K/s    0B      170%  -- -62%-30%             -2%   -2%
       aura  21.27K/s    0B      274%160%   ---48%             -4%   -3%
        ham  10.38K/s    0B      562%328% 204%  --             -8%   -7%

symfony/routing 900.48/s 786K 6481%3793%2362%1153% -- -82%
klein 741.87/s 262K 7867%4604%2867%1399% 121% --

================================== Bar Chart ==================================

    php array  58.37K/s | ████████████████████████████████████████████████████████████  |
          pux  34.16K/s | ███████████████████████████████████                           |
         aura  21.27K/s | █████████████████████                                         |
          ham  10.38K/s | ██████████                                                    |

symfony/routing 900.48/s | |
klein 741.87/s | |

============================== System Information ==============================

PHP Version: 5.5.6
CPU Brand String: Intel(R) Core(TM) i5-3427U CPU @ 1.80GHz

Reply to this email directly or view it on GitHub.

Wow. Look at Aura go! Nice work @pmjones and @harikt!

Owner

c9s commented Jan 14, 2014

Yeah if you focus on the dispatching speed, the extension can gain about 1.5x-2x improvement. actually the extension is to reduce the overhead for requests from apache2 or nginx.

harikt commented Jan 14, 2014

👍 nice .

pmjones commented Jan 14, 2014

Damn, I didn't know we were in the running. That's sure nice to see.

Having said that, I have to caution against reading too much into this. The routing system is probably not a bottleneck for most apps, and even the fastest routing system is not likely to have a huge effect on the end-to-end performance of an app. But even so, it's nice to have something to compare to. :-)

Maybe I'm just slow, but I'm unable to run the benchmark using https://github.com/c9s/router-benchmark.

Steps:

  • Cloned the repo
  • git submodule update --init
  • Ran composer install in the directories which included composer.json

When I try to run php benchmark.php, here's the error I get:

PHP Fatal error:  require(): Failed opening required 'SimpleBench/Task.php' (snipped)

There's not a composer.json for router-benchmark, I can't find SimpleBench on packagist, and I'm not doing any better with the Google.

What am I missing? (/me anticipates discovering I've made a really dumb, very obvious mistake)

harikt commented Jan 14, 2014

@jeremykendall seems this is what you are looking https://github.com/c9s/php-SimpleBench

Thanks @harikt.

Reviewing the SimpleBench install instructions, I see:

$ git clone .....
$ onion -d bundle
$ php profiling/magic.php

Suggestion: detailed install instructions 😃

harikt commented Jan 14, 2014

too bad @jeremykendall you never heard of onion ;) https://github.com/c9s/onion

Owner

c9s commented Jan 14, 2014

version 1.3.1 is just released, and we made a lot of performance improvement on the extension, here is the result of latest revision:

n=10000
Runing php array - . 138796.45654569/s
Runing pux - . 124982.98519026/s
Runing klein - . 1801.5070399717/s
Runing ham - . 13566.734991391/s
Runing aura - . 39657.986477172/s
Runing symfony/routing - . 1934.2415677861/s
                     Rate   Mem php array pux aura ham symfony/routing klein
      php array  138.8K/s    0B        ---90% -28% -9%             -1%   -1%
            pux 124.98K/s    0B      111%  -- -31%-10%             -1%   -1%
           aura  39.66K/s    0B      349%315%   ---34%             -4%   -4%
            ham  13.57K/s    0B     1023%921% 292%  --            -14%  -13%
symfony/routing   1.93K/s  786K     7175%6461%2050%701%              --  -93%
          klein    1.8K/s  262K     7704%6937%2201%753%            107%    --
================================== Bar Chart ==================================
        php array  138.8K/s | ████████████████████████████████████████████████████████████  |
              pux 124.98K/s | ██████████████████████████████████████████████████████        |
             aura  39.66K/s | █████████████████                                             |
              ham  13.57K/s | █████                                                         |
  symfony/routing   1.93K/s |                                                               |
            klein    1.8K/s |                                                               |
============================== System Information ==============================
PHP Version: 5.5.6
CPU Brand String: Intel(R) Core(TM) i5-3427U CPU @ 1.80GHz
With XDebug Extension.

Geez. Seems like a root composer.json and composer install would be enough :-/

Owner

c9s commented Jan 14, 2014

@jeremykendall,

although pear is pretty old, you may run:

pear channel-discover pear.corneltek.com
pear install corneltek/SimpleBench

to install the simple bench package.

Owner

c9s commented Jan 14, 2014

And you need to run git submodules to install the router dependencies:

git submodule init
git submodule update

I don't mean to be flip, but this would be a hell of a lot easier with one single composer.json file in the root of the project. Benchmark installation instructions would become:

  • git clone https://github.com/c9s/router-benchmark
  • cd router-benchmark
  • composer install
  • cd code
  • php benchmark.php

Just my two cents.

Owner

c9s commented Jan 14, 2014

@jeremykendall haha you're right, the repo structure is pretty rough, the installation should be like that.

harikt commented Jan 15, 2014

I agree with @jeremykendall here. So different versions can also be tested 👍 .

Owner

c9s commented Sep 12, 2015

php-r3 was added to php-router-benchmark repo, and it's the new baseline of all routers. :)

https://github.com/c9s/php-router-benchmark

I am closing this issue since it's included in tyler's php-router-benchmark repo.

c9s closed this Sep 12, 2015

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment