Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Benchmarks for \Pipeline\Simple #2

Open
wants to merge 11 commits into
base: master
Choose a base branch
from
Open

Conversation

sanmai
Copy link

@sanmai sanmai commented Mar 13, 2018

Benchmarks for \Pipeline\Simple, for a plain generators, and additional improvements.

It can be seen that generators are usually five times slower than plain arrays.

On my fairly old machine I get this:

Iterating over 100 ints ........
-----------------------
  PHP       [for]             0.00002 sec   x1.7 (+67%)
  PHP       [array functions] 0.00001 sec   x1.0 (100%)
  YaLinqo                     0.00007 sec   x5.9 (+492%)
  Ginq                        0.00024 sec   x21.3 (+2032%)
  Ginq      [generator]       0.00006 sec   x5.6 (+464%)
  Pinq                        0.00011 sec   x9.9 (+890%)
  Pipeline                    0.00003 sec   x2.3 (+126%)
  Pipeline  [generator]       0.00007 sec   x5.9 (+490%)

Generating array of 100 integers ........
--------------------------------
  PHP       [for]             0.00003 sec   x2.2 (+116%)
  PHP       [array functions] 0.00002 sec   x1.0 (100%)
  PHP       [xrange]          0.00007 sec   x4.3 (+330%)
  YaLinqo                     0.00009 sec   x5.7 (+473%)
  Ginq                        0.00029 sec   x19.0 (+1804%)
  Pinq                        0.00010 sec   x6.8 (+576%)
  Pipeline  [range]           0.00003 sec   x1.8 (+79%)
  Pipeline  [xrange]          0.00007 sec   x4.7 (+367%)

Generating lookup of 100 floats, calculate sum .......
----------------------------------------------
  PHP                         0.00026 sec   x1.0 (100%)
  YaLinqo                     0.00097 sec   x3.8 (+281%)
  YaLinqo   [string lambda]   0.00085 sec   x3.3 (+233%)
  Ginq                        0.00608 sec   x23.8 (+2284%)
  Pinq                        * Not implemented
  Pipeline                    0.00081 sec   x3.2 (+218%)
  Pipeline  [optimized]       0.00051 sec   x2.0 (+99%)

Counting values in arrays .......
-------------------------
  PHP       [for]             0.00009 sec   x1.0 (100%)
  PHP       [array functions] 0.00015 sec   x1.7 (+67%)
  YaLinqo                     0.00020 sec   x2.2 (+122%)
  YaLinqo   [string lambda]   0.00014 sec   x1.6 (+63%)
  Ginq                        0.00038 sec   x4.3 (+333%)
  Pinq                        0.00034 sec   x3.8 (+284%)
  Pipeline                    0.00033 sec   x3.8 (+277%)

Counting values in arrays deep ......
------------------------------
  PHP       [for]             0.00014 sec   x1.0 (100%)
  PHP       [array functions] 0.00075 sec   x5.4 (+445%)
  YaLinqo                     0.00164 sec   x12.0 (+1097%)
  Ginq                        0.00327 sec   x23.8 (+2278%)
  Pinq                        0.00483 sec   x35.2 (+3418%)
  Pipeline                    0.00295 sec   x21.5 (+2050%)

Filtering values in arrays ........
--------------------------
  PHP       [for]             0.00008 sec   x1.0 (100%)
  PHP       [yield]           0.00012 sec   x1.4 (+43%)
  PHP       [array functions] 0.00016 sec   x2.0 (+96%)
  YaLinqo                     0.00023 sec   x2.8 (+181%)
  YaLinqo   [string lambda]   0.00018 sec   x2.2 (+124%)
  Ginq                        0.00073 sec   x8.9 (+786%)
  Pinq                        0.00034 sec   x4.1 (+315%)
  Pipeline                    0.00032 sec   x3.9 (+287%)

Filtering values in arrays deep .......
-------------------------------
  PHP       [for]             0.00036 sec   x1.0 (100%)
  PHP       [array functions] 0.00098 sec   x2.7 (+175%)
  YaLinqo                     0.00285 sec   x8.0 (+695%)
  YaLinqo   [string lambda]   0.00280 sec   x7.8 (+681%)
  Ginq                        0.00761 sec   x21.3 (+2025%)
  Pinq                        0.00624 sec   x17.4 (+1644%)
  Pipeline                    0.00337 sec   x9.4 (+843%)

Sorting arrays of strings .......
-------------------------
  PHP                         0.00008 sec   x1.0 (100%)
  YaLinqo                     0.00011 sec   x1.5 (+47%)
  YaLinqo   [sort flags]      0.00010 sec   x1.3 (+32%)
  YaLinqo   [callback]        0.00235 sec   x30.3 (+2931%)
  Ginq                        0.00248 sec   x31.9 (+3090%)
  Pinq                        * Not implemented
  Pipeline                    * Not implemented

Sorting arrays of objects .........
-------------------------
  PHP       [multisort]       0.00001 sec   x1.0 (100%)
  PHP       [usort]           0.00003 sec   x2.4 (+137%)
  YaLinqo                     0.00009 sec   x6.8 (+578%)
  YaLinqo   [string lambda]   0.00009 sec   x6.3 (+525%)
  YaLinqo   [sort flags]      0.00008 sec   x6.0 (+499%)
  Ginq                        0.00037 sec   x26.5 (+2550%)
  Ginq      [property path]   0.00104 sec   x74.6 (+7364%)
  Pinq                        0.00026 sec   x18.8 (+1779%)
  Pipeline                    * Not implemented

Joining arrays .......
--------------
  PHP                         0.00002 sec   x1.0 (100%)
  YaLinqo                     0.00019 sec   x7.7 (+668%)
  YaLinqo   [string lambda]   0.00015 sec   x6.0 (+504%)
  Ginq                        0.00028 sec   x11.2 (+1023%)
  Ginq      [property path]   0.00045 sec   x18.2 (+1719%)
  Pinq                        0.00128 sec   x52.4 (+5137%)
  Pipeline                    0.00021 sec   x8.4 (+736%)

Aggregating arrays .........
------------------
  PHP       [for]             0.00015 sec   x1.6 (+57%)
  PHP       [array functions] 0.00036 sec   x3.8 (+277%)
  PHP       [optimized]       0.00009 sec   x1.0 (100%)
  YaLinqo                     0.00070 sec   x7.4 (+639%)
  YaLinqo   [string lambda]   0.00050 sec   x5.2 (+423%)
  Ginq                        0.00195 sec   x20.6 (+1961%)
  Ginq      [property path]   0.00682 sec   x72.1 (+7106%)
  Pinq                        0.00208 sec   x22.0 (+2100%)
  Pipeline                    0.00020 sec   x2.1 (+111%)

Aggregating arrays custom .......
-------------------------
  PHP       [for]             0.00002 sec   x1.0 (100%)
  PHP       [array functions] 0.00010 sec   x5.7 (+467%)
  YaLinqo                     0.00015 sec   x8.9 (+793%)
  YaLinqo   [string lambda]   0.00010 sec   x6.1 (+506%)
  Ginq                        0.00015 sec   x9.1 (+809%)
  Pinq                        0.00065 sec   x38.4 (+3743%)
  Pipeline                    0.00014 sec   x8.4 (+744%)

Process data from ReadMe example .......
--------------------------------
  PHP       [multisort]       0.00016 sec   x1.0 (100%)
  PHP       [usort]           0.00098 sec   x6.0 (+502%)
  YaLinqo                     0.00122 sec   x7.5 (+646%)
  YaLinqo   [string lambda]   0.00100 sec   x6.1 (+511%)
  Ginq                        0.01167 sec   x71.3 (+7030%)
  Pinq                        0.00429 sec   x26.2 (+2524%)
  Pipeline                    * Not implemented

Done!

On some cases \Pipeline\Simple beats YaLinqo but I'd only take it as a pleasant surprise: it wasn't intended for plain arrays processing.

@Athari
Copy link
Owner

Athari commented Mar 13, 2018

While Simple/Pipeline solves the same basic problem of PHP's array functions having horrible API, it's very barebones, so it doesn't have a place in the comparison of full LINQ implementations. Of course, most LINQ functions can be implemented through map-reduce, which your library seems to cover, but many examples become too verbose or plain impossible (like sorting or transforming keys). I'll leave the pull request open though, so that if anyone is interested they can check it out.

Your benchmark results are kinda weird. YaLinqo's string lambdas are in most cases faster than anonymous functions in the same YaLinqo tests, which can never happen. As for the tests in which your library performes better, there're clear reasons for this.

In case of "Generating lookup of 100 floats, calculate sum", you just skip some transformations which can't be implemented with Simple/Pipeline (it works only with values, not keys), this is the reason it's faster.

In case of "Aggregating arrays", you only have the optimized version which skips 3/4 of calculations. Now that I think about it, I should just remove the array-optimized case, as it skews the results of the whole test group.

@sanmai
Copy link
Author

sanmai commented Mar 13, 2018

Frankly, I was thinking of my task at hand: Simple/Pipeline need some sort of performance review, and if you would accept this PR I would just link to this repository and be done with it. Else I would have to write my own benchmarks. Is there anything that can change you mind?

In what PHP version were you testing? I used Debian’s stock PHP 7.0.

@Athari
Copy link
Owner

Athari commented Mar 13, 2018

@sanmai My requirements are:

  1. More or less comprehensive LINQ port, not just 5 basic functions. I don't want half of tests to display "Not implemented".
  2. Fixing benchmarks I mentioned above so they test the same code. Not much point in comparison if the logic is different.

Both requirements are beyond the scope of your library, if I understand correctly.

Anyway, I don't see anything wrong with pointing your users to your fork of my becnchmark project. I'm unlikely to update the project beyond updating dependencies, unless some major things happen to PHP which considerably affect the performance. And you're unlikely to expand your library much beyond what you already have. Overall, both can be considered static.

Now, if I understood the aim of your project incorrectly and you actually plan to implement a comprehensive "collection pipeline", not just basic map-reduce functionality, then feel free to ping me when you expand your library.

The latest run looks to be on PHP 5.5 on Windows. Oh well, I guess it needs updating then. PHP 7.0 severely affected the performance.

@sanmai
Copy link
Author

sanmai commented Mar 14, 2018

There you go - now with a fancy new LazyLINQ.

$ php benchmark.php 10000
Preparing data with 10000 items...	Done

Iterating over 10000 ints .......
-------------------------
  PHP       [for]             0.00105 sec   x2.0 (+102%)
  PHP       [array functions] 0.00052 sec   x1.0 (100%)
  YaLinqo                     0.01058 sec   x20.3 (+1927%)
  Ginq                        0.03240 sec   x62.1 (+6109%)
  Ginq      [generator]       0.00869 sec   x16.7 (+1566%)
  Pinq                        0.01018 sec   x19.5 (+1851%)
  LazyLINQ                    0.01066 sec   x20.4 (+1943%)

Generating array of 10000 integers .......
----------------------------------
  PHP       [for]             0.00216 sec   x2.2 (+119%)
  PHP       [array functions] 0.00099 sec   x1.0 (100%)
  PHP       [xrange]          0.00934 sec   x9.5 (+846%)
  YaLinqo                     0.01176 sec   x11.9 (+1092%)
  Ginq                        0.04492 sec   x45.5 (+4452%)
  Pinq                        0.01216 sec   x12.3 (+1132%)
  LazyLINQ                    0.01113 sec   x11.3 (+1028%)

Generating lookup of 10000 floats, calculate sum ......
------------------------------------------------
  PHP                         0.02568 sec   x1.0 (100%)
  YaLinqo                     0.12027 sec   x4.7 (+368%)
  YaLinqo   [string lambda]   0.10560 sec   x4.1 (+311%)
  Ginq                        0.48790 sec   x19.0 (+1800%)
  Pinq                        * Not implemented
  LazyLINQ                    0.11003 sec   x4.3 (+329%)

Counting values in arrays .......
-------------------------
  PHP       [for]             0.01817 sec   x1.0 (100%)
  PHP       [array functions] 0.03126 sec   x1.7 (+72%)
  YaLinqo                     0.04211 sec   x2.3 (+132%)
  YaLinqo   [string lambda]   0.03458 sec   x1.9 (+90%)
  Ginq                        0.07055 sec   x3.9 (+288%)
  Pinq                        0.06462 sec   x3.6 (+256%)
  LazyLINQ                    0.06631 sec   x3.6 (+265%)

Counting values in arrays deep ......
------------------------------
  PHP       [for]             0.02262 sec   x1.0 (100%)
  PHP       [array functions] 0.13204 sec   x5.8 (+484%)
  YaLinqo                     0.25359 sec   x11.2 (+1021%)
  Ginq                        0.53037 sec   x23.4 (+2244%)
  Pinq                        0.77256 sec   x34.1 (+3315%)
  LazyLINQ                    0.50367 sec   x22.3 (+2126%)

Filtering values in arrays ........
--------------------------
  PHP       [for]             0.02018 sec   x1.0 (100%)
  PHP       [yield]           0.02720 sec   x1.3 (+35%)
  PHP       [array functions] 0.03437 sec   x1.7 (+70%)
  YaLinqo                     0.05061 sec   x2.5 (+151%)
  YaLinqo   [string lambda]   0.04452 sec   x2.2 (+121%)
  Ginq                        0.13876 sec   x6.9 (+588%)
  Pinq                        0.06721 sec   x3.3 (+233%)
  LazyLINQ                    0.07101 sec   x3.5 (+252%)

Filtering values in arrays deep .......
-------------------------------
  PHP       [for]             0.06776 sec   x1.0 (100%)
  PHP       [array functions] 0.19821 sec   x2.9 (+193%)
  YaLinqo                     0.45135 sec   x6.7 (+566%)
  YaLinqo   [string lambda]   0.43517 sec   x6.4 (+542%)
  Ginq                        1.31743 sec   x19.4 (+1844%)
  Pinq                        0.97927 sec   x14.5 (+1345%)
  LazyLINQ                    0.95334 sec   x14.1 (+1307%)

Sorting arrays of strings .......
-------------------------
  PHP                         0.02675 sec   x1.0 (100%)
  YaLinqo                     0.02794 sec   x1.0 (+4%)
  YaLinqo   [sort flags]      0.02942 sec   x1.1 (+10%)
  YaLinqo   [callback]        0.81934 sec   x30.6 (+2963%)
  Ginq                        0.83279 sec   x31.1 (+3013%)
  Pinq                        * Not implemented
  LazyLINQ                    * Not implemented

Sorting arrays of objects .........
-------------------------
  PHP       [multisort]       0.00109 sec   x1.0 (100%)
  PHP       [usort]           0.02361 sec   x21.7 (+2068%)
  YaLinqo                     0.00651 sec   x6.0 (+498%)
  YaLinqo   [string lambda]   0.00430 sec   x4.0 (+295%)
  YaLinqo   [sort flags]      0.00432 sec   x4.0 (+297%)
  Ginq                        0.24023 sec   x220.5 (+21954%)
  Ginq      [property path]   0.84033 sec   x771.5 (+77047%)
  Pinq                        0.01937 sec   x17.8 (+1678%)
  LazyLINQ                    * Not implemented

Joining arrays .......
--------------
  PHP                         0.01807 sec   x1.0 (100%)
  YaLinqo                     0.03775 sec   x2.1 (+109%)
  YaLinqo   [string lambda]   0.02893 sec   x1.6 (+60%)
  Ginq                        0.02805 sec   x1.6 (+55%)
  Ginq      [property path]   0.04789 sec   x2.7 (+165%)
  Pinq                        0.19092 sec   x10.6 (+957%)
  LazyLINQ                    0.04061 sec   x2.2 (+125%)

Aggregating arrays .........
------------------
  PHP       [for]             0.02801 sec   x1.3 (+30%)
  PHP       [array functions] 0.06618 sec   x3.1 (+206%)
  PHP       [optimized]       0.02162 sec   x1.0 (100%)
  YaLinqo                     0.10824 sec   x5.0 (+401%)
  YaLinqo   [string lambda]   0.08056 sec   x3.7 (+273%)
  Ginq                        0.29904 sec   x13.8 (+1283%)
  Ginq      [property path]   1.02472 sec   x47.4 (+4639%)
  Pinq                        0.31002 sec   x14.3 (+1334%)
  LazyLINQ                    0.15868 sec   x7.3 (+634%)

Aggregating arrays custom .......
-------------------------
  PHP       [for]             0.00862 sec   x1.0 (100%)
  PHP       [array functions] 0.02273 sec   x2.6 (+164%)
  YaLinqo                     0.02758 sec   x3.2 (+220%)
  YaLinqo   [string lambda]   0.02034 sec   x2.4 (+136%)
  Ginq                        0.02852 sec   x3.3 (+231%)
  Pinq                        0.09963 sec   x11.6 (+1055%)
  LazyLINQ                    0.02715 sec   x3.1 (+215%)

Zip arrays .....
----------
  PHP       [for]             0.01361 sec   x1.0 (100%)
  YaLinqo                     * Not implemented
  Ginq                        0.09369 sec   x6.9 (+589%)
  Pinq                        * Not implemented
  LazyLINQ                    0.07873 sec   x5.8 (+479%)

Process data from ReadMe example .......
--------------------------------
  PHP       [multisort]       0.04346 sec   x1.0 (100%)
  PHP       [usort]           0.40580 sec   x9.3 (+834%)
  YaLinqo                     0.19375 sec   x4.5 (+346%)
  YaLinqo   [string lambda]   0.15897 sec   x3.7 (+266%)
  Ginq                        3.84582 sec   x88.5 (+8750%)
  Pinq                        0.59705 sec   x13.7 (+1274%)
  LazyLINQ                    * Not implemented

Done!

PHP 7.0 is reaching EOL later this year, so I'd say you're a little late to the party with your PHP 5.5.

Also I added a valid .travis.yml if you would ever wish to run benchmarks on different PHP versions on Travis.

@sanmai
Copy link
Author

sanmai commented Mar 8, 2020

Ping @Athari. You mind anything against last update? You wanted a more-less comprehensive LINQ port, and here you have it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants