performance tracker

[faassen]

I thought it would be fun to track performance.

This was the decoupling branch as of a while ago:

dispatch 0 args
2.50691318512
dispatch 1 args
4.27959012985
dispatch 2 args
6.44276189804
dispatch 3 args
7.31519389153
dispatch 4 args
8.34504699707
Plain func 0 args
0.161817789078
Plain func 4 args
0.481851100922

Here's some older benchmark of the arg extraction code (where @taschini already took out argextract, but not yet completely):

dispatch 0 args
1.25924301147
dispatch 1 args
2.03821897507
dispatch 2 args
3.58315300941
dispatch 3 args
4.08871388435
dispatch 4 args
4.67021179199
Plain func 0 args
0.159023046494
Plain func 4 args
0.48424077034

As of today, it's:

dispatch 0 args
1.30515289307
dispatch 1 args
1.78144598007
dispatch 2 args
3.02587485313
dispatch 3 args
3.37043809891
dispatch 4 args
3.69515585899
Plain func 0 args
0.163196086884
Plain func 4 args
0.484107017517

So we currently we're less than 10 times slower than plain Python.

[taschini]

We should have it as a tox task in Reg.

[faassen]

Is the tox task for local testing or for the CI? I'm not sure whether the CI environment is consistent enough in load for it to be worthwhile to do performance testing on it.

[faassen]

Experimenting with a dict based cache (instead of a LRU cache):

dispatch 0 args
0.832697153091
dispatch 1 args
1.17894697189
dispatch 2 args
2.40593910217
dispatch 3 args
2.73063993454
dispatch 4 args
3.10167312622
Plain func 0 args
0.160401105881
Plain func 4 args
0.480713844299

[taschini]

Experimenting with a dict based cache (instead of a LRU cache):

Quite a difference.

Is the tox task for local testing or for the CI? I'm not sure whether the CI environment is consistent enough in load for it to be worthwhile to do performance testing on it.

I think you can just read it in terms of multiples of the Plain func 0 args line.

[faassen]

Ah, yeah, we could normalize it by the 0 args line to get some idea at least. @henri-hulski are you interested in looking into this? You need to modify perf.py, just create a new copy of that under a different name so I can continue running the old perf.py for a while.

[taschini]

Can you update the perf after bc3afd1?

[faassen]

I updated this with the current master, so that includes your cache improvements and call improvements. Quite a boost again:

dispatch 0 args
0.477177858353
dispatch 1 args
0.804775953293
dispatch 2 args
1.78332781792
dispatch 3 args
2.1213991642
dispatch 4 args
2.46204090118
Plain func 0 args
0.159456014633
Plain func 4 args
0.48091506958

[henri-hulski]

@faassen I can add it to tox.
But not sure what @taschini means with

I think you can just read it in terms of multiples of the Plain func 0 args line.

[taschini]

In the sense that instead of looking at the times as measured in seconds, you could look at them in relative terms, as multiples of the time for the "Plain func 0 args"

[henri-hulski]

Ok. I think I got it.

[henri-hulski]

@faassen I named it tox_perf.py and added it to tox.

[faassen]

Okay, cool, that gives a nice idea of how fast/slow Reg is! But I'm glad you retained perf.py as absolute numbers are good. Also I can use it with Python 2 as that's what I've been doing the previous tests with.

I'm a bit surprised the KeyExtractor removal has had no effect on performance whatsoever.

dispatch 0 args
0.477218866348
dispatch 1 args
0.790820837021
dispatch 2 args
1.79506707191
dispatch 3 args
2.14167904854
dispatch 4 args
2.54557490349
Plain func 0 args
0.156985998154
Plain func 4 args
0.478663921356

[taschini]

I'm a bit surprised the KeyExtractor removal has had no effect on performance whatsoever.

Because match_argname (the only of the four predicates to be used in the benchmark) has not been using NameKeyExtractor (a close relative of KeyExtractor) since 1b5af1b, thirteen days ago.

Removing KeyExtractor (a1061ff) has affected the other three predicates: match_key, match_instance and match_class.

[faassen]

Latest master:

dispatch 0 args
0.468906164169
dispatch 1 args
0.759228944778
dispatch 2 args
1.11962509155
dispatch 3 args
1.35791301727
dispatch 4 args
2.06766414642
Plain func 0 args
0.173716068268
Plain func 4 args
0.483229160309