New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Benchmark cse on rather small expressions #38

Merged
merged 2 commits into from Jun 19, 2017

Conversation

Projects
None yet
2 participants
@bjodah
Member

bjodah commented Jun 14, 2017

This takes 16 ms on sympy-0.7.6

bjodah added a commit to bjodah/sympy that referenced this pull request Jun 15, 2017

Add dummify kwargs to cse()
It was hoped for that dummifying would help with performance.
See: sympy/sympy_benchmarks#38
However, that does not seem to be the case.

@bjodah bjodah changed the title from Benchmark cse on a rather small expression with floats to Benchmark cse on a rather small expressions Jun 19, 2017

@bjodah

This comment has been minimized.

Show comment
Hide comment
@bjodah

bjodah Jun 19, 2017

Member

I've turned the original benchmark to a minimal reproducible parametrized test. It clearly shows the performance regression while being on the order of ~1s even for the slow code:

[ 33.33%] · For sympy commit hash 9670ff89:
[ 33.33%] ·· Building for virtualenv-py3.5-fastcache-mpmath......
[ 33.33%] ·· Benchmarking virtualenv-py3.5-fastcache-mpmath
[ 66.67%] ··· Running cse.TimeCSE.time_cse                                                                                                                                                                        ok
[ 66.67%] ···· 
               ========= ==========
                 param1            
               --------- ----------
                 (2, 8)   39.93ms  
                 (3, 8)   41.45ms  
                 (4, 8)   42.01ms  
                 (2, 9)   85.49ms  
                (2, 10)   455.51ms 
                (2, 11)    1.00s   
               ========= ==========

[ 66.67%] · For sympy commit hash fbb29636:
[ 66.67%] ·· Building for virtualenv-py3.5-fastcache-mpmath.....
[ 66.67%] ·· Benchmarking virtualenv-py3.5-fastcache-mpmath
[100.00%] ··· Running cse.TimeCSE.time_cse                                                                                                                                                                        ok
[100.00%] ···· 
               ========= ========
                 param1          
               --------- --------
                 (2, 8)   4.54ms 
                 (3, 8)   4.79ms 
                 (4, 8)   4.90ms 
                 (2, 9)   4.93ms 
                (2, 10)   5.53ms 
                (2, 11)   5.83ms 
               ========= ========
Member

bjodah commented Jun 19, 2017

I've turned the original benchmark to a minimal reproducible parametrized test. It clearly shows the performance regression while being on the order of ~1s even for the slow code:

[ 33.33%] · For sympy commit hash 9670ff89:
[ 33.33%] ·· Building for virtualenv-py3.5-fastcache-mpmath......
[ 33.33%] ·· Benchmarking virtualenv-py3.5-fastcache-mpmath
[ 66.67%] ··· Running cse.TimeCSE.time_cse                                                                                                                                                                        ok
[ 66.67%] ···· 
               ========= ==========
                 param1            
               --------- ----------
                 (2, 8)   39.93ms  
                 (3, 8)   41.45ms  
                 (4, 8)   42.01ms  
                 (2, 9)   85.49ms  
                (2, 10)   455.51ms 
                (2, 11)    1.00s   
               ========= ==========

[ 66.67%] · For sympy commit hash fbb29636:
[ 66.67%] ·· Building for virtualenv-py3.5-fastcache-mpmath.....
[ 66.67%] ·· Benchmarking virtualenv-py3.5-fastcache-mpmath
[100.00%] ··· Running cse.TimeCSE.time_cse                                                                                                                                                                        ok
[100.00%] ···· 
               ========= ========
                 param1          
               --------- --------
                 (2, 8)   4.54ms 
                 (3, 8)   4.79ms 
                 (4, 8)   4.90ms 
                 (2, 9)   4.93ms 
                (2, 10)   5.53ms 
                (2, 11)   5.83ms 
               ========= ========
@moorepants

This comment has been minimized.

Show comment
Hide comment
@moorepants

moorepants Jun 19, 2017

Member

This isn't for small expressions but we do have one other cse test: http://www.moorepants.info/misc/sympy-asv/#large_exprs.TimeLargeExpressionOperations.time_cse and there was a recent regression that I made the PR author aware of. Nothing has happened to it since.

Member

moorepants commented Jun 19, 2017

This isn't for small expressions but we do have one other cse test: http://www.moorepants.info/misc/sympy-asv/#large_exprs.TimeLargeExpressionOperations.time_cse and there was a recent regression that I made the PR author aware of. Nothing has happened to it since.

@moorepants

This comment has been minimized.

Show comment
Hide comment
@moorepants

moorepants Jun 19, 2017

Member

This is fine to merge.

Member

moorepants commented Jun 19, 2017

This is fine to merge.

@bjodah

This comment has been minimized.

Show comment
Hide comment
@bjodah

bjodah Jun 19, 2017

Member

Thanks,

I just checked what happens when SYMPY_CACHE_SIZE=25000:

[ 33.33%] · For sympy commit hash 9670ff89:
[ 33.33%] ·· Building for virtualenv-py3.5-fastcache-mpmath......
[ 33.33%] ·· Benchmarking virtualenv-py3.5-fastcache-mpmath
[ 66.67%] ··· Running cse.TimeCSE.time_cse                                                                                                                                                                        ok
[ 66.67%] ···· 
               =============== ==========
                nexprs_nterms            
               --------------- ----------
                    (2, 8)      42.28ms  
                    (3, 8)      41.32ms  
                    (4, 8)      42.02ms  
                    (2, 9)      84.79ms  
                   (2, 10)      159.80ms 
                   (2, 11)      324.80ms 
               =============== ==========

[ 66.67%] · For sympy commit hash fbb29636:
[ 66.67%] ·· Building for virtualenv-py3.5-fastcache-mpmath......
[ 66.67%] ·· Benchmarking virtualenv-py3.5-fastcache-mpmath
[100.00%] ··· Running cse.TimeCSE.time_cse                                                                                                                                                                        ok
[100.00%] ···· 
               =============== ========
                nexprs_nterms          
               --------------- --------
                    (2, 8)      4.66ms 
                    (3, 8)      4.83ms 
                    (4, 8)      4.92ms 
                    (2, 9)      4.95ms 
                   (2, 10)      5.65ms 
                   (2, 11)      6.03ms 
               =============== ========

so the regression is still there.

Member

bjodah commented Jun 19, 2017

Thanks,

I just checked what happens when SYMPY_CACHE_SIZE=25000:

[ 33.33%] · For sympy commit hash 9670ff89:
[ 33.33%] ·· Building for virtualenv-py3.5-fastcache-mpmath......
[ 33.33%] ·· Benchmarking virtualenv-py3.5-fastcache-mpmath
[ 66.67%] ··· Running cse.TimeCSE.time_cse                                                                                                                                                                        ok
[ 66.67%] ···· 
               =============== ==========
                nexprs_nterms            
               --------------- ----------
                    (2, 8)      42.28ms  
                    (3, 8)      41.32ms  
                    (4, 8)      42.02ms  
                    (2, 9)      84.79ms  
                   (2, 10)      159.80ms 
                   (2, 11)      324.80ms 
               =============== ==========

[ 66.67%] · For sympy commit hash fbb29636:
[ 66.67%] ·· Building for virtualenv-py3.5-fastcache-mpmath......
[ 66.67%] ·· Benchmarking virtualenv-py3.5-fastcache-mpmath
[100.00%] ··· Running cse.TimeCSE.time_cse                                                                                                                                                                        ok
[100.00%] ···· 
               =============== ========
                nexprs_nterms          
               --------------- --------
                    (2, 8)      4.66ms 
                    (3, 8)      4.83ms 
                    (4, 8)      4.92ms 
                    (2, 9)      4.95ms 
                   (2, 10)      5.65ms 
                   (2, 11)      6.03ms 
               =============== ========

so the regression is still there.

@bjodah bjodah changed the title from Benchmark cse on a rather small expressions to Benchmark cse on rather small expressions Jun 19, 2017

@bjodah bjodah merged commit 444d9af into sympy:master Jun 19, 2017

1 check was pending

continuous-integration/travis-ci/pr The Travis CI build is in progress
Details

@bjodah bjodah deleted the bjodah:cse-floats branch Jun 19, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment