Performance regression between 1.2.0.4 and 1.2.0.5 #57

Closed
wrvn opened this Issue Feb 1, 2013 · 12 comments

Projects

None yet

5 participants

@wrvn
wrvn commented Feb 1, 2013

I have noticed huge performance regression in latest hashable library.

For example, this program (http://hpaste.org/81663) runs almost 3 times slower with hashable 1.2.0.5 - 190 seconds on my pc vs 70 seconds with hashable 1.2.0.4.

This was tested on windows x86, ghc 7.6.2

@bos
Collaborator
bos commented Feb 3, 2013

This is at least in part due to a performance regression between GHC 7.6.1 and 7.6.2.

Under 7.6.1, hashable 1.2.0.5 is slower than 1.2.0.4, but GHC 7.6.2 increases the margin by a large amount.

Times below were measured on OS X, with a 64-bit GHC.

         1.2.0.4  1.2.0.5
7.6.1       5.4       8.8
7.6.2       5.4      13.1
@tibbe
Owner
tibbe commented Mar 19, 2013

Here's the hashable benchmark numbers for 1.2.0.5, using GHC 7.6.1 and 7.6.2. I don't see a big enough difference to explain the numbers @bos gave above.

7.6.1:

benchmarking hash/String/5
mean: 313.4638 ns, lb 306.8101 ns, ub 323.2874 ns, ci 0.950
std dev: 40.79769 ns, lb 30.48706 ns, ub 54.19728 ns, ci 0.950
found 12 outliers among 100 samples (12.0%)
  7 (7.0%) high mild
  5 (5.0%) high severe
variance introduced by outliers: 87.286%
variance is severely inflated by outliers

benchmarking hash/String/8
mean: 418.8892 ns, lb 406.3645 ns, ub 438.1718 ns, ci 0.950
std dev: 78.34058 ns, lb 55.07190 ns, ub 103.3493 ns, ci 0.950
found 12 outliers among 100 samples (12.0%)
  2 (2.0%) high mild
  10 (10.0%) high severe
variance introduced by outliers: 93.605%
variance is severely inflated by outliers

benchmarking hash/String/11
mean: 674.0043 ns, lb 652.0246 ns, ub 701.0030 ns, ci 0.950
std dev: 124.3176 ns, lb 105.1755 ns, ub 142.8914 ns, ci 0.950
found 15 outliers among 100 samples (15.0%)
  15 (15.0%) high mild
variance introduced by outliers: 93.594%
variance is severely inflated by outliers

benchmarking hash/String/40
mean: 1.875777 us, lb 1.815115 us, ub 1.949752 us, ci 0.950
std dev: 344.6218 ns, lb 290.1250 ns, ub 410.6420 ns, ci 0.950
found 14 outliers among 100 samples (14.0%)
  13 (13.0%) high mild
  1 (1.0%) high severe
variance introduced by outliers: 93.591%
variance is severely inflated by outliers

benchmarking hash/String/128
mean: 4.854730 us, lb 4.830086 us, ub 4.881466 us, ci 0.950
std dev: 131.1657 ns, lb 117.6051 ns, ub 146.6917 ns, ci 0.950
variance introduced by outliers: 20.964%
variance is moderately inflated by outliers

benchmarking hash/String/512
mean: 19.13274 us, lb 18.78863 us, ub 19.87256 us, ci 0.950
std dev: 2.449709 us, lb 1.394656 us, ub 4.672181 us, ci 0.950
found 9 outliers among 100 samples (9.0%)
  5 (5.0%) high mild
  4 (4.0%) high severe
variance introduced by outliers: 86.279%
variance is severely inflated by outliers

benchmarking hash/String/2^20
mean: 45.80936 ms, lb 43.40311 ms, ub 48.64469 ms, ci 0.950
std dev: 13.40587 ms, lb 11.74395 ms, ub 15.32905 ms, ci 0.950
found 1 outliers among 100 samples (1.0%)
variance introduced by outliers: 97.825%
variance is severely inflated by outliers

7.6.2:

benchmarking hash/String/5
mean: 299.3452 ns, lb 296.3480 ns, ub 304.6419 ns, ci 0.950
std dev: 19.92880 ns, lb 12.91624 ns, ub 29.86216 ns, ci 0.950
found 11 outliers among 100 samples (11.0%)
  6 (6.0%) high mild
  5 (5.0%) high severe
variance introduced by outliers: 62.556%
variance is severely inflated by outliers

benchmarking hash/String/8
mean: 398.7295 ns, lb 396.8855 ns, ub 401.6021 ns, ci 0.950
std dev: 11.58449 ns, lb 8.578799 ns, ub 19.36588 ns, ci 0.950
found 4 outliers among 100 samples (4.0%)
  3 (3.0%) high mild
  1 (1.0%) high severe
variance introduced by outliers: 23.843%
variance is moderately inflated by outliers

benchmarking hash/String/11
mean: 570.8202 ns, lb 568.7652 ns, ub 573.2731 ns, ci 0.950
std dev: 11.43334 ns, lb 9.848201 ns, ub 14.44369 ns, ci 0.950
found 2 outliers among 100 samples (2.0%)
  2 (2.0%) high mild
variance introduced by outliers: 13.237%
variance is moderately inflated by outliers

benchmarking hash/String/40
mean: 1.646568 us, lb 1.637878 us, ub 1.657170 us, ci 0.950
std dev: 48.94198 ns, lb 41.57890 ns, ub 58.41102 ns, ci 0.950
found 4 outliers among 100 samples (4.0%)
  4 (4.0%) high mild
variance introduced by outliers: 24.804%
variance is moderately inflated by outliers

benchmarking hash/String/128
mean: 4.878967 us, lb 4.852378 us, ub 4.917509 us, ci 0.950
std dev: 162.2143 ns, lb 122.5275 ns, ub 245.5877 ns, ci 0.950
found 4 outliers among 100 samples (4.0%)
  2 (2.0%) high mild
  2 (2.0%) high severe
variance introduced by outliers: 28.728%
variance is moderately inflated by outliers

benchmarking hash/String/512
mean: 18.44862 us, lb 18.36460 us, ub 18.56844 us, ci 0.950
std dev: 507.1797 ns, lb 389.7598 ns, ub 732.2280 ns, ci 0.950
found 4 outliers among 100 samples (4.0%)
  2 (2.0%) high mild
  2 (2.0%) high severe
variance introduced by outliers: 21.906%
variance is moderately inflated by outliers

benchmarking hash/String/2^20
mean: 44.43636 ms, lb 42.07600 ms, ub 47.23868 ms, ci 0.950
std dev: 13.08253 ms, lb 11.38578 ms, ub 14.44831 ms, ci 0.950
found 25 outliers among 100 samples (25.0%)
  21 (21.0%) high mild
  4 (4.0%) high severe
variance introduced by outliers: 97.827%
variance is severely inflated by outliers
@andreasabel

Agda-2.3.3 (darcs version) runs more than 10x slower on "make library-test" with hashable-1.2.0.10 in comparison with hashable-1.1.2.5. Usually, the standard library compiles in 4 min, with hashable-1.2.0.10 it takes more than half an hour.

Took me a whole day to track down this performance bug to hashable, I thought it was one of my last patches. You owe me a beer!

For now, I guess the workaround is to constrain to hashable < 1.2.

Cheers,
Andreas

@asr
Contributor
asr commented Jul 11, 2013

Hi Andreas,

For now, I guess the workaround is to constrain to hashable < 1.2.

I am getting the usual times with hashable-1.2.0.6 and hashable-1.2.0.7. I need more time to test the other versions.

@andreasabel

Hi Johan,

Thanks for looking into this. I forgot to mention I am using ghc-7.4.1.
A fellow Agda developer has confirmed the 10x slowdown with the current hackage version of hashable, he is probably on a more recent ghc.

@asr
Contributor
asr commented Jul 11, 2013

make library-test numbers for Agda 2.3.3 using GHC 7.6.3:

  • hashable-1.1.2.5

    real 2m56.332s
    user 2m55.330s
    sys 0m0.670s

  • hashable-1.2.0.6

    real 3m20.012s
    user 3m15.470s
    sys 0m1.100s

  • hashable-1.2.0.7 and hashable-1.2.0.8

    Similar to hashable-1.2.0.6

  • hashable-1.2.0.9 (~8x slower respect to hashable-1.1.2.5)

    real 22m31.635s
    user 22m29.550s
    sys 0m1.020s

  • hashable-1.2.0.10 (~8x slower respect to hashable-1.1.2.5)

    real 22m24.140s
    user 22m21.870s
    sys 0m1.160s

@bos
Collaborator
bos commented Jul 12, 2013

Agda folks (@andreasabel and @asr), do you have a smaller test case - one that runs in seconds, instead of many minutes?

@andreasabel

In the Agda repo, there is the top-level directory benchmark with some benchmarks with a Makefile, so you can just run "make" there. Maybe they will be significant enough for testing hashable, but I have not tried.

@asr
Contributor
asr commented Jul 12, 2013

Tested with GHC 7.6.3.

$ darcs get --lazy http://code.haskell.org/Agda
$ cd Agda/benchmark/ac

Timing with hashable-1.1.2.5:

$ time agda --ignore-interfaces AC.agda
real 0m1.248s
user 0m1.210s
sys  0m0.030s

Timing with hashable-1.2.0.10:

$ time agda --ignore-interfaces AC.agda
real 0m2.466s
user 0m2.440s
sys  0m0.020s
@tibbe tibbe added a commit that closed this issue Sep 13, 2013
@tibbe Revert instances to their 1.1 implementations
This addresses a number of performance and correctness issues:

 * Fixed #57.
 * Fixed #59.
 * Fixed #66.
 * Fixes #72.
4064b9f
@tibbe tibbe closed this in 4064b9f Sep 13, 2013
@asr
Contributor
asr commented Sep 16, 2013

Thanks for fixing the issue. Just for the record, the command make library-test has the expected behaviour using hashable-1.2.1.0 in Agda 2.3.3.

@tibbe
Owner
tibbe commented Sep 16, 2013

@asr Thanks for checking!

@andreasabel

Awesome, thanks for fixing this issue!

@JasonGross JasonGross pushed a commit to JasonGross/Agda that referenced this issue Jan 13, 2014
@andreasabel andreasabel Restrict to hashable-1.1.x to prevent later versions which make Agda …
…library-test 10x slower.

Ignore-this: a4c686ed2b8e1b6b0238ef1ff63aa128
Performance bug confirmed on ghc-7.4.1 with hashable-1.2.0.10.
See also tibbe/hashable#57.

darcs-hash:20130711193301-cc7ee-bf8b07b9377b777b477abba07dba4c3ffe157297
3723a66
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment