Benchmarking Streamly backend #54

Magalame · 2019-05-08T07:21:55Z

See #35
I'm pretty sure my modifications of analyze.cabal are overkill at some point, but it should get the benchmark running.

Using the recommended flags by Streamly, I have for benchmarking runtime: stack ghc -- -package streamly-0.6.0 -package vector-0.12.0.3 --make benchStreamly.hs -main-is speed -O2 -fspec-constr-recursive=10

Giving:

benchmarking Tests/Vec
time                 9.290 ms   (9.126 ms .. 9.406 ms)
                     0.998 R²   (0.997 R² .. 0.999 R²)
mean                 9.782 ms   (9.658 ms .. 9.911 ms)
std dev              334.4 μs   (296.9 μs .. 384.8 μs)
variance introduced by outliers: 12% (moderately inflated)

benchmarking Tests/Stream
time                 6.049 ms   (5.908 ms .. 6.162 ms)
                     0.997 R²   (0.996 R² .. 0.999 R²)
mean                 5.728 ms   (5.667 ms .. 5.807 ms)
std dev              205.3 μs   (168.8 μs .. 263.7 μs)
variance introduced by outliers: 15% (moderately inflated)

benchmarking Tests/takeRowsVec 1000
time                 10.02 ms   (9.894 ms .. 10.19 ms)
                     0.999 R²   (0.997 R² .. 1.000 R²)
mean                 9.920 ms   (9.864 ms .. 10.01 ms)
std dev              192.2 μs   (143.0 μs .. 248.1 μs)

benchmarking Tests/takeRowsVec 500
time                 4.683 ms   (4.606 ms .. 4.769 ms)
                     0.998 R²   (0.997 R² .. 0.999 R²)
mean                 5.047 ms   (4.984 ms .. 5.112 ms)
std dev              202.9 μs   (172.4 μs .. 231.3 μs)
variance introduced by outliers: 21% (moderately inflated)

benchmarking Tests/takeRowsVec 200
time                 2.412 ms   (2.334 ms .. 2.468 ms)
                     0.995 R²   (0.993 R² .. 0.996 R²)
mean                 2.233 ms   (2.204 ms .. 2.271 ms)
std dev              110.0 μs   (85.48 μs .. 130.7 μs)
variance introduced by outliers: 33% (moderately inflated)

benchmarking Tests/takeRowsVec 100
time                 1.487 ms   (1.470 ms .. 1.505 ms)
                     0.999 R²   (0.999 R² .. 1.000 R²)
mean                 1.485 ms   (1.478 ms .. 1.492 ms)
std dev              23.60 μs   (20.56 μs .. 30.05 μs)

benchmarking Tests/takeRowsStream 1000
time                 5.987 ms   (5.950 ms .. 6.045 ms)
                     0.999 R²   (0.999 R² .. 1.000 R²)
mean                 5.997 ms   (5.967 ms .. 6.027 ms)
std dev              88.64 μs   (72.50 μs .. 119.3 μs)

benchmarking Tests/takeRowsStream 500
time                 3.232 ms   (3.213 ms .. 3.257 ms)
                     1.000 R²   (1.000 R² .. 1.000 R²)
mean                 3.223 ms   (3.213 ms .. 3.236 ms)
std dev              34.91 μs   (29.45 μs .. 43.75 μs)

benchmarking Tests/takeRowsStream 200
time                 1.578 ms   (1.546 ms .. 1.618 ms)
                     0.992 R²   (0.989 R² .. 0.995 R²)
mean                 1.852 ms   (1.798 ms .. 1.906 ms)
std dev              192.1 μs   (171.4 μs .. 219.9 μs)
variance introduced by outliers: 72% (severely inflated)

benchmarking Tests/takeRowsStream 100
time                 1.039 ms   (1.026 ms .. 1.052 ms)
                     0.999 R²   (0.999 R² .. 1.000 R²)
mean                 1.032 ms   (1.027 ms .. 1.036 ms)
std dev              16.18 μs   (12.57 μs .. 21.83 μs)

For memory use, we just change the target: stack ghc -- -package streamly-0.6.0 -package vector-0.12.0.3 --make benchStreamly.hs -main-is weight -O2 -fspec-constr-recursive=10

yielding

Case                Allocated  GCs
Vec                 1,876,984    1
Stream              1,997,512    1
takeRowsVec 500     1,047,752    1
takeRowsVec 200     1,039,688    1
takeRowsVec 100     1,039,712    1
takeRowsStream 500  1,148,048    1
takeRowsStream 200  1,087,264    1
takeRowsStream 100  1,055,528    1

It's very possible I've let a foolish mistake somewhere. However Streamly does seem promising in terms of speed, at the cost of a slight increase in memory consumption.

Update from origin

Magalame · 2019-05-08T13:47:36Z

ping @ocramz

Magalame · 2019-05-08T13:49:29Z

Also the break in the tests seem unrelated to the PR. It looks like dh-core/dh-core has a dependency conflict, which is surprising since everything was fine at the last commit

ocramz

It's good, but I'd rather have a couple minor changes before merging. Thanks!

ocramz · 2019-05-08T08:54:51Z

analyze/analyze.cabal

        text ,
        unordered-containers ,
        vector ,  
+        weigh ,


criterion and weigh only need to be in the "library" section only if you want to run benchmarks in GHCi, this is why I put some imports under "DEBUG". They will all be removed from the "library" section when we release this.

Indeed! I didn't think it was actually going to be merged, I thought it was just to treat the benchmark as a separate issue!

ocramz · 2019-05-08T08:56:44Z

analyze/test/benchStreamly.hs

+{-# LANGUAGE GADTs #-}
+
+
+module Main where


could you move this into a new bench folder? test should be just for tests.

ocramz · 2019-05-08T08:56:59Z

analyze/test/benchStreamly.hs

+import qualified Weigh as W
+
+
+-- implementation details required


what does this mean?

I meant that below this line are the back-end details required to use Streamly

ocramz · 2019-05-08T13:51:03Z

analyze/test/benchStreamly.hs

+n = 1500
+
+testKeys :: Vector Text
+testKeys = unsafePerformIO $ V.replicateM n $ liftM (T.pack . take 10 . randomRs ('a','z')) newStdGen


What is unsafePerformIO for, here and in the following?

@ocramz Using unsafePerformIO the value of testKeys and testData is determined only once (and so we avoid the overhead of creating a new set of data for each function to test. Although a nicer way to do this would be to have said functions accept the data as input arguments, and then bind the data once in speed and in weight

…into bench-streamly

ocramz

OK, the only thing I'm concerned about is that Travis is red now. ~~Could you fix that?~~

I can't think of why this is broken now. Need some sleep first

Magalame · 2019-05-09T14:28:30Z

@ocramz
I found the reason why:
dh-core -> analyze -> criterion -> statistics-0.15.0.0 -> DLA-0.1.0.0
but now I'm not sure why analyze compile on itself

Anyway, since DLA is only used only from statistics-0.15.0.0, I'll just fix criterion to use an older version

somewhat related to #52

analyze -> criterion -> statistics-15.0.0.0 (from lts) -> DLA-0.1.0.0, while package locally is 0.2.0.0 so, we use statistics 0.14.0.2, which is allowed by criterion, and doesn't import DLA

Magalame · 2019-05-09T15:21:57Z

Also I removed the import of Debug.Trace I had inadvertently left in the source code in #53

Magalame · 2019-05-09T15:28:36Z

And it works!

ocramz · 2019-05-09T15:42:46Z

Very nice, thank you!

ocramz · 2019-05-09T21:07:08Z

Hi @Magalame , I just noticed that you forgot a few things from this PR.
First of all, the code wasn't called from anywhere; I missed this because I usually assume one runs stack bench continuously when working on a benchmark.

Please see these two commits I made to fix it :
4f25eb9
d3a5830

I'm glad to include your contributions, but please be more careful and submit fully-working code next time.

Magalame · 2019-05-09T22:06:42Z

@ocramz I'm very sorry to hear that, I'm fairly inexperienced with stack/Haskell tooling in general so I might miss some things, but I'll be extra careful

Magalame and others added 5 commits April 22, 2019 22:42

Merge pull request #1 from DataHaskell/master

8ccbf76

Update from origin

Benchmark comparing the performances of Vector with Streamly

f4fc50f

Added some comments, removed superfluous modifications

251141f

Changed back to lts-11.22

982232e

Downgrading streamly version because of breaks

20184e9

ocramz reviewed May 8, 2019

View reviewed changes

Magalame and others added 4 commits May 8, 2019 10:17

reorganised .cabal

a052765

Moved benchmark to its own folder

4a002d8

renamed benchmark

1eb0743

Merge branch 'bench-streamly' of https://github.com/Magalame/dh-core …

b93e67b

…into bench-streamly

ocramz requested changes May 8, 2019

View reviewed changes

Magalame and others added 2 commits May 9, 2019 10:34

deleted binary

d6b3a2f

attempt to fix build

2e58a14

analyze -> criterion -> statistics-15.0.0.0 (from lts) -> DLA-0.1.0.0, while package locally is 0.2.0.0 so, we use statistics 0.14.0.2, which is allowed by criterion, and doesn't import DLA

Remove unused Debug.Trace

efc0675

ocramz merged commit 5eb9a47 into DataHaskell:master May 9, 2019

ocramz mentioned this pull request May 9, 2019

analyze : evaluate streaming for RFrame #35

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Benchmarking Streamly backend #54

Benchmarking Streamly backend #54

Magalame commented May 8, 2019 •

edited

Magalame commented May 8, 2019

Magalame commented May 8, 2019

ocramz left a comment

ocramz May 8, 2019

Magalame May 8, 2019

ocramz May 8, 2019

Magalame May 8, 2019

ocramz May 8, 2019

Magalame May 8, 2019

ocramz May 8, 2019

Magalame May 8, 2019 •

edited

ocramz left a comment •

edited

Magalame commented May 9, 2019 •

edited

Magalame commented May 9, 2019

Magalame commented May 9, 2019

ocramz commented May 9, 2019

ocramz commented May 9, 2019

Magalame commented May 9, 2019

		import qualified Weigh as W


		-- implementation details required

Benchmarking Streamly backend #54

Benchmarking Streamly backend #54

Conversation

Magalame commented May 8, 2019 • edited

Magalame commented May 8, 2019

Magalame commented May 8, 2019

ocramz left a comment

Choose a reason for hiding this comment

ocramz May 8, 2019

Choose a reason for hiding this comment

Magalame May 8, 2019

Choose a reason for hiding this comment

ocramz May 8, 2019

Choose a reason for hiding this comment

Magalame May 8, 2019

Choose a reason for hiding this comment

ocramz May 8, 2019

Choose a reason for hiding this comment

Magalame May 8, 2019

Choose a reason for hiding this comment

ocramz May 8, 2019

Choose a reason for hiding this comment

Magalame May 8, 2019 • edited

Choose a reason for hiding this comment

ocramz left a comment • edited

Choose a reason for hiding this comment

Magalame commented May 9, 2019 • edited

Magalame commented May 9, 2019

Magalame commented May 9, 2019

ocramz commented May 9, 2019

ocramz commented May 9, 2019

Magalame commented May 9, 2019

Magalame commented May 8, 2019 •

edited

Magalame May 8, 2019 •

edited

ocramz left a comment •

edited

Magalame commented May 9, 2019 •

edited