Skip to content

Commit

Permalink
add trie-simple benchmarks
Browse files Browse the repository at this point in the history
  • Loading branch information
@ocramz
ocramz committed May 18, 2019
1 parent 70ea065 commit 881be8a
Show file tree
Hide file tree
Showing 6 changed files with 132 additions and 81 deletions.
187 changes: 111 additions & 76 deletions README.md
View file
Expand Up @@ -4,133 +4,168 @@

Performance shootout of various prefix tree ("trie") implementations.

"space" benchmarking performed with `weigh` and "time" benchmarks were done with `criterion`.
"space" benchmarking performed with `weigh` and "time" benchmarks done with `criterion`.

Currently comparing two unoptimized implementations taken (but slightly modified) from didactic blogposts (https://alexandersgreen.wordpress.com/2010/09/13/prefix-trees-in-haskell/ and https://blog.jle.im/entry/tries-with-recursion-schemes.html) and two available on Hackage ([`generic-trie`](https://hackage.haskell.org/package/generic-trie) and [`bytestring-trie`](https://hackage.haskell.org/package/bytestring-trie)).
Currently comparing two unoptimized implementations taken (but slightly modified) from didactic blogposts (https://alexandersgreen.wordpress.com/2010/09/13/prefix-trees-in-haskell/ and https://blog.jle.im/entry/tries-with-recursion-schemes.html) and three available on Hackage ([`generic-trie`](https://hackage.haskell.org/package/generic-trie), [`bytestring-trie`](https://hackage.haskell.org/package/bytestring-trie), and [`trie-simple`](https://hackage.haskell.org/package/trie-simple)).

`generic-trie` seems to be the best choice, at least for a "lookup - fromList" pair, but I was curious to see how very diverse implementation techniques, notably one based on recursion schemes and another using an arrow type internally, lead to different space and time scaling behaviours.

Note : I am still somewhat inexperienced with Haskell benchmarking so the numbers below should be taken with a grain of salt.

Contributing : I would love to receive pull requests with extended and/or improved benchmarks.
## Contributing

Pull requests that extend and/or improve these benchmarks are very welcome.

Results from a laptop GHCi session :

Results obtained on a 2015 MacBook Pro GHCi session :


```
Benchmark space: RUNNING...
AG
Case Allocated GCs
small 1,208 0
medium 12,960 0
large 133,008 0
small 1,360 0
medium 13,112 0
large 133,752 0
JL
Case Allocated GCs
small 5,816 0
medium 60,240 0
large 637,824 0
small 5,928 0
medium 61,824 0
large 637,056 0
generic-trie
Case Allocated GCs
small 4,320 0
medium 24,752 0
large 222,320 0
small 6,416 0
medium 25,608 0
large 218,344 0
bytestring-trie
Case Allocated GCs
small 2,432 0
medium 52,712 0
large 2,600,152 2
medium 52,656 0
large 2,598,008 2
trie-simple
Case Allocated GCs
small 2,992 0
medium 201,784 0
large 18,895,048 18
Benchmark space: FINISH
```

```
Benchmark time: RUNNING...
benchmarking AG/small
time 181.5 ns (176.1 ns .. 185.9 ns)
0.996 R² (0.994 R² .. 0.997 R²)
mean 178.4 ns (174.9 ns .. 181.7 ns)
std dev 10.86 ns (9.374 ns .. 12.77 ns)
variance introduced by outliers: 77% (severely inflated)
time 175.6 ns (174.5 ns .. 177.3 ns)
1.000 R² (0.999 R² .. 1.000 R²)
mean 176.4 ns (175.5 ns .. 178.1 ns)
std dev 4.243 ns (2.521 ns .. 7.114 ns)
variance introduced by outliers: 34% (moderately inflated)
benchmarking AG/medium
time 5.185 μs (5.007 μs .. 5.362 μs)
0.993 R² (0.991 R² .. 0.995 R²)
mean 5.289 μs (5.155 μs .. 5.431 μs)
std dev 451.8 ns (388.5 ns .. 611.0 ns)
variance introduced by outliers: 83% (severely inflated)
time 5.278 μs (5.249 μs .. 5.316 μs)
0.999 R² (0.999 R² .. 1.000 R²)
mean 5.320 μs (5.267 μs .. 5.535 μs)
std dev 306.5 ns (102.7 ns .. 669.0 ns)
variance introduced by outliers: 69% (severely inflated)
benchmarking AG/large
time 103.6 μs (101.1 μs .. 106.6 μs)
0.993 R² (0.989 R² .. 0.996 R²)
mean 105.6 μs (103.4 μs .. 107.9 μs)
std dev 7.921 μs (6.453 μs .. 9.934 μs)
variance introduced by outliers: 71% (severely inflated)
time 87.55 μs (86.39 μs .. 88.67 μs)
0.999 R² (0.999 R² .. 1.000 R²)
mean 87.88 μs (86.85 μs .. 90.20 μs)
std dev 4.967 μs (1.965 μs .. 8.544 μs)
variance introduced by outliers: 59% (severely inflated)
benchmarking JL/small
time 950.7 ns (914.4 ns .. 986.9 ns)
0.991 R² (0.988 R² .. 0.994 R²)
mean 968.5 ns (937.7 ns .. 1.003 μs)
std dev 117.4 ns (90.40 ns .. 164.0 ns)
variance introduced by outliers: 92% (severely inflated)
time 1.060 μs (1.055 μs .. 1.065 μs)
0.999 R² (0.999 R² .. 1.000 R²)
mean 1.074 μs (1.061 μs .. 1.115 μs)
std dev 70.53 ns (22.19 ns .. 141.5 ns)
variance introduced by outliers: 78% (severely inflated)
benchmarking JL/medium
time 15.88 μs (15.37 μs .. 16.39 μs)
0.994 R² (0.992 R² .. 0.996 R²)
mean 16.00 μs (15.62 μs .. 16.40 μs)
std dev 1.332 μs (1.165 μs .. 1.573 μs)
variance introduced by outliers: 80% (severely inflated)
time 15.38 μs (15.27 μs .. 15.52 μs)
0.998 R² (0.997 R² .. 0.999 R²)
mean 15.65 μs (15.46 μs .. 15.90 μs)
std dev 756.3 ns (545.8 ns .. 1.075 μs)
variance introduced by outliers: 57% (severely inflated)
benchmarking JL/large
time 102.2 μs (99.34 μs .. 104.8 μs)
0.995 R² (0.993 R² .. 0.997 R²)
mean 98.88 μs (97.04 μs .. 100.9 μs)
std dev 6.480 μs (5.649 μs .. 7.350 μs)
variance introduced by outliers: 65% (severely inflated)
time 89.09 μs (87.90 μs .. 90.15 μs)
0.997 R² (0.994 R² .. 0.999 R²)
mean 91.01 μs (88.69 μs .. 97.55 μs)
std dev 11.52 μs (5.425 μs .. 24.76 μs)
variance introduced by outliers: 88% (severely inflated)
benchmarking generic-trie/small
time 540.2 ns (524.5 ns .. 559.1 ns)
0.994 R² (0.991 R² .. 0.997 R²)
mean 564.8 ns (552.7 ns .. 579.3 ns)
std dev 43.96 ns (34.92 ns .. 58.14 ns)
variance introduced by outliers: 84% (severely inflated)
time 469.9 ns (468.0 ns .. 472.9 ns)
1.000 R² (0.999 R² .. 1.000 R²)
mean 475.3 ns (471.0 ns .. 487.6 ns)
std dev 23.10 ns (7.637 ns .. 49.55 ns)
variance introduced by outliers: 66% (severely inflated)
benchmarking generic-trie/medium
time 6.480 μs (6.294 μs .. 6.653 μs)
0.994 R² (0.991 R² .. 0.997 R²)
mean 6.411 μs (6.252 μs .. 6.612 μs)
std dev 576.4 ns (485.7 ns .. 785.2 ns)
variance introduced by outliers: 84% (severely inflated)
time 6.506 μs (6.400 μs .. 6.647 μs)
0.998 R² (0.996 R² .. 0.999 R²)
mean 6.563 μs (6.476 μs .. 6.725 μs)
std dev 400.6 ns (242.9 ns .. 685.1 ns)
variance introduced by outliers: 71% (severely inflated)
benchmarking generic-trie/large
time 69.79 μs (68.15 μs .. 72.17 μs)
0.993 R² (0.990 R² .. 0.997 R²)
mean 73.61 μs (71.47 μs .. 76.05 μs)
std dev 7.236 μs (5.887 μs .. 9.019 μs)
variance introduced by outliers: 82% (severely inflated)
time 69.24 μs (68.48 μs .. 70.16 μs)
0.998 R² (0.995 R² .. 0.999 R²)
mean 70.59 μs (69.17 μs .. 76.47 μs)
std dev 8.090 μs (2.274 μs .. 17.78 μs)
variance introduced by outliers: 86% (severely inflated)
benchmarking bytestring-trie/small
time 353.1 ns (344.1 ns .. 363.8 ns)
0.994 R² (0.993 R² .. 0.996 R²)
mean 361.5 ns (350.2 ns .. 383.7 ns)
std dev 51.29 ns (28.04 ns .. 90.24 ns)
variance introduced by outliers: 95% (severely inflated)
time 319.3 ns (317.2 ns .. 322.8 ns)
0.999 R² (0.998 R² .. 1.000 R²)
mean 322.1 ns (319.5 ns .. 326.4 ns)
std dev 10.69 ns (7.791 ns .. 13.87 ns)
variance introduced by outliers: 48% (moderately inflated)
benchmarking bytestring-trie/medium
time 4.707 μs (4.543 μs .. 4.858 μs)
0.992 R² (0.990 R² .. 0.995 R²)
mean 4.729 μs (4.606 μs .. 4.890 μs)
std dev 464.3 ns (388.5 ns .. 685.7 ns)
variance introduced by outliers: 87% (severely inflated)
time 4.354 μs (4.292 μs .. 4.448 μs)
0.997 R² (0.994 R² .. 1.000 R²)
mean 4.371 μs (4.324 μs .. 4.479 μs)
std dev 225.9 ns (146.2 ns .. 369.1 ns)
variance introduced by outliers: 64% (severely inflated)
benchmarking bytestring-trie/large
time 99.09 μs (94.92 μs .. 102.1 μs)
0.991 R² (0.988 R² .. 0.995 R²)
mean 97.70 μs (95.20 μs .. 100.7 μs)
std dev 9.242 μs (7.809 μs .. 11.49 μs)
variance introduced by outliers: 80% (severely inflated)
time 90.30 μs (89.40 μs .. 91.22 μs)
0.999 R² (0.999 R² .. 1.000 R²)
mean 89.86 μs (89.33 μs .. 90.66 μs)
std dev 2.082 μs (1.396 μs .. 3.239 μs)
variance introduced by outliers: 19% (moderately inflated)
benchmarking trie-simple/small
time 402.4 ns (387.5 ns .. 422.7 ns)
0.984 R² (0.970 R² .. 0.998 R²)
mean 395.3 ns (385.9 ns .. 415.2 ns)
std dev 43.83 ns (25.89 ns .. 70.51 ns)
variance introduced by outliers: 92% (severely inflated)
benchmarking trie-simple/medium
time 33.17 μs (32.77 μs .. 33.71 μs)
0.999 R² (0.998 R² .. 1.000 R²)
mean 33.23 μs (33.04 μs .. 33.54 μs)
std dev 814.7 ns (537.0 ns .. 1.141 μs)
variance introduced by outliers: 23% (moderately inflated)
benchmarking trie-simple/large
time 19.79 ms (18.73 ms .. 21.76 ms)
0.969 R² (0.921 R² .. 0.998 R²)
mean 20.43 ms (19.77 ms .. 21.49 ms)
std dev 1.875 ms (1.201 ms .. 2.877 ms)
variance introduced by outliers: 41% (moderately inflated)
Benchmark time: FINISH
```
8 changes: 6 additions & 2 deletions bench/Space.hs
View file
Expand Up @@ -3,7 +3,7 @@ module Main where
import Weigh (Grouped (..), Weight (..), Weigh, mainWith, wgroup, func, io, commas, weighResults)
import System.Random.MWC.Probability (withSystemRandom)

import Bench.Trie (randTrieInputString, randTrieInputBS, ag, jl, gt, bt)
import Bench.Trie (randTrieInputString, randTrieInputBS, ag, jl, gt, bt, ts)


main :: IO ()
Expand All @@ -30,5 +30,9 @@ main = withSystemRandom $ \g -> do
wgroup "bytestring-trie" $ do
func "small" bt bssmall
func "medium" bt bsmedium
func "large" bt bslarge
func "large" bt bslarge
wgroup "trie-simple" $ do
func "small" ts ssmall
func "medium" ts smedium
func "large" ts slarge

9 changes: 7 additions & 2 deletions bench/Time.hs
View file
Expand Up @@ -14,7 +14,7 @@ import Criterion.Monad (withConfig)
-- import Control.Monad.Primitive
import System.Random.MWC.Probability (withSystemRandom)

import Bench.Trie (randTrieInputString, randTrieInputBS, ag, jl, gt, bt)
import Bench.Trie (randTrieInputString, randTrieInputBS, ag, jl, gt, bt, ts)


main :: IO ()
Expand Down Expand Up @@ -45,7 +45,12 @@ main = withSystemRandom $ \g -> do
bench "small" $ whnf bt bssmall
, bench "medium" $ whnf bt bsmedium
, bench "large" $ whnf bt bslarge
]
] ,
bgroup "trie-simple" [
bench "small" $ whnf ts ssmall
, bench "medium" $ whnf ts smedium
, bench "large" $ whnf ts slarge
]
]


Expand Down
6 changes: 5 additions & 1 deletion src/Bench/Trie.hs
View file
Expand Up @@ -13,7 +13,7 @@ import qualified Data.Trie.JustinLe as JL (Trie, fromList, lookup)

import qualified "generic-trie" Data.GenericTrie as GT (Trie, fromList, lookup, TrieKey)
import qualified "bytestring-trie" Data.Trie as BST

import qualified "trie-simple" Data.Trie.Map as TST (TMap, fromList, lookup)

letters :: [Char]
letters = ['a' .. 'z']
Expand Down Expand Up @@ -72,3 +72,7 @@ gt (kxs, k) = GT.lookup k $ GT.fromList kxs
-- | bytestring-trie
bt :: ([(B.ByteString, a)], B.ByteString) -> Maybe a
bt (kxs, k) = BST.lookup k $ BST.fromList kxs

-- | trie-simple
ts :: Ord c => ([([c], a)], [c]) -> Maybe a
ts (kxs, k) = TST.lookup k $ TST.fromList kxs
2 changes: 2 additions & 0 deletions stack.yaml
View file
Expand Up @@ -6,6 +6,8 @@ packages:
extra-deps:
- generic-trie-0.3.1@sha256:203cfc0086372abaf99ffb4b4b565eba8a28e6020f1104ec479cd96d0c3baeb7
- bytestring-trie-0.2.5.0@sha256:96498959cf2af0e3f1f3dfb526b78502c9fa8f21255a3506938ee2a26e61d653
- trie-simple-0.4.1.1@sha256:ff23f3092a53f1275180f32d2475d4009a33939eb04a2869a1308e5e80e8651b
- containers-0.5.11.0@sha256:28ad7337057442f75bc689315ab4ec7bdf5e6b2c39668f306672cecd82c02798

# Override default flag values for local packages and extra-deps
# flags: {}
Expand Down
1 change: 1 addition & 0 deletions trie-perf.cabal
View file
Expand Up @@ -36,6 +36,7 @@ library
, generic-trie
, mwc-probability
, primitive
, trie-simple
if flag(Time)
exposed-modules: Bench.Trie.Time
build-depends: criterion >= 1.5 && < 1.6,
Expand Down

0 comments on commit 881be8a

Please sign in to comment.