Skip to content

Commit

Permalink
PLT-9014 Added CLI interface.
Browse files Browse the repository at this point in the history
  • Loading branch information
@bwbush
bwbush committed Jan 5, 2024
1 parent 064c7d7 commit 7cc7c24
Show file tree
Hide file tree
Showing 5 changed files with 145 additions and 25 deletions.
110 changes: 110 additions & 0 deletions marlowe-benchmark/ReadMe.md
View file
@@ -0,0 +1,110 @@
# Benchmarks for Marlowe Runtime

The `marlowe-benchmark` tool runs a series of benchmarks against Marlowe Runtime services.


## Usage

```bash
marlowe-benchmark --help
```

```console
marlowe-benchmark : execute Marlowe Runtime benchmarks

Usage: marlowe-benchmark [--version] [--host HOST] [--port PORT] [--config FILE]

This command-line tool for executing benchmarks for Marlowe Runtime.

Available options:
-h,--help Show this help text
--version Show version
--host HOST Host for Marlowe proxy service.
--port PORT Port for Marlowe proxy service.
--config FILE Name of benchmark configuration file.
```


## Configuration

The optional JSON configuration for benchmarking specifies the number clients run in parallel during benchmarking and the quantity of data processed.

| JSON Key | Haskell Type | Description |
|-------------------------|--------------|---------------------------------------------------------------------|
| `headerSyncParallelism` | `Int` | Number of parallel clients for `HeaderSync` protocol. |
| `headerMaxContracts` | `Int` | Maximum number of contracts to be read by each `HeaderSync` client. |
| `bulkParallelism` | `Int` | Number of parallel clients for the `BulkSync` protocol. |
| `bulkPageSize` | `Word8` | Number of blocks to fetch at a time for the `BulkSync` clients. |
| `bulkMaxBlocks` | `Int` | Maximum number of blocks to fetch for each `BulkSync` client. |
| `syncParallelism` | `Int` | Number of parallel clients for `Sync` protocol. |
| `syncBatchSize` | `Int` | Number of contracts to be read by each `Sync` client. |
| `queryParallelism` | `Int` | Number of parallel clients for `Query` protocol. |
| `queryBatchSize` | `Int` | Number of queries to be executed by each `Query` client. |
| `queryPageSize` | `Int` | Page size for each `Query` client. |



## Example output

The output is lines of JSON, with one report per benchmarking client.

```bash
marlowe-benchmark
```

```console
{"blocksPerSecond":11.771785516655209,"contractsPerSecond":15.269010744861106,"metric":"HeaderSync","seconds":574.169482653}
{"blocksPerSecond":11.770062718147265,"contractsPerSecond":15.266776128125029,"metric":"HeaderSync","seconds":574.253524544}
{"blocksPerSecond":11.771827242209914,"contractsPerSecond":15.269064866467572,"metric":"HeaderSync","seconds":574.167447494}
{"blocksPerSecond":11.771792207230698,"contractsPerSecond":15.269019423108674,"metric":"HeaderSync","seconds":574.16915632}
{"applyInputsPerSecond":942.8365324905556,"blocksPerSecond":978.2612577741164,"createsPerSecond":180.32465581089704,"metric":"BulkSync","seconds":37.487940679,"withdrawsPerSecond":20.806691055103503}
{"applyInputsPerSecond":945.7512794506212,"blocksPerSecond":981.2855190633082,"createsPerSecond":180.88212332964207,"metric":"BulkSync","seconds":37.372405164,"withdrawsPerSecond":20.871014230343317}
{"applyInputsPerSecond":927.177585961421,"blocksPerSecond":962.0139654820538,"createsPerSecond":177.3297632224984,"metric":"BulkSync","seconds":38.121068213,"withdrawsPerSecond":20.461126525672892}
{"applyInputsPerSecond":949.2986468390368,"blocksPerSecond":984.9661699116706,"createsPerSecond":181.5605843155153,"metric":"BulkSync","seconds":37.23275085,"withdrawsPerSecond":20.949298190251767}
{"contractsPerSecond":1.6611365871553612,"metric":"Sync","seconds":308.222697615,"stepsPerSecond":6.787300274080108}
{"contractsPerSecond":1.6320142779350748,"metric":"Sync","seconds":313.722745519,"stepsPerSecond":6.830872260966534}
{"contractsPerSecond":1.6615895324437109,"metric":"Sync","seconds":308.138676853,"stepsPerSecond":6.759943351719238}
{"contractsPerSecond":1.6544344963387472,"metric":"Sync","seconds":309.471303417,"stepsPerSecond":6.750221997757115}
{"contractsPerSecond":3072.685366781302,"metric":"Query","pagesPerSecond":12.266908616099643,"queriesPerSecond":0.35048310331713267,"query":"No policy ID","seconds":45.651273481}
{"contractsPerSecond":3083.3628083568774,"metric":"Query","pagesPerSecond":12.309535564331096,"queriesPerSecond":0.3517010161237456,"query":"No policy ID","seconds":45.493186731}
{"contractsPerSecond":3066.656834056088,"metric":"Query","pagesPerSecond":12.242841244663293,"queriesPerSecond":0.3497954641332369,"query":"No policy ID","seconds":45.741016224}
{"contractsPerSecond":3087.258954561369,"metric":"Query","pagesPerSecond":12.325089929240097,"queriesPerSecond":0.35214542654971703,"query":"No policy ID","seconds":45.435773955}
```

Tools like `jq` and `dasel` can convert the output to CSV files.

```bash
marlowe-benchmark > results.json
for x in Sync BulkSync HeaderSync Query
do
echo
jq -s 'map(select(.metric == "'$x'"))' result.json \
| dasel -r json -w csv --csv-comma $'\t'
done
```

```console
contractsPerSecond metric seconds stepsPerSecond
1.661137 Sync 308.222698 6.787300
1.632014 Sync 313.722746 6.830872
1.661590 Sync 308.138677 6.759943
1.654434 Sync 309.471303 6.750222

applyInputsPerSecond blocksPerSecond createsPerSecond metric seconds withdrawsPerSecond
942.836532 978.261258 180.324656 BulkSync 37.487941 20.806691
945.751279 981.285519 180.882123 BulkSync 37.372405 20.871014
927.177586 962.013965 177.329763 BulkSync 38.121068 20.461127
949.298647 984.966170 181.560584 BulkSync 37.232751 20.949298

blocksPerSecond contractsPerSecond metric seconds
11.771786 15.269011 HeaderSync 574.169483
11.770063 15.266776 HeaderSync 574.253525
11.771827 15.269065 HeaderSync 574.167447
11.771792 15.269019 HeaderSync 574.169156

contractsPerSecond metric pagesPerSecond queriesPerSecond query seconds
3072.685367 Query 12.266909 0.350483 No policy ID 45.651273
3083.362808 Query 12.309536 0.351701 No policy ID 45.493187
3066.656834 Query 12.242841 0.349795 No policy ID 45.741016
3087.258955 Query 12.325090 0.352145 No policy ID 45.435774
```
12 changes: 5 additions & 7 deletions marlowe-benchmark/app/Language/Marlowe/Runtime/Benchmark/Query.hs
View file
Expand Up @@ -16,10 +16,8 @@ import Data.List.Split (chunksOf)
import Data.Time.Clock (NominalDiffTime)
import Data.Time.Clock.POSIX (getPOSIXTime)
import GHC.Generics (Generic)
import Language.Marlowe.Protocol.Query.Client (MarloweQueryClient)
import qualified Language.Marlowe.Protocol.Query.Client as Query
import Language.Marlowe.Protocol.Query.Types (ContractFilter)
import qualified Language.Marlowe.Protocol.Query.Types as Query
import Language.Marlowe.Protocol.Query.Client (MarloweQueryClient, getContractHeaders)
import Language.Marlowe.Protocol.Query.Types (ContractFilter, Order (Ascending), Page (..), Range (..))
import Language.Marlowe.Runtime.Client (runMarloweQueryClient)
import UnliftIO (forConcurrently)

Expand Down Expand Up @@ -98,9 +96,9 @@ benchmark
-> MarloweQueryClient m Statistics
-- ^ Action to run the benchmark.
benchmark start pageSize initial@Statistics{queries} cFilter =
let accumulate = (. Query.getContractHeaders cFilter) . (=<<) . handleNextPage
let accumulate = (. getContractHeaders cFilter) . (=<<) . handleNextPage
handleNextPage stats Nothing = pure stats
handleNextPage stats@Statistics{pages, contracts} (Just Query.Page{..}) =
handleNextPage stats@Statistics{pages, contracts} (Just Page{..}) =
do
now <- liftIO getPOSIXTime
let stats' =
Expand All @@ -112,4 +110,4 @@ benchmark start pageSize initial@Statistics{queries} cFilter =
case nextRange of
Nothing -> pure stats'
Just range -> stats' `accumulate` range
in initial{queries = queries + 1} `accumulate` Query.Range Nothing 0 pageSize Query.Ascending
in initial{queries = queries + 1} `accumulate` Range Nothing 0 pageSize Ascending
8 changes: 4 additions & 4 deletions marlowe-benchmark/app/Language/Marlowe/Runtime/Benchmark/Sync.hs
View file
Expand Up @@ -21,11 +21,11 @@ import Data.Time.Clock.POSIX (getPOSIXTime)
import Data.Type.Equality ((:~:) (Refl))
import GHC.Generics (Generic)
import Language.Marlowe.Protocol.Sync.Client (
ClientStFollow (ClientStFollow, recvMsgContractFound, recvMsgContractNotFound),
ClientStIdle (SendMsgDone, SendMsgRequestNext),
ClientStInit (SendMsgFollowContract),
ClientStFollow (..),
ClientStIdle (..),
ClientStInit (..),
ClientStNext (..),
ClientStWait (SendMsgCancel),
ClientStWait (..),
MarloweSyncClient (MarloweSyncClient),
)
import Language.Marlowe.Runtime.Client (runMarloweSyncClient)
Expand Down
38 changes: 24 additions & 14 deletions marlowe-benchmark/app/Main.hs
View file
@@ -1,5 +1,3 @@
{-# LANGUAGE LambdaCase #-}

-- | Execute Benchmarks.
module Main (
-- * Entry point
Expand All @@ -8,22 +6,34 @@ module Main (

import Data.Aeson (eitherDecodeFileStrict')
import Data.Default (def)
import Data.Version (showVersion)
import Language.Marlowe.Runtime.Benchmark (measure)
import Language.Marlowe.Runtime.Client (connectToMarloweRuntime)
import System.Environment (getArgs)
import Paths_marlowe_benchmark (version)

import qualified Options.Applicative as O

-- | Execute the benchmarks.
main :: IO ()
main =
do
(host, port, config) <-
getArgs >>= \case
[] -> pure ("localhost", 3700, def)
[h] -> pure (h, 3700, def)
[h, p] -> pure (h, read p, def)
[h, p, c] ->
eitherDecodeFileStrict' c >>= \case
Right c' -> pure (h, read p, c')
Left e -> error e
_ -> error "USAGE: marlowe-benchmark [host [port [configfile]]]"
connectToMarloweRuntime host port $ measure config
(host, port, config) <- O.execParser commandParser
config' <-
case config of
Nothing -> pure def
Just config'' -> either error id <$> eitherDecodeFileStrict' config''
connectToMarloweRuntime host (fromIntegral port) $ measure config'

commandParser :: O.ParserInfo (String, Int, Maybe FilePath)
commandParser =
let commandOptions =
(,,)
<$> O.strOption (O.long "host" <> O.value "localhost" <> O.metavar "HOST" <> O.help "Host for Marlowe proxy service.")
<*> O.option O.auto (O.long "port" <> O.value 3700 <> O.metavar "PORT" <> O.help "Port for Marlowe proxy service.")
<*> (O.optional . O.strOption) (O.long "config" <> O.metavar "FILE" <> O.help "Path to the benchmark configuration file.")
in O.info
(O.helper <*> (O.infoOption (showVersion version) $ O.long "version" <> O.help "Show version") <*> commandOptions)
( O.fullDesc
<> O.progDesc "This command-line tool executes benchmarks for Marlowe Runtime."
<> O.header "marlowe-benchmark : execute Marlowe Runtime benchmarks"
)
2 changes: 2 additions & 0 deletions marlowe-benchmark/marlowe-benchmark.cabal
View file
Expand Up @@ -66,6 +66,7 @@ executable marlowe-benchmark
Language.Marlowe.Runtime.Benchmark.HeaderSync
Language.Marlowe.Runtime.Benchmark.Query
Language.Marlowe.Runtime.Benchmark.Sync
Paths_marlowe_benchmark

build-depends:
, aeson
Expand All @@ -75,6 +76,7 @@ executable marlowe-benchmark
, data-default
, marlowe-client
, marlowe-runtime:{marlowe-runtime, discovery-api, history-api, sync-api}
, optparse-applicative
, split
, time
, unliftio

0 comments on commit 7cc7c24

Please sign in to comment.