Benchmark performance of a Hydra Head

[ch1bo]

Why

Users want to know what is capable with Hydra in certain use cases.

What

For that, we want to extend our current benchmark suite with a concrete scenario to measure latency & throughput of actual Hydra Head configuration and transaction structure. The results shall be collected and made available on a continuous basis.

A scenario description should always include

• Hydra node topology - number of nodes & clients
• UTXO structure - what is committed into the Head & processed in the Head
• Transaction structure - what kind of transactions to be processed in the Head

The first scenario which we want to cover was contributed by @Trymyrt below: #186 (comment)

How

• Run the existing benchmarks
• Create micro benchmarks for individual stages of processing tx Add micro benchmarks for serialising and applying Transactions #884
• Update the benchmark generating a dataset representing the scenario and using CBOR tx submission
• Make scenarios comparable easily
  • 🤔 What's this already?
• Discuss results with early adopters to validate communication (surprises etc.)
  • Early results of 50-100tps range appear way off expectations set by previous communication of 1000tps
• Acquire and publish benchmark results like we do it for the transaction costs (on PR, into docs)
  • dataset information (it likely will be generated on each run)
  • summarize average confirmation time
  • Don't point the public at this (yet)

TBD

• Use our dedicated github runner? or shall we try with standard runners and see our standard deviation?

[ch1bo]

Please leave a comment if you are particularly interested in some scenarios!

[ch1bo]

We need to sharpen this. It will be quite hard to make this actually usable for more than one specific use case. However, I think this is quite useful to establish a baseline. Marking this as an idea until we have clarified concrete steps.

[Trymyrt]

Description of benchmarking use case:

• Hydra node topology - number of nodes & clients
  Single node, centrally operated
• UTXO structure - what is committed into the Head & processed in the Head
  The initial commit can be pretty minimal/none. The idea is that the operator opens the head, does most of the validation off-chain and use Hydra as more of a record keeping tool. Let's assume a single "state" UTxO goes in and is used to close.
• Transaction structure - what kind of transactions to be processed in the Head
  The transactions going on in the head would primarily be broken up into 2 main transactions:

1. Mint user tokens
   When a user performs an action which is validated off-chain (for example send funds to a wallet or smart contract on main net to "top up" their account), a user token is minted. Each interacting user would have their own associated token, and the amounts would be linked to the amount they have committed. The mint tx would likely have an optional input that is the user token UTxO to make sure that each user only have a single UTxO
2. Burn user tokens
   The user input UTxO containing X tokens, would be split into B (tokens to be burned) and Y (remainder, ie X-B). The amount burned is based on the off-chain logic.

It's a pretty general framework which is very much the purpose, the point of this application is not to create the most verifiable and permissionless system in the world, but to really lean into centralization and lax security to boost scalability/throughput as much as at it would make any sense while still calling it a blockchain application.

[ch1bo]

Thanks @Trymyrt for helping conretizing this. I can't promise when we'll address your scenario, but it makes this thing more relevant!

[abailly-iohk]

@Trymyrt So the scenario is basically a single client, posting one tx at a time consuming and producing a single UTxO, to a single Hydra-node? We normally compute the tps for confirmed txs, eg. transactions which are part of a snapshot. This is mostly what users or apps would want because it's only when it's confirmed (eg. in a block) that a tx is definitely "adopted".

[abailly-iohk]

@Trymyrt I ran the existing benchmark with 1 node/1 client and a bunch of txs and here are some results:

Starting benchmark
Seeding network
Initializing Head
Comitting initialUTxO from dataset
HeadIsOpen
Client 1 (node 0): 0/3000 (0.00%)
Client 1 (node 0): 417/3000 (13.90%)
Client 1 (node 0): 837/3000 (27.90%)
Client 1 (node 0): 1251/3000 (41.70%)
Client 1 (node 0): 1658/3000 (55.27%)
Client 1 (node 0): 2067/3000 (68.90%)
Client 1 (node 0): 2478/3000 (82.60%)
Client 1 (node 0): 2792/3000 (93.07%)
All transactions confirmed. Sweet!
Closing the Head
Finalizing the Head
Writing results to: /tmp/bench-eed48b5feffb6ec6/results.csv
Confirmed txs: 3000
Average confirmation time: 1.2410800227999999e-2
Confirmed below 1 sec: 99.96666666666667%
Load test on 1 local nodes in /tmp/bench-eed48b5feffb6ec6 [✔]

The latency of 12.4 s translates to a throughput of 80tps.

One thing we, or even you, could easily do is to have a custom set of transactions as a workload should you want more accurate figures for specific tx types (eg. with minting). The Dataset structure is relatively simple, it's an initial transaction (representing the CollectCom) and then a list of ClientDataset representing the workload of parallel clients submitting txs in sequence (and waiting for each to be confirmed).

[abailly-iohk]

Here is the graph for 15k transactions running on my MBP M2 laptop (slightly faster than a VM). Results are similar, a bit over 100 tps

[abailly-iohk]

Running a micro-benchmark to measure the time it takes to apply a transaction to the ledger, I get the following for 10,000 txs

Building benchmark 'micro' for hydra-node-0.11.0..
Running 1 benchmarks...
Benchmark micro: RUNNING...
benchmarking Cardano Ledger/Apply Tx
time                 1.691 s    (1.579 s .. 1.777 s)
                     1.000 R²   (0.998 R² .. 1.000 R²)
mean                 1.693 s    (1.684 s .. 1.709 s)
std dev              15.92 ms   (324.7 μs .. 19.84 ms)
variance introduced by outliers: 19% (moderately inflated)

This means applying a single tx takes about 170 microsecond. There's of course a significant overhead in the whole roundtrip process of Submitting a tx, applying it to the L2 ledger, creating a snapshot, returning the snapshot confirmed if only because of serialisation and network communications overhead.

[abailly-iohk]

I am experimenting with changes to the serialisation (Eg. using CBOR instead of JSON in the API Server), but facing the same low-level issue I faced earlier with the client disconnecting abruptly while waiting to receive notification of the close.

[abailly-iohk]

After a long struggle, it seems we finally are onto something. Here is a run that demonstrates about 1000tps for single head/single node/single client case with some bottlenecks removed:

And here is typical duration trace of main events/effects in the hydra-node:

{"effect":"ReqTx","id":"acf84e1a3ed08e14c2b5ee946031d47daf557cb646d51b39ab2b13c0bb335ad6","tag":"TraceEffect","timestamp":"2023-06-08T11:41:27.005417379Z","us":2.735}
{"event":"NewTx","id":"acf84e1a3ed08e14c2b5ee946031d47daf557cb646d51b39ab2b13c0bb335ad6","tag":"TraceEvent","timestamp":"2023-06-08T11:41:27.005414383Z","us":6.603}
{"effect":"TxValid","id":"acf84e1a3ed08e14c2b5ee946031d47daf557cb646d51b39ab2b13c0bb335ad6","tag":"TraceEffect","timestamp":"2023-06-08T11:41:27.00561477Z","us":4.048}
{"event":"ReqTx","id":"acf84e1a3ed08e14c2b5ee946031d47daf557cb646d51b39ab2b13c0bb335ad6","tag":"TraceEvent","timestamp":"2023-06-08T11:41:27.005431055Z","us":188.695}
{"event":"ReqSn","id":"acf84e1a3ed08e14c2b5ee946031d47daf557cb646d51b39ab2b13c0bb335ad6","tag":"TraceEvent","timestamp":"2023-06-08T11:41:27.005621153Z","us":165.83}
{"effect":"SnapshotConfirmed","id":"acf84e1a3ed08e14c2b5ee946031d47daf557cb646d51b39ab2b13c0bb335ad6","tag":"TraceEffect","timestamp":"2023-06-08T11:41:27.013200489Z","us":4.759}

Contention came mainly from 1. the way we persist the state, 2. lack of actual parallelism, and 3. inefficient logging of full state.

[abailly-iohk]

Latest run of benchmark without any change in Server API serialisation:

persistence state	persistence output	outcome log	tx confirmation (ms)
Y	Y	Y	31
N	Y	Y	2.2
N	N	Y	1.9
N	N	N	0.9

=> No need to switch to CBOR now in Client <-> Server communications