Add micro benchmarks for serialising and applying Transactions #884

ghost · 2023-05-22T07:01:52Z

Following discussions on #186 this PR adds a few micro-benchmarks to clarify what is the performance bottleneck for applying transactions to Hydra Head. Here are the results on a recent MBP with M2 chip:

Running 1 benchmarks...
Benchmark micro: RUNNING...
benchmarking Cardano Ledger/Apply Tx
time                 281.6 μs   (280.6 μs .. 282.7 μs)
                     1.000 R²   (1.000 R² .. 1.000 R²)
mean                 281.3 μs   (280.9 μs .. 281.9 μs)
std dev              1.536 μs   (969.4 ns .. 2.170 μs)

benchmarking Cardano Ledger/Serialize NewTx (JSON)
time                 63.98 μs   (63.63 μs .. 64.61 μs)
                     1.000 R²   (0.999 R² .. 1.000 R²)
mean                 63.83 μs   (63.69 μs .. 64.44 μs)
std dev              743.5 ns   (165.1 ns .. 1.671 μs)

benchmarking Cardano Ledger/Serialize NewTx (CBOR)
time                 1.164 μs   (1.163 μs .. 1.166 μs)
                     1.000 R²   (1.000 R² .. 1.000 R²)
mean                 1.164 μs   (1.163 μs .. 1.166 μs)
std dev              3.917 ns   (2.369 ns .. 5.681 ns)

benchmarking Cardano Ledger/Deserialize NewTx (JSON)
time                 115.0 μs   (114.7 μs .. 115.1 μs)
                     1.000 R²   (1.000 R² .. 1.000 R²)
mean                 114.6 μs   (114.5 μs .. 114.8 μs)
std dev              391.6 ns   (286.9 ns .. 582.4 ns)

benchmarking Cardano Ledger/Deserialize NewTx (CBOR-in-JSON)
time                 11.72 μs   (11.70 μs .. 11.74 μs)
                     1.000 R²   (1.000 R² .. 1.000 R²)
mean                 11.70 μs   (11.69 μs .. 11.71 μs)
std dev              36.75 ns   (27.25 ns .. 57.05 ns)

Benchmark micro: FINISH

Interestingly, deserialising a JSON transaction takes about half the time of applying it so it seems we have some progression margin to improve the overall head performance.

CHANGELOG updated or not needed
[?] Documentation updated or not needed
Haddocks updated or not needed
No new TODOs introduced or explained herafter

github-actions · 2023-05-22T07:22:56Z

Test Results

307 tests ±0 301 ✔️ ±0 20m 49s ⏱️ +12s
104 suites ±0     6 💤 ±0
    5 files ±0     0 ❌ ±0

Results for commit 2119d9c. ± Comparison against base commit b7e90a6.

♻️ This comment has been updated with latest results.

github-actions · 2023-05-22T07:27:35Z

Transactions Costs

Sizes and execution budgets for Hydra protocol transactions. Note that unlisted parameters are currently using arbitrary values and results are not fully deterministic and comparable to previous runs.

Metadata
Generated at	2023-05-25 12:39:31.447689398 UTC
Max. memory units	14000000
Max. CPU units	10000000000
Max. tx size (kB)	16384

Script summary

Name	Hash	Size (Bytes)
νInitial	e5cc20df4a2216e706b3d00b59fbb15a7cf12dbd28d271d4a8cf6d04	4336
νCommit	47c102d5f95a0648b4065f2b8bff59d3e34536a82ee7b0d42df73123	2124
νHead	9fe3a5c4d826f9475368e1e24c15bf22f4df19893cce2689d3c0564a	9492
μHead	7bec671467e923281c92e94257931913be106e217270a7b97076cb9b*	4148

The minting policy hash is only usable for comparison. As the script is parameterized, the actual script is unique per Head.

Cost of Init Transaction

Parties	Tx size	% max Mem	% max CPU	Min fee ₳
1	4741	15.10	5.95	0.52
2	4944	17.19	6.73	0.56
3	5152	17.97	6.97	0.57
5	5562	24.59	9.54	0.66
10	6586	33.33	12.72	0.80
37	12127	98.76	37.46	1.75

Cost of Commit Transaction

Currently only one UTxO per commit allowed (this is about to change soon)

UTxO	Tx size	% max Mem	% max CPU	Min fee ₳
1	600	15.29	5.93	0.34

Cost of CollectCom Transaction

Parties	UTxO (bytes)	Tx size	% max Mem	% max CPU	Min fee ₳
1	57	816	27.42	10.68	0.48
2	114	1138	42.86	16.84	0.67
3	169	1458	61.14	24.18	0.88
4	226	1783	80.83	32.14	1.11

Cost of Close Transaction

Parties	Tx size	% max Mem	% max CPU	Min fee ₳
1	673	19.03	8.65	0.40
2	805	19.98	9.56	0.42
3	969	21.67	10.94	0.45
5	1299	24.46	13.45	0.50
10	2124	31.43	19.73	0.64
50	8733	87.17	69.99	1.74

Cost of Contest Transaction

Parties	Tx size	% max Mem	% max CPU	Min fee ₳
1	677	24.35	10.47	0.45
2	884	26.56	12.20	0.49
3	1005	27.78	13.21	0.51
5	1327	31.20	15.95	0.58
10	2161	39.77	22.80	0.73
45	7936	99.74	70.74	1.82

Cost of Abort Transaction

Some variation because of random mixture of still initial and already committed outputs.

Parties	Tx size	% max Mem	% max CPU	Min fee ₳
1	4856	22.24	9.34	0.61
2	5180	36.24	15.44	0.78
3	5502	53.13	22.83	0.99
4	5825	73.18	31.58	1.23
5	6148	95.88	41.52	1.49

Cost of FanOut Transaction

Involves spending head output and burning head tokens. Uses ada-only UTxO for better comparability.

Parties	UTxO	UTxO (bytes)	Tx size	% max Mem	% max CPU	Min fee ₳
5	0	0	4765	8.67	3.57	0.46
5	1	57	4796	10.06	4.39	0.47
5	5	285	4945	15.64	7.69	0.55
5	10	569	5124	22.61	11.82	0.64
5	20	1136	5480	36.57	20.08	0.83
5	30	1706	5846	50.53	28.34	1.02
5	40	2274	6202	64.49	36.60	1.21
5	50	2848	6564	78.46	44.87	1.40
5	65	3700	7102	99.42	57.28	1.68

ch1bo

Having these benchmarks will become handy.

Besides the comments below, should we run them in CI?

ch1bo · 2023-05-22T18:18:29Z

hydra-node/hydra-node.cabal

@@ -239,6 +238,27 @@ benchmark tx-cost
    -- NOTE(SN): should fix HLS choking on PlutusTx plugin
    ghc-options: -fplugin-opt PlutusTx.Plugin:defer-errors

+benchmark micro
+  import:         project-config
+  hs-source-dirs: exe/micro-bench


We usually keep benchmarks in the bench/ directory within a package.

Should: use bench/ as a source directory

Fair enough. The other benchmark we have is in the exe directory hence I followed suit. Happy to move them both.

You're right. tx-cost is different than the other 3 benchmarks we have on the repository. I'd say we should move all to bench/

ch1bo · 2023-05-22T18:21:50Z

hydra-node/exe/micro-bench/Main.hs

+      }
+
+defaultGlobals :: Ledger.Globals
+defaultGlobals =


Should: not create another location for fixtures

This and other symbols in here seems to have been copied from the hydra-node tests component. I'm not sure what weighs more, but maybe we want to have only a single set of fixtures? The epochInfo and pparams fixtures are coming from the hydra-node library already. Maybe we can create a more clear fixtures module which would make things less redundant and still not accidentally used?

Definitely copy-pasted, happy to move those to a dedicated module.

ghost · 2023-05-23T06:06:22Z

re-publication of benchmarks: It's trivial to output an html page so we could definitely do it, with the usual caveat those benchmarks are only meaningful to compare things as absolute values produced from running on a random VM environment are probably meaninglees.

ch1bo · 2023-05-23T07:00:17Z

re-publication of benchmarks: It's trivial to output an html page so we could definitely do it, with the usual caveat those benchmarks are only meaningful to compare things as absolute values produced from running on a random VM environment are probably meaninglees.

I don't mean to publish them, maybe just run them in CI first. But that would increase duration of it. That was just a thought, I'm fine with not doing anything about it.

ghost · 2023-05-23T08:00:23Z

running them without publishing or acting on them in some way is not very useful. I have added a job to publish them which should not add significant delay to the CI.

ch1bo

Approved, might need some fixing with recently changed way of assembling docs.

That way, we don't have to move the things later... although I'm not happy with the bench having to know too much about the doc structure but...

ghost requested review from ch1bo and ffakenz May 22, 2023 07:02

ch1bo requested changes May 22, 2023

View reviewed changes

ghost force-pushed the abailly-iohk/micro-benchmarks branch from 248d837 to 708fc48 Compare May 23, 2023 07:56

ghost requested a review from ch1bo May 23, 2023 08:00

ghost force-pushed the abailly-iohk/micro-benchmarks branch from 476659b to a089192 Compare May 23, 2023 09:10

ch1bo self-assigned this May 24, 2023

ch1bo approved these changes May 24, 2023

View reviewed changes

ffakenz approved these changes May 25, 2023

View reviewed changes

ch1bo assigned pgrange and unassigned ch1bo May 25, 2023

ch1bo mentioned this pull request May 25, 2023

Benchmark performance of a Hydra Head #186

Closed

6 tasks

abailly added 9 commits May 25, 2023 13:50

Add micro-benchmark assessing Tx application time

e2a6149

Add serialisation time to micro-benchmark

1116049

Move benchmarks to bench/ directory

f9cb3ca

Move Globals and default ledger env to shared Fixture module

e54f121

Publish micro-benchmarks results

eac885e

Fix micro benchmark run in Nix

f9d9be1

Fix publication of micro-benchmarks results

27b4ad0

Unneeded package in micro

4aed618

Cleanup micro benchmark's build

4de5005

pgrange force-pushed the abailly-iohk/micro-benchmarks branch from e4ae9ab to 83f46de Compare May 25, 2023 12:07

Let's put benchmark straight in the target directory

2119d9c

That way, we don't have to move the things later... although I'm not happy with the bench having to know too much about the doc structure but...

pgrange force-pushed the abailly-iohk/micro-benchmarks branch from 83f46de to 2119d9c Compare May 25, 2023 12:28

pgrange merged commit ed0c22d into master May 25, 2023

pgrange deleted the abailly-iohk/micro-benchmarks branch May 25, 2023 12:56

ch1bo added this to the 0.11.0 milestone Jun 27, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add micro benchmarks for serialising and applying Transactions #884

Add micro benchmarks for serialising and applying Transactions #884

ghost commented May 22, 2023

github-actions bot commented May 22, 2023 •

edited

Loading

github-actions bot commented May 22, 2023 •

edited

Loading

ch1bo left a comment

ch1bo May 22, 2023

ghost May 22, 2023

ch1bo May 23, 2023

ch1bo May 22, 2023

ghost May 22, 2023

ghost commented May 23, 2023

ch1bo commented May 23, 2023

ghost commented May 23, 2023

ch1bo left a comment

Add micro benchmarks for serialising and applying Transactions #884

Add micro benchmarks for serialising and applying Transactions #884

Conversation

ghost commented May 22, 2023

github-actions bot commented May 22, 2023 • edited Loading

Test Results

github-actions bot commented May 22, 2023 • edited Loading

Transactions Costs

Script summary

Cost of Init Transaction

Cost of Commit Transaction

Cost of CollectCom Transaction

Cost of Close Transaction

Cost of Contest Transaction

Cost of Abort Transaction

Cost of FanOut Transaction

ch1bo left a comment

Choose a reason for hiding this comment

ch1bo May 22, 2023

Choose a reason for hiding this comment

ghost May 22, 2023

Choose a reason for hiding this comment

ch1bo May 23, 2023

Choose a reason for hiding this comment

ch1bo May 22, 2023

Choose a reason for hiding this comment

ghost May 22, 2023

Choose a reason for hiding this comment

ghost commented May 23, 2023

ch1bo commented May 23, 2023

ghost commented May 23, 2023

ch1bo left a comment

Choose a reason for hiding this comment

github-actions bot commented May 22, 2023 •

edited

Loading

github-actions bot commented May 22, 2023 •

edited

Loading