Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use a single client per node in benchmarks #910

Merged
merged 1 commit into from
Jun 9, 2023

Conversation

abailly-iohk
Copy link
Contributor

A while ago we introduced the withNewClient function to submit transactions in the benchmark because we wanted to increase concurrency and be able to load the server with more than one client. We ended up dropping that capability from the benchmarks but kept having a separate connection between the main body of the benchmark and the transaction submission process.

This actually introduces a subtle bug that does only surface for sufficiently large loads. Here is what's happening:

  1. The benchmark opens a connection to control the head
  2. For each separate list of transaction in the workload, we open a connection and starts sending txs and reading results
  3. Because all Server outputs are broadcast to all clients, the first connection is also sent the messages but does not read it
  4. After a while, which depends on parameters of the machine, the TCP buffer is full and the TCP protocol triggers a close of the server -> client part of the connection
  5. The client -> server part is still valid so the client can send the close once the transaction processing ends
  6. But as soon as we want to receive something, we start reading stale messages accumulated in some buffer and then it the closed connection

  • CHANGELOG updated or not needed
  • Documentation updated or not needed
  • Haddocks updated or not needed
  • No new TODOs introduced or explained herafter

A while ago we introduced the `withNewClient` function to submit
transactions in the benchmark because we wanted to increase
concurrency and be able to load the server with more than one
client. We ended up dropping that capability from the benchmarks but
kept having a separate connection between the main body of the
benchmark and the transaction submission process.

This actually introduces a subtle bug that does only surface for
sufficiently large loads. Here is what's happening:
1. The benchmark opens a connection to control the head
2. For each separate list of transaction in the workload, we open a
connection and starts sending txs and reading results
3. Because all Server outputs are broadcast to all clients, the first
connection is also sent the messages but does not read it
4. After a while, which depends on parameters of the machine, the TCP
buffer is full and the TCP protocol triggers a close of the server ->
client part of the connection
5. The client -> server part is still valid so the client can send the
close once the transaction processing ends
6. But as soon as we want to receive something, we start reading stale
messages accumulated in some buffer and then it the closed connection
@abailly-iohk abailly-iohk requested review from pgrange and ch1bo and removed request for pgrange June 6, 2023 17:04
@ch1bo ch1bo requested review from pgrange, v0d1ch and ffakenz June 6, 2023 17:05
@github-actions
Copy link

github-actions bot commented Jun 6, 2023

Transactions Costs

Sizes and execution budgets for Hydra protocol transactions. Note that unlisted parameters are currently using arbitrary values and results are not fully deterministic and comparable to previous runs.

Metadata
Generated at 2023-06-06 17:13:02.347537111 UTC
Max. memory units 14000000
Max. CPU units 10000000000
Max. tx size (kB) 16384

Script summary

Name Hash Size (Bytes)
νInitial 2212a4ee618434b9b2f366d7c330dbdfb5c7072e793a850fd0de6ddd 4294
νCommit 69e1ccf9ad73dc6d37a5bc8de5aec86f3c4c1710fe5fd334e0e16b18 2124
νHead 8ae095dca4d14a1b8edffb37faa6c84ec60340fbf389a62f027e0b76 9355
μHead 33642a45c7fbb955ce1704ef09229bb211bf9af9980530db28c6aafe* 4148
  • The minting policy hash is only usable for comparison. As the script is parameterized, the actual script is unique per Head.

Cost of Init Transaction

Parties Tx size % max Mem % max CPU Min fee ₳
1 4739 13.97 5.49 0.51
2 4947 16.27 6.35 0.55
3 5152 19.50 7.61 0.59
5 5559 22.97 8.86 0.64
10 6587 36.20 13.91 0.83
37 12123 97.09 36.76 1.73

Cost of Commit Transaction

This is using ada-only outputs for better comparability.

UTxO Tx size % max Mem % max CPU Min fee ₳
1 596 14.98 5.74 0.34
2 784 19.57 7.70 0.40
3 970 24.66 9.84 0.46
5 1350 36.07 14.56 0.61
10 2280 71.73 28.85 1.04
13 2846 98.03 39.15 1.35

Cost of CollectCom Transaction

Parties UTxO (bytes) Tx size % max Mem % max CPU Min fee ₳
1 57 814 27.93 10.85 0.49
2 114 1142 43.16 16.89 0.67
3 170 1455 61.97 24.41 0.89
4 225 1783 82.10 32.49 1.13

Cost of Close Transaction

Parties Tx size % max Mem % max CPU Min fee ₳
1 640 18.88 8.42 0.39
2 671 17.76 7.45 0.38
3 969 21.67 10.94 0.45
5 1299 24.46 13.45 0.50
10 2120 31.42 19.73 0.64
50 8725 86.87 69.87 1.74

Cost of Contest Transaction

Parties Tx size % max Mem % max CPU Min fee ₳
1 676 24.34 10.47 0.45
2 833 26.49 12.01 0.49
3 1006 27.77 13.21 0.51
5 1335 31.20 15.95 0.58
10 2168 40.20 22.97 0.74
45 7928 99.74 70.74 1.81

Cost of Abort Transaction

Some variation because of random mixture of still initial and already committed outputs.

Parties Tx size % max Mem % max CPU Min fee ₳
1 4857 22.45 9.40 0.61
2 5174 36.59 15.53 0.79
3 5496 53.73 22.99 0.99
4 5817 73.63 31.65 1.23
5 6142 95.94 41.43 1.49

Cost of FanOut Transaction

Involves spending head output and burning head tokens. Uses ada-only UTxO for better comparability.

Parties UTxO UTxO (bytes) Tx size % max Mem % max CPU Min fee ₳
5 0 0 4769 8.66 3.57 0.46
5 1 57 4801 10.06 4.39 0.47
5 5 285 4945 15.64 7.69 0.55
5 10 569 5120 22.61 11.82 0.64
5 20 1137 5481 36.56 20.07 0.83
5 30 1708 5845 50.52 28.33 1.02
5 40 2279 6209 64.49 36.60 1.21
5 50 2847 6567 78.46 44.87 1.40
5 65 3700 7101 99.42 57.28 1.68

@github-actions
Copy link

github-actions bot commented Jun 6, 2023

Test Results

319 tests  +11   313 ✔️ +11   19m 28s ⏱️ + 1m 5s
108 suites +  4       6 💤 ±  0 
    6 files   +  1       0 ±  0 

Results for commit b1cc6b5. ± Comparison against base commit 70a9967.

@abailly-iohk abailly-iohk requested review from v0d1ch and ffakenz and removed request for v0d1ch and ffakenz June 8, 2023 12:55
@v0d1ch v0d1ch merged commit 714f0e4 into master Jun 9, 2023
34 checks passed
@v0d1ch v0d1ch deleted the abailly-iohk/single-client-per-node branch June 9, 2023 09:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants