Clean experimental CI

[pgrange]

We can't maintain two CI so let's ditch the experimentation.

Conclusion in a nutshell:

• most of the time both nix and non-nix CI perform similarly with an advantage to the nix one
• once in a while the non-nix CI will take 1h30 to build (upgrading ghc for instance) because it's the time that is needed to build our project from scratch without relying on some nix cache pre-built binaries so this was expected
• the non nix version was expected to build faster on an incremental commit in a branch as it would build only a small change when the nix version would rebuild the whole module but it's not clear and it happens that most of the build time for the non nix version is devoted to setting up the github runner
• the non-nix version has the advantage of using standard cabal tools and documenting and assessing a non-nix installation procedure which is interesting for non-nix people but it might well be the case that no sane haskell developer would not use nix to build their project.

Several alternatives to improve CI performance:

• have one single build step in github for build, test and benchmark everything as it would drastically reduce all the need for cache download and software install. It could also benefit to nix CI.
• use docker to have one single image ready to build hydra (see this project). But then integration between github ci and docker being what it is, the result looks a bit contrived.

Both CI have run in July and august. Comparing the numbers, we can see two abnormal compilation time for the non-nix CI. This is related to the situation where the CI cache is not ok and everything needs to be built again and it happens that the CI needs more or less 1h30 to build our project from scratch. It’s interesting to note that this happened only twice during the experiment.

Let’s remove these two pics so that we can better compare both CI performance. Then we can see that there is not an order of magnitude of difference between both CI but, still, nix almost always outperforms the non nix CI a bit. Although it might seem obvious it’s actually not as nix version needs to download binaries form nix cache and we observed quite some variability in the past with the nix cache performance.

When we look at some non-nix CI execution we can see that the runner setup can be often very time consuming compared to the compilation itself. See for instance this build job where we can see that 1’33 are spent setting up and tearing down the job and only 12 seconds in cabal build:

[github-actions]

Test Results

358 tests  ±0   353 ✔️ ±0   20m 43s ⏱️ - 1m 58s
121 suites ±0       5 💤 ±0 
    6 files   ±0       0 ❌ ±0 

Results for commit c6697d0. ± Comparison against base commit b020e30.

[github-actions]

Transactions Costs

Sizes and execution budgets for Hydra protocol transactions. Note that unlisted parameters are currently using arbitrary values and results are not fully deterministic and comparable to previous runs.

Metadata
Generated at	2023-09-13 06:04:25.838197988 UTC
Max. memory units	14000000
Max. CPU units	10000000000
Max. tx size (kB)	16384

Script summary

Name	Hash	Size (Bytes)
νInitial	3ffaf6b87df35cb01a52eb23032b8f0b1a2a3ad3acf0930abc9c833a	4150
νCommit	e4c32d6dc83b2917aa7805571f30437ad98b6d20d821d34d45943755	2093
νHead	8a43c1c4d5cb60c212e7aa540932f311cb914a1b6104f0f36a2aaaf0	8845
μHead	efd460b736e8155861e909d6507760b24a5c28717591c6e98c26b104*	4187

• The minting policy hash is only usable for comparison. As the script is parameterized, the actual script is unique per Head.

Cost of Init Transaction

Parties	Tx size	% max Mem	% max CPU	Min fee ₳
1	4784	11.77	4.63	0.49
2	4983	14.14	5.53	0.52
3	5190	16.47	6.42	0.56
5	5601	21.71	8.43	0.63
10	6625	33.47	12.90	0.81
37	12160	97.60	37.29	1.74

Cost of Commit Transaction

This is using ada-only outputs for better comparability.

UTxO	Tx size	% max Mem	% max CPU	Min fee ₳
1	599	12.75	5.01	0.32
2	787	16.49	6.69	0.37
3	975	20.37	8.42	0.42
5	1347	28.55	12.04	0.53
10	2274	51.35	21.96	0.82
18	3772	95.35	40.55	1.38

Cost of CollectCom Transaction

Parties	UTxO (bytes)	Tx size	% max Mem	% max CPU	Min fee ₳
1	57	823	23.73	9.44	0.45
2	114	1144	38.91	15.55	0.63
3	168	1454	54.06	21.79	0.81
4	228	1777	68.10	27.72	0.98
5	283	2095	93.53	38.01	1.27

Cost of Close Transaction

Parties	Tx size	% max Mem	% max CPU	Min fee ₳
1	699	19.22	8.83	0.40
2	898	20.66	10.28	0.43
3	1140	22.39	11.87	0.47
5	1580	26.10	15.13	0.54
10	2571	34.58	22.71	0.71
50	10652	98.80	81.84	2.01

Cost of Contest Transaction

Parties	Tx size	% max Mem	% max CPU	Min fee ₳
1	677	22.22	9.78	0.43
2	907	24.57	11.60	0.47
3	1193	27.01	13.64	0.52
5	1625	30.66	16.84	0.59
10	2682	40.22	24.97	0.77
43	9363	98.97	75.84	1.91

Cost of Abort Transaction

Some variation because of random mixture of still initial and already committed outputs.

Parties	Tx size	% max Mem	% max CPU	Min fee ₳
1	5014	21.57	9.28	0.61
2	5464	36.58	15.91	0.80
3	5898	54.84	23.98	1.03
4	6343	76.35	33.48	1.29

Cost of FanOut Transaction

Involves spending head output and burning head tokens. Uses ada-only UTxO for better comparability.

Parties	UTxO	UTxO (bytes)	Tx size	% max Mem	% max CPU	Min fee ₳
5	0	0	4803	9.06	3.81	0.46
5	1	57	4836	9.85	4.39	0.47
5	5	285	4984	15.45	7.73	0.55
5	10	568	5158	21.92	11.69	0.64
5	20	1140	5523	35.30	19.78	0.82
5	30	1708	5883	48.27	27.71	1.00
5	40	2277	6242	61.31	35.67	1.18
5	50	2847	6603	74.19	43.56	1.35
5	69	3930	7287	99.69	58.99	1.70