Build aarch64-darwin binaries #1182

ch1bo · 2023-11-28T11:09:23Z

🍎 Adds a workflow to build binaries on custom MacOS with Apple Silicon runner (aarch64-darwin system)

🍎 ~~Packages binary with runtime dependencies into a single archive so users can use DYLD_FALLBACK_LIBRARY_PATH to run the binaries (without further installing)~~

🍎 Use static linking for third-party libraries on all darwin builds and do some patching (courtesy of @angerman) when packing them up (in a `nix build .#release)

Example builds:

Linux x86_64: https://github.com/input-output-hk/hydra/actions/runs/7037060630
MacOS aarch64: https://github.com/input-output-hk/hydra/actions/runs/7037060637

CHANGELOG updated or not needed
Documentation updated or not needed
Haddocks updated or not needed
No new TODOs introduced or explained herafter

github-actions · 2023-11-28T11:22:07Z

Test Results

373 tests ±0 368 ✔️ ±0 20m 48s ⏱️ -54s
127 suites ±0     5 💤 ±0
    5 files ±0     0 ❌ ±0

Results for commit d43f41c. ± Comparison against base commit 94d936c.

♻️ This comment has been updated with latest results.

github-actions · 2023-11-28T11:22:10Z

Transactions Costs

Sizes and execution budgets for Hydra protocol transactions. Note that unlisted parameters are currently using arbitrary values and results are not fully deterministic and comparable to previous runs.

Metadata
Generated at	2023-12-01 16:44:30.18916715 UTC
Max. memory units	14000000
Max. CPU units	10000000000
Max. tx size (kB)	16384

Script summary

Name	Hash	Size (Bytes)
νInitial	4868d5365af5120ae0b3c93b819d3452a3cbdcc98595da2a7ae765b5	4069
νCommit	171a1e6bdbc8aa96d957a65b3f505517386af06ba265e3f784741f67	2050
νHead	e89b0c4a6155bac2434d1e500bd49c155b2b56744ccf5a0efa72a82e	9092
μHead	6849328242b5912ad218f134378e6baff11f3e74f7e36dcf8e13d53e*	4062

The minting policy hash is only usable for comparison. As the script is parameterized, the actual script is unique per Head.

Cost of Init Transaction

Parties	Tx size	% max Mem	% max CPU	Min fee ₳
1	4586	11.01	4.37	0.47
2	4791	12.97	5.11	0.50
3	4987	15.19	5.96	0.54
5	5394	19.62	7.65	0.60
10	6396	30.71	11.88	0.77
41	12626	99.54	38.15	1.79

Cost of Commit Transaction

This is using ada-only outputs for better comparability.

UTxO	Tx size	% max Mem	% max CPU	Min fee ₳
1	534	12.22	4.81	0.31
2	724	15.93	6.48	0.36
3	906	19.77	8.20	0.41
5	1282	27.87	11.80	0.52
10	2224	50.58	21.69	0.81
18	3725	94.20	40.16	1.36

Cost of CollectCom Transaction

Parties	UTxO (bytes)	Tx size	% max Mem	% max CPU	Min fee ₳
1	57	480	22.20	8.87	0.41
2	114	590	33.93	13.69	0.55
3	170	700	50.23	20.31	0.73
4	226	810	65.71	26.75	0.91
5	282	920	86.17	35.16	1.14

Cost of Close Transaction

Parties	Tx size	% max Mem	% max CPU	Min fee ₳
1	491	17.33	8.00	0.37
2	693	19.00	9.61	0.40
3	873	20.74	11.23	0.44
5	1222	23.67	14.11	0.50
10	2106	32.15	22.06	0.66
50	8948	95.33	82.98	1.91

Cost of Contest Transaction

Parties	Tx size	% max Mem	% max CPU	Min fee ₳
1	583	21.33	9.71	0.42
2	757	23.14	11.35	0.45
3	862	24.44	12.53	0.48
5	1326	28.81	16.40	0.56
10	2159	37.63	24.26	0.72
44	7806	96.68	77.44	1.83

Cost of Abort Transaction

Some variation because of random mixture of still initial and already committed outputs.

Parties	Tx size	% max Mem	% max CPU	Min fee ₳
1	4520	20.10	8.74	0.57
2	4711	33.98	14.96	0.74
3	4875	50.57	22.40	0.93
4	5078	70.13	31.17	1.16
5	5212	92.96	41.35	1.43

Cost of FanOut Transaction

Involves spending head output and burning head tokens. Uses ada-only UTxO for better comparability.

Parties	UTxO	UTxO (bytes)	Tx size	% max Mem	% max CPU	Min fee ₳
5	0	0	4416	9.21	3.88	0.45
5	1	57	4452	9.91	4.42	0.46
5	5	284	4595	15.56	7.79	0.53
5	10	569	4775	22.18	11.81	0.62
5	20	1140	5137	34.92	19.66	0.80
5	30	1710	5498	48.14	27.70	0.98
5	40	2276	5854	60.99	35.59	1.16
5	50	2847	6215	73.98	43.55	1.34
5	70	3979	6927	99.97	59.46	1.69

End-To-End Benchmark Results

This page is intended to collect the latest end-to-end benchmarks results produced by Hydra's Continuous Integration system from the latest master code.

Please take those results with a grain of salt as they are currently produced from very limited cloud VMs and not controlled hardware. Instead of focusing on the absolute results, the emphasis should be on relative results, eg. how the timings for a scenario evolve as the code changes.

Generated at 2023-12-01 16:37:02.91255008 UTC

3-nodes Scenario

A rather typical setup, with 3 nodes forming a Hydra head.

Number of nodes	3
Number of txs	900
Avg. Confirmation Time (ms)	25.084668988
P99	142.8214130799999ms
P95	37.6838706ms
P50	19.879728999999998ms
Number of Invalid txs	0

Baseline Scenario

This scenario represents a minimal case and as such is a good baseline against which to assess the overhead introduced by more complex setups. There is a single hydra-node d with a single client submitting single input and single output transactions with a constant UTxO set of 1.

Number of nodes	1
Number of txs	300
Avg. Confirmation Time (ms)	3.978869120
P99	5.4740360599999995ms
P95	4.7481434ms
P50	3.868373ms
Number of Invalid txs	0

ghost · 2023-11-29T07:15:32Z

Seems like the produced binaries are not runnable, I have successfully retrieved the produced package but the binaries crash even when given the DYLD_FALLBACK_LIBRARY_PATH path to find their libraries.

ghost

Not sure about the nix magic but the rest LGTM

ghost

Seems like the archive is published with twice the .zip extension

ch1bo · 2023-12-01T10:29:48Z

Seems like the archive is published with twice the .zip extension

Was only double the extension. Fixed now e14bc41

ghost · 2023-12-01T14:39:08Z

I guess the hydra errors are spurious? We have observed those already in the past IIRC

v0d1ch · 2023-12-01T16:07:11Z

Are we ok with darwin builds failing here? Do we want to support it going forward or we are hopping onto silicon train? :)

Using ldd we can find and copy the runtime dependencies and later use DYLD_FALLBACK_LIBRARY_PATH to point there.

We do not want a directory inside and it was not packaging correctly multiple derivations.

This ensures the resulting derivation has the same stdenv and downstream functions can infer the targetPlatform correctly, e.g. packaging.nix

Tried to use a somewhat consistent and not confusing (not too technical) naming. That is, no system triplets and something users would understand on the top level.

Upload artifact is always zipping the paths provided, so we need to unpack the release archive first.

ch1bo self-assigned this Nov 29, 2023

ch1bo force-pushed the abailly-iohk/add-custom-aarch64-darwin-runner branch 6 times, most recently from edcbb2d to 2339640 Compare November 29, 2023 18:39

ch1bo marked this pull request as ready for review November 29, 2023 18:46

ch1bo requested a review from a team November 29, 2023 18:47

ghost approved these changes Nov 30, 2023

View reviewed changes

ghost suggested changes Nov 30, 2023

View reviewed changes

ghost mentioned this pull request Dec 1, 2023

Fix URLs to download software and add some help text #1191

Merged

4 tasks

ghost self-requested a review December 1, 2023 10:41

ghost approved these changes Dec 1, 2023

View reviewed changes

ch1bo assigned ch1bo and unassigned ch1bo Dec 1, 2023

ghost force-pushed the abailly-iohk/add-custom-aarch64-darwin-runner branch from e14bc41 to fa21fa3 Compare December 1, 2023 14:10

abailly and others added 7 commits December 1, 2023 17:33

Configure job for darwin

dfdca80

Dedicated build-only job for Darwin

75d9dfa

Rename build step

8e8a3c3

Add some documentation to workflows

7e2cde9

Collect runtime libraries from nix store

622bfd4

Using ldd we can find and copy the runtime dependencies and later use DYLD_FALLBACK_LIBRARY_PATH to point there.

Use otool instead

e31026f

Minor typo

8e12d1a

ch1bo added 8 commits December 1, 2023 17:33

Modify packaging.nix to produce our zip format

28d3b9a

We do not want a directory inside and it was not packaging correctly multiple derivations.

Update workflows to use .#release and .#release-static

191f711

Use mkDerivation instaed of runCommand in embedRevision

25c1a5c

This ensures the resulting derivation has the same stdenv and downstream functions can infer the targetPlatform correctly, e.g. packaging.nix

Use artifact names based on branch/tag

f2521f2

Create versioned artifacts

f8cd8f6

Renamed workflows

343ced9

Tried to use a somewhat consistent and not confusing (not too technical) naming. That is, no system triplets and something users would understand on the top level.

Fix github artifacts

465deb5

Upload artifact is always zipping the paths provided, so we need to unpack the release archive first.

Avoid .zip.zip in artifact download

d43f41c

ch1bo force-pushed the abailly-iohk/add-custom-aarch64-darwin-runner branch from fa21fa3 to d43f41c Compare December 1, 2023 16:33

ch1bo merged commit 6d9163d into master Dec 1, 2023

ch1bo deleted the abailly-iohk/add-custom-aarch64-darwin-runner branch December 1, 2023 16:48

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Build aarch64-darwin binaries #1182

Build aarch64-darwin binaries #1182

ch1bo commented Nov 28, 2023 •

edited

Loading

github-actions bot commented Nov 28, 2023 •

edited

Loading

github-actions bot commented Nov 28, 2023 •

edited

Loading

ghost commented Nov 29, 2023

ghost left a comment

ghost left a comment

ch1bo commented Dec 1, 2023

ghost commented Dec 1, 2023

v0d1ch commented Dec 1, 2023

Build aarch64-darwin binaries #1182

Build aarch64-darwin binaries #1182

Conversation

ch1bo commented Nov 28, 2023 • edited Loading

github-actions bot commented Nov 28, 2023 • edited Loading

Test Results

github-actions bot commented Nov 28, 2023 • edited Loading

Transactions Costs

Script summary

Cost of Init Transaction

Cost of Commit Transaction

Cost of CollectCom Transaction

Cost of Close Transaction

Cost of Contest Transaction

Cost of Abort Transaction

Cost of FanOut Transaction

End-To-End Benchmark Results

3-nodes Scenario

Baseline Scenario

ghost commented Nov 29, 2023

ghost left a comment

Choose a reason for hiding this comment

ghost left a comment

Choose a reason for hiding this comment

ch1bo commented Dec 1, 2023

ghost commented Dec 1, 2023

v0d1ch commented Dec 1, 2023

ch1bo commented Nov 28, 2023 •

edited

Loading

github-actions bot commented Nov 28, 2023 •

edited

Loading

github-actions bot commented Nov 28, 2023 •

edited

Loading