Upgrade to cardano-node 8 #1015

v0d1ch · 2023-08-09T16:22:54Z

Updates the cardano-node in the development environment and on CI to 8.1.2
Updates cardano-api, which we use to connect to the node to 8.11.0
Consequently, needs to update cardano-ledger and plutus to suitable versions
Consequently, a lot of changes are required due to changed interfaces and moved modules in cardano-api and cardano-ledger, new versioning of serialization, versioned compilation in plutus
Uses the mainnet cost model from early August 2023 to evaluate scripts, as the builtin testing cost model is not reliable.
We fixed our golden files as we had an incorrect serialization on key witnesses. Everything else worked!
Fixes the demo by increasing fuel of the demo as it was already too low on master (not enough collateral to fan out)

CHANGELOG updated or not needed
Documentation updated or not needed
Haddocks updated or not needed
No new TODOs introduced or explained hereafter

Also tests against existence of this version. Configuration files unchanged still.

Seemingly some cost model parameter names changed.

The object-based serialization is (currently?) unavailable when querying the parameters from the network, so we need to resort to the unlabelled array items to make sure the same one is used.

The latest cardano-api packages were pushed to CHaP, so we can drop the source-repository package.

- Upgrade hydra-cardano-api dependencies to satisfy constraints. + Newest cardano-ledger works with up to plutus-ledger-api 1.7.0.0 + Newest cardano-api requires text greater than 2 - Remove ouroboros-consensus-cardano-test dependency This is currently problematic as it is only available in version 0.3, which depends on a lib in ouroboros-consensus-protocol package.

This is needed after index-state updates and slightly improved the CONTRIBUTING.md about this.

A new variant of libsodium via the overlay is needed.

GHC 9.2.8 is not available in vanilla nixpkgs

It seems like the SI unit variants (using seconds) of io-sim compatible classes moved to si-timers.

This is to be replaced by cardano-ledger-mary and cardano-ledger-allegra packages.

Because base16 changed its interface between versions.

This is available now upstream. Not with the UTF8 decoding, but this should not matter in our case.

The cardano-shelly-ma got split into cardano-shelley-allegra and cardano-shelley-mary including the module names.

They only provide deserialization instances for their new variant of cardano-binary.

This is only used in one place and the ledger now has a getMinFeeTx itself.

TxGovernanceActions and TxVotes were added.

Like the other constructors of cardano-api, we also have a pattern which provides the BundledProtocolParameters for the latest era (without type parameter).

The fact that toLedgerPParams now can fail, had us push the problem of not being able to create ledger params from the cardano-api params into the 'pparams' fixture. The whole function becomes a bit contrived now, but alternatives include adding lenses as dependencies.

This is consistent with newGlobals and makes call sites simpler as we don't handle the error cases to this exceptional situation.

Seemingly this changed from one to another ledger version and our ToJSON instance for a Tx is relying on the upstream cbor instances.

Our generator, after the upstream library updates, is not producing enough ada only outputs. We changed it so that the distribution of ada only outputs in the generator is producing these half of the time. Also remove some unneeded imports in Fixture module

We want to make sure that generated txs do not produce too large values in the outputs so we can actually cover the fees.

Fix the brick version to 0.73 Use the new File constructor

As the ledger changed the mint field to only being a multi-asset (minting lovelace does not make sense), but cardano-api still has everything as 'Value', we add a conversion function to create values from just Ledger.MultiAsset.

The hydra-test-utils also hard-codes pparams so we should also use mainnet parameters there. Obviously, the duplication between this and hydra-node fixtures should be removed, but we are not doing that in this PR.

github-actions · 2023-08-10T11:30:25Z

Test Results

361 tests +2 356 ✔️ +2 18m 58s ⏱️ - 4m 53s
121 suites +2     5 💤 ±0
    6 files ±0     0 ❌ ±0

Results for commit 1aca89e. ± Comparison against base commit ca76fed.

This pull request removes 12 and adds 14 tests. Note that renamed tests count towards both.

Hydra.Ledger.Cardano ‑ CBOR encoding of Tx
Hydra.Ledger.Cardano ‑ Roundtrip to and from Ledger
Hydra.Ledger.Cardano ‑ Same TxId before/after JSON encoding
Hydra.Ledger.Cardano ‑ applies valid transaction
Hydra.Ledger.Cardano ‑ applies valid transaction serialised from JSON
Hydra.Ledger.Cardano ‑ should parse a Cardano.UTxO
Hydra.Ledger.Cardano ‑ should parse a Tx
Hydra.Ledger.Cardano/JSON encoding of (Tx BabbageEra) ‑ allows to encode values with aeson and read them back
Hydra.Ledger.Cardano/JSON encoding of (Tx BabbageEra) ‑ produces the same JSON as is found in golden/Tx BabbageEra.json
Hydra.Ledger.Cardano/JSON encoding of (UTxO' (TxOut CtxUTxO BabbageEra)) ‑ allows to encode values with aeson and read them back
…

Hydra.Ledger.Cardano/Generators/genValue ‑ produces realistic values
Hydra.Ledger.Cardano/Tx ‑ JSON encoding of Tx according to schema
Hydra.Ledger.Cardano/Tx ‑ Roundtrip CBOR encoding
Hydra.Ledger.Cardano/Tx ‑ Roundtrip to and from Ledger
Hydra.Ledger.Cardano/Tx ‑ Same TxId before/after JSON encoding
Hydra.Ledger.Cardano/Tx ‑ parses a specific Tx
Hydra.Ledger.Cardano/Tx/JSON encoding of (Tx BabbageEra) ‑ allows to encode values with aeson and read them back
Hydra.Ledger.Cardano/Tx/JSON encoding of (Tx BabbageEra) ‑ produces the same JSON as is found in golden/Tx BabbageEra.json
Hydra.Ledger.Cardano/UTxO ‑ JSON encoding of UTxO according to schema
Hydra.Ledger.Cardano/UTxO ‑ parses a specific UTxO
…

♻️ This comment has been updated with latest results.

We need to avoid specialization of 'length' to make this compile

This has bitten multiple projects lately. See linked issue for the full reason why.

github-actions · 2023-08-10T13:19:27Z

Transactions Costs

Sizes and execution budgets for Hydra protocol transactions. Note that unlisted parameters are currently using arbitrary values and results are not fully deterministic and comparable to previous runs.

Metadata
Generated at	2023-08-10 15:13:09.66785774 UTC
Max. memory units	14000000
Max. CPU units	10000000000
Max. tx size (kB)	16384

Script summary

Name	Hash	Size (Bytes)
νInitial	3ffaf6b87df35cb01a52eb23032b8f0b1a2a3ad3acf0930abc9c833a	4150
νCommit	e4c32d6dc83b2917aa7805571f30437ad98b6d20d821d34d45943755	2093
νHead	8508839dfce39b6be65c018ce124ab549238e1b2ed08fb6588e5601f	8799
μHead	7e939052f278f7721cb1f98c3f2ca646827df9314a7e0d0168765b49*	4055

The minting policy hash is only usable for comparison. As the script is parameterized, the actual script is unique per Head.

Cost of Init Transaction

Parties	Tx size	% max Mem	% max CPU	Min fee ₳
1	4649	11.26	4.47	0.48
2	4855	13.27	5.24	0.51
3	5056	15.59	6.14	0.54
5	5468	19.55	7.67	0.60
10	6494	29.06	11.34	0.75
45	13668	99.81	38.68	1.84

Cost of Commit Transaction

This is using ada-only outputs for better comparability.

UTxO	Tx size	% max Mem	% max CPU	Min fee ₳
1	596	12.67	4.97	0.31
2	787	16.49	6.69	0.37
3	975	20.28	8.39	0.42
5	1350	28.46	12.01	0.53
10	2287	51.44	21.99	0.83
18	3784	95.26	40.52	1.38

Cost of CollectCom Transaction

Parties	UTxO (bytes)	Tx size	% max Mem	% max CPU	Min fee ₳
1	57	815	24.45	9.71	0.45
2	114	1136	37.89	15.18	0.62
3	170	1454	53.11	21.44	0.80
4	224	1770	69.31	28.18	0.99
5	281	2093	86.61	35.44	1.20

Cost of Close Transaction

Parties	Tx size	% max Mem	% max CPU	Min fee ₳
1	693	17.33	8.10	0.38
2	919	19.02	9.66	0.41
3	1167	20.71	11.26	0.45
5	1524	24.23	14.38	0.52
10	2651	31.85	21.67	0.68
50	10723	95.75	80.95	1.98

Cost of Contest Transaction

Parties	Tx size	% max Mem	% max CPU	Min fee ₳
1	722	22.60	10.11	0.44
2	958	24.99	11.95	0.48
3	1125	26.07	13.09	0.50
5	1595	30.13	16.55	0.58
10	2494	38.22	23.56	0.74
43	9168	98.83	75.29	1.89

Cost of Abort Transaction

Some variation because of random mixture of still initial and already committed outputs.

Parties	Tx size	% max Mem	% max CPU	Min fee ₳
1	4904	21.66	9.32	0.61
2	5336	36.46	15.87	0.79
3	5790	54.49	23.85	1.02
4	6274	76.86	33.68	1.29

Cost of FanOut Transaction

Involves spending head output and burning head tokens. Uses ada-only UTxO for better comparability.

Parties	UTxO	UTxO (bytes)	Tx size	% max Mem	% max CPU	Min fee ₳
5	0	0	4676	9.05	3.80	0.46
5	1	58	4709	10.10	4.50	0.47
5	5	285	4852	15.35	7.69	0.54
5	10	566	5032	22.17	11.79	0.64
5	20	1139	5390	35.29	19.78	0.82
5	30	1707	5751	48.42	27.77	0.99
5	40	2276	6110	61.56	35.77	1.17
5	50	2848	6467	74.44	43.66	1.35
5	69	3927	7151	99.68	58.98	1.70

We should investigate the costs for Hydra transactions, they seem quite expensive now?

This is the current protocol version on mainnet, so we use that for evaluating scripts.

ch1bo

What a feat, good work @ffakenz @v0d1ch 💪

v0d1ch assigned v0d1ch, ch1bo and ffakenz Aug 9, 2023

v0d1ch changed the title ~~Ensemble/update dependencies~~ Use cardano-node 8 Aug 9, 2023

v0d1ch changed the title ~~Use cardano-node 8~~ Upgrade to cardano-node 8 Aug 9, 2023

ch1bo and others added 18 commits August 10, 2023 09:19

Update cardano-node and cardano-cli to 8.1.2

cf9111f

Also tests against existence of this version. Configuration files unchanged still.

Add conway genesis files and update node config

45f9cd5

Seemingly some cost model parameter names changed.

Use preprod cost model in our devnet

c83ecb7

The object-based serialization is (currently?) unavailable when querying the parameters from the network, so we need to resort to the unlabelled array items to make sure the same one is used.

Update to cardano-api 8.x via CHaP

1e179e5

The latest cardano-api packages were pushed to CHaP, so we can drop the source-repository package.

Bump hackage and CHaP flake inputs

4fbdb4e

This is needed after index-state updates and slightly improved the CONTRIBUTING.md about this.

Update iohk-nix

d07ae26

A new variant of libsodium via the overlay is needed.

Fix cabalOnly nix shell

ef4f0af

GHC 9.2.8 is not available in vanilla nixpkgs

Re-export si-timers next to io-classes in hydra-prelude

379989d

It seems like the SI unit variants (using seconds) of io-sim compatible classes moved to si-timers.

Update cardano-ledger-shelley-ma dependencies

63da073

This is to be replaced by cardano-ledger-mary and cardano-ledger-allegra packages.

Replace base16 for base16-bytestring

f9d1104

Because base16 changed its interface between versions.

Fix type family invocation

b956c2b

Remove UsingRawBytesHex from hydra-cardano-api

1795377

This is available now upstream. Not with the UTF8 decoding, but this should not matter in our case.

Fix imports of Allegra/Mary ledger modules

4e8c336

The cardano-shelly-ma got split into cardano-shelley-allegra and cardano-shelley-mary including the module names.

Use cardano-ledger-binary to genTxIn

4986abe

They only provide deserialization instances for their new variant of cardano-binary.

Switch ledger imports in hydra-cardano-api

2c2c9f3

Update hydra-cardano-api in its usage of cardano-ledger packages

1692b62

Remove totalExecutionCost from hydra-cardano-api

b64c443

This is only used in one place and the ledger now has a getMinFeeTx itself.

ch1bo force-pushed the ensemble/update-dependencies branch 2 times, most recently from 4844e65 to 13b5c87 Compare August 10, 2023 07:26

ch1bo added 5 commits August 10, 2023 09:27

Update latest era TxBodyContent in hydra-cardano-api

6438d19

TxGovernanceActions and TxVotes were added.

Use timout from Prelude in hydra-test-utils

d0f61ab

Fix deprecated HSpec formatter in hydra-test-utils

d6ff4c8

Add BundledProtocolParameteters latest era pattern

41e9d3c

Like the other constructors of cardano-api, we also have a pattern which provides the BundledProtocolParameters for the latest era (without type parameter).

v0d1ch and others added 11 commits August 10, 2023 12:54

Convert to ledger PParams directly in defaultLedgerEnv

65c2f8f

Make newLedgerEnv monadic

46b80f4

This is consistent with newGlobals and makes call sites simpler as we don't handle the error cases to this exceptional situation.

Change all cbor encoded key witnesses

792c43e

Seemingly this changed from one to another ledger version and our ToJSON instance for a Tx is relying on the upstream cbor instances.

Resize the generated txs in WalletSpec

0232e3b

We want to make sure that generated txs do not produce too large values in the outputs so we can actually cover the fees.

Fix compilation for hydra-tui

b7a08fc

Fix the brick version to 0.73 Use the new File constructor

Remove all warnings

d483800

Use transMintValue to extract minted Value in Mutation

5d3f2d4

Add fromLedgerMultiAsset to hydra-cardano-api

0d36472

As the ledger changed the mint field to only being a multi-asset (minting lovelace does not make sense), but cardano-api still has everything as 'Value', we add a conversion function to create values from just Ledger.MultiAsset.

Also use mainnet parameters in plutus-.. tests

bc22d82

The hydra-test-utils also hard-codes pparams so we should also use mainnet parameters there. Obviously, the duplication between this and hydra-node fixtures should be removed, but we are not doing that in this PR.

Regenerate plutus scripts and update changelog

590519b

ch1bo force-pushed the ensemble/update-dependencies branch from 4a9fb49 to 590519b Compare August 10, 2023 10:56

ch1bo added 2 commits August 10, 2023 15:07

Fix plutus-tx compilation of plutus-merkle-tree

43bad57

We need to avoid specialization of 'length' to make this compile

Make haskell.nix the last overlay in project.nix

c543989

This has bitten multiple projects lately. See linked issue for the full reason why.

ch1bo force-pushed the ensemble/update-dependencies branch from 063a475 to adf991a Compare August 10, 2023 13:08

v0d1ch force-pushed the ensemble/update-dependencies branch 2 times, most recently from 5f47f73 to 1d9b826 Compare August 10, 2023 14:09

ch1bo force-pushed the ensemble/update-dependencies branch from d1bc392 to 28c9cf7 Compare August 10, 2023 14:23

ch1bo and others added 5 commits August 10, 2023 17:01

Update demo to have more fuel

727218b

We should investigate the costs for Hydra transactions, they seem quite expensive now?

Fix the link to download cardano-node 8.1.2

c40238a

Compile plutus-merkle-tree and plutuc-cbor for PLC100

e3dd829

Update protocol parameters fixture to protocol version 8

a0af7c2

This is the current protocol version on mainnet, so we use that for evaluating scripts.

Use fromString instead of unsafe unsafeDeserialiseFromRawBytes16

1aca89e

ch1bo force-pushed the ensemble/update-dependencies branch from 32a0c07 to 1aca89e Compare August 10, 2023 15:01

ch1bo approved these changes Aug 10, 2023

View reviewed changes

ch1bo linked an issue Aug 10, 2023 that may be closed by this pull request

Upgrade to cardano-node 8 #1007

Closed

4 tasks

ch1bo merged commit e8b06f1 into master Aug 10, 2023
24 of 25 checks passed

ch1bo deleted the ensemble/update-dependencies branch August 10, 2023 15:52

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Upgrade to cardano-node 8 #1015

Upgrade to cardano-node 8 #1015

v0d1ch commented Aug 9, 2023 •

edited by ch1bo

github-actions bot commented Aug 10, 2023 •

edited

github-actions bot commented Aug 10, 2023 •

edited

ch1bo left a comment

Upgrade to cardano-node 8 #1015

Upgrade to cardano-node 8 #1015

Conversation

v0d1ch commented Aug 9, 2023 • edited by ch1bo

github-actions bot commented Aug 10, 2023 • edited

Test Results

github-actions bot commented Aug 10, 2023 • edited

Transactions Costs

Script summary

Cost of Init Transaction

Cost of Commit Transaction

Cost of CollectCom Transaction

Cost of Close Transaction

Cost of Contest Transaction

Cost of Abort Transaction

Cost of FanOut Transaction

ch1bo left a comment

Choose a reason for hiding this comment

v0d1ch commented Aug 9, 2023 •

edited by ch1bo

github-actions bot commented Aug 10, 2023 •

edited

github-actions bot commented Aug 10, 2023 •

edited