Update to the newest ledger #4349

lehins · 2023-02-08T11:42:16Z

Description

Fixes invalid ExtLedgerState serialization. Produced CBOR encoding contained a list without a length, thus it was not decodable as arbitrary CBOR.

Integration of latest cardano-ledger and cardano-base.

Update iohk-nix sources to get latest libsodium needed for cardano-crypto-praos
Account for introduction of versioned CBOR in ledger. The new interface EncCBOR/DecCBOR is very similar to the plain ToCBOR/FromCBOR, with helper functions to go between them.
Remove usage of MemoBytes for Header, since Header cannot be parameterized on era and MemoBytes now requires it.
Account for addition of Shelley translation context: FromByronTranslationContext
Account for change of parametrization of ShelleyGenesis from era to crypto
Account for switch of PParams and other types from HasFields to lenses
Account for various fields getting renamed
Account for the fact that ConwayGenesis has to also include AlonzoGenesis fields, because otherwise CanStartFromGenesis can't work. We need to figure out the whole business of TranslationContext and Genesis, it is a bit messy right now IMHO.
Update golden tests, which were needed due to:
- Fixes to PParams serialization
- Changes of default protocol versions in emptyPParams in order eras
- Fixes to test examples
- Addition of a new field with deposits to the ledger state
- Fix to ExtLedgerState serialization
- Changes to Conway types
Remove some arbitrary instances (such as Slotting for example), since they are now imported from ledger. Notable instances are also UTCTime and NominalDiffTime, that now are imported from quickcheck-instances. In order to preserve older semantics standalone generators for those types were retained and used in delayNextSlot property test. Also guarded that against an "infinite" loop with within.

Blocked by IntersectMBO/cardano-ledger#3282 and CHaPS releases

Checklist

Branch
- Commit sequence broadly makes sense
- Commits have useful messages
- The documentation has been properly updated
- New tests are added if needed and existing tests are updated
- Any changes affecting Consensus packages must have an entry in the appropriate changelog.d directory created using scriv. If in doubt, see the Consensus release process.
- If this branch changes Network and has any consequences for downstream repositories or end users, said changes must be documented in interface-CHANGELOG.md
- If serialization changes, user-facing consequences (e.g. replay from genesis) are confirmed to be intentional.
Pull Request
- Self-reviewed the diff
- Useful pull request description at least containing the following information:
  - What does this PR change?
  - Why these changes were needed?
  - How does this affect downstream repositories and/or end-users?
  - Which ticket does this PR close (if any)? If it does, is it linked?
- Reviewer requested

cabal.project

ouroboros-consensus-cardano-tools/src/Cardano/Tools/DBAnalyser/Block/Cardano.hs

.github/PULL_REQUEST_TEMPLATE.md

disassembler

I've reviewed this and while long, i don't see anything controversial. It's mainly adopting to the new pparams syntax and the changes to enc/Dec serialization functions for the versioned cbor. While i agree not touching the odd protver in tools is smart, please do open a GH issue to track the oddity so the team can fix it at a later date or clarify why the numbers vary in older eras.

dnadales

Thank you for the great PR description 🙌

LGTM. However, I can't answer @lehins 's questions.

ouroboros-consensus-cardano-test/src/Test/ThreadNet/Infra/ShelleyBasedHardFork.hs

ouroboros-consensus-cardano-test/src/Test/ThreadNet/TxGen/Cardano.hs

ouroboros-consensus-cardano-tools/src/Cardano/Api/KeysByron.hs

ouroboros-consensus-cardano-tools/src/Cardano/Node/Protocol/Cardano.hs

ouroboros-consensus-shelley/src/Ouroboros/Consensus/Shelley/Node/TPraos.hs

ouroboros-consensus-shelley/src/Ouroboros/Consensus/Shelley/ShelleyHFC.hs

ouroboros-consensus-test/test-consensus/Test/Consensus/BlockchainTime/Simple.hs

…`CBOR`

* Update iohk-nix sources to get latest `libsodium` needed for `cardano-crypto-praos` * Account for introduction of versioned CBOR in ledger. The new interface `EncCBOR`/`DecCBOR` is very similar to the plain `ToCBOR`/`FromCBOR`, with helper functions to go between them. * Remove usage of `MemoBytes` for `Header`, since `Header` cannot be parameterized on `era` and `MemoBytes` now requires it. * Account for addition of Shelley translation context: `FromByronTranslationContext` * Account for change of parametrization of `ShelleyGenesis` from `era` to `crypto` * Account for switch of `PParams` and other types from `HasFields` to lenses * Account for various fields getting renamed * Account for the fact that ConwayGenesis has to also include AlonzoGenesis fields, because otherwise `CanStartFromGenesis` can't work. We need to figure out the whole business of TranslationContext and Genesis, it is a bit messy right now IMHO. * Update golden tests, which were needed due to: * Fixes to PParams serialization * Changes of default protocol versions in emptyPParams in order eras * Fixes to test examples * Addition of a new field with deposits to the ledger state * Fix to `ExtLedgerState` serialization * Changes to Conway types * Remove some arbitrary instances (such as Slotting for example), since they are now imported from ledger. Notable instances are also `UTCTime` and `NominalDiffTime`, that now are imported from `quickcheck-instances`. In order to preserve older semantics standalone generators for those types were retained and used in `delayNextSlot` property test. Also guarded that against an "infinite" loop with `within`.

lehins · 2023-03-02T23:48:55Z

bors r+

4349: Update to the newest ledger r=lehins a=lehins # Description Fixes invalid `ExtLedgerState` serialization. Produced CBOR encoding contained a list without a length, thus it was not decodable as arbitrary CBOR. Integration of latest `cardano-ledger` and `cardano-base`. * Update iohk-nix sources to get latest `libsodium` needed for `cardano-crypto-praos` * Account for introduction of versioned CBOR in ledger. The new interface `EncCBOR`/`DecCBOR` is very similar to the plain `ToCBOR`/`FromCBOR`, with helper functions to go between them. * Remove usage of `MemoBytes` for `Header`, since `Header` cannot be parameterized on `era` and `MemoBytes` now requires it. * Account for addition of Shelley translation context: `FromByronTranslationContext` * Account for change of parametrization of `ShelleyGenesis` from `era` to `crypto` * Account for switch of `PParams` and other types from `HasFields` to lenses * Account for various fields getting renamed * Account for the fact that ConwayGenesis has to also include AlonzoGenesis fields, because otherwise `CanStartFromGenesis` can't work. We need to figure out the whole business of TranslationContext and Genesis, it is a bit messy right now IMHO. * Update golden tests, which were needed due to: * Fixes to PParams serialization * Changes of default protocol versions in emptyPParams in order eras * Fixes to test examples * Addition of a new field with deposits to the ledger state * Fix to `ExtLedgerState` serialization * Changes to Conway types * Remove some arbitrary instances (such as Slotting for example), since they are now imported from ledger. Notable instances are also `UTCTime` and `NominalDiffTime`, that now are imported from `quickcheck-instances`. In order to preserve older semantics standalone generators for those types were retained and used in `delayNextSlot` property test. Also guarded that against an "infinite" loop with `within`. Blocked by IntersectMBO/cardano-ledger#3282 and CHaPS releases Co-authored-by: Alexey Kuleshevich <alexey.kuleshevich@iohk.io> Co-authored-by: teodanciu <teodora.danciu@tweag.io>

iohk-bors · 2023-03-03T02:51:48Z

Timed out.

lehins · 2023-03-03T03:07:50Z

bors r+

iohk-bors · 2023-03-03T05:49:39Z

Build succeeded:

IntersectMBO/ouroboros-network#4349 changed/fixed the encoding of `PParams`, which broke compatibility of older clients with Node 8.0. This PR restores compatibility, by making the en-/decoding version-dependent. See the commit message for some implementation details. Also, note how the golden files changed due to this PR: - Pre-Alonzo serialization did not change. - Alonzo and Babbage changed, but only for `CardanoNodeToClientVersion <= 10`; these are enabled by `NodeToClient <= 14`, which are the currently released node-to-client versions. - Note that no golden files changed for `CardanoNodeToClientVersion{11,12}` (which are enabled by `NodeToClientV_{15,16}`). `NodeToClientV_15` will be released in Node 8.1, and indeed, we want to use the new and fixed encoding when this version is negotiated.

bartfrenk mentioned this pull request Feb 8, 2023

Remove Shelley genesis info from the ledger configuration #4091

Closed

6 tasks

lehins force-pushed the lehins/update-ledger branch 2 times, most recently from 2bac21b to aa8ec69 Compare February 14, 2023 21:27

lehins changed the title ~~Lehins/update ledger~~ Update to the newest ledger Feb 15, 2023

lehins commented Feb 16, 2023

View reviewed changes

cabal.project Outdated Show resolved Hide resolved

lehins force-pushed the lehins/update-ledger branch 2 times, most recently from dc85175 to 85d20b3 Compare February 16, 2023 19:26

lehins commented Feb 16, 2023

View reviewed changes

ouroboros-consensus-cardano-tools/src/Cardano/Tools/DBAnalyser/Block/Cardano.hs Show resolved Hide resolved

lehins force-pushed the lehins/update-ledger branch from 9ad99e1 to db06e19 Compare February 16, 2023 20:29

lehins marked this pull request as ready for review February 16, 2023 20:30

lehins requested review from a team, nfrisby, dnadales, coot and bolt12 as code owners February 16, 2023 20:30

lehins force-pushed the lehins/update-ledger branch from db06e19 to f8247e7 Compare February 16, 2023 21:36

dnadales reviewed Feb 17, 2023

View reviewed changes

.github/PULL_REQUEST_TEMPLATE.md Outdated Show resolved Hide resolved

disassembler approved these changes Feb 17, 2023

View reviewed changes

dnadales approved these changes Feb 17, 2023

View reviewed changes

lehins force-pushed the lehins/update-ledger branch 2 times, most recently from 4975d7a to 8981498 Compare February 21, 2023 15:46

lehins mentioned this pull request Feb 22, 2023

WIP Split Crypto into standard Crypto and HeaderCrypto IntersectMBO/cardano-ledger#3262

Closed

6 tasks

lehins force-pushed the lehins/update-ledger branch 2 times, most recently from dad1400 to 3214721 Compare February 23, 2023 18:54

andreabedini mentioned this pull request Feb 24, 2023

Bump haskell.nix #4364

Closed

lehins force-pushed the lehins/update-ledger branch from 240bbf0 to 31a34a3 Compare February 28, 2023 12:46

lehins added 4 commits March 2, 2023 16:56

Add .nix-shell-cabal.project to .gitignore

898c027

Fix ExtLedgerState serialization, make sure it is encoded as valid …

327ead9

…`CBOR`

Update all dependencies and pin ledger versions to newly released

2ff62a2

Apply stylish-haskell

cb15887

lehins force-pushed the lehins/update-ledger branch from dc3c027 to cb15887 Compare March 2, 2023 14:00

Reformat nix files using nixfmt

f6bc60d

teodanciu mentioned this pull request Mar 2, 2023

Update iohk-nix sources in preparation for updating to latest ledger #4416

Closed

10 tasks

iohk-bors bot merged commit 17889be into master Mar 3, 2023

iohk-bors bot deleted the lehins/update-ledger branch March 3, 2023 05:49

dnadales mentioned this pull request Mar 8, 2023

Keep backwards compatibility with previous ledger snapshot version #4434

Closed

This was referenced May 22, 2023

Restore en-/decoding compatibility for GetCurrentPParams IntersectMBO/ouroboros-consensus#95

Merged

Test that our CBOR encoders actually produce valid CBOR IntersectMBO/ouroboros-consensus#96

Closed

lehins mentioned this pull request Aug 8, 2023

Use cardano-ledger-binary serialization #4240

Closed

amesgen mentioned this pull request Nov 30, 2023

Use proper CBOR for ExtLedgerState IntersectMBO/ouroboros-consensus#631

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update to the newest ledger #4349

Update to the newest ledger #4349

lehins commented Feb 8, 2023 •

edited

Loading

disassembler left a comment •

edited

Loading

dnadales left a comment

lehins commented Mar 2, 2023

iohk-bors bot commented Mar 3, 2023

lehins commented Mar 3, 2023

iohk-bors bot commented Mar 3, 2023

Update to the newest ledger #4349

Update to the newest ledger #4349

Conversation

lehins commented Feb 8, 2023 • edited Loading

Description

Checklist

disassembler left a comment • edited Loading

Choose a reason for hiding this comment

dnadales left a comment

Choose a reason for hiding this comment

lehins commented Mar 2, 2023

iohk-bors bot commented Mar 3, 2023

lehins commented Mar 3, 2023

iohk-bors bot commented Mar 3, 2023

lehins commented Feb 8, 2023 •

edited

Loading

disassembler left a comment •

edited

Loading