Shelley: avoid expensive equality check of the LedgerView #2512
Conversation
There was a problem hiding this comment.
After enabling profiling-detail: all-functions in my cabal.project.local, my profiling reports became much more complete (no more fn_3 (indiv) taking the majority of the time 🙄). I found that this equality was a major time/alloc sink.
I think this, in combination with IntersectMBO/cardano-ledger#1775 will improve validation speed significantly. For example, my profile of db-validator on the first Shelley era went from ~3 min to 20s (that's with PR 1775 applied).
| = SL.lvProtParams oldLedgerView /= SL.lvProtParams newLedgerView || | ||
| SL.lvGenDelegs oldLedgerView /= SL.lvGenDelegs newLedgerView |
There was a problem hiding this comment.
@nc6 Do these two values ever change within an epoch or also only at the epoch boundary?
If the latter, then we can improve/simplify things even further.
There was a problem hiding this comment.
Protocol parameters only change at an epoch boundary. Genesis delegates may change within an epoch.
There was a problem hiding this comment.
Great, updated. Thanks!
We check whether the `LedgerView` changed after (re)applying each block, to see whether we should store a snapshot of the old one. The `LedgerView` includes the overlay schedule (a map with 432,000 entries!) and the pool distribution (a map with an entry for each stakepool). Comparing these two data structures for equality is very costly. Since the overlay schedule and the pool distribution only change when transitioning to a new epoch, we can use the epoch number as a proxy for the equality check of those two data structures. The protocol parameters can also only change at the epoch boundary. The genesis delegates may change within an epoch.
mrBliss
force-pushed
the
mrBliss/avoid-ledgerview-equality
branch
from
cdbd5e9
to
8e7b5e2
Compare
|
bors merge |
Highlights: * IntersectMBO/cardano-ledger#1784 * IntersectMBO/cardano-ledger#1785 * IntersectMBO/cardano-ledger#1779 * IntersectMBO/ouroboros-network#2512 * Remove unused source-repository-package cardano-sl-x509 (this package required the `ip < 1.5` constraint which conflicted with the `primitive >= 0.7.1.0` lower bound in `cardano-ledger-specs`). * Remove unused http-client dependency from stack.yaml * Remove duplicate from cabal.project
Highlights: * IntersectMBO/cardano-ledger#1784 * IntersectMBO/cardano-ledger#1785 * IntersectMBO/cardano-ledger#1779 * IntersectMBO/cardano-ledger#1785 * IntersectMBO/cardano-ledger#1742 * IntersectMBO/ouroboros-network#2512 * Remove unused source-repository-package cardano-sl-x509 (this package required the `ip < 1.5` constraint which conflicted with the `primitive >= 0.7.1.0` lower bound in `cardano-ledger-specs`). * Remove unused http-client dependency from stack.yaml * Remove duplicate from cabal.project
1696: Update dependencies r=mrBliss a=mrBliss Highlights: * IntersectMBO/cardano-ledger#1784 * IntersectMBO/cardano-ledger#1785 * IntersectMBO/cardano-ledger#1779 * IntersectMBO/cardano-ledger#1785 * IntersectMBO/cardano-ledger#1742 * IntersectMBO/ouroboros-network#2512 Co-authored-by: Thomas Winant <thomas@well-typed.com> Co-authored-by: Samuel Leathers <samuel.leathers@iohk.io>
|
Note to myself: when we run into a similar situation again, i.e., having to check the equality of huge data structures, and there isn't a cheap proxy like in this case, there's always: https://hackage.haskell.org/package/containers-0.6.3.1/docs/src/Utils.Containers.Internal.PtrEquality.html#ptrEq 😄 |
1696: Update dependencies r=mrBliss a=mrBliss Highlights: * IntersectMBO/cardano-ledger#1784 * IntersectMBO/cardano-ledger#1785 * IntersectMBO/cardano-ledger#1779 * IntersectMBO/cardano-ledger#1785 * IntersectMBO/cardano-ledger#1742 * IntersectMBO/ouroboros-network#2512 Co-authored-by: Thomas Winant <thomas@well-typed.com> Co-authored-by: Samuel Leathers <samuel.leathers@iohk.io>
We check whether the
LedgerViewchanged after (re)applying each block, to seewhether we should store a snapshot of the old one. The
LedgerViewincludes theoverlay schedule (a map with 432,000 entries!) and the pool distribution (a map
with an entry for each stakepool). Comparing these two data structures for
equality is very costly. Since the overlay schedule and the pool distribution
only change when transitioning to a new epoch, we can use the epoch number as a
proxy for the equality check of those two data structures.
The other two data structures, i.e.,
lvProtParamsandlvGenDelegs, are smalland can be checked time.