generalize db-converter #2375
Merged
generalize db-converter #2375
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
kderme
force-pushed
the
kderme/generalize-db-converter
branch
from
7fbcc60 to
97a6fda
Compare
mrBliss
reviewed
Jul 9, 2020
kderme
force-pushed
the
kderme/generalize-db-converter
branch
from
97a6fda to
b7ed0ad
Compare
|
I tested this on a small part of mainnet and it seems to work ok. However I noticed the validation lasted way longer when the cardano flags where enabled (instead of simple Byron). Outputs with Cardano: Notice 0:16.81 vs 3:28.27 ! |
kderme
commented
Jul 13, 2020
This is worrying! This is reproducible, right? Not just a fluke? Can you try replacing the last The next step would be to start profiling. |
mrBliss
added a commit
that referenced
this pull request
Jul 13, 2020
In #2375, we discovered that validating a Byron chain took 10x as long using `CardanoBlock` than using `ByronBlock`. Looking at the profiling report, we see it being dominated by: ``` exp' src/Shelley/Spec/NonIntegral.hs:(171,1)-(175,43) 52.1 56.7 ln' src/Shelley/Spec/NonIntegral.hs:(159,1)-(162,27) 22.7 26.1 ``` Which is strange, as we're validating a Byron-only chain! It turns out that these two functions are called by `mkActiveSlotCoeff`, which is called as part of `mkShelleyLedgerConfig`, which we are calling in `completeLedgerConfig`. This happens for *each* block. To fix this, we replace the partial ledger config of Shelley with a non-partial one containing a dummy `EpochInfo Identity`. In `completeLedgerConfig` we replace the dummy `EpochInfo Identity` with the correct one. This means we only call `mkShelleyLedgerConfig` once and cache its result.
We found the cause. See #2390 for a fix. |
mrBliss
reviewed
Jul 13, 2020
mrBliss
added a commit
that referenced
this pull request
Jul 13, 2020
In #2375, we discovered that validating a Byron chain took 10x as long using `CardanoBlock` than using `ByronBlock`. Looking at the profiling report, we see it being dominated by: ``` exp' src/Shelley/Spec/NonIntegral.hs:(171,1)-(175,43) 52.1 56.7 ln' src/Shelley/Spec/NonIntegral.hs:(159,1)-(162,27) 22.7 26.1 ``` Which is strange, as we're validating a Byron-only chain! It turns out that these two functions are called by `mkActiveSlotCoeff`, which is called as part of `mkShelleyLedgerConfig`, which we are calling in `completeLedgerConfig`. This happens for *each* block. To fix this, we replace the partial ledger config of Shelley with a non-partial one containing a dummy `EpochInfo Identity`. In `completeLedgerConfig` we replace the dummy `EpochInfo Identity` with the correct one. This means we only call `mkShelleyLedgerConfig` once and cache its result.
iohk-bors bot
added a commit
that referenced
this pull request
Jul 13, 2020
2390: Avoid recomputing the ShelleyLedgerConfig r=mrBliss a=mrBliss In #2375, we discovered that validating a Byron chain took 10x as long using `CardanoBlock` than using `ByronBlock`. Looking at the profiling report, we see it being dominated by: ``` exp' src/Shelley/Spec/NonIntegral.hs:(171,1)-(175,43) 52.1 56.7 ln' src/Shelley/Spec/NonIntegral.hs:(159,1)-(162,27) 22.7 26.1 ``` Which is strange, as we're validating a Byron-only chain! It turns out that these two functions are called by `mkActiveSlotCoeff`, which is called as part of `mkShelleyLedgerConfig`, which we are calling in `completeLedgerConfig`. This happens for *each* block. To fix this, we replace the partial ledger config of Shelley with a non-partial one containing a dummy `EpochInfo Identity`. In `completeLedgerConfig` we replace the dummy `EpochInfo Identity` with the correct one. This means we only call `mkShelleyLedgerConfig` once and cache its result. Co-authored-by: Thomas Winant <thomas@well-typed.com>
kderme
force-pushed
the
kderme/generalize-db-converter
branch
from
10be7d5 to
0a8f3e2
Compare
mrBliss
reviewed
Jul 14, 2020
mrBliss
reviewed
Jul 14, 2020
kderme
force-pushed
the
kderme/generalize-db-converter
branch
from
0a8f3e2 to
4b79937
Compare
mrBliss
approved these changes
Jul 14, 2020
|
bors merge |
|
bors cancel |
|
I just realised this will break https://github.com/input-output-hk/ouroboros-network/blob/master/nix/validate-mainnet.nix We should update that too. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
#2369