ImmutableDB: allow to lookup blocks by slot, leverage in db-{analyser,truncater} #1143
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
amesgen
force-pushed
the
amesgen/db-immutaliser
branch
2 times, most recently
from
e01abc7
to
9939044
Compare
amesgen
force-pushed
the
amesgen/immdb-slots
branch
from
42e0000
to
10fb20c
Compare
dnadales
approved these changes
Jun 14, 2024
ouroboros-consensus-cardano/src/unstable-cardano-tools/Cardano/Tools/DBAnalyser/Analysis.hs
Outdated
Show resolved
Hide resolved
amesgen
force-pushed
the
amesgen/db-immutaliser
branch
from
9939044
to
443e350
Compare
amesgen
force-pushed
the
amesgen/immdb-slots
branch
from
10fb20c
to
42c61f1
Compare
See later commits for how this is useful in db-{analyser,truncater}.
amesgen
force-pushed
the
amesgen/immdb-slots
branch
from
42c61f1
to
7877e43
Compare
This also slightly changes the semantics: Previously, it would truncate to the
block with the largest slot that is smaller or equal to the slot argument. Now,
it will only truncate to the (non-EBB) block in exactly that slot, and fail
otherwise.
In fact, the new behavior more closely corresponds to the CLI description:
--truncate-after-slot SLOT_NUMBER
The slot number of the intended new tip of the chain
after truncation
amesgen
force-pushed
the
amesgen/immdb-slots
branch
from
7877e43
to
5747d3c
Compare
facundominguez
pushed a commit
that referenced
this pull request
Jun 25, 2024
…,truncater} (#1143) This PR adds a new function to the internal ImmutableDB API: ```haskell -- | Get the hash of the block in the given slot. If the slot contains both -- an EBB and a non-EBB, return the hash of the non-EBB. getHashForSlot :: SlotNo -> m (Maybe (HeaderHash blk)) ``` It is then used in two ways: - In db-analyser: Many analyses do not actually need a ledger state to perform their work. This PR adds support for avoiding this redundant and long work on startup. However, the existing ImmutableDB streaming API needs a point (previously read from the ledger state), not just a slot to start streaming. This is where `getHashForSlot` comes in. - In db-truncater: Previously, truncating after a slot took linear time (by iterating over the entire database). With `getHashForSlot`, it is easy to now change it to take constant time. Note that the behavior changes slightly, see the commit message for details.
facundominguez
pushed a commit
that referenced
this pull request
Jun 28, 2024
…,truncater} (#1143) This PR adds a new function to the internal ImmutableDB API: ```haskell -- | Get the hash of the block in the given slot. If the slot contains both -- an EBB and a non-EBB, return the hash of the non-EBB. getHashForSlot :: SlotNo -> m (Maybe (HeaderHash blk)) ``` It is then used in two ways: - In db-analyser: Many analyses do not actually need a ledger state to perform their work. This PR adds support for avoiding this redundant and long work on startup. However, the existing ImmutableDB streaming API needs a point (previously read from the ledger state), not just a slot to start streaming. This is where `getHashForSlot` comes in. - In db-truncater: Previously, truncating after a slot took linear time (by iterating over the entire database). With `getHashForSlot`, it is easy to now change it to take constant time. Note that the behavior changes slightly, see the commit message for details.
This was referenced Aug 1, 2024
github-merge-queue bot
pushed a commit
that referenced
this pull request
Aug 2, 2024
Closes #1202 This PR reverts the behavioral change of #1143, specifically 5747d3c. Concretely, `--truncate-after-slot slotNo` will now remove all blocks with a slot number higher than `slotNo` in the ImmutableDB, but does not require that a block with exactly that slot number exists. This is convenient eg for truncating all blocks after an epoch without having to find out the exact slot of the last block in the epoch just before. At the same time, the run time is still much faster than before #1143: We iteratively check all slot numbers descending from the given one, and truncate to the first point that is in the ImmutableDB. As realistic ImmutableDBs are only somewhat sparse (active slot coefficient is `f = 1/20`), this should be very fast (ie still constant time in the length of the chain if we consider the slot distance between any two adjacent blocks to be bounded). In addition, we explicitly check whether the given argument is beyond the tip of the ImmutableDB, and immediately exit (successfully) in that case.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR adds a new function to the internal ImmutableDB API:
It is then used in two ways:
In db-analyser: Many analyses do not actually need a ledger state to perform their work. This PR adds support for avoiding this redundant and long work on startup. However, the existing ImmutableDB streaming API needs a point (previously read from the ledger state), not just a slot to start streaming. This is where
getHashForSlotcomes in.In db-truncater: Previously, truncating after a slot took linear time (by iterating over the entire database). With
getHashForSlot, it is easy to now change it to take constant time. Note that the behavior changes slightly, see the commit message for details.