Provide low-level API for db-sync

[jasagredo]

The integration work of the first version of UTxO-HD with db-sync was intended to avoid using tables and just carrying an EmptyMK ledger state all over the execution, with the full UTxO set inside, just as before UTxO-HD.

However, it turns out that our API is too restrictive as it requires ValuesMK for ticking and applying blocks, which will then be stowed. It is perfectly fine for Consensus' concerns but not for this db-sync integration as stowing/unstowing the full UTxO set duplicates the map on each step and therefore memory consumption explodes.

We should provide lower level alternatives to the tickThenReapply function that don't expect to have the tables unstowed, i.e.

tickThenReapply' :: ... -> LedgerState blk EmptyMK -> LedgerState blk EmptyMK

Possibly this just means factoring out some of the things we do (the stowing and unstowing of values) into a separate function, something like tickThenReapplyWithTables or some such, to get:

tickThenReapplyWithTables :: ... -> LedgerState blk ValuesMK -> LedgerState blk ValuesMK
tickThenReapplyWithTables = unstow . tickThenReapply' . stow

(Probably with some applyDiffs in between).

db-sync has a half-working integration which uses ouroboros-network at ea8f657533b64ad3494d3695c945c8fb24c6a0f1.

I think we should just start a branch there and do whatever we need to do to provide this API, but even maybe only in the packages needed by db-sync, to not get into trouble fighting the tests, as this is exploratory integration work. In particular db-sync uses:

• ouroboros-consensus
• ouroboros-consensus-diffusion
• ouroboros-consensus-protocol
• ouroboros-consensus-byron
• ouroboros-consensus-shelley
• ouroboros-consensus-cardano

[kderme]

Thanks! Fyi this was my attempt to use the existing api https://github.com/input-output-hk/cardano-db-sync/blob/e841d95123d25f054ef355a5734bc123bb6dcae1/cardano-db-sync/src/Cardano/DbSync/LedgerState/ConsensusApi.hs#L30. It had performance issues, because of the need to unstow and stow back the ledger state.

[dnadales]

This depends on #99