CIP-0078? | Extended Local Chain Sync Protocol

[erikd]

(current CIP draft in branch)

[AndrewWestberg]

I believe implementing this CIP would solve: IntersectMBO/cardano-node#3691

[JaredCorduan]

Should we specify that the user may wish to request one of the three (labeled mark, set, go) snap shots? Is the live, up-to-date distribution ever needed (though note that this one changes block by block)?

[AndrewWestberg]

I think yes. It is very useful to request only 1 of mark, set, go. @papacarp at pooltool needs mark. At dripdropz, I need just the go snapshot.

[kderme]

db-sync extracts slices of ~2000 per block from the set snashot starting from the epoch boundary. The size may be bigger based on the expected number of blocks in the epoch.

https://github.com/input-output-hk/cardano-db-sync/blob/master/cardano-db-sync/src/Cardano/DbSync/Era/Shelley/Generic/StakeDist.hs#L44

[erikd]

For the sake of simpilicity, I really think this should be push only from the node to the consumer..

The live up-to-date distribution is not currently needed by db-sync.

As for which one (and making sure I have this correct, mark is the stake distributions that will be used in current_epoch + 2, set in current_epoch + 1 and set in the current_epoch). My understanding is that set is not guaranteed stable in the first part of an epoch, but I would suggest this is the one we want, but only as soon as it is stable.

[abailly-iohk]

We need both mark and set in mithril for example. Different use cases will require different sets so providing all of them would just make things simpler because AFAIK they are maintained together in the Ledger?

[papacarp]

As for which one (and making sure I have this correct, mark is the stake distributions that will be used in current_epoch + 2, set in current_epoch + 1 and set in the current_epoch). My understanding is that set is not guaranteed stable in the first part of an epoch, but I would suggest this is the one we want, but only as soon as it is stable.

Slight correction here:
mark = current_epoch + 1 (this is next epochs stake distribution)
set = current_epoch (this is the stake distribution being used CURRENTLY to make blocks)
go = current_epoch - 1 (this is used for reward calculations)

mark snapshot would not be guaranteed until after the stability window I presume but it would be good to pull whenever the consumer wants as each application will have different accuracy requirements.

[erikd]

Sorry, @papacarp, about half the time I look at the mark|set|go terminology I get it wrong :/.

[JaredCorduan]

Should we specify an event on the epoch boundary that lists the stake credentials which appeared in the reward calculation, but were de-registered when the reward were given out?

[AndrewWestberg]

I'm not sure on this one. @erikd would know if db-sync needs it in that format.

[kderme]

Should we specify an event on the epoch boundary that lists the stake credentials which appeared in the reward calculation, but were de-registered when the reward were given out?

I believe this exists already. It's called RestrainedRewards

[JaredCorduan]

Currently, @erikd and @kderme depend on the ledger events, and indeed they do use an event to sort this problem out: https://github.com/input-output-hk/cardano-ledger/blob/fd806798eb68e904435720dbcea47def27c1a0fc/eras/shelley/impl/src/Cardano/Ledger/Shelley/Rules/NewEpoch.hs#L264-L267

[erikd]

Yes, there currently is an event like this and we need to keep it.

[dcoutts]

Remember that the node does not store old ledger states, nor ledger events. Thus it cannot provide information from the ledger state for chain sync clients that need to start at the beginning of the chain. In principle it could only provide such information for the recent parts of the chain (and could only do so at the cost of considerable complexity in the node).

[kderme]

A possible solution is to extend the ChainDB with a key-value db, where key is the block hash and value is a binary blob of the ledger events. A new pair will be inserted for each applied block and the extended chainsync server will query the db and append the events to the block. Rollbacks would cause a pair delete.

Maintaining the db would be optional, so it won't affect normal operation for SPOs/wallets. Of course it would add complexity in the code. I guess another challenge is to reach consensus on the exact content and format of events.

[dcoutts]

The content and format of ledger events is very much not stable. Such a table would need to be reconstructed on every node upgrade, whenever the ledger events change. Then there's the size of such a table. It'll be big.

Again this all adds complexity.

The alternative is that the Cardano API makes it convenient to manage the ledger state client side. For resource use, the solution is that the Cardano API eventually migrates to use the same on-disk ledger state storage scheme as the node itself uses (once that functionality is stable within the node).

[erikd]

The alternative is that the Cardano API makes it convenient to manage the ledger state client side.

The size of ledger state already makes it highly inconvenient and db-sync maintaining its own copy makes db-sync fragile,difficult to maintain and a complete pain in the neck for anyone operating db-sync.

[abailly-iohk]

Providing the events as they are produced, without storing them, could be a relatively simple first step: The events are already emitted by the ledger. This won't solve all the problems and would still require some mechanism to prime db-sync when it starts from scratch. To address this latter point I have suggested writing some tool, outside of db-sync, that could populate it from and existing stored ledger state. Then db-sync would be able to catch-up with the events stream from this point. Note that this could require restarting the node from that specific point in time to ensure events are correctly emitted.

Another snag to take care of are rollbacks: I don't know how ledger currently handle events, and whether or not those events are invertible. If this were the case then rollbacks could be handled very gracefully by any events client by un-applying the events. Otherwise, extra care would be needed within db-sync or any client.

[abailly-iohk]

The alternative is that the Cardano API makes it convenient to manage the ledger state client side. For resource use, the solution is that the Cardano API eventually migrates to use the same on-disk ledger state storage scheme as the node itself uses

That's definitely desirable, eg. to have some tool or family of tools able to work with the node's raw storage and particularly ledger state, and this would be useful as suggested in my previous comment as a way to prime db-sync or any other client.
This implies defining some public storage schema and versioning it but I understand the ledger team is already working on this.

[abailly-iohk]

typo: missing closing paren

[abailly-iohk]

We need both mark and set in mithril for example. Different use cases will require different sets so providing all of them would just make things simpler because AFAIK they are maintained together in the Ledger?

[abailly-iohk]

An alternative solution that won't solve the duplicate use of RAM but would solve the duplicate use of code would be to pack consensus + ledger into a "Slim node" that works in read-only mode with an existing ChainDB, that would be packaged and built along side the node.

[abailly-iohk]

Then db-sync or any client could use that "ghost node" to replay blocks from any point in time, generate events, query ledger state and what not. I am even surprised such a tool does not already exist: Is this not somewhat akin to what db-analyser do?

[kderme]

Can't this idea be extended even more to solve the duplicate RAM problem? What if db-sync maintained its own ChainDB, connected to peers itself, ran ouroboros etc? That way it doesn't need a separate trusted node process, there is no replication, no need for protocol extension, deployment is greatly simplified etc. At the same time this adds no complexity to the node that @dcoutts understandably wants to avoid, as this complexity will live in the db-sync repository.

[abailly-iohk]

Actually, this is an idea that popped into my head recently: Make the full node's logic available as a library instead of only as an executable. Then db-sync could just embed the node's logic, and provided there's some way to plug "decorators" here and there, could just tap on node processing stuff, or poke at the current ledger state when it needs, etc...

[KtorZ]

Another alternative -- and straightforward -- solution would be to also simply "dump" information required to the calculation of implicit stuff (such as rewards) into some file as the node processes them. And either, document the format of that file or provide a thin api-layer to query its content.

Having just that would solve many problems down the line; client applications will almost always be connected to a local node and that node will have to do this work anyway. So instead of duplicating that work; simply record the information when its done and gives way to access that information later on. It could be a mere .csv or .json file recording the stake distribution on each epoch transition.

[erikd]

Oh, ok, so you would not need to run a node on a machine running db-sync at all. Not sure how this compares to mu proposal in terms of amount of work or complexity.

[abailly-iohk]

It's probably (much) more work so I'd not consider this as an alternative for this particular CIP

[erikd]

Funny, in the last company I worked for, everything was written as a library first (makes testing much easier) and then thin wrapper was added to make it into an executable.

[NetWalker108]

Then db-sync or any client could use that "ghost node" to replay blocks from any point in time, generate events, query ledger state and what not. I am even surprised such a tool does not already exist: Is this not somewhat akin to what db-analyser do?

Is this similar to what Santiago from TxPipe proposed a while back called Dolos, a Data Node?

[abailly-iohk]

I don't know, I wasn't there a while back ;) But I guess this idea is pretty much obvious and not novel so I am pretty sure other people have thought of iti.

[abailly-iohk]

A possible implementation first step would be to write a prototype funnelling existing events (or only one of them for simplicity's sake because I assume there would be need to implement serialisation) through the consensus code down to the network stack, in order to properly understand the impact of this change.

[kderme]

The alternative is that the Cardano API makes it convenient to manage the ledger state client side. For resource use, the solution is that the Cardano API eventually migrates to use the same on-disk ledger state storage scheme as the node itself uses

Is this about reusing the same utxo db that the node uses by client processes through an extended api or about replicating the utxo-db for the client?

[abailly-iohk]

Note there already exist tools that "decrypt" the ledger state and can interact with a DB: https://github.com/input-output-hk/cardano-ledger/tree/master/libs/ledger-state

[dcoutts]

@kderme I mean the Cardano API reusing the same underlying library to manage the ledger state on disk.

It could not reuse the same instance of the on-disk state, simply because that state only corresponds to the end of the chain, but the chain sync protocol allows syncing from the beginning.

[kderme]

Right, integrating and replicating utxo-hd on the client side would reduce memory but it would be very heavy on IOPS requirements. Running on a vm 2 x utxo-hd instances plus a very IO heavy Postgres for db-sync is not really ideal. UTXO-hd is still great and will benefit db-sync users even without integrating it, by cutting total memory usage almost in half.

API can definitely provide convenience, but it cannot reduce recourses usage. This would require changes on the protocol level.

[dcoutts]

Yes, fundamentally the ledger state either has to be in memory or on disk, and the latter costs IOPS.

And given that you need to be able to sync from genesis, then that involves two copies of the ledger state somewhere (whether it's client or server side).

The idea to only use ledger events and never directly access the ledger state is interesting, but it's still quite unstable (so would require replaying the chain and ledger state on every upgrade) and we don't yet have any real idea of how big the ledger events will be. We know it's big because it at least includes all of the reward payouts for every stake address every epoch.

[eyeinsky]

The Marconi indexer[1] would also benefit from having (parts of) ledger state available, and as outlined in the CIP it would be beneficial not to have two identical ledger states living in memory. Getting ledger state from the node would save any indexer needing to compute it on the client side.

---

From the comments above, an ideal situation would be what @abailly-iohk and @kderme said:

An alternative solution that won't solve the duplicate use of RAM but would solve the duplicate use of code would be to pack consensus + ledger into a "Slim node" that works in read-only mode with an existing ChainDB

What if db-sync maintained its own ChainDB, connected to peers itself, ran ouroboros etc? That way it doesn't need a separate trusted node process

I.e have db-sync, Marconi or any other blockchain indexer act as a passive node which could discover peers and synchronize similar to how the node does it now. To do this node would need to be a library that provided the code to implement it (the API being perhaps a stream of blocks and a fold function over it that would give ledger state at every block).

For now, the short term approach for Marconi for indexers that require info from ledger state is to reconstruct it on the client side -- much like dbsync does, and with the same downsides.

As an aside, the blockhain and folding over it into some desirable data structure sounds much like event sourcing, in event sourcing's terms ledger state would be called a "read model".

[1] Indexer is any program that picks out from the blockchain any info that a dApp developer might care about and filters, folds and/or stores it.

[rphair]

While all the other technical reviews are being addressed, I would happily see this promoted to a Candidate CIP especially because of the promise of a long awaited answer to IntersectMBO/cardano-node#3691 (as already mentioned).

[KtorZ]

I think there's little use-case for the UTxO state or the protocol parameters because these are straightforward to obtain from the chain-sync protocol itself.

However, anything that regards rewards is indeed a pain in the *** to work with; because rewards are implicit in the protocol. I'd love to see a quick justification here (instead of "basically" 😅 ...) stating just that. There's no way for a client application to provide reliable information on rewards without maintaining the entire ledger state; for one needs to know what everyone owns in order to calculate rewards snapshots on each era.

[erikd]

I am pretty sure that the protocol parameters are not available on chain, only the voting for the changes. Yes, the ledger rules can be applied to whatever voting is seen on chain but that is way more fragile than getting the parameters from the source.

Agree, that nobody in their right mind would want the UTxO state.

[JaredCorduan]

the protocol parameters are available on chain, NewEpochState -> EpochState -> PParams.

[erikd]

It would be really nice to have a function to pull the protocol parameters out of ledger state so that I did not have to go digging around inside it.

[JaredCorduan]

you're in luck, I have one right here: nesEs . esPp

[erikd]

I would call that "digging into the internals".

I am asking for an officially maintained function that is part of an official API.

[kderme]

The lack of a quality api is indeed an issue, but a much bigger issues imo, is the need to replicate resources outside the node, especially for ledger state data. An api cannot help there, this requires improvements on the protocol level. Api and protocol are two different topics.

[KtorZ]

Another alternative -- and straightforward -- solution would be to also simply "dump" information required to the calculation of implicit stuff (such as rewards) into some file as the node processes them. And either, document the format of that file or provide a thin api-layer to query its content.

Having just that would solve many problems down the line; client applications will almost always be connected to a local node and that node will have to do this work anyway. So instead of duplicating that work; simply record the information when its done and gives way to access that information later on. It could be a mere .csv or .json file recording the stake distribution on each epoch transition.

[KtorZ]

This has always been said to be "impossible" because the ledger does not record past events and wouldn't be able to replay any of those events that are too far in the past.

Wouldn't it make sense to start by recording the events somewhere?

[erikd]

That is effectively part of this proposal.

[KtorZ]

What is this "Scientia program" about?

[erikd]

It was a proposal put forward by John Woods during his temporary tenure at IOG.

[eyeinsky]

@KtorZ The output document isn't public, but it was a user research report of what chain indexers exist, which and why people use, pros, cons etc.

[KtorZ]

If it isn't public, I don't see why it is mentioned here 👍

[rphair]

marked as Waiting for Author because only the first few comments & reviews have been engaged with so far (as per CIP meeting today)

[KtorZ]

As discussed in today's editor meeting:

• The proposal as it stands today is a bit thin w.r.t to the solution being proposed, but it goes a great extend in detailing the problem and the use-cases. Hence it feels like the proposal would be better framed as a CPS perhaps?

• Someone also hinted to the fact that the problem described in the problem seems to be overlapping with the problems that "Dolos" from TxPipe is aiming to solve. @scarmuega offered to have a look at the proposal and chime in.

[abailly-iohk]

Perhaps we can have a quick discussion with @scarmuega and @erikd to clarify potential solution(s) and see whether or not it would make sense to transform this CIP in a CPS? There's a relatively straightforward implementation path that implies making the existing LedgerEvents available through a mini-protocol in the existing consensus code, possibly both when node is synced and db-sync is just following the chain, and through an offline/CLI tool when priming a db-sync from scratch.

[scarmuega]

After going through the whole thread, I must say that I agree with the motivation of the CIP (or CPS) and with the overall theme of moving to a more event-driven architecture.

IMHO, client-side data integrations should be implemented exclusively using the event-sourcing pattern.

In the context of this pattern, the term "command" refers to a request or instruction to perform a specific action. In contrast, an "event" is a record of something that has already happened. Integration between components of this architecture happens by sharing "events" (not commands).

In Cardano, we have this tradition of building data integration clients by parsing and re-playing commands (blocks / txs), assuming that the end-result will match the correct view of the ledger. This is only possible given the deterministic nature of Cardano, but is very inefficient and prone to errors since it requires re-implementing a lot of the domain logic.

I believe that downstream components should rely on a stream of structured domain events that convey the result of applying blocks to the ledger. AFAIK, this is what @abailly-iohk is proposing.

I know that Oura and Scrolls could be drastically simplified by using this approach, I assume that other tools such as DB-Sync, Carp, Ogmios, Kupo, etc would benefit in a similar fashion. Eager to learn, but I can't think of any specific client-side use case that benefits from having the raw cbor block as data-source (as opposed to a stream of structured domain events).

Regarding implementation strategies, there're many valid options with different levels of efforts attached to it. There's some overlap with the effort that TxPipe is doing for Dolos. This project is still on an early stage, we could revisit the scope and see how to align it with the goals described by this PR.

Happy to join a meeting for further discussion.

[eyeinsky]

Please also consider what I wrote just above as well. :) (here, not in the comment in this thread)

[eyeinsky]

@scarmuega This might sound very haskell-specific, but through local chain-sync you can easily get parsed blocks, not CBOR. For other languages, writing a parser is a problem though.

Regarding event-sourcing, the fundamental sequence of events you get are blocks/transactions. If there are other types of info a data integration client needs, then they can map and fold those into the events they need.

For resumption (when that is what "business" needs :)), the event-sourcing answer seems to be to occasionally snapshot the data structure that is being folded, and then compute onwards/backwards from them by the use of events (much like I and P frames work in video compression).

[abailly-iohk]

Thanks for chiming in @scarmuega. Yes, what you describe is exactly what I have in mind and I think a desirable future for the node: Have it be as lean as possible, and distribute information to whoever is interested in it, in both forms: The commands (blocks, txs) and the Events (change to UTxO set, to SD, rewards, ...).

[scarmuega]

@scarmuega This might sound very haskell-specific, but through local chain-sync you can easily get parsed blocks, not CBOR. For other languages, writing a parser is a problem though.

Agreed, we could consider the parsing issue as a nice-to-have, but given that we're talking about client-side integration, we should not be constrained by the hard networking requirements of the node-to-node protocol and maybe provide a developer-friendly interface. In fact, this is what Ogmios does, which IMO is what the node should provide by default.

Regarding event-sourcing, the fundamental sequence of events you get are blocks/transactions. If there are other types of info a data integration client needs, then they can map and fold those into the events they need.

This is the core of my argument. The folding is already happening at the node. The "data" available as a result of this folding is much richer than the Block / Tx in itself.

For example, a domain event representing an already folded Tx could:

• provide the input UTxOs as part of the payload (IMO, huge win for dev experience)
• explicitly show which inputs were consumed (either collateral or normal)
• explicitly show which outputs where produced (either collateral return or normal)
• embed reference scripts as part of the payload
• de-reference Byron pointer addresses

Stake reward calculation is another topic that could be improved. AFAIK (happy to be corrected), the results from reward calculations at the epoch boundary could also be exposed as events, something that is not easily accesible at the moment.

For resumption (when that is what "business" needs :)), the event-sourcing answer seems to be to occasionally snapshot the data structure that is being folded, and then compute onwards/backwards from them by the use of events (much like I and P frames work in video compression).

Nice analogy (fun fact, my last job involved working with video compression for AI). My argument is that we're trying to improve the last mile integration (Node-to-Client). Following your analogy, I see it as the HDMI cable that transports decoded video ready for consumption.

Of course, all of the above can be computed from blocks / txs, but it adds an unnecessary burden on the client side.

[abailly-iohk]

@eyeinsky This is a heavily overloaded terminology but I think that @scarmuega is right: From the point of the view of the Ledger and its state, blocks are actually commands and events are whatever gets changed in the ledger from applying those commands. Of course, from the point of view of the consensus, blocks and txs are more events because they represent "past events" that happened (eg. there's been a consensus on them).
The determinism of Ledger rules implies the mapping between the stream of commands (blocks) and the stream of events (changes to state) is injective (It's not bijective because different ordering of blocks might still lead to the same state I think).
For most users, events in this sense are thus more important because they represent the changes to the state they care about (UTxO, rewards, Statkes, ...) that have been produced according to the Ledger rules. If all you have is the stream of commands, then you'd have to recreate the rules to compute the state, a process which is guaranteed to be error prone given the complexity of those rules.

[KtorZ]

One important aspect to consider in this design is also that some events being mentioned here cannot generally be "re-generated" after the facts because the ledger only maintains an aggregated view of the chain. So, some events that may be emitted while syncing (like, rewards distribution on a particular epoch) can't necessarily be streamed again on demand from a client. Unless the ledger/consensus start also storing those events (which sounds like doable and like a fairly low hanging fruit).

[eyeinsky]

If I understand correctly, only blocks can be streamed again from an arbitrary location. To start having a valid ledger state from the middle of the chain means that there needs to be some snapshot of it. Not sure how large a ledger state is to serialize and store, but if storing all of them is too much, then having a periodic snapshot is the way to go. I.e, for getting ledger state (or ledger events) for any given chain point, find the latest snapshot before that, apply blocks until reaching the desired point.

I think fold function that makes ledger essentially boils down to a type signature like this: Block -> LedgerState -> f (LedgerState, LedgerEvents). (Don't know what the f is, but it should be something that allows for failure)

And another info point: both dbsync and foldBlocks (in cardano-api:Cardano.Api.LedgerState) don't actually reimplement the ledger rules but import that functionality from the ledger (e.g here). They do add at thin layer of code though, so mistakes are possible there.

[abailly-iohk]

only blocks can be streamed again from an arbitrary location.

If you also have the state before that the location.

having a periodic snapshot is the way to go

This is something the node does: It periodically snapshots (2 by default but that's configurable IIRC) the full ledger state so that in case of crashes, it does not have to rebuild the full state from scratch.

$ ls -l mainnet/db/ledger/
total 3981772
-rw-rw-r-- 1 curry curry 2038646827 Dec  8 15:49 78904636
-rw-rw-r-- 1 curry curry 2038676305 Dec  8 17:01 78909272

both dbsync and foldBlocks (in cardano-api:Cardano.Api.LedgerState) don't actually reimplement the ledger rules

Sure, but they need to maintain the full ledger state in memory (or serialise/deserialise it) to be able to foldBlocks

[KtorZ]

@abailly-iohk": If you also have the state before that the location.

Not sure what you mean by that? Blocks are stored in the immutableDB and can be fetched back at anytime, provided that clients can provide points on chain corresponding to those blocks. So in principle, there's no need to hold on the ledger state whatsoever to access blocks.

@abailly-iohk: This is something the node does: It periodically snapshots

Sounds like the way to go perhaps? Increasing the number of snapshots and enabling client applications to also query from those snapshots? In the end, applying blocks is relatively fast. Even ten thousands blocks take only a couple of seconds to process; so having frequent but sparse snapshots sounds interesting; especially if the snapshot frequency can be tweaked on startup (so that client applications connected to a local node can ask for more snapshots than block producers who probably care less).

@abailly-iohk: Sure, but they need to maintain the full ledger state in memory (or serialise/deserialise it) to be able to foldBlocks

Only if they seek to maintain implicit properties of the chain; like rewards (or really anything connected to rewards like the full stake distribution). Applications like kupo can maintain the entire UTxO-set without resorting to storing the ledger state at all (nor does it even maintain the UTxO set in plain memory). And, it can also maintain partial view of the UTxO set and provide access to any transaction metadata; and that is because everything in the UTxO set is the explicit consequence of a block application. There's no need to know what is the current ledger state in order to identify spent inputs or produced outputs.

It is however necessary to know that full ledger state to calculate the stake distribution at the end of an epoch and to know how much rewards should be given to each stakeholders. There's no explicit manifestation of a reward distribution in blocks. And this is the root of all problems described in this proposal.

---

Note that for a while, I've been considering doing something a bit silly which is to simply fork the cardano-node and add a little functionality to dump rewards distribution periodically (on each epoch snapshot) into a file on disk. While arguably inelegant, this would allow any application to consume this information without needing to maintain a full ledger state.
Anything else at this stage is fairly easy to obtain by processing blocks (including protocol parameters, pool registrations, assets etc..)

[abailly-iohk]

(I hate having discussions in GH PR 🙄 )

Not sure what you mean by that? Blocks are stored in the immutableDB and can be fetched back at anytime, provided that clients can provide points on chain corresponding to those blocks. So in principle, there's no need to hold on the ledger state whatsoever to access blocks.

Sure but blocks represent changes to the state of the ledger so to understand what's the result of applying a block you need a Ledger state. If you don't have one, you need to replay the entire chain of blocks from genesis.

Sounds like the way to go perhaps? Increasing the number of snapshots and enabling client applications to also query from those snapshots?

Yes, I agree. But snapshots are supposed to be "private", eg. the serialisation format is not standardised so it's only useful within the confines of the existing ledger haskell code. BTW, it's already possible to increase the frequency of snapshots through the --snapshot-interval. The number of snapshots is not exposed as a parameter but it should not be hard to change it from the default 2.

In the end, applying blocks is relatively fast. Even ten thousands blocks take only a couple of seconds to process; so having frequent but sparse snapshots sounds interesting;

Agreed.

Only if they seek to maintain implicit properties of the chain; like rewards (or really anything connected to rewards like the full stake distribution).

Not sure what you mean by implicit properties but if I look at the cardano-ledger specs, I can see (or could see if the link was still up...) there's quite a lot of rules that are defined which I definitely would not want to reimplement partially. Using raw blocks info works for UTxO but not for other stuff like rewards or stake distribution, and possibly other aspects of the ledger I am overlooking.

It is however necessary to know that full ledger state to calculate the stake distribution at the end of an epoch and to know how much rewards should be given to each stakeholders. There's no explicit manifestation of a reward distribution in blocks. And this is the root of all problems described in this proposal.

And why I and others are suggesting this problem should be solved using an Event sourced approach whic nicely complements the command sourced nature of the blockchain,

Note that for a while, I've been considering doing something a bit silly which is to simply fork the cardano-node and add a little functionality to dump rewards distribution periodically

That's the kind of things a standalone cardano-node-lib would make it easy to do.

[KtorZ]

(*) what I mean by 'implicit properties' are elements that can only determined by calculations that requires the full ledger state and for which there's no direct triggers in blocks. The best example being the distribution of rewards. There's absolutely nothing explicitly recorded on the chain that indicates what rewards were sent to what account because rewards aren't carried by transactions.

Implicit properties only exists because all ledgers agree on how to calculate them from a state. They aren't the direct consequence of a command, but rather an implicitly agreed-upon information which can only be derived from the entire ledger state.

[kderme]

In the end, applying blocks is relatively fast. Even ten thousands blocks take only a couple of seconds to process; so having frequent but sparse snapshots sounds interesting;

With ledger states snapshots the problem still remains though: parsing them means memory is duplicated, since node and clients need a copy. That's why my suggestion was to store ledger events in a dedicated db #375 (comment).

Ledger events, documented in https://github.com/input-output-hk/cardano-ledger/blob/master/docs/LedgerEvents.md, include things that are impossible (like historic rewards, stake distribution) or hard to calculate without the ledger state (epoch params, deposits, refunds). They are smaller than the whole ledger state, but still tell the full story.

[abailly-iohk]

(*) what I mean by 'implicit properties' are elements that can only determined by calculations that requires the full ledger state and for which there's no direct triggers in blocks.

That's exactly the definition of command sourcing: What the system persists is a list of commands and the state is constructed by applying the stream of commands on an initial state. This works iff the apply function is deterministic which fortunately (and intentionally) is the case for Cardano.

Implicit properties only exists because all ledgers agree on how to calculate them from a state. They aren't the direct
consequence of a command,

They are the direct consequence of interpreting a "command" in certain ways. Note that the same stream of commands could lead to different states given different interpreters.

but rather an implicitly agreed-upon information which can only be derived from the entire ledger state.

It's perfectly possible a command works on part of a ledger state. To use your previous example, we could have a rewardsCalculator interpreter that would only care about "commands" insofar as they apply to rewards (I am not an expert on rewards calculation so I don't know how much state this represents but this is certainly not the full state).

Note this contrasts with the situation in event sourcing where, to keep running the same example, the event stored would include some events about how the rewards are changed so that there would not be any need to know the reward rules to compute those rewards.

To makes things more complicated, it's perfectly possible to view a stream of events as commands and run them through an interpreter that would for example, compute some "derived property" as you name it from the stream of events/commands, be it the number of bytes modified, or the distribution of time between 2 blocks, etc.