Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[WIP] Relative Kernels #19

Draft
wants to merge 4 commits into
base: master
from

Conversation

@antiochp
Copy link
Member

commented Aug 9, 2019

@antiochp antiochp self-assigned this Aug 9, 2019

@tromp

This comment has been minimized.

Copy link

commented Aug 9, 2019

A few days ago while pondering relative kernels, I thought it might be useful to introduce reference kernels. A relative kernel would only be able to refer to a reference kernel. Which would require a slightly higher fee. That means there would be much fewer kernels to refer to, which would reduce resource usage of the necessary indexing data structures.

But then I realized such a distinction on kernel types is a privacy leak, so I scrapped the idea...

Anyway, thought I'd write it down for future reference:-)

@tromp

This comment has been minimized.

Copy link

commented Aug 9, 2019

Regarding

A "relative lock height" kernel would need an additional 32 bytes of data to store a reference

It only needs 8 bytes, namely the leaf or full index of the reference kernel in the kernel MMR.
When computing the kernel_sig_msg, this index is used to serialize the actual referenced kernel.
When verifying the kernel, this index is needed anyway to do the relative height check.

We could allow the broadcasted tx to have the full 32 byte reference kernel, and let the miners convert it to an 8 byte index. But normally you would only broadcast such a tx if you see the references kernel being confirmed, so it may be simpler to require the broadcaster to set the index. Then only the slate would have the 32 byte reference kernel.

@antiochp

This comment has been minimized.

Copy link
Member Author

commented Aug 12, 2019

It only needs 8 bytes, namely the leaf or full index of the reference kernel in the kernel MMR.

Yes. We have discussed this before I think? And then I forgot about it...
The slight complexity is who is responsible for converting indices to commitments, as you say.
Let me think about this a bit more and we can add this to the RFC.

But normally you would only broadcast such a tx if you see the references kernel being confirmed, so it may be simpler to require the broadcaster to set the index.

Agreed. I think this is the conclusion we came to previously. The parties transacting (via a slate) would require the full excess commitment. But once the tx is built and ready to broadcast we can translate these to the more compact MMR indices.

The kernel excess would be required when verifying the signature, but verifiers can quickly lookup the excess based via the provided MMR index.

The one edge-case here that we need to think through. If we do support a height of 0 for a relative lock height then we can potentially build a tx that references a kernel that does not yet exist in the MMR, it will be added in the same block as the new tx. In this situation we would not yet have a MMR index for the referenced kernel. Maybe we do not allow this and the relative height must be non-zero?

If we do allow relative height of 0 then we would potentially also need to handle the case where k1 references k2 and k2 itself has a circular reference back to k1. Maybe we do not want to go down the path of permitting this...

The big benefit of requiring relative height to be non-zero is the fact that a referenced kernel must have a lower MMR index than the kernel referencing it. This would not always be the case for two txs within the same block as no guarantees are made about the order of kernels within a block.

@tromp

This comment has been minimized.

Copy link

commented Aug 12, 2019

Indeed this was discussed before.
A downside is that kernel_sig_msg is no longer a function of just the kernel features, as it depends on the kernel MMR. We probably have to accept the ugliness of passing that as an argument to kernel_sig_msg...

@antiochp

This comment has been minimized.

Copy link
Member Author

commented Aug 12, 2019

The other complexity here is how to deal with forks and reorgs if we are passing the MMR index as part of the tx (and not the referenced kernel excess).
If block 100 puts kernel k1 at pos 100 but block 100' puts k1 at pos 101 then some nodes potentially see a subsequent relative height locked tx as invalid based on what appears to be an incorrect kernel index.

So we may need to reference kernels by the excess commitment up to the point where they are actually added to a block (by the miner).

@tromp

This comment has been minimized.

Copy link

commented Aug 12, 2019

Ah yes, you're right; replacement of the commitment by an index can only happen when the later kernel is added to a block (as the past is immutable from this block's viewpoint); i.e. by a miner.

@antiochp

This comment has been minimized.

Copy link
Member Author

commented Aug 13, 2019

Ah yes, you're right; replacement of the commitment by an index can only happen when the later kernel is added to a block (as the past is immutable from this block's viewpoint); i.e. by a miner.

Yes. I remember some of the earlier conversations now (one reason why we should be documenting these proposals as RFCs...) As long as txs are re-orderable (i.e. before inclusion in a block) we need to explicitly reference the excess commitment itself.

And we do need to be careful with handling reorgs and forks from the perspective of the txpool as a reorg can order previously accepted txs differently - txs with relative lock heights can re-enter the txpool and we need to revalidate the lock height criteria carefully.

@tromp

This comment has been minimized.

Copy link

commented Aug 25, 2019

In the top comment I wrote:

That means there would be much fewer kernels to refer to, which would reduce resource usage of the necessary indexing data structures.

After more careful thought, there is another resource we should be more concerned about.
That is the effort needed in the Initial Block Download. For Grin to be truly scalable, we should aim the IBD effort to be at most linear in UTXO set size, not in total number of transactions. So how can we hope to achieve that?
In my opinion, the most promising approach is through the use of proof techniques such as STARKs
(Scalable Transparent ARguments-of-Knowledge). The idea is to accompany the UTXO set with some proof that there exists some valid kernel history, consistent with all block headers, that validates said UTXO set. It is currently quite feasible to construct such proofs on a beefy machine for around a million kernels, and with future hardware and software improvements this is expected to go up by another 2 orders of magnitude. We could aim to build one STARK proof for every year of kernel history.

BUT this approach would be severely handicapped by kernels that can freely reference older kernels.
Performing an IBD with a STARK proof of kernel history, only provides knowledge of kernels beyond the STARK timeframe, which can be as short as the STATE_SYNC_THRESHOLD of two days.

It may therefore be wise to limit the relative height to that same horizon.
If we then want to break up the whole block chain history into shorter segments that can be handled by STARKs, then the segments would only need to overlap by this horizon, in order to validate all relative height constraints.

Actually, I find 2 days to be a little short, and would prefer to change STATE_SYNC_THRESHOLD to match CUT_THROUGH_HORIZON, i.e. one week.

Can anyone think of a use case of relative kernels where one week would be considered insufficient?

@antiochp

This comment has been minimized.

Copy link
Member Author

commented Aug 28, 2019

Ignoring specifics around STARKs for a second - it sounds like the issue could be restated to -

Do we want to allow arbitrary relative heights between relative kernels and the kernels they reference? And if we want to limit this to a shorter period what should the limit be?

Intuitively it makes sense to have some kind of limit and not simply allow arbitrary relative heights here.

Some follow up questions -

  • Would it be easier to introduce a more restrictive limit at a later date?
    • Can we do this via a soft fork?
  • Or would it be easier to start off with one that we then decide is overly restrictive - can we relax it at a later date?
    • Does this need to be a hard fork?
@tromp

This comment has been minimized.

Copy link

commented Aug 28, 2019

Do we want to allow arbitrary relative heights between relative kernels and the kernels they reference?

I would argue that we do.

And if we want to limit this to a shorter period what should the limit be?

I think a week is quite reasonable. With payment channels, monitoring the chain for illicit close transactions is a minimal burden with a 1 week window. I suspect very few people would want to wait longer than a week to reclaim funds from a channel closure. Most will probably set it shorter.
Btw, a one week limite allows the relative height to fit in 16 bits...

Would it be easier to introduce a more restrictive limit at a later date? Can we do this via a soft fork?

Yes, a graph-rate majority of miners can enforce shorter limits, by considering longer limits invalid.

Or would it be easier to start off with one that we then decide is overly restrictive - can we relax it at a later date? Does this need to be a hard fork?

That is indeed a hard-fork, as all nodes would need to upgrade to accept the longer limits.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
2 participants
You can’t perform that action at this time.