Add support for sending compact slates

[yeastplume]

Very WIP at the moment, with the goal being to allow a 'compact slate' mode where the sender only provides their calculated excess instead of all of their inputs/outputs. Will fill out with more detail as we go.

Notes so far:

• The recipient won't be able to verify the fee in this case
• Sender will need an extra step at some point in the workflow to repopulate the slate's transaction with their outputs. Have to figure out whether this should mean an extra explicit call in the workflow of if there's anywhere this step can be logically inserted into the existing calls.
• BPs can either be stored in the transaction context, or just regenerated at repopulation time

Changes to slate to support mimimalization:

• tx field becomes an Option
• tx field is omitted from the slate if it is None (null)
• tx field and enclosed inputs/outputs do not need to be included in the first leg of a transaction exchange. (All inputs/outputs naturally need to be present at time of posting).
• num_participants becomes an Option
• num_participants may be omitted from the slate if it is None (null), if num_participants is omitted, its value is assumed to be 2
• lock_height becomes an Option
• lock_height may be omitted from the slate if it is None (null) if lock_height is omitted, it's value is assumed to be 0
• ttl_cutoff_height may be omitted from the slate if it is None (null),
• payment_proof may be omitted from the slate if it is None (null),
• message is removed from participant_info entries
• message_sig is removed from participant_info entries
• id is removed from participant_info entries. Parties can identify themselves via private keys stored in the transaction context. The initiator must also record whether the transaction was intended to be an invoice in the context
• part_sig may be omitted from a participant_info entry if it has not yet been filled out
• receiver_signature may be omitted from payment_proof if it has not yet been filled out

[DavidBurkett]

It was discussed in the past that this would be an RFC. I still think that would be a good idea so we could all weigh in on what belongs in the slate.

[yeastplume]

Yes, (perhaps should have been clearer that) that's the plan. I need to have a play around with this and get something working, so the intention is to introduce the RFC alongside this PR once I've formed an opinion on the best approach.

[yeastplume]

With the compacted transaction workflow working, the current state of a compact slate is:

{                                                                                                                                                                                                                                                                                                               
  "version_info": {                                                                                                                                                                                                                                                                                                            
    "version": 4,                                                                                                                                                                                                                                                                                                              
    "orig_version": 4,                                                                                                                                                                                                                                                                                                         
    "block_header_version": 3                                                                                                                                                                                                                                                                                                  
  },                                                                                                                                                           
  "num_participants": 2,                                                                                                                                       
  "id": "106340df-0854-4fa2-9655-1ad5fda3a24b",                                                                                                                
  "tx": null,                                                                                                                                                                                                                                                                                                                  
  "excess": "0814a621184c6190dccc5f70c0edace9ea95e06e8c9fdd3fccc10298a33939bdb5",                                                                                                                                                                                                                                              
  "is_compact": true,                                                                                                                                          
  "amount": "60000000000",                                                                                                                                                                                                                                                                                                     
  "fee": "0",                                                                                                                                                                                                                                                                                                                  
  "height": "14",                                                                                                                                                                                                                                                                                                              
  "lock_height": "0",                                                                                                                                                                                                                                                                                                          
  "ttl_cutoff_height": null,                                                                                                                                                                                                                                                                                                   
  "participant_data": [                                                                                                                                                                                                                                                                                                        
    {                                                                                                                                                                                                                                                                                                                          
      "id": "1",                                                                                                                                               
      "public_blind_excess": "02e2dcb9cde8a1e536dc1ddf3fe16ec7c270928792c300d9ce080ca052c3138e19",                                                             
      "public_nonce": "025be5ec2a55e8c082c004e3459660f1a63cf5d6c230e343069163d0b8b9ce4fd3",                                                                                                                                                                                                                                    
      "part_sig": null,                                                                                                                                                                                                                                                                                                        
      "message": null,                                                                                                                                                                                                                                                                                                         
      "message_sig": null                                                                                                                                                                                                                                                                                                      
    }                                                                                                                                                          
  ],                                                                                                                                                           
  "payment_proof": null                                                                                                                                        
}

Summarizing next steps based on discussions in the wallet dev transaction building channel.

• Slates should not output fields if they're null
• orig_version is no longer necessary
• message and message_sig can go altogether
• num_participants can be assumed to be 2 if the field doesn't exist
• payment_proof and ttl_cutoff_height can remain, if desired, (but do not have to exist in the slate)
• If we assume all V4 slates start off as compacted, we can lose the is_compact field

Next steps would be to reduce the initial compacted slate to something like the following:

{                                                                                                                                                                                                                                                                                                               
  "version_info": {                                                                                                                                                                                                                                                                                                            
    "version": 4,                                                                                                                                                                                                                                                                                                              
    "block_header_version": 3                                                                                                                                                                                                                                                                                                  
  },                                                                                                                                                           
  "id": "106340df-0854-4fa2-9655-1ad5fda3a24b",                                                                                                                
  "excess": "0814a621184c6190dccc5f70c0edace9ea95e06e8c9fdd3fccc10298a33939bdb5",                                                                                                                                                                                                                                              
  "amount": "60000000000",                                                                                                                                                                                                                                                                                                     
  "fee": "400",                                                                                                                                                                                                                                                                                                                  
  "participant_data": [                                                                                                                                                                                                                                                                                                        
    {                                                                                                                                                                                                                                                                                                                          
      "id": "1",                                                                                                                                               
      "public_nonce": "025be5ec2a55e8c082c004e3459660f1a63cf5d6c230e343069163d0b8b9ce4fd3",                                                                                                                                                                                                                                    
    }                                                                                                                                                          
  ]                                                                                                                                                           
}

[yeastplume]

H

[yeastplume]

Latest state of Slate:

{
  "version_info": {
    "version": 4,
    "block_header_version": 3
  },
  "id": "c10f8321-a328-4d11-9a43-c446dc942fdd",
  "amount": "1000000000",
  "fee": "8000000",
  "height": "437088",
  "participant_data": [
    {
      "public_blind_excess": "02ef37e5552a112f829e292bb031b484aaba53836cd6415aacbdb8d3d2b0c4bb9a",
      "public_nonce": "03c78da42179b80bd123f5a39f5f04e2a918679da09cad5558ebbe20953a82883e"
    }
  ]
}

And with a payment proof:

{
  "version_info": {
    "version": 4,
    "block_header_version": 3
  },
  "id": "77fcf666-2a91-4bde-b76c-2d43c3b4aff5",
  "amount": "1000000000",
  "fee": "8000000",
  "height": "437100",
  "participant_data": [
    {
      "public_blind_excess": "03097ec1d0d1d013814734b71db39a6d20ec0a4f85e32b3ac83a17afc1da1f2e5f",
      "public_nonce": "03cf874b8722648ee023932a62cba85acbbe26b5944f3a53b54963b180f8ff0794"
    }
  ],
  "payment_proof": {
    "sender_address": "7e008eb593ba17d116e282d6267a3c6aad87b910933ad34dfa4d7d2c92b6ba31",
    "receiver_address": "3a425bd5da5f0f78593251ede7fad0ecf7a95679d84b2cb405255d97ce068234"
  }
}

I've updated the top comment with the fuller list of slate changes, last couple of outstanding questions are:

• Is height necessary?
• Is version_info.block_header_version still necessary? I believe we use it during the period between forks

[lehnberg]

Nice work documenting all this and providing samples @yeastplume!

I'm putting on a nitpick hat and trying to be "clever" with ways to reduce this even further. I suspect some of this cleverness may be a bit too clever for my own good here, so please forgive me if these are stupid suggestions.

Ideas

Re-considering version_info

In your non-payment proof slate example above,

"version_info": {
    "version": 4,
    "block_header_version": 3
  }

Takes up ~16% of the slate (66/407 characters). I note you are asking whether block_header_version is even required any more, but if we assume it is, could this data instead be represented as

"version" : 4.3

Where 4 is version and 3 is block_header_version? This would be 15 characters, i.e. 77% less characters, taking up 4.2% of the reduced slate size instead (15/356 characters).

Reconsidering id

"id": "77fcf666-2a91-4bde-b76c-2d43c3b4aff5"

Takes up 10.8% of the slate (44/407 characters). How is this ID derived? Correct me if I'm wrong, but the purpose of this ID is to identify the slate in question and associate it with an open transaction in the sender's/recipient's wallet. Is a 36-character string justified for uniqueness or might we be able to derive a smaller UID here?

Reconsidering public_ keys

If we can assume that transacting parties will never be sharing their private excess or nonce, might we be able to drop the public_ in public_blind_excess and public_nonce keys?

Reconsidering participant_ keys

Similarly, if we can assume that only participants will be sharing data, is it possible to drop participant_ in participant_data without losing any information?

Reconsidering indentation

Okay, I suspect you'll hate me for this, but just want to point out that assuming a space takes up space in serialisation, just by removing indentation we'll gain 10.8% (363/407), without sacrificing a lot of readability imo:

{
"version_info": {
"version": 4,
"block_header_version": 3
},
"id": "c10f8321-a328-4d11-9a43-c446dc942fdd",
"amount": "1000000000",
"fee": "8000000",
"height": "437088",
"participant_data": [
{
"public_blind_excess": "02ef37e5552a112f829e292bb031b484aaba53836cd6415aacbdb8d3d2b0c4bb9a",
"public_nonce": "03c78da42179b80bd123f5a39f5f04e2a918679da09cad5558ebbe20953a82883e"
}
]
}

Questions

1. What is height used for in this context?
2. How does payment_proof{} work in a transaction with 3 or more participants? Assuming two receivers, are they distinguished as receiver_address1, receiver_address2, or differently?

[yeastplume]

"version" : 4.3

Yep, if we decide we need to leave the block version in something like this will work fine. Otherwise it just becomes "version": 4

Reconsidering id

I would like to keep UUID, but we can de/ser shorter UUIDs into the slate:

https://github.com/seigert/shorter-uuid-rs

Fine with all of the field renaming.

Reconsidering indentation

This is just being output with pretty-printing for reference. They can also be output without whitespace for more compactness, and the whitespace can be re-added for printing at any stage if needed.

[yeastplume]

See the RFC for continued discussion of compacting the Slate:

mimblewimble/grin-rfcs#49

This particular PR is already far larger than I'd like it to be, but it implements the parts of the RFC that required some experimentation and validation. Before this one gets much bigger, would prefer to clean it up, merge once we're a little way into the RFC process, then implement the rest in further PRs.

[yeastplume]

Ready for review/merge now provided there isn't major pushback on the RFC (which there doesn't appear to be just yet). Apologies again for the length, but further changes towards compacting the slate and conforming to the RFC should be contained in much shorter PRs.

[yeastplume]

See mimblewimble/rust-secp256k1-zkp#68 for cause of current test failures, will fix once new version of libsecp is integrated into Grin.

[yeastplume]

These directives and the Serialize trait on the Slate and included structs aren't necessary (only versioned slates are ever de/serialized), will be cleaning this up in a further PR

[yeastplume]

Created a separate branch 4.0.0/compact_slates to contain compact slate work. Will be merging PRs freely into that branch to try and retain some review atomicity, and also won't be concerned with tests passing until the final PR (particularly the JSON-RPC doctests, which will all be updated at the end).