EVM-friendly TipSet and ECChain formats #216

Stebalien · 2024-05-14T03:50:37Z

This patch implements the changes described in
filecoin-project/FIPs#1004.

Unfortunately, go-f3 now needs to be aware of the internal structure of TipSet objects as it sends them over the wire in one format, but signs another. This effectively reverts #149.

Specific changes:

On the wire (and in finality certificates), TipSet objects are now directly cbor-marshaled within the payload instead of first being marshaled to byte strings.
Instead of directly covering the entire tipset list with the payload signature:
1. TipSet objects are "marshaled for signing" per the FIP change proposed above.
2. These marshaled tipset objects are stuffed into a merkle tree.
3. The merkle tree root is signed (along with the payload's other fields), again according the proposed changes.

fixes #166
fixes #150

gpbft/chain.go

gpbft/chain_test.go

This patch implements the changes described in filecoin-project/FIPs#1004. Unfortunately, go-f3 now needs to be aware of the internal structure of `TipSet` objects as it sends them over the wire in one format, but signs another. This effectively reverts #149. Specific changes: 1. On the wire (and in finality certificates), `TipSet` objects are now directly cbor-marshaled within the payload instead of first being marshaled to byte strings. 2. Instead of directly covering the entire tipset list with the payload signature: 1. `TipSet` objects are "marshaled for signing" per the FIP change proposed above. 2. These marshaled tipset objects are stuffed into a merkle tree. 3. The merkle tree root is signed (along with the payload's other fields), again according the proposed changes. fixes #166

gpbft/gpbft.go

gpbft/chain.go

Stebalien · 2024-05-14T04:24:39Z

A few TODOs (needs a few more tests) but should otherwise be ready for review.

Stebalien · 2024-05-14T04:37:05Z

Hm. Apparently this is killing performance (probably all the hashing). We may need to make that pluggable (e.g., for testing).

Stebalien · 2024-05-14T16:19:04Z

Welp. Sha256 is 5x faster on my machine.

Stebalien · 2024-05-14T16:47:16Z

If I look at the total time for TestRepeat:

before - 6.3s
keccak merkle-tree - 41s
sha2 merkle-tree - 19s
sha2 merkle-tree + no tipset hashing - 13.6s

anorth

Partial review, not done yet

gpbft/chain.go

anorth · 2024-05-14T23:28:12Z

gpbft/chain.go

+
+func (ts *TipSet) MarshalForSigning() []byte {
+	var buf bytes.Buffer
+	_ = cbg.WriteByteArray(&buf, ts.TipSet)


Why is this using cbg? Just buf.Write?

Because https://github.com/filecoin-project/FIPs/blob/b98ae5f0e9c0b033279f29fa5c507e457f826947/FIPS/fip-0086.md?plain=1#L307

I'm actually not sure where the wires got crossed in the FEVM FIP. IMO, it should have either been CBOR([Cid, Cid, Cid]) or Raw(Cid || Cid || Cid). Instead we got CBOR(Cid || Cid || Cid).

Honestly, I'm tempted to change it to CBOR([Cid, Cid, Cid]) because then we'd be able to dereference the CIDs. E.g.,$TIPSET_CID/0 would refer to the CID of the first block. But if we did that, we'd probably (eventually) want a small FIP to make FEVM use the new format.

gpbft/chain.go

codecov-commenter · 2024-05-15T23:41:53Z

Codecov Report

Attention: Patch coverage is 59.78261% with 111 lines in your changes are missing coverage. Please review.

Project coverage is 71.55%. Comparing base (9828418) to head (0064000).
Report is 2 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #216      +/-   ##
==========================================
- Coverage   73.19%   71.55%   -1.64%     
==========================================
  Files          30       31       +1     
  Lines        2130     2338     +208     
==========================================
+ Hits         1559     1673     +114     
- Misses        446      535      +89     
- Partials      125      130       +5

Files	Coverage Δ
sim/adversary/decide.go	`80.00% <100.00%> (ø)`
sim/adversary/repeat.go	`85.00% <100.00%> (ø)`
sim/adversary/withhold.go	`84.46% <100.00%> (-1.95%)`	⬇️
sim/ec.go	`54.78% <100.00%> (ø)`
sim/ecchain_gen.go	`87.69% <100.00%> (-0.37%)`	⬇️
sim/host.go	`100.00% <100.00%> (ø)`
sim/options.go	`76.00% <100.00%> (ø)`
sim/signing/bls.go	`80.95% <100.00%> (+2.00%)`	⬆️
sim/signing/fake.go	`89.18% <100.00%> (+6.21%)`	⬆️
sim/tipset_gen.go	`100.00% <100.00%> (ø)`
... and 4 more

Stebalien · 2024-05-15T23:48:29Z

sim/signing/bls.go

@@ -46,3 +46,7 @@ func (b *BLSBackend) GenerateKey() (gpbft.PubKey, any) {
 	b.signersByPubKey[string(pubKeyB)] = blssig.SignerWithKeyOnG1(pubKeyB, priv)
 	return pubKeyB, priv
 }
+
+func (b *BLSBackend) MarshalPayloadForSigning(nn gpbft.NetworkName, p *gpbft.Payload) []byte {
+	return p.MarshalForSigning(nn)


I'm actually marshaling here because, in this case, I assume we don't really care about performance.

Stebalien · 2024-05-15T23:49:08Z

sim/signing/fake.go

@@ -84,3 +85,37 @@ func (s *FakeBackend) VerifyAggregate(payload, aggSig []byte, signers []gpbft.Pu
 	}
 	return nil
 }
+
+func (v *FakeBackend) MarshalPayloadForSigning(nn gpbft.NetworkName, p *gpbft.Payload) []byte {


This skips all hashing. We'll eventually hash with sha256 when signing, but only doing that once is much faster.

Stebalien · 2024-05-15T23:50:11Z

gpbft/api.go

@@ -92,4 +92,6 @@ type Host interface {
 	Signer
 	Verifier
 	DecisionReceiver
+
+	MarshalPayloadForSigning(*Payload) []byte


I added this as a new function on the base Host because it doesn't really fit on either the Signer or the Verifier. I can also create a new SignatureOps (or something) interface that combines Signer, Verifier, and MarshalPayloadForSigning.

I merged it into a Signatures interface. Open to suggestions on better names.

Bikeshedding on name; alternative suggestions:

Signing,

SignerVerifier

(don't hate me) separate all the sign related stuff into their own package called signing and call the interface Scheme then in gpbft it would read `signing.Scheme

I'll leave 3 for a future patch. Between 1, 2, and the current name... I don't have any preference.

Stebalien · 2024-05-15T23:50:35Z

gpbft/gpbft.go

@@ -400,7 +405,7 @@ func (i *instance) validateMessage(msg *GMessage) error {
 	}

 	// Check vote signature.
-	sigPayload := msg.Vote.MarshalForSigning(i.participant.host.NetworkName())
+	sigPayload := i.participant.host.MarshalPayloadForSigning(&msg.Vote)


This turned out to be "not terrible" because we already needed the host for the network name.

Stebalien · 2024-05-15T23:51:46Z

gpbft/participant_test.go

+	pt.host.On("NetworkName").Return(pt.networkName).Maybe()
+	pt.host.On("MarshalPayloadForSigning", mock.AnythingOfType("*gpbft.Payload")).
+		Return([]byte(gpbft.DOMAIN_SEPARATION_TAG + ":" + pt.networkName))



Is this the right way to do this? Basically:

We may not call NetworkName now, because that's handled inside of Host.MarshalPayloadForSigning.

Whenever we call Sign, we call Host.MarshalPayloadForSigning.

Into a single Signatures interface.

anorth

Looks solid. I suggest waiting for one more peer review on this one, though.

merkle/merkle_test.go

Kubuxu

Late but LGTM

Stebalien force-pushed the steb/merklize branch 2 times, most recently from 6d72d02 to efdd9e8 Compare May 14, 2024 04:13

Stebalien commented May 14, 2024

View reviewed changes

gpbft/chain.go Outdated Show resolved Hide resolved

gpbft/chain.go Show resolved Hide resolved

gpbft/chain_test.go Show resolved Hide resolved

Stebalien force-pushed the steb/merklize branch from efdd9e8 to a12ab48 Compare May 14, 2024 04:16

Stebalien force-pushed the steb/merklize branch from a12ab48 to d5c87aa Compare May 14, 2024 04:18

Stebalien commented May 14, 2024

View reviewed changes

gpbft/gpbft.go Show resolved Hide resolved

gpbft/chain.go Show resolved Hide resolved

gpbft/chain.go Show resolved Hide resolved

Stebalien marked this pull request as ready for review May 14, 2024 04:24

Stebalien requested a review from Kubuxu May 14, 2024 04:24

Stebalien added 2 commits May 14, 2024 08:45

test marshal payload for signing

e5f7256

slightly optimize hashing

c6f8484

Merge branch 'main' into steb/merklize

4bd81c7

Stebalien force-pushed the steb/merklize branch from 60d7dfb to 4bd81c7 Compare May 14, 2024 16:33

add a worst-case benchmark for marshalling payloads

c61a864

anorth reviewed May 14, 2024

View reviewed changes

Stebalien added 3 commits May 14, 2024 20:09

address feedback

5083f66

Fake marshaling for testing

5a0888f

Merge branch 'main' into steb/merklize

79d71cc

Stebalien commented May 15, 2024

View reviewed changes

Stebalien added 3 commits May 16, 2024 08:29

pre-allocate when marshaling tipsets

602c56e

combine signer/verifier and MarshalPayloadForSignature

0bf9cbf

Into a single Signatures interface.

tipset marshal for signing test

d58a231

anorth approved these changes May 16, 2024

View reviewed changes

merkle/merkle_test.go Show resolved Hide resolved

Test bad merkle-tree paths

0064000

masih approved these changes May 17, 2024

View reviewed changes

Stebalien added this pull request to the merge queue May 17, 2024

Merged via the queue into main with commit 837ffde May 17, 2024
3 checks passed

Stebalien deleted the steb/merklize branch May 17, 2024 15:13

Kubuxu reviewed May 20, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

EVM-friendly TipSet and ECChain formats #216

EVM-friendly TipSet and ECChain formats #216

Stebalien commented May 14, 2024 •

edited

Loading

Stebalien commented May 14, 2024

Stebalien commented May 14, 2024

Stebalien commented May 14, 2024

Stebalien commented May 14, 2024

anorth left a comment

anorth May 14, 2024

Stebalien May 15, 2024

Stebalien May 15, 2024

codecov-commenter commented May 15, 2024 •

edited

Loading

Stebalien May 15, 2024

Stebalien May 15, 2024

Stebalien May 15, 2024

Stebalien May 16, 2024

masih May 16, 2024

Stebalien May 16, 2024

Stebalien May 15, 2024

Stebalien May 15, 2024

anorth left a comment

Kubuxu left a comment

EVM-friendly TipSet and ECChain formats #216

EVM-friendly TipSet and ECChain formats #216

Conversation

Stebalien commented May 14, 2024 • edited Loading

Stebalien commented May 14, 2024

Stebalien commented May 14, 2024

Stebalien commented May 14, 2024

Stebalien commented May 14, 2024

anorth left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov-commenter commented May 15, 2024 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

anorth left a comment

Choose a reason for hiding this comment

Kubuxu left a comment

Choose a reason for hiding this comment

Stebalien commented May 14, 2024 •

edited

Loading

codecov-commenter commented May 15, 2024 •

edited

Loading