Construct Collation and Merklize Chunks into chunkRoot #133

roveneliah · 2018-05-23T17:35:22Z

This is a PR referencing #125 to construct collation instances from serialized blobs as well as merklizing the blobs to get the chunkRoot.

Details

deserialize body into tx list and pass list into NewCollation function
created wrapper Chunks in order to use types.DeriveSha function
merklize chunks in SaveBody

terencechain

Couple fixes on the comments. Are we covering these changes in the tests?

terencechain · 2018-05-23T17:42:38Z

sharding/collation.go

+
+
+// Chunks is a wrapper around a chunk array to implement DerivableList,
+// which allows us to Merklize the chunks into the chunkRoot


missing a period after chunkRoot

terencechain · 2018-05-23T17:44:14Z

sharding/shard.go

-	col := NewCollation(header, body, nil)
+	txs, err := Deserialize(body)
+	if err != nil {
+		return nil, fmt.Errorf("cannot deserialize body", err)


"cannot deserialize body:"

terencechain · 2018-05-23T17:44:38Z

sharding/shard.go

@@ -78,7 +79,11 @@ func (s *Shard) CollationByHash(headerHash *common.Hash) (*Collation, error) {
 	}
 	// TODO: deserializes the body into a txs slice instead of using


Take off the todo

terencechain · 2018-05-23T17:47:56Z

don't forget to add package name to the commit message

…er/go-ethereum into construct-collation

terencechain · 2018-05-23T20:52:27Z

lint and tests failed, we should look into that

prestonvanloon

Tests?

prestonvanloon · 2018-05-24T01:55:56Z

sharding/shard.go

-	// right now we will just take the raw keccak256 of the body until #92 is merged.
-	chunkRoot := common.BytesToHash(body)
+	// check if body is empty and throw error.
+	if body == nil {


This only checks if the body is nil, did you also want to check len(body) == 0? I think that is the true condition for an empty body.

rauljordan · 2018-05-26T18:27:18Z

Any update on this @elihanover? Do you have a timeframe for finishing the PR?

roveneliah · 2018-05-26T21:15:35Z

Yes, focusing back on this tomorrow

roveneliah · 2018-05-27T21:50:51Z

// CalculateChunkRoot updates the collation header's chunk root based on the body.
func (c *Collation) CalculateChunkRoot() {
	// TODO: this needs to be based on blob serialization.
	// For proof of custody we need to split chunks (body) into chunk + salt and
	// take the merkle root of that.
	chunkRoot := common.BytesToHash(c.body)
	c.header.data.ChunkRoot = &chunkRoot
}

Could someone point me to any resources on "salt"? Not sure how that fits in.

… into construct-collation

nisdas · 2018-05-28T02:38:59Z

I think salt is referring to Proof of Custody. You can take a look at #112 for context on it

…er/go-ethereum into construct-collation

terencechain · 2018-05-28T05:02:49Z

Here is another good thread on Proof of Custody:
https://ethresear.ch/t/enforcing-windback-validity-and-availability-and-a-proof-of-custody/949

Utilize Proof of Custody and construct chunk root with salt may be out of the scope for this PR. You are more than welcome to skip that and open another PR down the line

terencechain · 2018-05-28T05:13:02Z

sharding/shard.go

+		return fmt.Errorf("body is empty")
+	}
+
+	chunks := Chunks(body) // wrapper allowing us to merklizing the chunks


you forgot a period after the comment

terencechain · 2018-05-28T05:13:14Z

sharding/shard.go

-	// right now we will just take the raw keccak256 of the body until #92 is merged.
-	chunkRoot := common.BytesToHash(body)
+	// check if body is empty and throw error.
+	if body == nil || len(body) == 0 { // is this check strict enough?


the check looks good to me

terencechain · 2018-05-28T05:15:35Z

sharding/shard.go

 	// nil as the third arg to MakeCollation.
-	col := NewCollation(header, body, nil)
+	txs, err := DeserializeBlobToTx(body)


Why does this have to do with construct collation and merklize chunks into chunk root?

roveneliah · 2018-06-01T16:19:00Z

@terenc3t just let me know. Either way, I'll work on that either today or this weekend.

roveneliah · 2018-06-07T14:45:00Z

@rauljordan thanks for the feedback. I'll make changes today.

prestonvanloon · 2018-06-08T23:19:12Z

sharding/collation_test.go

+
+	c.CalculateChunkRoot()
+	salt := []byte{1, 0x9f}
+	fmt.Printf("%x\n", c.header.data.ChunkRoot)


Please remove all of these print statements from this test

…er/go-ethereum into construct-collation

rauljordan

Very nice, thank you for doing this @elihanover!

terencechain · 2018-06-13T14:11:51Z

LGTM. Thanks for the PR @elihanover!

prestonvanloon · 2018-06-13T14:31:05Z

sharding/shard.go

-	// right now we will just take the raw keccak256 of the body until #92 is merged.
-	chunkRoot := common.BytesToHash(body)
+	if len(body) == 0 {
+		return fmt.Errorf("body is empty")


Please do not use fmt if not formatting. Use errors.New

roveneliah · 2018-06-13T16:45:58Z

Pulled the most recent code and ran the whisperv6 test. It worked for me, so not sure why it's failing with Travis..

terencechain · 2018-06-13T17:29:54Z

Restarted Travis. It's flaky

rawfalafel · 2018-06-14T03:43:13Z

sharding/collation.go

+}
+
+// ConvertBackToTx converts raw blobs back to their original transactions.
+func ConvertBackToTx(rawBlobs []utils.RawBlob) ([]*types.Transaction, error) {


There's an identical function, convertRawBlobToTx below

Good catch. Let's delete one of them

Construct Collation and Merklize Chunks into chunkRoot

Construct Collation and Merklize Chunks into chunkRoot Former-commit-id: 1c70ba6

Construct Collation and Merklize Chunks into chunkRoot Former-commit-id: d4ed64c3d3802f1bfc330d6d4c7ee18109cd3f85 [formerly 1c70ba6] Former-commit-id: 1ab7be7

* Init Week0 * Added week 0 updates * republish hackmd public link * Week 1 and name updates * EPF Week 2 Updates --------- Co-authored-by: Mário Havel <61149543+taxmeifyoucan@users.noreply.github.com>

roveneliah added 3 commits May 22, 2018 13:37

deserialize into txn list

af225dc

sharding: merklize body into chunkroot and save in db

6ab9e0e

Merge branch 'master' into construct-collation

4c2e070

terencechain requested changes May 23, 2018

View reviewed changes

roveneliah added 2 commits May 23, 2018 10:51

sharding: error and comment formatting

41a7c06

Merge branch 'construct-collation' of https://www.github.com/elihanov…

11026d4

…er/go-ethereum into construct-collation

prestonvanloon reviewed May 24, 2018

View reviewed changes

rauljordan changed the title ~~Construct collation and merklize chunks into chunkRoot~~ Construct Collation and Merklize Chunks into chunkRoot May 24, 2018

prestonvanloon added this to the Ruby milestone May 25, 2018

roveneliah added 3 commits May 27, 2018 15:03

sharding: chunkRoot based on seralized blobs across package and tests

a4e7f99

Merge branch 'master' into construct-collation

e89d86a

Merge branch 'master' of https://github.com/prysmaticlabs/geth-sharding…

ac93120

… into construct-collation

roveneliah added 2 commits May 27, 2018 20:20

sharding: correct deserialize function used

d30e6e6

Merge branch 'construct-collation' of https://www.github.com/elihanov…

19096b3

…er/go-ethereum into construct-collation

terencechain reviewed May 28, 2018

View reviewed changes

roveneliah added 6 commits May 27, 2018 23:46

sharding: comments

a749717

sharding: gofmt

7398ca3

sharding: removed redundant condition

c7c3ef1

Merge branch 'master' into construct-collation

2a7f98e

Merge branch 'master' into construct-collation

db1ac93

Merge branch 'master' into construct-collation

71777f8

roveneliah added 4 commits June 7, 2018 09:51

sharding: clearer error messages, removed unnecessary lines

257486a

sharding: BytesToChunks takes body as parameter

5c58544

linting

05fd109

Merge branch 'master' into construct-collation

94093e0

prestonvanloon requested changes Jun 8, 2018

View reviewed changes

roveneliah added 6 commits June 12, 2018 10:11

sharding: treated rlp error similar to transactions

c5be442

sharding: fix merge conflict

c30afe1

Merge branch 'construct-collation' of https://www.github.com/elihanov…

525a64b

…er/go-ethereum into construct-collation

Merge branch 'master' into construct-collation

d6f0763

Merge branch 'construct-collation' of https://www.github.com/elihanov…

6322c2c

…er/go-ethereum into construct-collation

Merge branch 'master' into construct-collation

4cdf98b

rauljordan approved these changes Jun 13, 2018

View reviewed changes

terencechain approved these changes Jun 13, 2018

View reviewed changes

prestonvanloon requested changes Jun 13, 2018

View reviewed changes

roveneliah added 2 commits June 13, 2018 08:21

sharding: corrected error type

92b39ac

Merge branch 'master' into construct-collation

c5895d8

Merge branch 'master' into construct-collation

2f84b58

prestonvanloon approved these changes Jun 14, 2018

View reviewed changes

rauljordan merged commit 0c8aac1 into prysmaticlabs:master Jun 14, 2018

rauljordan mentioned this pull request Jun 14, 2018

Revert "Construct Collation and Merklize Chunks into chunkRoot" #180

Closed

rawfalafel reviewed Jun 14, 2018

View reviewed changes

prestonvanloon pushed a commit that referenced this pull request Jul 22, 2018

Merge pull request #133 from elihanover/construct-collation

1c70ba6

Construct Collation and Merklize Chunks into chunkRoot

prestonvanloon pushed a commit that referenced this pull request Jul 22, 2018

Merge pull request #133 from elihanover/construct-collation

1ab7be7

Construct Collation and Merklize Chunks into chunkRoot Former-commit-id: 1c70ba6

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Construct Collation and Merklize Chunks into chunkRoot #133

Construct Collation and Merklize Chunks into chunkRoot #133

roveneliah commented May 23, 2018

terencechain left a comment

terencechain May 23, 2018

terencechain May 23, 2018

terencechain May 23, 2018

terencechain commented May 23, 2018

terencechain commented May 23, 2018

prestonvanloon left a comment

prestonvanloon May 24, 2018

rauljordan commented May 26, 2018

roveneliah commented May 26, 2018

roveneliah commented May 27, 2018

nisdas commented May 28, 2018 •

edited

Loading

terencechain commented May 28, 2018

terencechain May 28, 2018

terencechain May 28, 2018

terencechain May 28, 2018

roveneliah commented Jun 1, 2018

roveneliah commented Jun 7, 2018

prestonvanloon Jun 8, 2018

rauljordan left a comment

terencechain commented Jun 13, 2018

prestonvanloon Jun 13, 2018

roveneliah commented Jun 13, 2018

terencechain commented Jun 13, 2018

rawfalafel Jun 14, 2018

terencechain Jun 14, 2018



		// Chunks is a wrapper around a chunk array to implement DerivableList,
		// which allows us to Merklize the chunks into the chunkRoot

		@@ -78,7 +79,11 @@ func (s Shard) CollationByHash(headerHash common.Hash) (*Collation, error) {
		}
		// TODO: deserializes the body into a txs slice instead of using

Construct Collation and Merklize Chunks into chunkRoot #133

Construct Collation and Merklize Chunks into chunkRoot #133

Conversation

roveneliah commented May 23, 2018

Details

terencechain left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

terencechain commented May 23, 2018

terencechain commented May 23, 2018

prestonvanloon left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rauljordan commented May 26, 2018

roveneliah commented May 26, 2018

roveneliah commented May 27, 2018

nisdas commented May 28, 2018 • edited Loading

terencechain commented May 28, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

roveneliah commented Jun 1, 2018

roveneliah commented Jun 7, 2018

Choose a reason for hiding this comment

rauljordan left a comment

Choose a reason for hiding this comment

terencechain commented Jun 13, 2018

Choose a reason for hiding this comment

roveneliah commented Jun 13, 2018

terencechain commented Jun 13, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nisdas commented May 28, 2018 •

edited

Loading