Implement Proposers Submitting Collations onto SMC #111

terencechain · 2018-05-10T17:28:01Z

This is a PR referencing #106 to implement a proposer service/protocol that can be used to create collation, sign collation, check SMC header availability and add header to the main chain

Context

To align proposer implementation with minimal sharding protocol. Proposers will be responsible for create collation, sign collation, check collation header availability on the mainchain, add collation header to the mainchain via SMC, and broadcast collation body to the shard network it lives on.

Requirements for Merge

Create a Proposer struct/interface
Define common methods such as create collation, sign collation, check header hasn't been added in a given period and shard, add header
Pass all the tests for the methods below

Functions Being Implemented

func (p *Proposer) CreateCollation(node.Node, shardId *big.Int, period *big.Int, txs []*types.Transaction) (*sharding.Collation, error) {}
func (p *Proposer) CheckHeaderInSMC(shardID *big.Int, period *big.Int) (bool, error) {}
func (p *Proposer) AddHeaderToSMC(header *CollationHeader) (error) {}

terencechain · 2018-05-18T17:12:28Z

The architecture of how proposer proposes collation will be based on #122

sharding: beginning proposer service tests

prestonvanloon · 2018-06-02T00:14:00Z

sharding/node/node.go

@@ -182,6 +183,13 @@ func (n *shardingNode) CreateTXOpts(value *big.Int) (*bind.TransactOpts, error)
 	}, nil
 }

+// Sign signs the hash of collationHeader contents by
+// using default account on keystore and returns signed signature.
+func (n *shardingNode) Sign(hash common.Hash) ([]byte, error) {


Do we not have tests for this file?

This will be addressed with #149

prestonvanloon · 2018-06-02T00:23:28Z

sharding/proposer/service_test.go

+	return nil
+}
+
+func (m *mockNode) Close() {


Yikes, there are ~75 lines of mockNode and many of these are unused.
Is there a better way to do this?
Can we declare an interface for the proposer expected methods?

Take a look at effective go interface definition: https://golang.org/doc/effective_go.html#interfaces
Interfaces in Go provide a way to specify the behavior of an object: if something can do this, then it can be used here.

What this means to me: we could compose some interface to explain what a proposer-compatible node looks like.

type proposeNode interface { SMC // SMC interface that has SMCTransactor() and SMCCaller() Account() (Account, err) CreateTXOpts() (...) }

This ~75 lines of setup with some methods that aren't used by this package make me think that we are not properly making use of interfaces. I.e. The consumer (sharding/proposer) should define the interface that it needs to properly function and the producer (sharding/node) simply provides a full struct.

Hopefully, that will reduce boilerplate on the test setup and provide clear isolation of responsibilities.

This will be addressed with #149

prestonvanloon · 2018-06-02T00:32:52Z

sharding/proposer/service_test.go

+		t.Errorf("Create collation failed: %v", err)
+	}
+	if collation.Header().Period().Cmp(big.NewInt(5)) != 0 {
+		t.Errorf("Incorrect collationd period, want 5, got %v ", collation.Header().Period())


prestonvanloon · 2018-06-02T00:34:34Z

sharding/proposer/service_test.go

+func TestCheckCollation(t *testing.T) {
+	backend, smc, err := setup()
+	if err != nil {
+		t.Fatalf("Failed to deploy SMC contract: %v", err)


Rather than repeating this in every test, can you just pass t to setup() and check this within the setup?

prestonvanloon · 2018-06-02T00:35:44Z

sharding/proposer/service_test.go

+		t.Errorf("Can not check header availability: %v", err)
+	}
+	if a {
+		t.Errorf("Check header avability should return: %v", !a)


prestonvanloon · 2018-06-02T00:35:50Z

sharding/proposer/service_test.go

+	// normal test case 2: check if we can add header for period 2, should return true.
+	a, err = checkHeaderAvailability(node, big.NewInt(0), big.NewInt(2))
+	if !a {
+		t.Errorf("Check header avability should return: %v", !a)


prestonvanloon · 2018-06-02T00:42:12Z

sharding/proposer/service_test.go

+	if collationFromSMC.Proposer != node.Account().Address {
+		t.Errorf("Incorrect proposer address, got %v", *collation.ProposerAddress())
+	}
+	if common.BytesToHash(collationFromSMC.ChunkRoot[:]) != *collation.Header().ChunkRoot() {


why do you need [:], what does that do?

[:] converts byte[32] to byte[]

why do you need it?

BytesToHash takes byte[] mean while ChunkRoot in SMC is byte[32]. I need to convert byte[32] to byte[]

prestonvanloon · 2018-06-02T00:44:57Z

sharding/proposer/service_test.go

+		t.Errorf("Can not check header availability: %v", err)
+	}
+	if a {
+		t.Errorf("Check header avability should return: %v", !a)


Also, we know that it should return a falsey value. Wouldn't it be better to report the actual value?

Ah. Good catch!

prestonvanloon · 2018-06-02T00:47:38Z

sharding/proposer/proposer.go

+	}
+
+	var chunkRoot [32]byte
+	copy(chunkRoot[:], collation.Header().ChunkRoot().Bytes())


Why do you need to make a copy here? Are you mutating it?

Not mutating it. I want to convert []byte (what .Bytes() returns) to [32]byte (what SMC takes). Any better idea on how to do that?

Does it work if you do this?

chunkRoot := collation.Header().ChunkRoot().Bytes()[:32]

This should reference the same array in-memory, so you aren't copying the data.

That doesn't work. I really don't want to copy data. I haven't found a nice way to do it without copying

What doesn't work about it? This should be a slice referencing the same internal array.

It's still []byte not [32]byte. Is this what you are suggesting?

array :=[]byte{0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,31} array32 := array[:32] fmt.Printf("%T", array32)

I think we can either use the copy function or by assigning array elements one by one but all of these copy data...

After more research, the Go language spec does not allow you to access "directly" the underlying, backing array of a slice. We can do that using package unsafe, but as its name hints: it's unsafe

var ( s []byte // slice a *[32]byte // pointer to array ) s = make([]byte, 32) a = byte32(s) func byte32(s []byte) (a *[32]byte) { if len(a) <= len(s) { a = (*[len(a)]byte)(unsafe.Pointer(&s[0])) } return a }

prestonvanloon · 2018-06-02T00:47:54Z

sharding/proposer/proposer.go

+	var chunkRoot [32]byte
+	copy(chunkRoot[:], collation.Header().ChunkRoot().Bytes())
+
+	_, err = n.SMCTransactor().AddHeader(txOps, collation.Header().ShardID(), collation.Header().Period(), chunkRoot)


Why are you throwing away the transaction? Did it succeed? Has it been mined?

I'll add a check to make sure tx succeeds

rauljordan · 2018-06-02T18:20:42Z

sharding/proposer/proposer.go

+// added to the main chain. There can only be one header per shard
+// per period, proposer should check if anyone else has added the header.
+// checkHeaderAvailability returns true if it's available, false if it's unavailable.
+func checkHeaderAvailability(n node.Node, shardId *big.Int, period *big.Int) (bool, error) {


I don't like the name of this method - it seems like it has to do with the whole data availability problem so it's confusing. Let's change to something else.

You are right. Changed it to checkHeaderSubmitted

rauljordan · 2018-06-07T01:43:03Z

With #149, we now have a simple entry point where we can add all the proposer goroutines. Make sure we leverage as much concurrency as we can for this within the Proposer.Start() function:

// Start the main loop for proposing collations.
func (p *Proposer) Start() error {
	log.Info("Starting proposer service")
	// TODO: Propose collations.
	return nil
}

Let's try to avoid the issue in the notary service as explained in #152.

prestonvanloon · 2018-06-07T03:07:11Z

I have some issues with the copy requirement conversions from []byte to [32]byte.

Is this our code that’s enforcing one or the other?
I feel like it’s wasteful to copy, but it is literately only 32 bytes.

If it’s upstream code, let’s just be sure to comment on the slice operations to fixed size array and vice versa. Specifically, I mean to explain why we do it in a few words. Future readers (like forgetful Preston) will read this code and say “what were they thinking?” and a comment would satisfy this.

Thanks

terencechain · 2018-06-07T06:02:41Z

Below is the code. ChunkRoot is always going to be a hash. The question is how do we convert hash to [32]byte (not []byte) without copying data. I haven't been able to find a good way. Any tips?

func (c *Collation) CalculateChunkRoot() {
	chunkRoot := common.BytesToHash(c.body)
	c.header.data.ChunkRoot = &chunkRoot
}

prestonvanloon · 2018-06-07T16:32:46Z

If its easier to copy, let's just do that for now. My only ask is for a comment to explain why

rauljordan · 2018-06-07T22:28:33Z

Looks good @terenc3t, just open an issue for the TODO

rauljordan · 2018-06-08T14:08:39Z

@prestonvanloon can we get a quick look on this to approve?

prestonvanloon

LGTM, sorry to hold up merging this!

sharding: beginning the proposer design of phase1

e29ba35

terencechain changed the title ~~sharding: beginning the proposer design of phase1~~ Implement proposer propose collation onto SMC May 10, 2018

terencechain self-assigned this May 10, 2018

terencechain added the in progress label May 10, 2018

terencechain added 2 commits May 11, 2018 14:40

sharding: addHeader and checkHeader functions

44d7515

sharding/proposer: beginning addHeader test

654dadc

terencechain added 2 commits May 20, 2018 18:18

sharding/proposer: getting latest master

79ec7ad

sharding/proposer: sync implementation w/ latest master

ce0ef29

rauljordan changed the title ~~Implement proposer propose collation onto SMC~~ Implement Proposers Submitting Collations onto SMC May 23, 2018

sharding/proposer: merge with master

1cc6a11

terencechain mentioned this pull request May 24, 2018

Optimize collation serializer and deserializer functions #134

Closed

terencechain added 6 commits May 24, 2018 20:59

Merge branch 'master' into proposer-phase1

1af0151

sharding/proposer: function to construct collation

fa5b3e4

sharding: merge with master

3ed3d5a

sharding: beginning proposer service tests

6343783

sharding: beginning proposer service tests

sharding/proposer: test for createCollation

e545ce0

sharding/proposer: Done with tests. 6 more test cases added

737255a

terencechain added Needs Review and removed In Progress labels Jun 1, 2018

terencechain added this to To do in Collation Proposals via automation Jun 1, 2018

terencechain added this to the Ruby milestone Jun 1, 2018

prestonvanloon requested review from prestonvanloon and rauljordan June 2, 2018 00:11

prestonvanloon requested changes Jun 2, 2018

View reviewed changes

rauljordan requested changes Jun 2, 2018

View reviewed changes

terencechain added 2 commits June 2, 2018 23:21

sharding/proposer: addressed review feedback

2c5b84e

sharding/proposer: add TransactionReceipt for tests

ff9f831

terencechain mentioned this pull request Jun 6, 2018

Implement Sharded TxPool #161

Closed

nisdas mentioned this pull request Jun 6, 2018

Create Common Testing Interface that all Sharding Actors Will be Able to Use #163

Closed

merge with master

bf31f17

sharding/proposer: refactor based on new architecture

cefb98a

sharding/proposer: added todos

0c8c4b8

rauljordan approved these changes Jun 7, 2018

View reviewed changes

prestonvanloon approved these changes Jun 9, 2018

View reviewed changes

terencechain added 2 commits June 9, 2018 08:56

Merge branch 'master' into proposer-phase1

5f2733f

sharding/proposer: resolve merge conflicts

b6e7978

terencechain merged commit 5791426 into prysmaticlabs:master Jun 10, 2018

Collation Proposals automation moved this from To do to Done Jun 10, 2018

rauljordan mentioned this pull request Jun 11, 2018

Propose collations onto SMC #106

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implement Proposers Submitting Collations onto SMC #111

Implement Proposers Submitting Collations onto SMC #111

terencechain commented May 10, 2018 •

edited

Loading

terencechain commented May 18, 2018

prestonvanloon Jun 2, 2018

terencechain Jun 4, 2018

prestonvanloon Jun 2, 2018

terencechain Jun 4, 2018

prestonvanloon Jun 2, 2018

prestonvanloon Jun 2, 2018

prestonvanloon Jun 2, 2018

prestonvanloon Jun 2, 2018

prestonvanloon Jun 2, 2018

terencechain Jun 2, 2018

prestonvanloon Jun 2, 2018

terencechain Jun 3, 2018

prestonvanloon Jun 2, 2018

terencechain Jun 3, 2018

prestonvanloon Jun 2, 2018

terencechain Jun 2, 2018

prestonvanloon Jun 2, 2018

terencechain Jun 3, 2018

prestonvanloon Jun 3, 2018

terencechain Jun 3, 2018 •

edited

Loading

terencechain Jun 4, 2018

prestonvanloon Jun 2, 2018

terencechain Jun 2, 2018

rauljordan Jun 2, 2018

terencechain Jun 3, 2018

rauljordan commented Jun 7, 2018

prestonvanloon commented Jun 7, 2018 •

edited

Loading

terencechain commented Jun 7, 2018

prestonvanloon commented Jun 7, 2018

rauljordan commented Jun 7, 2018

rauljordan commented Jun 8, 2018

prestonvanloon left a comment

Implement Proposers Submitting Collations onto SMC #111

Implement Proposers Submitting Collations onto SMC #111

Conversation

terencechain commented May 10, 2018 • edited Loading

Context

Requirements for Merge

Functions Being Implemented

terencechain commented May 18, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

terencechain Jun 3, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rauljordan commented Jun 7, 2018

prestonvanloon commented Jun 7, 2018 • edited Loading

terencechain commented Jun 7, 2018

prestonvanloon commented Jun 7, 2018

rauljordan commented Jun 7, 2018

rauljordan commented Jun 8, 2018

prestonvanloon left a comment

Choose a reason for hiding this comment

terencechain commented May 10, 2018 •

edited

Loading

terencechain Jun 3, 2018 •

edited

Loading

prestonvanloon commented Jun 7, 2018 •

edited

Loading