New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
WIP: Shard Chain Type With Useful Methods for Notary/Proposer Clients #100
Conversation
sharding/shard.go
Outdated
|
||
// Shard base struct. | ||
type Shard struct { | ||
shardDB *shardKV |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can you change this to a common interface? Right now, it only accepts a database of struct shardKV, so it’s not very swappable.
sharding/shard.go
Outdated
// MakeShard creates an instance of a Shard struct given a shardID. | ||
func MakeShard(shardID *big.Int) *Shard { | ||
// Swappable - can be makeShardLevelDB, makeShardSparseTrie, etc. | ||
shardDB := makeShardKV() |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Shouldn’t this be a function argument? Why does a shard know which database to initialize? It feels a bit out of the responsibilities of a initializing a shard.
I can imagine that a nontrivial database has many arguments so this method will quickly get out of control.
sharding/collation.go
Outdated
|
||
// ShardID is the identifier for a shard. | ||
func (c *Collation) ShardID() *big.Int { return c.header.shardID } | ||
// Hash returns the hash of a collation's entire contents. Useful for tests. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is this only useful for tests? Maybe move to the testing package
sharding/shard_test.go
Outdated
shard := MakeShard(big.NewInt(3)) | ||
|
||
if err := shard.ValidateShardID(header); err == nil { | ||
t.Fatalf("Shard ID validation incorrect. Function should throw error when shardID's do not match. want=%d. got=%d", header.ShardID().Int64(), shard.ShardID().Int64()) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This should be Errorf. You should only fatal when the test cannot possibly proceed forward.
Please change this everywhere in this test
sharding/db.go
Outdated
fmt.Printf("Map: %v\n", sb.kv) | ||
fmt.Printf("Key: %v\n", k) | ||
fmt.Printf("Val: %v\n", sb.kv[k]) | ||
fmt.Printf("Ok: %v\n", ok) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Remove all of this printing. If you feel that you really need it, use ethereum log with the appropriate level (debug).
sharding/collation.go
Outdated
// Transactions returns an array of tx's in the collation. | ||
func (c *Collation) Transactions() []*types.Transaction { return c.transactions } | ||
// Body returns the collation's byte body. | ||
func (c *Collation) Body() []byte { return c.body } |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Shouldn’t this return a pointer to the byte array?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why would it? In case it's nil?
sharding/db.go
Outdated
fmt.Printf("Val: %v\n", sb.kv[k]) | ||
fmt.Printf("Ok: %v\n", ok) | ||
if !ok { | ||
return nil, fmt.Errorf("Key Not Found") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Change to (“key not found %v”, k)
sharding/shard.go
Outdated
} | ||
|
||
// GetHeaderByHash of collation. | ||
func (s *Shard) GetHeaderByHash(hash *common.Hash) (*CollationHeader, error) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Please remove “Get” here and everywhere https://golang.org/doc/effective_go.html#Getters
sharding/shard_test.go
Outdated
// if collation.Hash() != dbCollation.Hash() { | ||
// t.Fatalf("Collations do not match. want=%v. got=%v", collation, dbCollation) | ||
// } | ||
// } |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Uncomment or delete this test. No point in checking in dead code
sharding/shard.go
Outdated
return common.BytesToHash([]byte(key)) | ||
} | ||
|
||
// dataAvailabilityLookupKey formats a string that will become a lookup key |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Wrong method name
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Still wrong!
sharding/collation.go
Outdated
) | ||
|
||
// Collation base struct. | ||
type Collation struct { | ||
header *CollationHeader | ||
body []byte | ||
transactions []*types.Transaction |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
+1. How are we handling a body and transactions?
Are we supposed to update the body every time we modify this array?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Wouldnt transactions be called chunks? There's no TransactionRoot in collation header, only ChunkRoot
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
transactions are actual deserialized tx's that will be helpful for our implementation
sharding/collation.go
Outdated
// Period the collation corresponds to. | ||
func (c *Collation) Period() *big.Int { return c.header.period } | ||
// Transactions returns an array of tx's in the collation. | ||
func (c *Collation) Transactions() []*types.Transaction { return c.transactions } |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Transactions here too. I thought that's replaced by chunks
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We still need them because we'll be deserializing the chunks into transactions. We'll need a place to store this deserialized data and keeping a tx's slice is good here.
sharding/db.go
Outdated
|
||
"github.com/ethereum/go-ethereum/common" | ||
) | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
+1, a highlevel comment on how this works would be nice
sharding/shard_test.go
Outdated
shard := MakeShard(big.NewInt(1)) | ||
|
||
if err := shard.SaveHeader(header); err != nil { | ||
t.Fatal(err) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Should the Fatal be more descriptive here? Like t.Fataf("Save header failed: %v", err)
sharding/shard_test.go
Outdated
// It's being saved, but the .Get func doesn't fetch the value...? | ||
dbHeader, err := shard.GetHeaderByHash(&hash) | ||
if err != nil { | ||
t.Fatal(err) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
More description for t.Fatal(err)
?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
A lot of questions about design choices.
Could you add some details to the pull request description where necessary?
sharding/collation.go
Outdated
) | ||
|
||
// Collation base struct. | ||
type Collation struct { | ||
header *CollationHeader | ||
body []byte | ||
transactions []*types.Transaction |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
+1. How are we handling a body and transactions?
Are we supposed to update the body every time we modify this array?
sharding/collation.go
Outdated
|
||
// DecodeRLP uses an RLP Stream to populate the data field of a collation header. | ||
func (h *CollationHeader) DecodeRLP(s *rlp.Stream) error { | ||
err := s.Decode(&h.data) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
return s.Decode(&h.data)
sharding/collation.go
Outdated
|
||
// CalculateChunkRoot updates the collation header's chunk root based on the body. | ||
func (c *Collation) CalculateChunkRoot() { | ||
// TODO: this needs to be based on blob serialization. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can you create a github issue for this?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
+1 There's a lot more to be done here. For proof of custody we need to split chunks (body) into chunk + salt and take the merkle root of that
sharding/db.go
Outdated
|
||
"github.com/ethereum/go-ethereum/common" | ||
) | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
+1, a highlevel comment on how this works would be nice
sharding/db.go
Outdated
@@ -0,0 +1,37 @@ | |||
package sharding |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can/should we move this to its own package?
sharding/shard.go
Outdated
hash := common.BytesToHash(key.Bytes()) | ||
collationHashBytes, err := s.shardDB.Get(hash) | ||
if err != nil { | ||
return nil, fmt.Errorf("no canonical collation set for period, shardID pair: %v", err) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This error seems to assume that the error was caused by non-existent data. See my other comment about this.
sharding/shard.go
Outdated
if err != nil { | ||
return nil, fmt.Errorf("no canonical collation set for period, shardID pair: %v", err) | ||
} | ||
collationHash := common.BytesToHash(collationHashBytes) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What happens when the data isn't there but there was no error? i.e. if collationHashBytes == nil
sharding/shard.go
Outdated
if err != nil { | ||
return nil, err | ||
} | ||
return &Collation{header: header, body: body}, nil |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
return &Collation{header, body}, nil
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Getting "too few values in struct initializer"
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
That doesn't sound right...
sharding/shard.go
Outdated
} | ||
|
||
// Uses the hash of the header as the key. | ||
hash := header.Hash() |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Do you need to declare this variable? It would be simpler as
s.shardDB.Put(header.Hash(), encoded)
sharding/shard.go
Outdated
if err != nil { | ||
return nil, err | ||
} | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Handle case for header == nil
?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is covered by the headerbyhash method. error means that the header was nil.
sharding/collation.go
Outdated
// ProposerAddress is the coinbase addr of the creator for the collation. | ||
func (c *Collation) ProposerAddress() *common.Address { | ||
return c.header.data.ProposerAddress | ||
} | ||
|
||
// AddTransaction adds to the collation's body of tx blobs. | ||
func (c *Collation) AddTransaction(tx *types.Transaction) { | ||
// TODO: Include blob serialization instead. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
github issue here too
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What exactly should the issue be @terenc3t ?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Use blob serialization instead of appending tx to c.transactions c.transactions = append(c.transactions, tx)
sharding/collation.go
Outdated
rlp.Encode(hw, h) | ||
hw.Sum(hash[:0]) | ||
return hash | ||
} | ||
|
||
// ShardID is the identifier for a shard. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This comment is not exactly accurate. It should be something like ShardID the collation belongs to
sharding/collation.go
Outdated
period *big.Int //the period number in which collation to be included. | ||
proposerAddress *common.Address //address of the collation proposer. | ||
proposerSignature []byte //the proposer's signature for calculating collation hash. | ||
data collationHeaderData |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
why put a nested struct collationHeaderData
?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is so that it acts as a read only property. You can only access the data via getters and cannot modify it
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ah! got it. Thanks for the explanations
sharding/collation.go
Outdated
|
||
// CalculateChunkRoot updates the collation header's chunk root based on the body. | ||
func (c *Collation) CalculateChunkRoot() { | ||
// TODO: this needs to be based on blob serialization. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
+1 There's a lot more to be done here. For proof of custody we need to split chunks (body) into chunk + salt and take the merkle root of that
sharding/db.go
Outdated
"github.com/ethereum/go-ethereum/common" | ||
) | ||
|
||
type shardKV struct { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
+1 Some basic tests for Get
, Has
, Put
and Delete
would be nice
sharding/shard.go
Outdated
return nil | ||
} | ||
|
||
// HeaderByHash of collation. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
can we make the function description comment more descriptive?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is looking really solid. Just a few comments for code optimization and to remove commented code.
sharding/collation.go
Outdated
// body represents the serialized blob of a collation's transactions. | ||
body []byte | ||
// transactions serves as a useful slice to store deserialized chunks from the | ||
// collation's body. Every time this transactions slice is updated, the serialized |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Please update comment as described above.
sharding/collation_test.go
Outdated
// TODO: this test needs to change as we will be serializing tx's into blobs | ||
// within the collation body instead. | ||
|
||
// func TestCollation_AddTransactions(t *testing.T) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Delete all of this?
if err != nil { | ||
return nil, fmt.Errorf("cannot fetch body by chunk root: %v", err) | ||
} | ||
// TODO: deserializes the body into a txs slice instead of using |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Please create github issue for this. It would be a nice place for someone to contribute after serialization is added.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'll create as soon as we merge
sharding/shard.go
Outdated
return nil, err | ||
} | ||
if body == nil { | ||
return nil, fmt.Errorf("no corresponding body with chunk root found: %v", chunkRoot.String()) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Change this to %s
and I think you don't have to call .String()
since go will do that automatically.
if val == nil { | ||
return false, fmt.Errorf("availability not set for header") | ||
} | ||
availability := *val |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I believe this makes a copy of the data. Can't you access val[0]
?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
cause val is type *[]byte
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yep! I am mistaken about copying the data. With a slice, like you have here, it is copying the internal slice pointers so this is not an expensive operation.
In fact, it is OK to pass in a slice as an argument since it's really just a set of 1 pointer and 2 values. (beginning, cap, len).
sharding/shard.go
Outdated
} | ||
// sets the key to be the canonical collation lookup key and val as RLP encoded | ||
// collation header. | ||
if err := s.shardDB.Put(key, encoded); err != nil { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Return this
|
||
// dataAvailabilityLookupKey formats a string that will become a lookup | ||
// key in the shardDB. | ||
func dataAvailabilityLookupKey(chunkRoot *common.Hash) common.Hash { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nevermind.
sharding/shard.go
Outdated
// dataAvailabilityLookupKey formats a string that will become a lookup | ||
// key in the shardDB. | ||
func dataAvailabilityLookupKey(chunkRoot *common.Hash) common.Hash { | ||
key := fmt.Sprintf("availability-lookup:%s", chunkRoot.String()) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't think you need to call .String()
it should be automatic for the formatter with %s
.
sharding/shard.go
Outdated
// of the shard for ease of use. | ||
func canonicalCollationLookupKey(shardID *big.Int, period *big.Int) common.Hash { | ||
str := "canonical-collation-lookup:shardID=%s,period=%s" | ||
key := fmt.Sprintf(str, shardID.String(), period.String()) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't think you need to call .String()
, it should be automatic.
sharding/shard_test.go
Outdated
} | ||
|
||
if canonicalHeaderHash.String() != headerHash.String() { | ||
t.Errorf("header hashes do not match. want=%v. got=%v", headerHash.String(), canonicalHeaderHash.String()) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Remove .String()
and change %v
to %s
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Great job on the well-written comments! Makes it really to understand the code
sharding/collation.go
Outdated
type collationHeaderData struct { | ||
ShardID *big.Int // the shard ID of the shard. | ||
ChunkRoot *common.Hash // the root of the chunk tree which identifies collation body. | ||
Period *big.Int // the period number in which collation to be Pncluded. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
s/Pncluded/included
sharding/collation.go
Outdated
// Hash takes the keccak256 of the collation header's contents. | ||
func (h *CollationHeader) Hash() (hash common.Hash) { | ||
hw := sha3.NewKeccak256() | ||
rlp.Encode(hw, h) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Makes more sense to encode h.data
instead of h
?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The best pull request I have seen all day. Great job
WIP: Shard Chain Type With Useful Methods for Notary/Proposer Clients
Former-commit-id: 74c85fc
Former-commit-id: 32c501e
Former-commit-id: 59b4763
WIP: Shard Chain Type With Useful Methods for Notary/Proposer Clients Former-commit-id: d7f0bdf
This is a PR referencing #99 to create a shard chain interface/struct that can be used across our various packages.
Context
This was worked on in Py-EVM ethereum/py-evm#570 and would also serve as a useful wrapper class to thin out the code we would otherwise have to write for notaries/proposers.
Requirements for Merge
The design for an actual db backend used in our sharding client is beyond the scope of this PR. This PR just uses a simple k-v store.
Design Rationale
In this PR, we include both a
transactions []*types.Transaction
field along with abody []byte
field in theCollation
struct.The reason for this is that
transactions
serves as a useful slice to store deserialized chunks from thecollation's body we can manipulate programmatically. Every time this transactions slice is updated, the serialize would need to be recalculated. This will be a useful property for proposers and executors in our sharding implementation down the line.
Functions Being Implemented