Fail to sync when a peer joins existing shard

[adlrocha]

When a peer joins a shard that has already mined a number of blocks, it doesn't seem to be syncing the previous state correctly. It successfully start listening to new blocks in the shard's pubsub topic but it throws the following error:

2021-10-05T12:26:56.423+0200    ERROR   chain   chain/sync.go:211       Received block with impossibly large height 163

Which means that it didn't synced with the shard.

We currently share the exchange.Client between the root chain and child shards. This may be related:

eudico/chain/sharding/sub.go

Line 52 in 8174c43

exchange exchange.Client //TODO: We may need to create a new one for every shard if syncing fails

[adlrocha]

Problem found. When we start a new shard we need to:

• Create a new exchange.Server that looks servers requests for the shard chain in other protocol handler

  • eudico/node/modules/services.go

    Lines 99 to 102 in 912548b

    	func RunChainExchange(h host.Host, svc exchange.Server) {
    	h.SetStreamHandler(exchange.BlockSyncProtocolID, svc.HandleStream) // old
    	h.SetStreamHandler(exchange.ChainExchangeProtocolID, svc.HandleStream) // new
    	}

  • eudico/node/builder_chain.go

    Line 91 in 912548b

    Override(new(exchange.Server), exchange.NewServer),

• Start a new exchange.Client for the shard that points to the right protocol handler.

  • eudico/node/builder_chain.go

    Line 87 in 912548b

    Override(new(exchange.Client), exchange.NewClient),

  • We'll probably need to make the client configurable so we can start a new client pointing to other protocol handler.

    • eudico/chain/exchange/client.go

      Line 405 in 912548b

      if len(supported) == 0 || (supported[0] != BlockSyncProtocolID && supported[0] != ChainExchangeProtocolID) {

• waitForSync before accepting new blocks in the shard.

[adlrocha]

After the above the shard is unable to sync. This is because peers learn what is the current heaviest tipset from the HelloService so we need to run a specific Hello protocol for shards that peer can run when joining a shard.

eudico/node/modules/services.go

Line 56 in 912548b

func RunHello(mctx helpers.MetricsCtx, lc fx.Lifecycle, h host.Host, svc *hello.Service) error {

eudico/node/builder_chain.go

Line 158 in 912548b

Override(RunHelloKey, modules.RunHello),

eudico/node/hello/hello.go

Line 56 in 912548b

func NewHelloService(h host.Host, cs *store.ChainStore, syncer *chain.Syncer, cons consensus.Consensus, pmgr peermgr.MaybePeerMgr) *Service {

[adlrocha]

hello protocol and chain sync fail because when a new peer joins the shard it generates the genesis by himself and it not deterministic (as I thought it was). The genesis blocks the two peers get to is not the same, making impossible the sync between the two chains.

We need to have the same genesis available for all peers in the shard. We can either:

• Implement a function that generates the genesis block deterministically so it is the same for all peers.
• Generate the genesis when the new shard is generated and make it available for all peers through the shard actor.

[adlrocha]

Fixed in 4309e51