Skip to content

Optimize synchronize_chain_state/download_certificates. #3902

Open
@afck

Description

@afck

These two ChainClient functions do almost the same thing, and both not in an optimal way:

  • synchronize_chain_state downloads the whole chain from all validators in parallel (wasteful) until a quorum of tasks have completed.
  • download_certificates goes through the validators sequentially in random order (bad if the first one is slow) and attempts to download the certificates only from that one.

This code should be deduplicated and optimized, so that the client is neither slowed down by the slowest validator, nor do we redundantly download everything from everyone.

Sketch of a Design

Create a task for each validator:

  • First, get the chain info to learn that validator's next_block_height.
  • Then repeatedly download a range of certificates. (See below.)

In another task, process the certificates in order, whenever they become available.

Stop downloading but finish processing the downloaded certificates as soon as there is a height H such that:

  • We have downloaded all certificates up to (excluding) H,
  • a quorum of validators has returned a chain info, and
  • Fewer than ⅓ of validators by weight reports a next_block_height > H.

To select a range to download for a validator, try to optimize, for example:

  • Always use a range that that validator actually claims to have (based on their next_block_height).
  • If available, choose a range that has not been assigned to any validator yet.
  • Otherwise choose a range that has been assigned to the lowest number of validators.
  • Among those, choose the range with the lowest height.

All of this requires a lot of coordination between the tasks; we should probably prefer channels over shared state if feasible.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    Status

    No status

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions