Checkpoint Sync 5/5 - end-to-end test

[kasey]

What type of PR is this?

Other

What does this PR do? Why is it needed?

This PR adds a step to the existing end-to-end test to perform retrieve a checkpoint state+block from one of the running nodes once all nodes are in their final synced state.

Which issues(s) does this PR fix?

Fixes #

Other notes for review

Due to the way the checkpoint sync period is calculated and the small number of epochs in the end-to-end, this test will currently always fetch the genesis state. It would probably be better for it to fetch HEAD-1, but that would require modifying ws calculation. Alternatively we could tweak the parameters of the checkpoint sync algorithm (like the safety coefficient) to try to shorten the interval to just an epoch or so, but I need to look at the math again to check how many epochs e2e would need to run for this to be feasible.

[michaelsproul]

I just built Prysm using this branch and tried to checkpoint sync against a Lighthouse node, but it failed due to the absence of the /eth/v1/beacon/weak_subjectivity endpoint.

[2022-03-23 14:39:04]  INFO requesting http://localhost:7052/eth/v1/beacon/weak_subjectivity
[2022-03-23 14:39:05] ERROR main: Error retrieving checkpoint origin state and block: unexpected API response for prysm-only weak subjectivity checkpoint API: code=405, url=http://localhost:7052/eth/v1/beacon/weak_subjectivity, body=response body:
{"code":405,"message":"METHOD_NOT_ALLOWED","stacktraces":[]}: did not receive 2xx response from API

I noticed that you try to detect a 404 from the BN and fallback to a backwards-compatible impl if it doesn't exist, but Lighthouse returns a 405 for non-existent HTTP methods (this is our bad, I've opened an issue here to track that: sigp/lighthouse#3112). It might be cool to handle the 405 in the meantime 🙏

After trying Lighthouse I tried Teku via Infura, and got a 401 Unauthorized, which I think is due to Prysm not passing along the project_id:project_secret (username and password) from the first part of the URL:

./beacon-chain --checkpoint-sync-url "https://project_id:project_secret@eth2-beacon-mainnet.infura.io"
...
[2022-03-23 14:47:42]  INFO requesting http://eth2-beacon-mainnet.infura.io/eth/v1/beacon/weak_subjectivity
[2022-03-23 14:47:45] ERROR main: Error retrieving checkpoint origin state and block: unexpected API response for prysm-only weak subjectivity checkpoint API: code=401, url=https://eth2-beacon-mainnet.infura.io:443/eth/v1/beacon/weak_subjectivity, body=response body:
project id required
: did not receive 2xx response from API

[rauljordan]

Hi @kasey curious about the status of this one and if there's anything I can do to help

[kasey]

I just built Prysm using this branch and tried to checkpoint sync against a Lighthouse node, but it failed due to the absence of the /eth/v1/beacon/weak_subjectivity endpoint.

[2022-03-23 14:39:04]  INFO requesting http://localhost:7052/eth/v1/beacon/weak_subjectivity
[2022-03-23 14:39:05] ERROR main: Error retrieving checkpoint origin state and block: unexpected API response for prysm-only weak subjectivity checkpoint API: code=405, url=http://localhost:7052/eth/v1/beacon/weak_subjectivity, body=response body:
{"code":405,"message":"METHOD_NOT_ALLOWED","stacktraces":[]}: did not receive 2xx response from API

I noticed that you try to detect a 404 from the BN and fallback to a backwards-compatible impl if it doesn't exist, but Lighthouse returns a 405 for non-existent HTTP methods (this is our bad, I've opened an issue here to track that: sigp/lighthouse#3112). It might be cool to handle the 405 in the meantime pray

After trying Lighthouse I tried Teku via Infura, and got a 401 Unauthorized, which I think is due to Prysm not passing along the project_id:project_secret (username and password) from the first part of the URL:

./beacon-chain --checkpoint-sync-url "https://project_id:project_secret@eth2-beacon-mainnet.infura.io"
...
[2022-03-23 14:47:42]  INFO requesting http://eth2-beacon-mainnet.infura.io/eth/v1/beacon/weak_subjectivity
[2022-03-23 14:47:45] ERROR main: Error retrieving checkpoint origin state and block: unexpected API response for prysm-only weak subjectivity checkpoint API: code=401, url=https://eth2-beacon-mainnet.infura.io:443/eth/v1/beacon/weak_subjectivity, body=response body:
project id required
: did not receive 2xx response from API

Hi @michaelsproul, thanks for trying this out! I opened a bug last week on the basic auth issue #10467 and plan to handle 405 equivalent to 404. Sorry this PR got a little stalled, I had to take some personal time off in the last week - I will ping you when these issues are resolved.

[kasey]

@michaelsproul we just merged PR #10723 which significantly changes the way our checkpoint sync works. Instead of trying to fetch the block from the beginning of the latest weak subjectivity period, we simply fetch the latest finalized block, working on the assumption that this is preferable in terms of sync speed. This is also more compatible with Infura, which fails to serve the state-by-slot request we need to accompany the checkpoint block, but does a fine job fetching genesis and finalized states + block by root. I was convinced by Adrian Sutton that this is generally just as safe as the other method given that users can confirm the fetched roots are canonical using a trusted beacon API. Please give this a try and let me know how it works for you.

[kasey]

Closing this PR as checkpoint sync behavior changed heavily in #10723 which includes its own e2e test.