Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Network ReqResp modularization #1930

Merged
merged 75 commits into from
Jan 8, 2021
Merged

Conversation

dapplion
Copy link
Contributor

@dapplion dapplion commented Jan 6, 2021

Refactors network reqResp code with the general goals of modularization and better maintainability. The overall logic is not changed with a few exceptions outlined below.

  • Do not rely on events to handle reqResp responses. Instead, the Sync registers a ReqRespHandler to network's ReqResp instance
    • Allows keeping only the app-level logic in Sync: given a request, yield a response.
    • Error handling is done exclusively in network's ReqResp, any error on any method will return error to the requester and close the stream
type ReqRespHandler = (
  method: Method, 
  requestBody: RequestBody,
  peerId: PeerId
) => AsyncIterable<ResponseBody>
  • De-deduplicate ssz-snappy codec logic. Allows to be recycled in requestEncode and responseEncode
    • Current organization allows to easily add another encoding strategy with minimal logic duplication
    • Use LodestarError style with rich documented errors
  • Abstract response timeout handling in separate function responseTimeoutsHandler
    • RESP_TIMEOUT waits for an entire response_chunk as indicated by the spec, instead of just a raw bytes chunk from libp2p
  • Only log in the main response or request handler, other functions either return or throw
    • Reduces the modules that need to be passed around and simplifies flow control
  • Document non-obvious code decisions and general function purpose with content from the p2p spec
  • For ssz list requests, limits the number of responses collected to the requested amount. Prevents a malicious node from sending us infinite blocks and DOS us.

This branch is currently syncing Pyrmont in lodestar-cloud-02 with a stable peer count. I has also synced Pyrmont on my local machine without issues.

Happy to jump on a call if more context is needed by a reviewer.

@github-actions github-actions bot added scope-networking All issues related to networking, gossip, and libp2p. Types labels Jan 6, 2021
@codeclimate
Copy link

codeclimate bot commented Jan 6, 2021

Code Climate has analyzed commit bc75d81 and detected 8 issues on this pull request.

Here's the issue category breakdown:

Category Count
Complexity 8

View more on Code Climate.

Copy link
Member

@wemeetagain wemeetagain left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great PR! I think this is long-overdue refactor of the req/resp code.

I checked this out and saw a lot of positive things:

  • significantly more comprehensive comments
  • encoding/decoding logic pulled into pure functions (rather than embedded in fns that return async generators)
  • more correct handling of req/resp timeout and response length limits
  • fully defined request/resp error codes/types
  • simplified/linearized req/resp flows

Things like the types can be pushed upstream (eg: it-pipe and libp2p) in the future.

@wemeetagain wemeetagain merged commit 49df11a into master Jan 8, 2021
@wemeetagain wemeetagain deleted the dapplion/network-modularization branch January 8, 2021 23:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
scope-networking All issues related to networking, gossip, and libp2p.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants