Allow rate limiting clients #1420
Labels
enhancement
New feature or request
Comments
|
Sounds good to me, would gladly accept a PR for this :) |
github-merge-queue bot
pushed a commit
to astriaorg/astria
that referenced
this issue
May 27, 2024
## Summary Limits the number of requests conductor sends to the Sequencer CometBFT endpoint to 100 per minute. ## Background During sync conductor can DOS Sequencer's CometBFT node by sending too many requests for commits and validator sets. With the batching logic introduced in #1049 there can be dozens (or more) blocks stored in each Celestia blob, each of which needs to be checked separately. With several blobs being fetched at once during, this can quickly spiral into hundreds (if not thousands) requests per minute. Note that only calls to `/commit` and `/validators` are rate limited, because there is currently no way to enforce this at the transport layer, see this issue: informalsystems/tendermint-rs#1420 However, the only other calls are to `/genesis` (once at startup), and `/abci_info` (every block-time period, usually every 2 seconds), which is rare enough to not need a rate limit. ## Changes - Use a tower `RateLimit` middleware around a tendermint-rs `HttpClient` only send up to 100 requests per minute. ## Breaking changes - Adds an environment variable `ASTRIA_CONDUCTOR_SEQUENCER_REQUESTS_PER_SECOND` to configure rate-limiting of requests sent to the Sequencer CometBFT node for verification of Sequencer block data fetched from Celestia blobs ## Testing This needs to be observed end-to-end, potentially letting conductor run for a very long time with only soft commits, and then turning firm commits on. ## Related Issues closes #1064 --------- Co-authored-by: Jordan Oroshiba <jordan@astria.org>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Description
I need to rate limit our tendermint-rpc
HttpClientto not DOS our cometbft node. Unfortunately,tendermint_rpc::client::http::Builderandtendermint_rpc::client::http::HttpClientdo not provide ways to set a tower middleware like itsRateLimitLayer.Prior art for how this could be achieved is for example by providing a
set_middlewaremethod as is done by jsonrpsee here: https://docs.rs/jsonrpsee-http-client/0.22.5/jsonrpsee_http_client/struct.HttpClientBuilder.html#method.set_http_middlewareInternally, they wrap their hyper client in the provided
ServiceBuilder: https://docs.rs/jsonrpsee-http-client/0.22.5/src/jsonrpsee_http_client/transport.rs.html#254I am going to try and work around it using the techniques described in this reqwest issue tracker here: seanmonstar/reqwest#491 (specifically using
tower::ServiceBuilder::service_fn), but that requires a lot of extra boilerplate at the application level.Definition of "done"
Being able to rate-limit outgoing requests over http. The general solution would be to allow setting arbitrary
tower::Services, but rate limiting would be great.The text was updated successfully, but these errors were encountered: