Shape RPC traffic to prevent DoS from heavy load

[benjamincburns]

Description

As an person running a Besu node, I want it to rate limit RPC requests when there are lots of pending requests that have not yet completed.

Today it's fairly easy for me to run operations over RPC that cause a node to become bogged down and stop tracking the network head, or worse, to throw OOM exceptions and hang. In enterprise environments it's likely that nodes will be shared across many services/users. In this sort of environment, we don't want a surge in RPC traffic from one user to crash the node or deny service to other users who are keeping their traffic within expected limits.

Acceptance Criteria

• RPC traffic can be submitted to Besu at any rate and Besu does not crash and does not stop tracking the head of the network.

Steps to Reproduce (Bug)

1. Spam Besu with a ton of Transactions
   or
2. Spam Besu with a ton of transaction trace requests
   or
3. Spam basu with a ton of eth_call requests

(there likely are plenty of other heavy RPC calls that will either cause crashes or cause the node to fall behind)

Expected behavior:
Node returns a rate limiting error response when it is too busy to process incoming RPC messages, or applies back-pressure on the request rte through some other means.

Actual behavior:
Node keeps trying to process RPC requests past the point where it can no longer maintain network state, or until it runs out of memory and hangs without crashing.

Frequency:
I expect in a large scale production environment this will be a common issue.

Recently I've seen this happen on our test nodes when a misconfiguration caused RPC traffic to be sent to a node that wasn't spec'd to handle the load that was being put on it.

I've also caused plenty of nodes to hang while running fixed-rate performance tests with Caliper.

Versions (Add all that apply)

• Software version: 1.4.X (1.4.5 is current release)

[non-fungible-nelson]

we now limit RPC with this #4965 which is user configurable with some sensible defaults.