Reliable relayer

[polymorpher]

The current relayer operates in a serial mode, of which some burst, concurrent requests are denied until the blockchain finishes processing a pending transaction. As more users are getting onboarded at the same time in offline events, there are requests to support burst concurrent API calls. The implementation can be non-trivial. For example, nonces need to be manually tracked. Some transactions can be dropped by the blockchain silently, or not mined for unknown reasons.

In the initial design, clients are encouraged to share a relayer with only a small group users, which is operated and set up based on their own usage pattern. The reason is that a faulty or malicious relayer may cause denial of service to the clients, and gas fees are paid by the relayer account. Thus, a relayer should be a semi-private, semi-public layer that serves a small group of users. The clients are supposed to be switching between multiple relayers based on performance and reliability. This was discussed in page 2 and 11 in https://docs.google.com/document/d/1dExGu2FBQCj0KF5iX6ww1RAZrNN_fbDb1rmwmHe1jbE/edit#

Since client implementations (e.g. Timeless 1wallet) increasingly rely on the existing test relayer instead of operating their own, the test relayer's capability must be expanded. Several functionalities should be implemented:

• custom nonce tracking per network (pending, latest, synchronization with blockchain)
• multiple relayer accounts to be sampled from for each transaction
• load balancing between multiple relayer accounts based on pending transactions of each
• backoff and retry submission of transactions for each relayer account, with valid replacement transaction parameters (e.g. increased gas price)

[polymorpher]

The core wiki should also be updated - many descriptions are now inaccurate given the vast amount of updates over the last few months

[polymorpher]

If time allows, #74 and #73, and related discussions in issues and PRs should also be addressed. They can be discussed in detail under a separate issue.

[polymorpher]

Fixed by
#261
#265

[polymorpher]

Further fixed by 6449e58...b1f186a

[polymorpher]

Removed documentation related items from this issue. Will create separate issues for those