Implement a lame duck mode for RPC services #4489
Assignees
Labels
A-RPC
Area: rpc
C-enhancement
Category: An issue proposing an enhancement or a PR with one.
Node
Node team
T-node
Team: issues relevant to the node experience team
Projects
Comments
bowenwang1996
added
C-enhancement
mina86
added a commit
to mina86/nearcore
that referenced
this issue
Aug 12, 2021
…tuple Rather than relying on positions in a tuple, change start_with_config function to return its data in a structure so that the elements can be named. This will also make it easier to add new return values and thanks to the use of ‘..’ callers won’t be affected. Issue: near#4489
mina86
added a commit
to mina86/nearcore
that referenced
this issue
Aug 13, 2021
With the default signal handlers actix_web HTTP servers uses, sending a SIGINT results in a forceful shutdown of the servers even if there are requests in flight. This is unfriendly to the clients which may as a result get a partial response or a broken connection. Disable the signal handlers and instead explicitly call `stop(true)` on the HTTP servers. This gives the workers time to finish their processing. As a result, if a client successfully connected to an RPC endpoint, they will get their response. (That is, of course, unless processing the request would take more than five seconds but that should’t be the case with any of the supported RPC calls). Fixes: near#4489
near-bulldozer bot
pushed a commit
that referenced
this issue
Aug 16, 2021
With the default signal handlers actix_web HTTP servers uses, sending a SIGINT results in a forceful shutdown of the servers even if there are requests in flight. This is unfriendly to the clients which may as a result get a partial response or a broken connection. Disable the signal handlers and instead explicitly call `stop(true)` on the HTTP servers. This gives the workers time to finish their processing. As a result, if a client successfully connected to an RPC endpoint, they will get their response. (That is, of course, unless processing the request would take more than five seconds but that should’t be the case with any of the supported RPC calls). Fixes: #4489
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
With #3266 done, the node handles SIGINT and SIGTERM somewhat gracefully, i.e. all the cleanup code is executed so the result is no longer effectively a crash. However, all processing is interrupted. Most notably, if a client sent an RPC request, if the node gets SIGINT it will stop the processing and close connection to the client.
This should be improved by adding a lame duck mode to RPC endpoints. That is, once SIGINT or SIGTERM is received, the node should stop accepting new requests on RPC endpoints, finish whatever it’s doing and only then exit.
Note that there’s similar situation with nodes communicating with each other when the node may drop a connection to another peer while working on some request. This is less of an issue since peers should expect and handle correctly unexpected network interruptions and peer crashes. However, RPC clients are usually less fault tolerant so it would be nice if the node handled all incoming requests properly.
The text was updated successfully, but these errors were encountered: