Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Replace epee http server with uWebSockets for RPC #1185

Closed
wants to merge 6 commits into from

Conversation

jagerman
Copy link
Member

@jagerman jagerman commented Jun 29, 2020

This replaces the NIH epee http server which does not work all that well with an external C++ library called uWebSockets.

Additionally it fixes #1161 and fixes long polling on dev (which is currently entirely commented out on dev since the RPC overhaul PR and just results in an infinite loop of spamming the server with requests).

Fundamentally this gives the following advantages:

  • Much less code to maintain
  • Just one thread for handling HTTP connections versus epee's pool of threads
  • Uses existing LokiMQ job server and existing thread pool for handling the actual tasks; they are processed/scheduled in the same "rpc" or "admin" queues as lokimq rpc calls. One notable benefit is that "admin" rpc commands get their own queue (and thus cannot be delayed by long rpc commands). Currently the lokimq threads and the http rpc thread pool and the p2p thread pool and the job queue thread pool and the dns lookup thread pool and... are all different thread pools; this is a step towards consolidating them.
  • Very little mutex contention (which has been a major problem with epee RPC in the past): there is one mutex (inside uWebSockets) for putting responses back into the thread managing the connection, and one guarding the long pollers list; everything internally gets handled through (lock-free) lokimq inproc sockets.
  • Faster RPC performance on average, and much better worst case performance. Epee's http interface seems to have some race condition that ocassionally stalls a request (even a very simple one) for a dozen or more seconds for no good reason.
  • Long polling is implemented here without blocking any threads -- instead we just do the checksum test and, if unchanged, queue it up to be checked and replied to later (by either a lokimq timer, or a callback from core on mempool changes).

The basic idea of how this works from a high level:

We launch a single thread to handle HTTP RPC requests and response data. This uWebSockets thread works something like the main "proxy" thread in LokiMQ: it never actually handles any logic; it only serves to shuttle data that arrives in a request which we immediately queue for some other worker thread, and then, at some later point, queue a reply back to that waiting connection. Everything is asynchronous and non-blocking here: the basic uWebSockets event loop just operates as things arrive, passes it off immediately, and goes back to waiting for the next thing to arrive.

The basic flow of a request is like this:

0. uWS thread -- listens on localhost:22023
1. incoming request arrives on localhost:22023
2. uWS fires callback, which injects the task into the LokiMQ job queue
3a. LokiMQ schedules it as an RPC job
3b. uWS thread goes back to waiting for data/new connections.
4. Some LokiMQ thread runs it, gets the result
5. Result gets passed back to the uWS thread
6. uWS thread takes the request and starts sending it (asynchronously) back to the requestor.

In more detail:

uWebSockets has registered has registered handlers for non-jsonrpc requests (legacy JSON or binary). If the port is restricted then admin commands get mapped to a "Access denied" response handler, otherwise public commands (and admin commands on an unrestricted port) go to the rpc command handler.

POST requests to /json_rpc have their own handler; this is a little different than the above because it has to parse the request before it can determine whether it is allowed or not, but once this is done it continues roughly the same as legacy/binary requests.

uWebSockets then listens on the given IP/port for new incoming requests, and starts listening for requests in a thread (we own this thread). When a request arrives, it fires the event handler for that request. (This may happen multiple times, if the client is sending a bunch of data in a POST request). Once we have the full request, we then queue the job in LokiMQ, putting it in the "rpc" or "admin" command categories. (The one practical different here is that "admin" is configured to be allowed to start up its own thread if all other threads are busy, while "rpc" commands are prioritized along with everything else.) LokiMQ then schedules this job along with native LokiMQ "rpc." or "admin." requests.

When a LMQ worker thread becomes available (potentially immediately when there are idle threads), the RPC command gets called in it and runs. Whatever output it produces (or error message, if it throws) then gets wrapped up in jsonrpc boilerplate (if necessary), and delivered to the uWebSockets thread to be sent in reply to that request.

uWebSockets picks up the data (when its event loop is ready for it) and sends whatever it can without blocking, then buffers whatever it couldn't send to be sent again in a later event loop iteration once the requestor can accept more data. (This part is outside lokid; we only have to give uWS the data and let it worry about delivery).


PR details:

Things removed from this PR:

  1. ssl settings; with this PR the HTTP RPC interface is plain-text. The previous default generated a self-signed certificate for the server on startup and then the client accepted any certificate. This is actually worse than unencrypted because it is entirely MITM-readable and yet might make people think that their RPC communication is encrypted.

uWebSockets does support HTTPS, and we could glue the existing options into it, but I'm not convinced it's worthwhile: it works much better to put HTTPS in a front-end proxy holding the certificate that proxies requests to the backend (which can then listen in restricted mode on some localhost port). One reason this is better is that it is much easier to reload and/or restart such a front-end server, while certificate updates with lokid require a full restart. Another reason is that you get an error page instead of a timeout if something is wrong with the backend. Yet another reason is that making LetsEncrypt certificates work with web servers is quite a bit simpler than making them work with a random daemon.

  1. HTTP Digest authentication. Digest authentication is obsolete (and was already obsolete when it got added to Monero). It's complicated and requires a fair chunk of code. HTTP-Digest was originally an attempt to provide a password authentication mechanism that does not leak the password in transit, but still required that the server know the password. It only has marginal value against replay attacks, though, and is made entirely obsolete by sending traffic over HTTPS instead. No client out there supports Digest but not also Basic auth, and so given the limited usefulness it seems pointless to support more than Basic auth for HTTP RPC login.

What's worse is that epee's HTTP Digest authentication is a terribly bloated implementation: it uses boost::spirit -- a recursive descent parser meant for building complex language grammars -- just to parse a single HTTP header for Digest auth. This is a big load of crap that should never have been accepted upstream, and that we should get rid of (even if we wanted to support Digest auth it takes less than 100 lines of code to do it when not using a recursive descent parser).

Still to do (in some follow-up, lower-priority PR):

  • Make wallet_rpc_server also use uWebSockets. This is less important, though, as it's typically used for a single client where the performance issues and thread contention don't really show up.
  • Delete the epee http serving code (which for now is still needed by the above).

- Rename hash_xor to `operator ^=`, so you can just write
    `hasha ^= hashb;`
- Replace static variables with C++17 inline constexprs.  This should
result in slightly smaller binaries because the linker can throw away
the duplicates, whereas right now each compilation unit gets its own
private copy.
With the current approach -fPIC wasn't getting used when building
external libraries.  I tried moving add_subdirectory(external) down
below where we set flags, but that results in a slew or warning because
it *also* turn on a bunch of warnings that aren't safe in the various
external code.
Because:

    boost::join(v | boost::adaptors::transformed([](uint64_t out){return std::to_string(out);}), " ")

is ugly as sin, while:

    tools::join(" ", v)

is nice and simple.

Also removes a few unnecessary boost iterator adaptor includes and uses.
This replaces the NIH epee http server which does not work all that well
with an external C++ library called uWebSockets.  Fundamentally this
gives the following advantages:

- Much less code to maintain
- Just one thread for handling HTTP connections versus epee's pool of
threads
- Uses existing LokiMQ job server and existing thread pool for handling
the actual tasks; they are processed/scheduled in the same "rpc" or
"admin" queues as lokimq rpc calls.  One notable benefit is that "admin"
rpc commands get their own queue (and thus cannot be delayed by long rpc
commands).  Currently the lokimq threads and the http rpc thread pool
and the p2p thread pool and the job queue thread pool and the dns lookup
thread pool and... are *all* different thread pools; this is a step
towards consolidating them.
- Very little mutex contention (which has been a major problem with epee
RPC in the past): there is one mutex (inside uWebSockets) for putting
responses back into the thread managing the connection; everything
internally gets handled through (lock-free) lokimq inproc sockets.
- Faster RPC performance on average, and much better worst case
performance.  Epee's http interface seems to have some race condition
that ocassionally stalls a request (even a very simple one) for a dozen
or more seconds for no good reason.
- Long polling gets redone here to no longer need threads; instead we
just store the request and respond when the thread pool, or else in a
timer (that runs once/second) for timing out long polls.

---

The basic idea of how this works from a high level:

We launch a single thread to handle HTTP RPC requests and response data.
This uWebSockets thread is essentially running an event loop: it never
actually handles any logic; it only serves to shuttle data that arrives
in a request to some other thread, and then, at some later point, to
send some reply back to that waiting connection.  Everything is
asynchronous and non-blocking here: the basic uWebSockets event loop
just operates as things arrive, passes it off immediately, and goes back
to waiting for the next thing to arrive.

The basic flow is like this:

    0. uWS thread -- listens on localhost:22023
    1. uWS thread -- incoming request on localhost:22023
    2. uWS thread -- fires callback, which injects the task into the LokiMQ job queue
    3. LMQ main loop -- schedules it as an RPC job
    4. LMQ rpc thread -- Some LokiMQ thread runs it, gets the result
    5. LMQ rpc thread -- Result gets queued up for the uWS thread
    6. uWS thread -- takes the request and starts sending it
       (asynchronously) back to the requestor.

In more detail:

uWebSockets has registered has registered handlers for non-jsonrpc
requests (legacy JSON or binary).  If the port is restricted then admin
commands get mapped to a "Access denied" response handler, otherwise
public commands (and admin commands on an unrestricted port) go to the
rpc command handler.

POST requests to /json_rpc have their own handler; this is a little
different than the above because it has to parse the request before it
can determine whether it is allowed or not, but once this is done it
continues roughly the same as legacy/binary requests.

uWebSockets then listens on the given IP/port for new incoming requests,
and starts listening for requests in a thread (we own this thread).
When a request arrives, it fires the event handler for that request.
(This may happen multiple times, if the client is sending a bunch of
data in a POST request).  Once we have the full request, we then queue
the job in LokiMQ, putting it in the "rpc" or "admin" command
categories.  (The one practical different here is that "admin" is
configured to be allowed to start up its own thread if all other threads
are busy, while "rpc" commands are prioritized along with everything
else.)  LokiMQ then schedules this, along with native LokiMQ "rpc." or
"admin." requests.

When a LMQ worker thread becomes available, the RPC command gets called
in it and runs.  Whatever output it produces (or error message, if it
throws) then gets wrapped up in jsonrpc boilerplate (if necessary), and
delivered to the uWebSockets thread to be sent in reply to that request.

uWebSockets picks up the data and sends whatever it can without
blocking, then buffers whatever it couldn't send to be sent again in a
later event loop iteration once the requestor can accept more data.
(This part is outside lokid; we only have to give uWS the data and let
it worry about delivery).

---

PR specifics:

Things removed from this PR:

1. ssl settings; with this PR the HTTP RPC interface is plain-text.  The
previous default generated a self-signed certificate for the server on
startup and then the client accepted any certificate.  This is actually
*worse* than unencrypted because it is entirely MITM-readable and yet
might make people think that their RPC communication is encrypted, and
setting up actual certificates is difficult enough that I think most
people don't bother.

uWebSockets *does* support HTTPS, and we could glue the existing options
into it, but I'm not convinced it's worthwhile: it works much better to
put HTTPS in a front-end proxy holding the certificate that proxies
requests to the backend (which can then listen in restricted mode on
some localhost port).  One reason this is better is that it is much
easier to reload and/or restart such a front-end server, while
certificate updates with lokid require a full restart.  Another reason
is that you get an error page instead of a timeout if something is wrong
with the backend.  Finally we also save having to generate a temporary
certificate on *every* lokid invocation.

2. HTTP Digest authentication.  Digest authentication is obsolete (and
was already obsolete when it got added to Monero).  HTTP-Digest was
originally an attempt to provide a password authentication mechanism
that does not leak the password in transit, but still required that the
server know the password.  It only has marginal value against replay
attacks, and is made entirely obsolete by sending traffic over HTTPS
instead.  No client out there supports Digest but *not* Basic auth, and
so given the limited usefulness it seems pointless to support more than
Basic auth for HTTP RPC login.

What's worse is that epee's HTTP Digest authentication is a terrible
implementation: it uses boost::spirit -- a recursive descent parser
meant for building complex language grammars -- just to parse a single
HTTP header for Digest auth.  This is a big load of crap that should
never have been accepted upstream, and that we should get rid of (even
if we wanted to support Digest auth it takes less than 100 lines of code
to do it when *not* using a recursive descent parser).

Still to do (in another PR):

- Make the same server side change to the wallet_rpc_server.  This is
lower priority because the wallet_rpc server isn't generally a
performance bottleneck (since it usually has just one client).
- Delete the (to-be-)unused epee code for serving http, https, and digest.
- Add libuv submodule (only needed and built on Windows)

- Temporarily switch uWebSockets to my github repo: I submitted some
mingw compilation fixes upstream; as soon as a new upstream release
comes out we will switch this back to the upstream repo.

- Switch BOOST_SCOPE_EXIT to LOKI_DEFER

- Don't compile `closefrom()` on Windows (it isn't used, and generates
an unused function warning).
@jagerman jagerman marked this pull request as ready for review July 17, 2020 03:37
@jagerman
Copy link
Member Author

Merged (via #1208)

@jagerman jagerman closed this Aug 17, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

1 participant