libevent-based http server #5677

Merged
merged 16 commits into from Sep 4, 2015

Conversation

Projects
None yet
@laanwj
Member

laanwj commented Jan 18, 2015

  • Replace usage of boost::asio with libevent2. boost::asio is not part of C++11, so unlike other boost there is no forwards-compatibility reason to stick with it. Together with #4738 (convert json_spirit to UniValue), this rids Bitcoin Core of the worst offenders with regard to compile-time slowness.
  • Replace spit-and-duct-tape http server with evhttp. Front-end http handling is handled by libevent, a work queue (with configurable depth and parallelism) is used to handle application requests.
  • Wrap HTTP request in C++ class; this makes the application code mostly HTTP-server-neutral
  • Refactor RPC to move all http-specific code to a separate file. Theoreticaly this can allow building without HTTP server but with another RPC backend, e.g. Qt's debug console (currently not implemented) or future RPC mechanisms people may want to use.
  • HTTP dispatch mechanism; services (e.g., RPC, REST) register which URL paths they want to handle.

By using a proven, high-performance asynchronous networking library (also used by Tor) and HTTP server, problems such as #5674, #5655, #344 should be avoided.

What works? bitcoind, bitcoin-cli, bitcoin-qt. Unit tests and RPC/REST tests pass. The aim for now is everything but SSL support.

Configuration options:

  • -rpcthreads: repurposed as "number of work handler threads". Still defaults to 4.
  • -rpcworkqueue: maximum depth of work queue. When this is reached, new requests will return a 500 Internal Error.
  • -rpctimeout: inactivity time, in seconds, after which to disconnect a client.
  • -debug=http: low-level http activity logging

(due to the separation of RPC and HTTP server, renaming these options may make sense, but I've kept this out of backwards compatiblity)

TODO:

  • Build system (currently hardcodes libraries, so this will definitely not pass Travis) (thanks @theuni)
  • REST and RPC register their own request handlers respectively
  • Qt debug console must register a RPCTimerInterface (to make timeouts in the debug console work with -server=0)
  • Interrupt/Shutdown flow needs to be cleaned up
  • [warn] event_active: event has no event_base set. appears sometimes to the console. Seems to be harmless, but it is weird (see @ajweiss comments)
@theuni

This comment has been minimized.

Show comment
Hide comment
@theuni

theuni Jan 20, 2015

Member

Very nice! Before looking over the work itself, I wanted to be sure that libevent was viable for all of our build targets.

See here for the build-system work. This should be enough to get Travis passing, I'd think:
https://github.com/theuni/bitcoin/commits/5677

Member

theuni commented Jan 20, 2015

Very nice! Before looking over the work itself, I wanted to be sure that libevent was viable for all of our build targets.

See here for the build-system work. This should be enough to get Travis passing, I'd think:
https://github.com/theuni/bitcoin/commits/5677

@laanwj

This comment has been minimized.

Show comment
Hide comment
@laanwj

laanwj Jan 20, 2015

Member

@cfields Will pull that in, thanks a lot!

Member

laanwj commented Jan 20, 2015

@cfields Will pull that in, thanks a lot!

@luke-jr

This comment has been minimized.

Show comment
Hide comment
@luke-jr

luke-jr Jan 20, 2015

Member

Hm, nothing special needed for longpolling?

Member

luke-jr commented Jan 20, 2015

Hm, nothing special needed for longpolling?

@laanwj

This comment has been minimized.

Show comment
Hide comment
@laanwj

laanwj Jan 20, 2015

Member

@luke-jr I don't think so. The current implementation should work. Of course it would be more optimal to release the worker thread while longpolling, and change "new block" it into an event-trigger, but I leave that as a challenge for later.

Member

laanwj commented Jan 20, 2015

@luke-jr I don't think so. The current implementation should work. Of course it would be more optimal to release the worker thread while longpolling, and change "new block" it into an event-trigger, but I leave that as a challenge for later.

@laanwj

This comment has been minimized.

Show comment
Hide comment
@laanwj

laanwj Jan 20, 2015

Member

The fail in "32-bit + dash" is strange

FAIL: qt/test/test_bitcoin-qt

I'm not sure how this can be affected at all (passes fine here), but I'll check.

On Win32/64 it still tries to link against libevent_pthread. IIRC there is no specific thread library for windows, evthread_use_windows_threads is part of the core library there.

/usr/bin/x86_64-w64-mingw32-ld: cannot find -levent_pthreads
Member

laanwj commented Jan 20, 2015

The fail in "32-bit + dash" is strange

FAIL: qt/test/test_bitcoin-qt

I'm not sure how this can be affected at all (passes fine here), but I'll check.

On Win32/64 it still tries to link against libevent_pthread. IIRC there is no specific thread library for windows, evthread_use_windows_threads is part of the core library there.

/usr/bin/x86_64-w64-mingw32-ld: cannot find -levent_pthreads
@theuni

This comment has been minimized.

Show comment
Hide comment
@theuni

theuni Jan 20, 2015

Member

Blah, sorry, missed that one.

Member

theuni commented Jan 20, 2015

Blah, sorry, missed that one.

@laanwj

This comment has been minimized.

Show comment
Hide comment
@laanwj

laanwj Jan 20, 2015

Member

It's easy to miss those sneaky qt unit tests. Windows passes now!

That leaves the 32-bit + dash case, which is not an intermittent issue

FAIL! : PaymentServerTests::paymentServerTests() Compared values are not the same
Actual (merchant):
Expected (QString("testmerchant.org")): testmerchant.org
Loc: [qt/test/paymentservertests.cpp(84)]

... no clue how this happens yet. The Qt tests don't use the RPC mechanism. My gut feeling is some interaction with OpenSSL which, absent verification, is now an indirect dependency through Qt? Not sure, and why it only happens in this test is a mystery to me.

Member

laanwj commented Jan 20, 2015

It's easy to miss those sneaky qt unit tests. Windows passes now!

That leaves the 32-bit + dash case, which is not an intermittent issue

FAIL! : PaymentServerTests::paymentServerTests() Compared values are not the same
Actual (merchant):
Expected (QString("testmerchant.org")): testmerchant.org
Loc: [qt/test/paymentservertests.cpp(84)]

... no clue how this happens yet. The Qt tests don't use the RPC mechanism. My gut feeling is some interaction with OpenSSL which, absent verification, is now an indirect dependency through Qt? Not sure, and why it only happens in this test is a mystery to me.

@theuni

This comment has been minimized.

Show comment
Hide comment
@theuni

theuni Jan 20, 2015

Member

@laanwj Yes, it has some interaction with qt:

  • ./configure --with-gui=qt4: fine.
  • ./configure --with-gui=qt5: fine.
  • make -C depends; ./configure --prefix=pwd/depends/x86_64-unknown-linux-gnu: busted
  • make -C depends USE_LINUX_STATIC_QT5=1; ./configure --prefix=pwd/depends/x86_64-unknown-linux-gnu: fine.
Member

theuni commented Jan 20, 2015

@laanwj Yes, it has some interaction with qt:

  • ./configure --with-gui=qt4: fine.
  • ./configure --with-gui=qt5: fine.
  • make -C depends; ./configure --prefix=pwd/depends/x86_64-unknown-linux-gnu: busted
  • make -C depends USE_LINUX_STATIC_QT5=1; ./configure --prefix=pwd/depends/x86_64-unknown-linux-gnu: fine.
@jonasschnelli

This comment has been minimized.

Show comment
Hide comment
@jonasschnelli

jonasschnelli Jan 20, 2015

Member

tested gitian build.
Binaries to test: https://builds.jonasschnelli.ch/pulls/5677/

Member

jonasschnelli commented Jan 20, 2015

tested gitian build.
Binaries to test: https://builds.jonasschnelli.ch/pulls/5677/

@jtimon

This comment has been minimized.

Show comment
Hide comment
@jtimon

jtimon Jan 21, 2015

Member

Concept ACK

Member

jtimon commented Jan 21, 2015

Concept ACK

@laanwj

This comment has been minimized.

Show comment
Hide comment
@laanwj

laanwj Jan 23, 2015

Member

Now that the code is stable it is time for some benchmarking.
I found a nice scriptable framework for HTTP benchmarking, wrk. Some results.
These benchmarks were taken at the default settings (4 worker threads, 16 depth work queue).

GET request to invalid URL

These are handled by evhttp itself, so this is the baseline.

$  ./wrk -t12 -c15 -d10s http://127.0.0.1:18332/inv
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   375.53us   66.86us   6.31ms   93.03%
    Req/Sec     2.67k   121.28     3.11k    80.98%
  303293 requests in 10.00s, 36.73MB read
  Non-2xx or 3xx responses:
    404: 303293
Requests/sec:  30343.08
Transfer/sec:      3.68MB

As expected, we get 404s.

GET request to /

These are dispatched to a worker thread. Some more latency is expected. In the worker thread these error out early, as GET is not a valid method for JSON RPC.

$ ./wrk -t12 -c15 -d10s http://127.0.0.1:18332/
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   472.96us   44.12us   1.28ms   93.31%
    Req/Sec     2.14k   127.02     2.33k    82.90%
  245025 requests in 10.00s, 41.59MB read
  Non-2xx or 3xx responses:
    405: 245025
Requests/sec:  24514.30
Transfer/sec:      4.16MB

As expected: lots of 405 (invalid method for URL) results.

RPC getgenerate requests

Post getgenerate requests. This is an extremely cheap RPC call.
Script:

wrk.method = "POST"
wrk.body   = "{\"method\":\"getgenerate\",\"params\":[],\"id\":1}\n"
wrk.headers["Content-Type"] = "application/json"
wrk.headers["Authorization"] = "Basic XXX"
$ ./wrk -t12 -c15 -d10s -s getgenerate.lua http://127.0.0.1:18332/
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   571.28us   59.77us   2.09ms   91.89%
    Req/Sec     1.82k   110.56     2.00k    80.32%
  204481 requests in 9.99s, 28.28MB read
  Non-2xx or 3xx responses:
Requests/sec:  20463.77
Transfer/sec:      2.83MB

All responses succesful.

RPC getinfo requests

Post getinfo requests. This is a more expensive RPC call.

wrk.method = "POST"
wrk.body   = "{\"method\":\"getinfo\",\"params\":[],\"id\":1}\n"
wrk.headers["Content-Type"] = "application/json"
wrk.headers["Authorization"] = "Basic XXX
$ ./wrk -t12 -c15 -d10s -s getinfo.lua http://127.0.0.1:18332/
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     1.69ms  572.87us  15.39ms   95.21%
    Req/Sec   626.46     81.57     1.00k    84.26%
  71245 requests in 9.99s, 37.17MB read
  Non-2xx or 3xx responses:
Requests/sec:   7128.48
Transfer/sec:      3.72MB

Although the performance goes down, no errors happen. The maximum queue depth is never reached.

RPC requests w/ invalid authentication

Post getinfo requests with invalid authentication. This will trigger a 250ms delay, and thus we can trigger worker queue-full conditions with enough threads.

wrk.method = "POST"
wrk.body   = "{\"method\":\"getinfo\",\"params\":[],\"id\":1}\n"
wrk.headers["Content-Type"] = "application/json"
wrk.headers["Authorization"] = "Basic YYY
$ ./wrk -t12 -c15 -d10s -s getinfo_unauth.lua http://127.0.0.1:18332/
Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   751.03ms  323.38us 752.15ms   81.82%
    Req/Sec     1.05      0.65     2.00     58.33%
  156 requests in 10.00s, 19.80KB read
  Non-2xx or 3xx responses:
    401: 156
Requests/sec:     15.60
Transfer/sec:      1.98KB

Increasing the load further, we can exceed the work queue depth:

$ ./wrk -t12 -c45 -d10s -s getinfo_unauth.lua http://127.0.0.1:18332/
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   204.35ms  462.08ms   1.25s    83.72%
    Req/Sec     2.42k     1.53k    6.00k    75.64%
  277072 requests in 10.00s, 43.59MB read
  Non-2xx or 3xx responses:
    401: 156
    500: **276916**
Requests/sec:  27706.71
Transfer/sec:      4.36MB

Looks good. Nothing unexpected, it's clear the evhttp is not the bottleneck, and the work queue works as expected. Will try this with the old asio-based HTTP server shortly.

Member

laanwj commented Jan 23, 2015

Now that the code is stable it is time for some benchmarking.
I found a nice scriptable framework for HTTP benchmarking, wrk. Some results.
These benchmarks were taken at the default settings (4 worker threads, 16 depth work queue).

GET request to invalid URL

These are handled by evhttp itself, so this is the baseline.

$  ./wrk -t12 -c15 -d10s http://127.0.0.1:18332/inv
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   375.53us   66.86us   6.31ms   93.03%
    Req/Sec     2.67k   121.28     3.11k    80.98%
  303293 requests in 10.00s, 36.73MB read
  Non-2xx or 3xx responses:
    404: 303293
Requests/sec:  30343.08
Transfer/sec:      3.68MB

As expected, we get 404s.

GET request to /

These are dispatched to a worker thread. Some more latency is expected. In the worker thread these error out early, as GET is not a valid method for JSON RPC.

$ ./wrk -t12 -c15 -d10s http://127.0.0.1:18332/
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   472.96us   44.12us   1.28ms   93.31%
    Req/Sec     2.14k   127.02     2.33k    82.90%
  245025 requests in 10.00s, 41.59MB read
  Non-2xx or 3xx responses:
    405: 245025
Requests/sec:  24514.30
Transfer/sec:      4.16MB

As expected: lots of 405 (invalid method for URL) results.

RPC getgenerate requests

Post getgenerate requests. This is an extremely cheap RPC call.
Script:

wrk.method = "POST"
wrk.body   = "{\"method\":\"getgenerate\",\"params\":[],\"id\":1}\n"
wrk.headers["Content-Type"] = "application/json"
wrk.headers["Authorization"] = "Basic XXX"
$ ./wrk -t12 -c15 -d10s -s getgenerate.lua http://127.0.0.1:18332/
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   571.28us   59.77us   2.09ms   91.89%
    Req/Sec     1.82k   110.56     2.00k    80.32%
  204481 requests in 9.99s, 28.28MB read
  Non-2xx or 3xx responses:
Requests/sec:  20463.77
Transfer/sec:      2.83MB

All responses succesful.

RPC getinfo requests

Post getinfo requests. This is a more expensive RPC call.

wrk.method = "POST"
wrk.body   = "{\"method\":\"getinfo\",\"params\":[],\"id\":1}\n"
wrk.headers["Content-Type"] = "application/json"
wrk.headers["Authorization"] = "Basic XXX
$ ./wrk -t12 -c15 -d10s -s getinfo.lua http://127.0.0.1:18332/
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     1.69ms  572.87us  15.39ms   95.21%
    Req/Sec   626.46     81.57     1.00k    84.26%
  71245 requests in 9.99s, 37.17MB read
  Non-2xx or 3xx responses:
Requests/sec:   7128.48
Transfer/sec:      3.72MB

Although the performance goes down, no errors happen. The maximum queue depth is never reached.

RPC requests w/ invalid authentication

Post getinfo requests with invalid authentication. This will trigger a 250ms delay, and thus we can trigger worker queue-full conditions with enough threads.

wrk.method = "POST"
wrk.body   = "{\"method\":\"getinfo\",\"params\":[],\"id\":1}\n"
wrk.headers["Content-Type"] = "application/json"
wrk.headers["Authorization"] = "Basic YYY
$ ./wrk -t12 -c15 -d10s -s getinfo_unauth.lua http://127.0.0.1:18332/
Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   751.03ms  323.38us 752.15ms   81.82%
    Req/Sec     1.05      0.65     2.00     58.33%
  156 requests in 10.00s, 19.80KB read
  Non-2xx or 3xx responses:
    401: 156
Requests/sec:     15.60
Transfer/sec:      1.98KB

Increasing the load further, we can exceed the work queue depth:

$ ./wrk -t12 -c45 -d10s -s getinfo_unauth.lua http://127.0.0.1:18332/
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   204.35ms  462.08ms   1.25s    83.72%
    Req/Sec     2.42k     1.53k    6.00k    75.64%
  277072 requests in 10.00s, 43.59MB read
  Non-2xx or 3xx responses:
    401: 156
    500: **276916**
Requests/sec:  27706.71
Transfer/sec:      4.36MB

Looks good. Nothing unexpected, it's clear the evhttp is not the bottleneck, and the work queue works as expected. Will try this with the old asio-based HTTP server shortly.

@laanwj

This comment has been minimized.

Show comment
Hide comment
@laanwj

laanwj Jan 23, 2015

Member

Old http server

Same steps as above, repeated with the old server as of commit 944c256.

GET request to invalid URL

$  ./wrk -t12 -c15 -d10s http://127.0.0.1:18332/inv
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   502.17us   94.60us   2.65ms   74.77%
    Req/Sec     1.89k   119.76     2.33k    83.38%
  214864 requests in 9.99s, 37.50MB read
  Responses:
    404: 214864
Requests/sec:  21502.28
Transfer/sec:      3.75MB

GET request to /

$ ./wrk -t12 -c15 -d10s http://127.0.0.1:18332/
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   502.58us   92.30us   2.07ms   73.39%
    Req/Sec     1.89k   118.97     2.22k    83.36%
  214269 requests in 9.99s, 103.40MB read
  Responses:
    401: 214269
Requests/sec:  21447.93
Transfer/sec:     10.35MB

RPC getgenerate requests

$ ./wrk -t12 -c15 -d10s -s getgenerate.lua http://127.0.0.1:18332/
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    93.68us   24.84us   1.27ms   89.23%
    Req/Sec    10.06k   616.30    11.78k    64.49%
  380211 requests in 10.00s, 78.32MB read
  Socket errors: connect 0, read 0, write 0, timeout 39
  Responses:
    200: 380211
Requests/sec:  38006.24
Transfer/sec:      7.83MB

RPC getinfo requests

$ ./wrk -t12 -c15 -d10s -s getinfo.lua http://127.0.0.1:18332/
Running 10s test @ http://127.0.0.1:18332/
  12 threads and 15 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   544.15us  187.74us   4.03ms   79.81%
    Req/Sec     1.89k   188.50     2.90k    63.09%
  71480 requests in 10.00s, 42.13MB read
  Socket errors: connect 0, read 0, write 0, timeout 39
  Responses:
    200: 71480
Requests/sec:   7144.83
Transfer/sec:      4.21MB

RPC requests w/ invalid authentication

$ ./wrk -t12 -c15 -d10s -s getinfo_unauth.lua http://127.0.0.1:18332/
  12 threads and 15 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   751.75ms  161.83us 752.03ms   73.91%
    Req/Sec     0.94      0.66     2.00     56.52%
  156 requests in 10.00s, 77.09KB read
  Responses:
    401: 156
Requests/sec:     15.60
Transfer/sec:      7.71KB
$ ./wrk -t12 -c45 -d10s -s getinfo_unauth.lua http://127.0.0.1:18332/
Running 10s test @ http://127.0.0.1:18332/
  12 threads and 45 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     2.19s   309.47ms   2.26s    95.24%
    Req/Sec     0.90      1.56     6.00     87.30%
  156 requests in 10.01s, 77.09KB read
  Socket errors: connect 0, read 0, write 0, timeout 24
  Responses:
    401: 156
Requests/sec:     15.59
Transfer/sec:      7.70KB

Summary

Testcase                      Old (r/s) New (r/s)
================================================
GET request to invalid URL    21502     30343
GET request to /              21448     24514
RPC getgenerate requests      38006*    20463
RPC getinfo requests           7144*     7128
RPC requests w/ invalid auth     16*       16

* with timeout errors.
  • The new server wins on the base http requests front. Even to /, which are dispatched to the worker threads.
  • The old server is currently much faster with getgenerate requests. I am curious why. Also: how can getgenerate requests be faster than simple GET /'s? I suspect a measurement error that has to do with the timeouts (edit: this is because the old server disconnects after errors, so those don't utilize keepalive).
  • getinfo requests are, as expected, ~ the same speed. Processing overhead dominates.
  • Same for unauthenticated requests. The requests take 250ms to handle so the number seen is exactly as expected.

Take into account that it's not entirely a fair comparison: the new http server can service a large number of connections at the same time, whereas the old server can have a maximum of four (or, -rpcthreads) and starves additional.connections (thanks to keep-alive). There is some overhead in multiplexing that is absent in a one-to-one scenario. This is also a purely a local benchmark. I/O bandwidth doesn't come into it, and the benchmark tool competes for the same CPU as the server.

Member

laanwj commented Jan 23, 2015

Old http server

Same steps as above, repeated with the old server as of commit 944c256.

GET request to invalid URL

$  ./wrk -t12 -c15 -d10s http://127.0.0.1:18332/inv
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   502.17us   94.60us   2.65ms   74.77%
    Req/Sec     1.89k   119.76     2.33k    83.38%
  214864 requests in 9.99s, 37.50MB read
  Responses:
    404: 214864
Requests/sec:  21502.28
Transfer/sec:      3.75MB

GET request to /

$ ./wrk -t12 -c15 -d10s http://127.0.0.1:18332/
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   502.58us   92.30us   2.07ms   73.39%
    Req/Sec     1.89k   118.97     2.22k    83.36%
  214269 requests in 9.99s, 103.40MB read
  Responses:
    401: 214269
Requests/sec:  21447.93
Transfer/sec:     10.35MB

RPC getgenerate requests

$ ./wrk -t12 -c15 -d10s -s getgenerate.lua http://127.0.0.1:18332/
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    93.68us   24.84us   1.27ms   89.23%
    Req/Sec    10.06k   616.30    11.78k    64.49%
  380211 requests in 10.00s, 78.32MB read
  Socket errors: connect 0, read 0, write 0, timeout 39
  Responses:
    200: 380211
Requests/sec:  38006.24
Transfer/sec:      7.83MB

RPC getinfo requests

$ ./wrk -t12 -c15 -d10s -s getinfo.lua http://127.0.0.1:18332/
Running 10s test @ http://127.0.0.1:18332/
  12 threads and 15 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   544.15us  187.74us   4.03ms   79.81%
    Req/Sec     1.89k   188.50     2.90k    63.09%
  71480 requests in 10.00s, 42.13MB read
  Socket errors: connect 0, read 0, write 0, timeout 39
  Responses:
    200: 71480
Requests/sec:   7144.83
Transfer/sec:      4.21MB

RPC requests w/ invalid authentication

$ ./wrk -t12 -c15 -d10s -s getinfo_unauth.lua http://127.0.0.1:18332/
  12 threads and 15 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   751.75ms  161.83us 752.03ms   73.91%
    Req/Sec     0.94      0.66     2.00     56.52%
  156 requests in 10.00s, 77.09KB read
  Responses:
    401: 156
Requests/sec:     15.60
Transfer/sec:      7.71KB
$ ./wrk -t12 -c45 -d10s -s getinfo_unauth.lua http://127.0.0.1:18332/
Running 10s test @ http://127.0.0.1:18332/
  12 threads and 45 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     2.19s   309.47ms   2.26s    95.24%
    Req/Sec     0.90      1.56     6.00     87.30%
  156 requests in 10.01s, 77.09KB read
  Socket errors: connect 0, read 0, write 0, timeout 24
  Responses:
    401: 156
Requests/sec:     15.59
Transfer/sec:      7.70KB

Summary

Testcase                      Old (r/s) New (r/s)
================================================
GET request to invalid URL    21502     30343
GET request to /              21448     24514
RPC getgenerate requests      38006*    20463
RPC getinfo requests           7144*     7128
RPC requests w/ invalid auth     16*       16

* with timeout errors.
  • The new server wins on the base http requests front. Even to /, which are dispatched to the worker threads.
  • The old server is currently much faster with getgenerate requests. I am curious why. Also: how can getgenerate requests be faster than simple GET /'s? I suspect a measurement error that has to do with the timeouts (edit: this is because the old server disconnects after errors, so those don't utilize keepalive).
  • getinfo requests are, as expected, ~ the same speed. Processing overhead dominates.
  • Same for unauthenticated requests. The requests take 250ms to handle so the number seen is exactly as expected.

Take into account that it's not entirely a fair comparison: the new http server can service a large number of connections at the same time, whereas the old server can have a maximum of four (or, -rpcthreads) and starves additional.connections (thanks to keep-alive). There is some overhead in multiplexing that is absent in a one-to-one scenario. This is also a purely a local benchmark. I/O bandwidth doesn't come into it, and the benchmark tool competes for the same CPU as the server.

@Diapolo

This comment has been minimized.

Show comment
Hide comment
@Diapolo

Diapolo Jan 23, 2015

Why are we dropping SSL support for RPC?

Diapolo commented Jan 23, 2015

Why are we dropping SSL support for RPC?

src/bitcoin-cli.cpp
+
+ // Synchronously look up hostname
+ struct evhttp_connection *evcon = evhttp_connection_base_new(base, NULL, host.c_str(), port); // XXX RAII
+ if (evcon == NULL)

This comment has been minimized.

@Diapolo

Diapolo Jan 23, 2015

Just to understand why are you explicitly using == NULL here and for base above just !var?

@Diapolo

Diapolo Jan 23, 2015

Just to understand why are you explicitly using == NULL here and for base above just !var?

This comment has been minimized.

@laanwj

laanwj Jan 23, 2015

Member

@Diapolo == NULL and !evcon are equivalent in this case so purely a matter to taste. The pull clearly states [PoC] as for proof-of-concept, please don't report all these minor things but only critical or high-level issues,

@laanwj

laanwj Jan 23, 2015

Member

@Diapolo == NULL and !evcon are equivalent in this case so purely a matter to taste. The pull clearly states [PoC] as for proof-of-concept, please don't report all these minor things but only critical or high-level issues,

src/bitcoin-cli.cpp
+ struct evkeyvalq *output_headers = evhttp_request_get_output_headers(req);
+ assert(output_headers);
+ evhttp_add_header(output_headers, "Host", host.c_str());
+ evhttp_add_header(output_headers, "Connection", "close");

This comment has been minimized.

@Diapolo

Diapolo Jan 23, 2015

Perhaps use some constant or struct for such strings and use a speaking name in the code?

@Diapolo

Diapolo Jan 23, 2015

Perhaps use some constant or struct for such strings and use a speaking name in the code?

@laanwj

This comment has been minimized.

Show comment
Hide comment
@laanwj

laanwj Jan 23, 2015

Member

@Diapolo Re: this pull: because I don't feel like implementing it.

In the longer term I (and many others) would also argue it is better to drop it:

  • Makes it possible to drop OpenSSL dependency from bitcoind completely, after secp256k1 verification is used. Simplifies the code overall.
  • I've never heard of anyone using SSL with RPC. It may also be better not to know; after all, this invites opening up the RPC port reachable to the internet or other untrusted networks. The limited amount of configurability of rpcssl almost guarantees this will be an insecure setup.
  • If you really need to use RPC remotely over an untrusted network, it is easy enough to set up stunnel or a SSH tunnel, or even e.g. OpenVPN, with the full power of those tools available you may have a chance of doing so securely.
Member

laanwj commented Jan 23, 2015

@Diapolo Re: this pull: because I don't feel like implementing it.

In the longer term I (and many others) would also argue it is better to drop it:

  • Makes it possible to drop OpenSSL dependency from bitcoind completely, after secp256k1 verification is used. Simplifies the code overall.
  • I've never heard of anyone using SSL with RPC. It may also be better not to know; after all, this invites opening up the RPC port reachable to the internet or other untrusted networks. The limited amount of configurability of rpcssl almost guarantees this will be an insecure setup.
  • If you really need to use RPC remotely over an untrusted network, it is easy enough to set up stunnel or a SSH tunnel, or even e.g. OpenVPN, with the full power of those tools available you may have a chance of doing so securely.
@jonasschnelli

This comment has been minimized.

Show comment
Hide comment
@jonasschnelli

jonasschnelli Jan 23, 2015

Member

I agree with @laanwj. SSL support could lead somebody to believe it's "save". IMO it currently a bad idea to expose bitcoind RPC to a public accessible area. Nevertheless, If one like to do this, he could still do a apache ssl enable reverse proxy to bitcoind's RPC.

Member

jonasschnelli commented Jan 23, 2015

I agree with @laanwj. SSL support could lead somebody to believe it's "save". IMO it currently a bad idea to expose bitcoind RPC to a public accessible area. Nevertheless, If one like to do this, he could still do a apache ssl enable reverse proxy to bitcoind's RPC.

@gmaxwell

This comment has been minimized.

Show comment
Hide comment
@gmaxwell

gmaxwell Jan 23, 2015

Member

Also SSL in the RPC massively increases the attack surface we have exposed (if you also expose it to the outside world) and we've had to push updates previously on account of it-- even though we believe its a feature virtually no one uses. As mentioned it can be better accomplished via stunnel (or any of several other tools)

Member

gmaxwell commented Jan 23, 2015

Also SSL in the RPC massively increases the attack surface we have exposed (if you also expose it to the outside world) and we've had to push updates previously on account of it-- even though we believe its a feature virtually no one uses. As mentioned it can be better accomplished via stunnel (or any of several other tools)

@gavinandresen

This comment has been minimized.

Show comment
Hide comment
@gavinandresen

gavinandresen Jan 23, 2015

Contributor

I agree, it was a mistake to add SSL support to the RPC (mea culpa-- I wrote the original version of that code).

Contributor

gavinandresen commented Jan 23, 2015

I agree, it was a mistake to add SSL support to the RPC (mea culpa-- I wrote the original version of that code).

@jonasschnelli

This comment has been minimized.

Show comment
Hide comment
@jonasschnelli

jonasschnelli Feb 4, 2015

Member

concept ACK.
needs rebase.

Member

jonasschnelli commented Feb 4, 2015

concept ACK.
needs rebase.

@arnuschky

This comment has been minimized.

Show comment
Hide comment
@arnuschky

arnuschky Feb 21, 2015

Contributor

If I might chime in on the SSL discussion: I am not sure whether it should be removed. In terms of integrating bitcoin core into a bigger system, I would expect that it offers SSL in order to protect credentials. This has not much to do with publicly available access, I habitually do this for any service that sends credentials over any network (and I think that this is quite advisable for a bitcoin wallet). Of course, setting up an encrypted tunnel might be an option, but it's cumbersome and error-prone.

I would maybe propose a different route: leave the core's RPC functions unauthenticated, and add authentication and SSL to the wallet.

Contributor

arnuschky commented Feb 21, 2015

If I might chime in on the SSL discussion: I am not sure whether it should be removed. In terms of integrating bitcoin core into a bigger system, I would expect that it offers SSL in order to protect credentials. This has not much to do with publicly available access, I habitually do this for any service that sends credentials over any network (and I think that this is quite advisable for a bitcoin wallet). Of course, setting up an encrypted tunnel might be an option, but it's cumbersome and error-prone.

I would maybe propose a different route: leave the core's RPC functions unauthenticated, and add authentication and SSL to the wallet.

@luke-jr

This comment has been minimized.

Show comment
Hide comment
@luke-jr

luke-jr Feb 21, 2015

Member

@arnuschky There is no secure way to expose RPC to an untrusted network, whether you use the wallet or not. Although perhaps with a libevent-based server maybe that changes...

Member

luke-jr commented Feb 21, 2015

@arnuschky There is no secure way to expose RPC to an untrusted network, whether you use the wallet or not. Although perhaps with a libevent-based server maybe that changes...

@jgarzik

This comment has been minimized.

Show comment
Hide comment
@jgarzik

jgarzik Feb 21, 2015

Contributor

@arnuschky The HTTP REST interface provides unauthenticated access to public blockchain data: https://github.com/bitcoin/bitcoin/blob/master/doc/REST-interface.md

Contributor

jgarzik commented Feb 21, 2015

@arnuschky The HTTP REST interface provides unauthenticated access to public blockchain data: https://github.com/bitcoin/bitcoin/blob/master/doc/REST-interface.md

@arnuschky

This comment has been minimized.

Show comment
Hide comment
@arnuschky

arnuschky Feb 22, 2015

Contributor

@luke-jr What are you referring to?

Sorry, it seems that I haven't made myself clear. I agree that it is advisable to get rid of the openssl dependency in bitcoin core. However, losing the SSL feature for wallet RPC is a bit annoying from the system integrator's point of view - it just makes deployment more complex. I don't agree with the black-and-white consideration of a "trusted" vs "untrusted" network. Even in a so-called "trusted" network, there might be bad actors (hoster, employees, compromised servers etc). Thus, running a wallet RPC access without encryption in a "trusted" network is a bad idea, IMO.

Of course, a tunnel or a reverse proxy is an option, but increases complexity of deployment and is error-prone. Furthermore, if the default is insecure people just run with it. I would even argue the opposite: disable unencrypted RPC (except on 127.0.0.1) - this is also what btcd does, and I find it quite sensible.

Maybe encrypted RPC could be broken out into the wallet with the upcoming modularization of the wallet code? On the long run, I find the design of btcd or for example the cryptonote coins quite sensible: a unauthenticated RPC for public blockchain data (what is now offered by the REST interface), and on a different port an authenticated, encrypted RPC for wallet access.

PS: sorry if I am taking this a bit OT here, maybe this should be discussed somewhere else

Contributor

arnuschky commented Feb 22, 2015

@luke-jr What are you referring to?

Sorry, it seems that I haven't made myself clear. I agree that it is advisable to get rid of the openssl dependency in bitcoin core. However, losing the SSL feature for wallet RPC is a bit annoying from the system integrator's point of view - it just makes deployment more complex. I don't agree with the black-and-white consideration of a "trusted" vs "untrusted" network. Even in a so-called "trusted" network, there might be bad actors (hoster, employees, compromised servers etc). Thus, running a wallet RPC access without encryption in a "trusted" network is a bad idea, IMO.

Of course, a tunnel or a reverse proxy is an option, but increases complexity of deployment and is error-prone. Furthermore, if the default is insecure people just run with it. I would even argue the opposite: disable unencrypted RPC (except on 127.0.0.1) - this is also what btcd does, and I find it quite sensible.

Maybe encrypted RPC could be broken out into the wallet with the upcoming modularization of the wallet code? On the long run, I find the design of btcd or for example the cryptonote coins quite sensible: a unauthenticated RPC for public blockchain data (what is now offered by the REST interface), and on a different port an authenticated, encrypted RPC for wallet access.

PS: sorry if I am taking this a bit OT here, maybe this should be discussed somewhere else

@sipa

This comment has been minimized.

Show comment
Hide comment
@sipa

sipa Mar 3, 2015

Member

Don't let this go unmaintained :)

Member

sipa commented Mar 3, 2015

Don't let this go unmaintained :)

@jonasschnelli

This comment has been minimized.

Show comment
Hide comment
@jonasschnelli

jonasschnelli Mar 5, 2015

Member

Needs rebase. I'm willing to help (testing, fixing, etc.).

Member

jonasschnelli commented Mar 5, 2015

Needs rebase. I'm willing to help (testing, fixing, etc.).

@jonasschnelli

This comment has been minimized.

Show comment
Hide comment
@jonasschnelli

jonasschnelli Mar 10, 2015

Member

I think this also solves #5526

Member

jonasschnelli commented Mar 10, 2015

I think this also solves #5526

@laanwj laanwj added this to the 0.11.0 milestone Mar 12, 2015

@laanwj laanwj modified the milestones: 0.11.0, 0.12.0 May 1, 2015

@jgarzik

This comment has been minimized.

Show comment
Hide comment
@jgarzik

jgarzik May 2, 2015

Contributor

Continued +1 on this.

Contributor

jgarzik commented May 2, 2015

Continued +1 on this.

@ajweiss

This comment has been minimized.

Show comment
Hide comment
@ajweiss

ajweiss May 5, 2015

Contributor

This was straightforward to rebase onto master. While playing with it, this message got printed to the console once:

[warn] event_active: event has no event_base set.

I haven't been able to reproduce it, but I'm banging on it now.

Contributor

ajweiss commented May 5, 2015

This was straightforward to rebase onto master. While playing with it, this message got printed to the console once:

[warn] event_active: event has no event_base set.

I haven't been able to reproduce it, but I'm banging on it now.

@ajweiss

View changes

src/init.cpp
+{
+ RPCServer::OnStopped(&OnRPCStopped);
+ RPCServer::OnPreCommand(&OnRPCPreCommand);
+ if (!StartHTTPServer(threadGroup))

This comment has been minimized.

@ajweiss

ajweiss May 5, 2015

Contributor

I think there's a complicated race condition here. StartHTTPServer() will bind the HTTP server and start listening, but if requests come in before the RPC/REST servers have called RegisterHTTPHandler(), their requests will result in 404s. Moreover, if the request comes in before eventBase is set in StartHTTPServer(), HTTPRequest::WriteReply() will attempt to queue an HTTPEvent with an invalid event_base, which results in the server dropping the request on the floor and not replying at all. (with an ugly libevent warning appearing on the console) You can reproduce this by adding a MilliSleep(5000) to the end of StartHTTPServer before it assigns eventBase. With this in place, regression tests will fail. (It's pretty sporadic without the sleep) You can catch the null eventBase by adding an assertion to HTTPRequest::WriteReply(), where you should see nStatus being set to 404. Hope this is helpful!

@ajweiss

ajweiss May 5, 2015

Contributor

I think there's a complicated race condition here. StartHTTPServer() will bind the HTTP server and start listening, but if requests come in before the RPC/REST servers have called RegisterHTTPHandler(), their requests will result in 404s. Moreover, if the request comes in before eventBase is set in StartHTTPServer(), HTTPRequest::WriteReply() will attempt to queue an HTTPEvent with an invalid event_base, which results in the server dropping the request on the floor and not replying at all. (with an ugly libevent warning appearing on the console) You can reproduce this by adding a MilliSleep(5000) to the end of StartHTTPServer before it assigns eventBase. With this in place, regression tests will fail. (It's pretty sporadic without the sleep) You can catch the null eventBase by adding an assertion to HTTPRequest::WriteReply(), where you should see nStatus being set to 404. Hope this is helpful!

This comment has been minimized.

@laanwj

laanwj May 7, 2015

Member

Thanks a lot for testing this! I'll try to track it down

@laanwj

laanwj May 7, 2015

Member

Thanks a lot for testing this! I'll try to track it down

@cheako

This comment has been minimized.

Show comment
Hide comment
@cheako

cheako May 15, 2015

I attempted to rebase master from the 2015_01_evhttpd branch and ran into a lot of merge conflicts that seemed to have nothing to do with libevent or http. I'm not an expert in git, but does this indicate a serious problem?

I've attempted with multiple -s options, but when completed I perform a diff master and I expected to see changes related to evhttpd but instead everything else seems to change.

cheako commented May 15, 2015

I attempted to rebase master from the 2015_01_evhttpd branch and ran into a lot of merge conflicts that seemed to have nothing to do with libevent or http. I'm not an expert in git, but does this indicate a serious problem?

I've attempted with multiple -s options, but when completed I perform a diff master and I expected to see changes related to evhttpd but instead everything else seems to change.

@jgarzik

This comment has been minimized.

Show comment
Hide comment
@jgarzik

jgarzik May 15, 2015

Contributor

@cheako Sometimes the easiest thing to do is cherry pick each commit onto a fresh tree, rather than trying a git merge.

Contributor

jgarzik commented May 15, 2015

@cheako Sometimes the easiest thing to do is cherry pick each commit onto a fresh tree, rather than trying a git merge.

@laanwj

This comment has been minimized.

Show comment
Hide comment
@laanwj

laanwj May 15, 2015

Member

@cheako This has become non-trivial to rebase, foremost due to a few REST changes and code moves.
I promise to get back to this after the 0.11.0 release bustle and #6121 is merged.

Member

laanwj commented May 15, 2015

@cheako This has become non-trivial to rebase, foremost due to a few REST changes and code moves.
I promise to get back to this after the 0.11.0 release bustle and #6121 is merged.

laanwj and others added some commits Aug 27, 2015

tests: GET requests cannot have request body, use POST in rest.py
Sending a request body with GET request is not valid in HTTP spec, and
not compatible with evhttpd.
evhttpd implementation
- *Replace usage of boost::asio with [libevent2](http://libevent.org/)*.
boost::asio is not part of C++11, so unlike other boost there is no
forwards-compatibility reason to stick with it. Together with #4738 (convert
json_spirit to UniValue), this rids Bitcoin Core of the worst offenders with
regard to compile-time slowness.

- *Replace spit-and-duct-tape http server with evhttp*. Front-end http handling
is handled by libevent, a work queue (with configurable depth and parallelism)
is used to handle application requests.

- *Wrap HTTP request in C++ class*; this makes the application code mostly
HTTP-server-neutral

- *Refactor RPC to move all http-specific code to a separate file*.
Theoreticaly this can allow building without HTTP server but with another RPC
backend, e.g. Qt's debug console (currently not implemented) or future RPC
mechanisms people may want to use.

- *HTTP dispatch mechanism*; services (e.g., RPC, REST) register which URL
paths they want to handle.

By using a proven, high-performance asynchronous networking library (also used
by Tor) and HTTP server, problems such as #5674, #5655, #344 should be avoided.

What works? bitcoind, bitcoin-cli, bitcoin-qt. Unit tests and RPC/REST tests
pass. The aim for now is everything but SSL support.

Configuration options:

- `-rpcthreads`: repurposed as "number of  work handler threads". Still
defaults to 4.

- `-rpcworkqueue`: maximum depth of work queue. When this is reached, new
requests will return a 500 Internal Error.

- `-rpctimeout`: inactivity time, in seconds, after which to disconnect a
client.

- `-debug=http`: low-level http activity logging
Implement RPCTimerHandler for Qt RPC console
Implement RPCTimerHandler for Qt RPC console, so that `walletpassphrase`
works with GUI and `-server=0`.

Also simplify HTTPEvent-related code by using boost::function directly.
Fix race condition between starting HTTP server thread and setting Ev…
…entBase()

Split StartHTTPServer into InitHTTPServer and StartHTTPServer to give
clients a window to register their handlers without race conditions.

Thanks @ajweiss for figuring this out.
libevent: Windows reuseaddr workaround in depends
Make it possible to reuse sockets.
This is necessary to make the RPC tests work in WINE.

laanwj added some commits Sep 4, 2015

doc: update deps in build-unix.md after libevent
Add libevent, change usage of libssl from "secure communication" to
"crypto" that's more accurate after RPC SSL support removed.
Revert "rpc-tests: re-enable rpc-tests for Windows"
This reverts commit bd30c3d.

Disable windows RPC tests for now. These should be re-enabled once a
suitable Wine version is used on Travis.

@laanwj laanwj merged commit d528025 into bitcoin:master Sep 4, 2015

1 check passed

continuous-integration/travis-ci/pr The Travis CI build passed
Details

laanwj added a commit that referenced this pull request Sep 4, 2015

Merge pull request #5677
d528025 Revert "rpc-tests: re-enable rpc-tests for Windows" (Wladimir J. van der Laan)
1e700c9 doc: update deps in build-unix.md after libevent (Wladimir J. van der Laan)
26c9b83 Move windows socket init to utility function (Wladimir J. van der Laan)
4be0b08 libevent: Windows reuseaddr workaround in depends (Cory Fields)
3a174cd Fix race condition between starting HTTP server thread and setting EventBase() (Wladimir J. van der Laan)
6d2bc22 Document options for new HTTP/RPC server in --help (Wladimir J. van der Laan)
be33f3f Implement RPCTimerHandler for Qt RPC console (Wladimir J. van der Laan)
57d85d9 doc: mention SSL support dropped for RPC in release notes (Wladimir J. van der Laan)
40b556d evhttpd implementation (Wladimir J. van der Laan)
ee2a42b tests: GET requests cannot have request body, use POST in rest.py (Wladimir J. van der Laan)
6e996d3 tests: fix qt payment test (Cory Fields)
3140ef9 build: build-system changes for libevent (Wladimir J. van der Laan)
a9af234 libevent: add depends (Cory Fields)
6a21dd5 Remove rpc_boostasiotocnetaddr test (Wladimir J. van der Laan)
8f9301c qa: Remove -rpckeepalive tests from httpbasics (Wladimir J. van der Laan)
51fcfc0 doc: remove documentation for rpcssl (Wladimir J. van der Laan)
@jonasschnelli

This comment has been minimized.

Show comment
Hide comment
@jonasschnelli

jonasschnelli Sep 4, 2015

Member

Post merge tested ACK

Member

jonasschnelli commented Sep 4, 2015

Post merge tested ACK

@paveljanik

This comment has been minimized.

Show comment
Hide comment
@paveljanik

paveljanik Sep 4, 2015

Contributor

Build notes should be extended for at least OS X/brew. I use macports here and:

port install libevent

was needed to get it compiled.

Contributor

paveljanik commented Sep 4, 2015

Build notes should be extended for at least OS X/brew. I use macports here and:

port install libevent

was needed to get it compiled.

@jonasschnelli

This comment has been minimized.

Show comment
Hide comment
@jonasschnelli

jonasschnelli Sep 4, 2015

Member

@paveljanik: just added libevent mentioning for osx in #6635

Member

jonasschnelli commented Sep 4, 2015

@paveljanik: just added libevent mentioning for osx in #6635

@jgarzik

This comment has been minimized.

Show comment
Hide comment
@jgarzik

jgarzik Sep 4, 2015

Contributor

post merge tested re-ACK

Contributor

jgarzik commented Sep 4, 2015

post merge tested re-ACK

@jtimon jtimon referenced this pull request Sep 4, 2015

Merged

Add ZeroMQ notifications #6103

@msgilligan msgilligan referenced this pull request in bitcoinj/bitcoinj Sep 20, 2015

Open

JSON-RPC Support #895

luke-jr added a commit to luke-jr/bitcoin that referenced this pull request Jan 10, 2016

luke-jr added a commit to luke-jr/bitcoin that referenced this pull request Jan 10, 2016

@unsystemizer unsystemizer referenced this pull request in CounterpartyXCP/counterparty-lib Feb 8, 2016

Open

KeyError issue #807

@dagurval dagurval referenced this pull request in bitcoinxt/bitcoinxt Nov 18, 2016

Merged

libevent-based http server #172

@str4d str4d referenced this pull request in zcash/zcash Mar 14, 2017

Merged

libevent-based http server #2176

@denravonska denravonska referenced this pull request in gridcoin-community/Gridcoin-Research Jun 4, 2018

Merged

Fix SSL RPC issues #1139

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment