Support multiple http endpoints #1137

huangminghuang · 2023-05-05T14:47:05Z

This PR implements the proposal https://github.com/eosnetworkfoundation/product/blob/main/api-http/proposals/http-plugin.md to support multiple endpoints for http plugins.

APIs are categorized into the following groups:

chain_ro

v1/chain/get_activated_protocol_features
v1/chain/get_block_info
v1/chain/get_block_header_state
v1/chain/get_account
v1/chain/get_code
v1/chain/get_code_hash
v1/chain/get_consensus_parameters
v1/chain/get_abi
v1/chain/get_raw_code_and_abi
v1/chain/get_raw_abi
v1/chain/get_table_rows
v1/chain/get_table_by_scope
v1/chain/get_currency_balance
v1/chain/get_currency_stats
v1/chain/get_producers
v1/chain/get_producer_schedule
v1/chain/get_scheduled_transactions
v1/chain/get_required_keys
v1/chain/get_transaction_id
v1/chain/send_read_only_transaction
v1/chain/get_accounts_by_authorizers
v1/chain/get_raw_block
v1/chain/get_block
v1/chain/get_block_header
v1/chain/get_transaction_status

chain_rw

v1/chain/compute_transaction
v1/chain/push_transaction
v1/chain/push_transactions
v1/chain/send_transaction
v1/chain/send_transaction2

db_size

v1/db_size/get

net_ro

v1/net/status
v1/net/connections

net_rw

v1/net/connect
v1/net/disconnect

producer_ro

v1/producer/paused
v1/producer/get_runtime_options
v1/producer/get_greylist
v1/producer/get_whitelist_blacklist
v1/producer/get_scheduled_protocol_feature_activations
v1/producer/get_supported_protocol_features
v1/producer/get_account_ram_corrections
v1/producer/get_unapplied_transactions

producer_rw

v1/producer/pause
v1/producer/resume
v1/producer/update_runtime_options
v1/producer/add_greylist_accounts
v1/producer/remove_greylist_accounts
v1/producer/set_whitelist_blacklist
v1/producer/schedule_protocol_feature_activations
v1/producer/get_integrity_hash

snapshot

v1/producer/get_snapshot_requests
v1/producer/create_snapshot
v1/producer/schedule_snapshot
v1/producer/unschedule_snapshot

trace_api

v1/trace_api/get_block
v1/trace_api/get_transaction_trace

prometheus

v1/prometheus/metrics

node

v1/node/get_supported_apis
v1/chain/get_info

User configuration knob

A single http-category-address can be used to configure all addresses in command line and ini file. The option can be used multiple times as needed.

http-server-address   = http-category-address
http-category-address = chain_ro,127.0.0.1:8081
http-category-address = chain_ro,:9090
http-category-address = chain_rw,[::1]:8082
http-category-address = net_ro,127.0.0.1:8081
http-category-address = net_rw,[::]:8083 
http-category-address = producer_ro,/tmp/absolute_unix_path.sock
http-category-address = producer_rw,./relative/unix_path.sock
http-category-address = snapshot,./relative/unix_path.sock

A single hostname:port specification can be used by multiple categories, as of the case for 127.0.0.1:8081 in above example. However, two specifications having the same port with different hostname strings are always considered as configuration error regardless whether they can be resolved into the same set of IP addresses. For example, the following configuration are invalid.

http-category-address = chain_ro,127.0.0.1:8081
http-category-address = chain_rw,localhost:8081 # error, 127.0.0.1 and localhost are different

For the node category, it is NOT user configurable because it will be available for all listened http addresses.

For backward compatibility, the HTTP category facility has to be opted in by specifying http-server-address = http-category-address explicitly.
When http-server-address = http-category-address, all the unspecified categories are considered disabled. In addition, http-server-address = http-category-address cannot be specified together with a non-empty unix-socket-path.

All existing configuration options in the http plugin apply to all handlers for all addresses; this includes the http-threads option. This means all handlers share the same thread pool.

server address format

server address can be in one of the following formats
- ipv4_address:port like 127.0.0.1:8080
- ipv6_address:port like [2001:db8:3c4d:15::1a2f:1a2b]:8080
- hostname:port like my.domain.com:8080
- unix socket path (must starts with '/' or './')
In the case of the hostname, the IP address is resolved by ARP, ALL resolved addresses will be listened to.
To listen to all interfaces for both IPv4 and IPv6, just use :8080. Notice that the behavior of [::]:8080 is dependent on whether the IPv4-mapped IPv6 is enabled in the system.

Resolves #965

plugins/http_plugin/http_plugin.cpp

plugins/http_plugin/include/eosio/http_plugin/beast_http_session.hpp

plugins/chain_api_plugin/chain_api_plugin.cpp

plugins/http_plugin/include/eosio/http_plugin/common.hpp

heifner · 2023-05-09T20:43:20Z

tests/CMakeLists.txt

@@ -241,6 +241,10 @@ add_test(NAME plugin_http_api_test COMMAND tests/plugin_http_api_test.py WORKING
 set_tests_properties(plugin_http_api_test PROPERTIES TIMEOUT 50)
 set_property(TEST plugin_http_api_test PROPERTY LABELS nonparallelizable_tests)

+add_test(NAME plugin_http_category_api_test COMMAND tests/plugin_http_api_test.py WORKING_DIRECTORY ${CMAKE_BINARY_DIR})
+set_tests_properties(plugin_http_category_api_test PROPERTIES TIMEOUT 50 ENVIRONMENT "PLUGIN_HTTP_TEST_CATEGORY=ON")
+set_property(TEST plugin_http_category_api_test PROPERTY LABELS nonparallelizable_tests)


How is different then the existing plugin_http_api_test, no new args are passed to the test.

I see this passes an ENVIRONMENT variable. Can we change that to an argument to the python test.

I tried several ways and failed. This is the only way that I can think of which doesn't require me to rewrite the entire test.

plugins/http_plugin/tests/unit_tests.cpp

heifner · 2023-05-11T13:07:45Z

Tests failing because keosd not launching:

info  2023-05-10T21:38:51.868 keosd     wallet_plugin.cpp:32          plugin_initialize    ] initializing wallet plugin
warn  2023-05-10T21:38:51.869 keosd     http_plugin.cpp:483           plugin_initialize    ] 13 N5boost10wrapexceptINS_12bad_any_castEEE: boost::bad_any_cast: failed conversion using boost::any_cast
rethrow boost::bad_any_cast: failed conversion using boost::any_cast: 
    {"what":"boost::bad_any_cast: failed conversion using boost::any_cast"}
    keosd  http_plugin.cpp:483 plugin_initialize

appbase: exception thrown during plugin "eosio::wallet_api_plugin" initialization.
boost::bad_any_cast: failed conversion using boost::any_cast

plugins/http_plugin/http_plugin.cpp

plugins/http_plugin/tests/unit_tests.cpp

plugins/http_plugin/http_plugin.cpp

plugins/http_plugin/include/eosio/http_plugin/common.hpp

greg7mdp · 2023-05-11T15:22:50Z

plugins/http_plugin/http_plugin.cpp

+            boost::system::error_code ec;
+            tcp::resolver             resolver(plugin_state->thread_pool.get_executor());
+            auto endpoints = resolver.resolve(host, port, boost::asio::ip::tcp::resolver::passive, ec);
+            EOS_ASSERT(!ec, chain::plugin_config_exception, "failed to resolve address: ${msg}", ("msg", ec.message()));


Just curious, why are we throwing here if the address is incorrect? Where is the intended try/catch block?

I believe main() should catch it.

Good point, should likely not throw from here.

This looks like it is only called from is_on_loopback which is only currently called from plugin_initialize. However It seems like it would be better for this to only log a warning and return false. Seems like loopback should always be resolvable so unable to resolve likely means not on loopback.

plugins/http_plugin/http_plugin.cpp

plugins/http_plugin/tests/unit_tests.cpp

tests/plugin_http_api_test.py

plugins/http_plugin/http_plugin.cpp

greg7mdp · 2023-05-12T13:08:56Z

I'm not understanding where we do the open()/bind()/listen() on the acceptor in this new version?

plugins/http_plugin/http_plugin.cpp

support multiple http endpoints

6fb23ca

huangminghuang requested a review from heifner May 5, 2023 14:47

fix for ubuntu 22.04

be4d98f

heifner added the OCI Work exclusive to OCI team label May 5, 2023

heifner added this to the Leap v5.0.0-rc1 milestone May 5, 2023

huangminghuang added 4 commits May 5, 2023 14:11

fix host header

cfbd1e5

conditional enable ipv6 test

444edc6

revise host validation

8b5645a

fix local_address

841896f

BenjaminGormanPMP requested a review from greg7mdp May 8, 2023 21:17

heifner requested changes May 9, 2023

View reviewed changes

greg7mdp reviewed May 10, 2023

View reviewed changes

plugins/http_plugin/tests/unit_tests.cpp Outdated Show resolved Hide resolved

plugins/http_plugin/tests/unit_tests.cpp Outdated Show resolved Hide resolved

plugins/http_plugin/tests/unit_tests.cpp Outdated Show resolved Hide resolved

address PR comments

216a96d

fix keosd

1d06df3

heifner reviewed May 11, 2023

View reviewed changes

plugins/http_plugin/http_plugin.cpp Outdated Show resolved Hide resolved

heifner reviewed May 11, 2023

View reviewed changes

plugins/http_plugin/http_plugin.cpp Outdated Show resolved Hide resolved

heifner reviewed May 11, 2023

View reviewed changes

plugins/http_plugin/http_plugin.cpp Show resolved Hide resolved

greg7mdp reviewed May 11, 2023

View reviewed changes

plugins/http_plugin/tests/unit_tests.cpp Outdated Show resolved Hide resolved

greg7mdp reviewed May 11, 2023

View reviewed changes

plugins/http_plugin/tests/unit_tests.cpp Outdated Show resolved Hide resolved

plugins/http_plugin/http_plugin.cpp Outdated Show resolved Hide resolved

greg7mdp reviewed May 11, 2023

View reviewed changes

more PR comment fix

0880013

greg7mdp reviewed May 11, 2023

View reviewed changes

plugins/http_plugin/tests/unit_tests.cpp Show resolved Hide resolved

jgiszczak reviewed May 11, 2023

View reviewed changes

tests/plugin_http_api_test.py Outdated Show resolved Hide resolved

greg7mdp reviewed May 11, 2023

View reviewed changes

greg7mdp requested changes May 12, 2023

View reviewed changes

plugins/http_plugin/http_plugin.cpp Outdated Show resolved Hide resolved

address PR comments

7cde470

greg7mdp requested changes May 12, 2023

View reviewed changes

plugins/http_plugin/http_plugin.cpp Outdated Show resolved Hide resolved

Merge remote-tracking branch 'origin/main' into huangminghuang/http

7133a66

huangminghuang added 3 commits May 12, 2023 16:12

log categories for listening addresses

e73fd5a

fix grammar

5d29893

address PR comments

eb0f174

heifner approved these changes May 15, 2023

View reviewed changes

greg7mdp approved these changes May 15, 2023

View reviewed changes

huangminghuang merged commit e429ba6 into main May 15, 2023
13 checks passed

huangminghuang deleted the huangminghuang/http branch May 15, 2023 14:16

heifner added the documentation Improvements or additions to documentation label May 16, 2023

bhazzard mentioned this pull request Jun 15, 2023

prometheus plugin cannot handle HTTP HEAD request #1085

Open

heifner mentioned this pull request Jun 22, 2023

LEAP 4.0.3 - Long startup time after upgrade to 4.0.3. #1331

Closed

heifner mentioned this pull request Sep 27, 2023

Prometheus exporter not working in 5.0 test build #1681

Closed

matthewdarwin mentioned this pull request Sep 27, 2023

Provide example config.ini that matches old behaviour for "Support multiple http endpoints" 5.0 #1684

Closed

This was referenced Sep 29, 2023

Remove prometheus-exporter-address option #1698

Merged

[5.0] Improve cleos error messages #1733

Merged

heifner mentioned this pull request Oct 9, 2023

[5.0 -> main] Improve cleos error messages #1738

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support multiple http endpoints #1137

Support multiple http endpoints #1137

huangminghuang commented May 5, 2023 •

edited by greg7mdp

heifner May 9, 2023

heifner May 11, 2023

huangminghuang May 11, 2023

heifner commented May 11, 2023

greg7mdp May 11, 2023

huangminghuang May 12, 2023

heifner May 12, 2023

heifner May 15, 2023

greg7mdp commented May 12, 2023

Support multiple http endpoints #1137

Support multiple http endpoints #1137

Conversation

huangminghuang commented May 5, 2023 • edited by greg7mdp

chain_ro

chain_rw

db_size

net_ro

net_rw

producer_ro

producer_rw

snapshot

trace_api

prometheus

node

User configuration knob

server address format

heifner May 9, 2023

Choose a reason for hiding this comment

heifner May 11, 2023

Choose a reason for hiding this comment

huangminghuang May 11, 2023

Choose a reason for hiding this comment

heifner commented May 11, 2023

greg7mdp May 11, 2023

Choose a reason for hiding this comment

huangminghuang May 12, 2023

Choose a reason for hiding this comment

heifner May 12, 2023

Choose a reason for hiding this comment

heifner May 15, 2023

Choose a reason for hiding this comment

greg7mdp commented May 12, 2023

huangminghuang commented May 5, 2023 •

edited by greg7mdp