[Merged by Bors] - node, api: split out private api and rework api configuration #4213

dshulyak · 2023-03-28T08:25:01Z

closes: #3883
needs: spacemeshos/api#226

      --grpc-json-listener string                     Socket for the grpc gateway for the list of services in grpc-public-services. If left empty - grpc gateway won't be enabled.
      --grpc-private-listener string                  Socket for the list of services specified in grpc-private-services. (default "127.0.0.1:9093")
      --grpc-private-services strings                 List of services that must be kept private or exposed only in secure environments. (default [smesher])
      --grpc-public-listener string                   Socket for the list of services specified in grpc-public-services. (default "0.0.0.0:9092")
      --grpc-public-services strings                  List of services that are safe to open for the network. (default [debug,gateway,global,mesh,transaction,node])
      --grpc-recv-msg-size int                        GRPC api recv message size (default 10485760)
      --grpc-send-msg-size int                        GRPC api send message size (default 10485760)

we support two different sockets where services can be enabled. public (default to 0.0.0.0:9092) and private (defaults to 127.0.0.1:9093). by default we run smesher on private, and everything else on public. this can be changed by option above.

`🤖 Generated by Copilot at 2abafcf`

This pull request refactors the API configuration and implementation to support multiple GRPC servers and services, and adds a new feature to allow smeshers to update and switch between PoET servers. It also updates the tests, logging, and presets to use the new configuration and services. It modifies the following files: api/config/config.go, api/grpcserver/grpc.go, api/grpcserver/grpcserver_test.go, api/grpcserver/http_server.go, api/grpcserver/node_service.go, api/grpcserver/smesher_service.go, api/grpcserver/transaction_service_test.go, cmd/base.go, cmd/node/node_test.go, cmd/node/node.go, activation/interface.go, activation/mocks.go, api/grpcserver/smesher_service_test.go, api/node.go, cmd/bootstrapper/generator_test.go, cmd/root.go, config/presets/fastnet.go, config/presets/testnet.go, go.mod, and systest/parameters/fastnet/smesher.json.

codecov · 2023-03-28T08:40:51Z

Codecov Report

Merging #4213 (2abafcf) into develop (5325c04) will increase coverage by 0.0%.
The diff coverage is 88.3%.

@@           Coverage Diff           @@
##           develop   #4213   +/-   ##
=======================================
  Coverage     76.3%   76.3%           
=======================================
  Files          240     240           
  Lines        25086   25078    -8     
=======================================
- Hits         19151   19145    -6     
+ Misses        4728    4726    -2     
  Partials      1207    1207

Impacted Files	Coverage Δ
api/grpcserver/node_service.go	`65.9% <ø> (-2.8%)`	⬇️
config/presets/fastnet.go	`100.0% <ø> (+3.1%)`	⬆️
config/presets/testnet.go	`100.0% <ø> (+3.1%)`	⬆️
api/grpcserver/http_server.go	`88.4% <60.0%> (-3.5%)`	⬇️
api/grpcserver/smesher_service.go	`63.9% <90.9%> (+1.7%)`	⬆️
api/config/config.go	`100.0% <100.0%> (+3.2%)`	⬆️
api/grpcserver/grpc.go	`88.8% <100.0%> (+7.8%)`	⬆️

... and 17 files with indirect coverage changes

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

brusherru · 2023-03-29T15:44:13Z

One important moment, NodeService contains some endpoints that should work well on a remote API: Status, Build, Version, Echo, StatusStream. However, the rest endpoints should be private or completely removed: UpdatePoetServers, ErrorStream, SyncStart (remove?), Shutdown (already deprecated as I know).

Instead of doing some complicated separation of endpoints within the Service, I propose to move them to another Service (e.g. AdminService). What do you think?

dshulyak · 2023-03-29T16:02:07Z

Instead of doing some complicated separation of endpoints within the Service, I propose to move them to another Service (e.g. AdminService). What do you think?

sure, if thats what you prefer i will move them

dshulyak · 2023-03-29T16:26:25Z

@brusherru actually does it makes sense if i remove Shutdown/StartSync completely, and move UpdatePoetServers to Smeshing api. and move Node to the list of public services?

brusherru · 2023-03-29T18:29:31Z

@dshulyak yes, sounds good to me! And much better than one more service! :)

dshulyak · 2023-03-31T07:21:51Z

@brusherru i moved stuff around the way we discussedd

countvonzero

please update the PR description. thanks

countvonzero · 2023-03-31T20:35:57Z

api/config/config.go

-		SmesherStreamInterval: defaultSmesherStreamInterval,
+		PublicServices:        []Service{Debug, Gateway, GlobalState, Mesh, Transaction, Node},
+		PublicListener:        "0.0.0.0:9092",
+		PrivateServices:       []Service{Smesher},


it almost feels like we should call this Admin

brusherru

LGTM

dshulyak · 2023-04-11T08:46:07Z

bors merge

closes: #3883 needs: spacemeshos/api#226 ``` --grpc-json-listener string Socket for the grpc gateway for the list of services in grpc-public-services. If left empty - grpc gateway won't be enabled. --grpc-private-listener string Socket for the list of services specified in grpc-private-services. (default "127.0.0.1:9093") --grpc-private-services strings List of services that must be kept private or exposed only in secure environments. (default [smesher]) --grpc-public-listener string Socket for the list of services specified in grpc-public-services. (default "0.0.0.0:9092") --grpc-public-services strings List of services that are safe to open for the network. (default [debug,gateway,global,mesh,transaction,node]) --grpc-recv-msg-size int GRPC api recv message size (default 10485760) --grpc-send-msg-size int GRPC api send message size (default 10485760) ``` we support two different sockets where services can be enabled. public (default to 0.0.0.0:9092) and private (defaults to 127.0.0.1:9093). by default we run smesher on private, and everything else on public. this can be changed by option above.

dshulyak · 2023-04-11T08:47:28Z

bors cancel

bors · 2023-04-11T08:47:29Z

Canceled.

dshulyak · 2023-04-11T13:03:32Z

bors merge

closes: #3883 needs: spacemeshos/api#226 ``` --grpc-json-listener string Socket for the grpc gateway for the list of services in grpc-public-services. If left empty - grpc gateway won't be enabled. --grpc-private-listener string Socket for the list of services specified in grpc-private-services. (default "127.0.0.1:9093") --grpc-private-services strings List of services that must be kept private or exposed only in secure environments. (default [smesher]) --grpc-public-listener string Socket for the list of services specified in grpc-public-services. (default "0.0.0.0:9092") --grpc-public-services strings List of services that are safe to open for the network. (default [debug,gateway,global,mesh,transaction,node]) --grpc-recv-msg-size int GRPC api recv message size (default 10485760) --grpc-send-msg-size int GRPC api send message size (default 10485760) ``` we support two different sockets where services can be enabled. public (default to 0.0.0.0:9092) and private (defaults to 127.0.0.1:9093). by default we run smesher on private, and everything else on public. this can be changed by option above.

bors · 2023-04-11T14:11:34Z

Build failed:

systest-status

dshulyak · 2023-04-11T14:41:21Z

bors merge

closes: #3883 needs: spacemeshos/api#226 ``` --grpc-json-listener string Socket for the grpc gateway for the list of services in grpc-public-services. If left empty - grpc gateway won't be enabled. --grpc-private-listener string Socket for the list of services specified in grpc-private-services. (default "127.0.0.1:9093") --grpc-private-services strings List of services that must be kept private or exposed only in secure environments. (default [smesher]) --grpc-public-listener string Socket for the list of services specified in grpc-public-services. (default "0.0.0.0:9092") --grpc-public-services strings List of services that are safe to open for the network. (default [debug,gateway,global,mesh,transaction,node]) --grpc-recv-msg-size int GRPC api recv message size (default 10485760) --grpc-send-msg-size int GRPC api send message size (default 10485760) ``` we support two different sockets where services can be enabled. public (default to 0.0.0.0:9092) and private (defaults to 127.0.0.1:9093). by default we run smesher on private, and everything else on public. this can be changed by option above.

bors · 2023-04-11T15:13:25Z

Build failed:

systest-status

dshulyak · 2023-04-11T15:18:34Z

bors merge

closes: #3883 needs: spacemeshos/api#226 ``` --grpc-json-listener string Socket for the grpc gateway for the list of services in grpc-public-services. If left empty - grpc gateway won't be enabled. --grpc-private-listener string Socket for the list of services specified in grpc-private-services. (default "127.0.0.1:9093") --grpc-private-services strings List of services that must be kept private or exposed only in secure environments. (default [smesher]) --grpc-public-listener string Socket for the list of services specified in grpc-public-services. (default "0.0.0.0:9092") --grpc-public-services strings List of services that are safe to open for the network. (default [debug,gateway,global,mesh,transaction,node]) --grpc-recv-msg-size int GRPC api recv message size (default 10485760) --grpc-send-msg-size int GRPC api send message size (default 10485760) ``` we support two different sockets where services can be enabled. public (default to 0.0.0.0:9092) and private (defaults to 127.0.0.1:9093). by default we run smesher on private, and everything else on public. this can be changed by option above.

bors · 2023-04-11T15:30:52Z

Build failed:

ci-status

dshulyak · 2023-04-11T16:04:51Z

bors merge

closes: #3883 needs: spacemeshos/api#226 ``` --grpc-json-listener string Socket for the grpc gateway for the list of services in grpc-public-services. If left empty - grpc gateway won't be enabled. --grpc-private-listener string Socket for the list of services specified in grpc-private-services. (default "127.0.0.1:9093") --grpc-private-services strings List of services that must be kept private or exposed only in secure environments. (default [smesher]) --grpc-public-listener string Socket for the list of services specified in grpc-public-services. (default "0.0.0.0:9092") --grpc-public-services strings List of services that are safe to open for the network. (default [debug,gateway,global,mesh,transaction,node]) --grpc-recv-msg-size int GRPC api recv message size (default 10485760) --grpc-send-msg-size int GRPC api send message size (default 10485760) ``` we support two different sockets where services can be enabled. public (default to 0.0.0.0:9092) and private (defaults to 127.0.0.1:9093). by default we run smesher on private, and everything else on public. this can be changed by option above.

bors · 2023-04-11T17:13:55Z

Build failed:

systest-status

dshulyak · 2023-04-11T18:00:17Z

bors merge

closes: #3883 needs: spacemeshos/api#226 ``` --grpc-json-listener string Socket for the grpc gateway for the list of services in grpc-public-services. If left empty - grpc gateway won't be enabled. --grpc-private-listener string Socket for the list of services specified in grpc-private-services. (default "127.0.0.1:9093") --grpc-private-services strings List of services that must be kept private or exposed only in secure environments. (default [smesher]) --grpc-public-listener string Socket for the list of services specified in grpc-public-services. (default "0.0.0.0:9092") --grpc-public-services strings List of services that are safe to open for the network. (default [debug,gateway,global,mesh,transaction,node]) --grpc-recv-msg-size int GRPC api recv message size (default 10485760) --grpc-send-msg-size int GRPC api send message size (default 10485760) ``` we support two different sockets where services can be enabled. public (default to 0.0.0.0:9092) and private (defaults to 127.0.0.1:9093). by default we run smesher on private, and everything else on public. this can be changed by option above.

bors · 2023-04-11T18:37:56Z

Pull request successfully merged into develop.

Build succeeded:

ci-status
systest-status

dshulyak added 4 commits March 28, 2023 10:51

rework configuration

43dea84

in system tests move all to public interface

6d209bf

add tests back

2897b7b

typo

4b279e9

dshulyak requested review from countvonzero, fasmat and poszu as code owners March 28, 2023 08:25

dshulyak requested review from brusherru and pigmej March 28, 2023 08:25

dshulyak added 4 commits March 31, 2023 09:17

update go mod to my commit

99bc409

Merge branch 'develop' into private-api

41f66c9

update as discussed

937e27a

move node service to public

dd5a0be

dshulyak added 2 commits March 31, 2023 10:22

tidy

fae8188

drop ActivationAPI from mocks

9a5722c

countvonzero approved these changes Mar 31, 2023

View reviewed changes

brusherru approved these changes Apr 10, 2023

View reviewed changes

brusherru mentioned this pull request Apr 11, 2023

Support separated public and private APIs spacemeshos/smapp#1213

Merged

pigmej approved these changes Apr 11, 2023

View reviewed changes

Merge branch 'develop' into private-api

dd95c88

fix things

f75a9b0

drop gateway from config

31bfd97

add smesher to config..

8bf8d9e

zero out maps/slices before updating them

2abafcf

bors bot changed the title ~~node, api: split out private api and rework api configuration~~ [Merged by Bors] - node, api: split out private api and rework api configuration Apr 11, 2023

bors bot closed this Apr 11, 2023

lrettig mentioned this pull request Apr 13, 2023

Major code cleanup spacemeshos/smcli#34

Merged

pigmej mentioned this pull request Apr 18, 2023

Split node api to private and public spacemeshos/pm#225

Closed

2 tasks

kacpersaw mentioned this pull request May 22, 2023

Support separate grpc apis spacemeshos/explorer-backend#51

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Merged by Bors] - node, api: split out private api and rework api configuration #4213

[Merged by Bors] - node, api: split out private api and rework api configuration #4213

dshulyak commented Mar 28, 2023 •

edited by pigmej

Loading

codecov bot commented Mar 28, 2023 •

edited

Loading

brusherru commented Mar 29, 2023

dshulyak commented Mar 29, 2023

dshulyak commented Mar 29, 2023

brusherru commented Mar 29, 2023

dshulyak commented Mar 31, 2023

countvonzero left a comment

countvonzero Mar 31, 2023

brusherru left a comment

dshulyak commented Apr 11, 2023

dshulyak commented Apr 11, 2023

bors bot commented Apr 11, 2023

dshulyak commented Apr 11, 2023

bors bot commented Apr 11, 2023

dshulyak commented Apr 11, 2023

bors bot commented Apr 11, 2023

dshulyak commented Apr 11, 2023

bors bot commented Apr 11, 2023

dshulyak commented Apr 11, 2023

bors bot commented Apr 11, 2023

dshulyak commented Apr 11, 2023

bors bot commented Apr 11, 2023

[Merged by Bors] - node, api: split out private api and rework api configuration #4213

[Merged by Bors] - node, api: split out private api and rework api configuration #4213

Conversation

dshulyak commented Mar 28, 2023 • edited by pigmej Loading

🤖 Generated by Copilot at 2abafcf

codecov bot commented Mar 28, 2023 • edited Loading

Codecov Report

brusherru commented Mar 29, 2023

dshulyak commented Mar 29, 2023

dshulyak commented Mar 29, 2023

brusherru commented Mar 29, 2023

dshulyak commented Mar 31, 2023

countvonzero left a comment

Choose a reason for hiding this comment

countvonzero Mar 31, 2023

Choose a reason for hiding this comment

brusherru left a comment

Choose a reason for hiding this comment

dshulyak commented Apr 11, 2023

dshulyak commented Apr 11, 2023

bors bot commented Apr 11, 2023

dshulyak commented Apr 11, 2023

bors bot commented Apr 11, 2023

dshulyak commented Apr 11, 2023

bors bot commented Apr 11, 2023

dshulyak commented Apr 11, 2023

bors bot commented Apr 11, 2023

dshulyak commented Apr 11, 2023

bors bot commented Apr 11, 2023

dshulyak commented Apr 11, 2023

bors bot commented Apr 11, 2023

dshulyak commented Mar 28, 2023 •

edited by pigmej

Loading

`🤖 Generated by Copilot at 2abafcf`

codecov bot commented Mar 28, 2023 •

edited

Loading