Boot time speed up & metrics #1

jorgecuesta · 2023-03-31T21:23:31Z

High-Level Summary

This update includes changes that allow for faster boot time and better overall performance & maintainability.

Mesh bootstrap time has been greatly reduced to almost 0, which improves overall performance and makes it easier to maintain your key files.
Mesh now supports hot reloading of keys and chains efficiently by deduplicating full nodes, which allows for faster updates and better overall performance.
Code has been split into smaller files to make it easier to navigate and modify, which helps with both development and maintenance.
The Relay, Relay Time, and Error metrics now use labels for chain id and name to differentiate between different chains and to help with identifying and categorizing the various metrics.
Misc bug fixes such as connection re-use and invalid sessions which should increase the performance of your mesh and full node.

Breaking Changes

This changes require you update both: Pocket Node & Mesh Node

Added /v1/private/mesh/check endpoint
Moved /v1/mesh/health to /v1/private/mesh/health
Removed /v1/private/mesh/servicer
Removed Prometheus metrics using chain id as part of the metric name
Removed authtoken query param from private mesh endpoints in favor of Authorization: <token> header.
- NOTE: No changes need on your side unless you have some tooling built over /v1/private/mesh/* endpoints

NOTE:

You must add the new private endpoints (check) and use the same code/image on both sides or your mesh node will not start.
You could remove the /v1/health endpoint from your proxy.

Configuration Changes

Added node_check_interval to configure the rate at a full Node is health checked.
Added log_chain_request and log_chain_response to avoid unnecessary debug logs.
Added chains_name_map and remote_chains_name_map to enhance exposed metrics with chain name.
Added metrics_moniker to help to identify a mesh node instance on metrics queries.
Added metrics_report_interval to configure the rate at metrics about workers are reported.
Added <servicer|chain>_rpc_max_idle_connections
Added <servicer|chain>_rpc_max_conns_per_host
Added <servicer|chain>_rpc_max_idle_conns_per_host
Added chain_drop_connections
Added client_rpc_read_timeout
Added client_rpc_read_header_timeout
Added client_rpc_write_timeout
Added chain_request_path_cleanup
Removed hot_reload_interval in favor of keys_hot_reload_interval and chains_hot_reload_interval as separated values
Rename worker pool options to specify a different set for Servicer or Metrics
Remove rpc_timeout in favor of: client_rpc_timeout and chain_rpc_timeout

Core Changes

New:

New keys structure
Deduplication of nodes (reduce bandwidth usage and faster boot time)
Allow control RPS of mesh to a node (using servicer_max_workers) due to how workers are used now
Chains check. Check against the Pocket node if the chains id on the chain file matches the one on Pocket Node.
Added a Worker for metrics
Chains name maps (local or remote) to enhance metrics
Added endpoint to allow mesh node check against servicer node (health, chains, addresses)
Added sanitization of the URL before call chain due to errors observed on chains like Avax/DFK sending \t characters on the path.

Fix:

Invalid sessions error
HTTP connection drops due to avoiding read response body (decreases CPU utilization and bandwidth)
Handle a few cases where after an error it keeps going without interrupting the flow, invalidating a relay/session.
Update chains using /v1/private/mesh/updatechains endpoint

Rework:

Relay, Relay Time, and Error metrics now use labels for chain id and name
Split Keys and Chains hot reload interval to allow them to work independently.
Moved the authtoken query parameter used to call private method to be passed using the HTTP header Authorization to avoid it being leaked on logs.
Split code in files to help development and readability of it.
Bootstrap time reduce to almost 0

Dependencies:

Added xsync to have better performance that the native for concurrent access.
Added gojsonschema to allow validate keys and chains name map format.
Added golang-set to manage set of values in arrays faster and async safe
Bump version github.com/alitto/pond from v1.8.1 to v1.8.3
Bump version github.com/prometheus/client_golang from v1.11.0 to v1.11.1

Bump Version

New version: RC-0.3.0

Docs

Update mesh.doc
Added missing links to dockerhub
Added Grafana dashboard to consume the new metrics.
Updated rpc-spec.yaml

Added different key format. Refactor connectivity checks. Refactor node/servicer internal structure of mesh to reduce amount of worker/cron instances. Refactor chains/keys reload.

app/cmd/rpc/mesh.go

nodiesBlade

Ah. I don't have much insights to add. File loading/configuration can only be done so concisely. As long as it works!

Added FullNode worker dynamic resize on servicers change. Updated servicers reload to only run the modification on maps when there is something new/removed.

jorgecuesta · 2023-04-04T17:05:01Z

@PoktBlade more changes on this, please check the latest commit

…e and better readability of the code without so many casts. Refactor fullNode.Servicer to be a map instead of a slice. Enhance a bit more the logs and bootstrap time information.

Added metrics config support. Refactor code to split in files. Bump pond version to 1.8.3 (patch). Clean up the code.

Update config to handle rpc timeout for different things like chains, client and pocket node calls with a different value.

…able by config file.

Ensure that http response body is read even on errored request to reuse connections.

Enhanced chains reload logs. Enhanced startup logs.

… so many edge cases and possible infinite goroutine spams. Added name property to nodes as optional key, if not set use the hostname of the node url. Added minWorker, maxWorker, maxCapacity to prometheus metrics collectors. Refactor minWorker, maxWorker and maxCapacity option in config. Bump default to a more real world value. Updated docs.

…pact on mesh code, but help node runners to keep internal track.

…/queries on prometheus. Added chains name map so those metrics could contain the chain name you wish. ChainsNameMap could work with a local file or remote endpoint (GET)

… error log. Does not affect the code but is unnecessary.

Added status_type and status_code labels to error metrics. Added internal, notify and chain error metrics.

Added metrics docs and basic geo-mesh grafana dashboard.

app/cmd/rpc/mesh/metrics.go

doc/specs/mesh.md

…ould be leaked on logs. Moved Grafana Dashboard to a file to easily compare/copy from raw github files. Updated mesh.md

Enhance docs based on blade suggestions.

…and the one dispatching to chains.

app/mesh.go

nodiesBlade · 2023-04-24T16:01:26Z

High Level Summary

This update includes changes that allow for faster boot time and better overall performance & maintainability.

Mesh bootstrap time has been greatly reduced to almost 0, which improves overall performance and makes it easier to maintain your key files.
Mesh now supports hot reloading of keys and chains efficiently by duplicating full nodes, which allows for faster updates and better overall performance.
Code has been split into smaller files to make it easier to navigate and modify, which helps with both development and maintenance.
The Relay, Relay Time, and Error metrics now use labels for chain id and name to differentiate between different chains and to help with identifying and categorizing the various metrics.
Misc bug fixes such as connection re-use and invalid sessions which should increase the performance of your mesh and full node.

Breaking Changes

This changes require you update both: Pocket Node & Mesh Node

Added /v1/private/mesh/check endpoint
Moved /v1/mesh/health to /v1/private/mesh/health
Removed /v1/private/mesh/servicer
Removed Prometheus metrics using chain id as part of the metric name

NOTE: You must add the new private endpoints (check & health), or your mesh node will not start.

Configuration Changes

Added node_check_interval to configure the rate at a full Node is health checked.
Added log_chain_request and log_chain_response to avoid unnecessary debug logs.
Added chains_name_map and remote_chains_name_map to enhance exposed metrics with chain name.
Added metrics_moniker to help to identify a mesh node instance on metrics queries.
Added metrics_report_interval to configure the rate at metrics about workers are reported.
Added support for Servicer and Chain http.Transport options:
<servicer|chain>_rpc_max_idle_connections
<servicer|chain>_rpc_max_conns_per_host
<servicer|chain>_rpc_max_idle_conns_per_host
Removed hot_reload_interval in favor of keys_hot_reload_interval and chains_hot_reload_interval as separated values
Rename worker pool options to specify a different set for Servicer or Metrics
Remove rpc_timeout in favor of: client_rpc_timeout and chain_rpc_timeout

Core Changes

New:

New keys structure
Deduplication of nodes (reduce bandwidth usage and faster boot time)
Allow control RPS of mesh to a node (using servicer_max_workers) due to how workers are used now
Chains check. Check against the Pocket node if the chains id on the chain file matches the one on Pocket Node.
Added a Worker for metrics
Chains name maps (local or remote) to enhance metrics
Added endpoint to allow mesh node check against servicer node (health, chains, addresses)

Fix:

Invalid sessions error
HTTP connection drops due to avoiding read response body (decreases CPU utilization and bandwidth)
Handle a few cases where after an error it keeps going without interrupting the flow, invalidating a relay/session.
Update chains using /v1/private/mesh/updatechains endpoint

Rework:

Relay, Relay Time, and Error metrics now use labels for chain id and name
Keys hot reload
Chains hot reload
Moved the authtoken query parameter used to call private method to be passed using the HTTP header Authorization to avoid it being leaked on logs.
Split code in files to help development and readability of it.
Bootstrap time reduce to almost 0

Dependencies:

Added xsync to have better performance that the native for concurrent access.
Added gojsonschema to allow validate keys and chains name map format.
Added golang-set to manage set of values in arrays faster and async safe
Bump version github.com/alitto/pond from v1.8.1 to v1.8.3
Bump version github.com/prometheus/client_golang from v1.11.0 to v1.11.1

Bump Version

New version: RC-0.3.0

Docs

Update mesh.doc
Added missing links to dockerhub
Added Grafana dashboard to consume the new metrics.
Updated rpc-spec.yaml

--

I made some changes to the release notes, could you elaborate more on the rework of:

2. Keys hot reload
3. Chains hot reload

nodiesBlade

LGTM.

I think this is an iterative step architecturally and code-wise to improve the quality of Mesh. This is obviously not its final form and plenty of more code quality & architectural improvements down the road.

amazing work @jorgecuesta , I imagine a future where the entire network is leveraging mesh in some form or way

Initial rework to speed up bootstrap times of mesh client.

491eaca

Added different key format. Refactor connectivity checks. Refactor node/servicer internal structure of mesh to reduce amount of worker/cron instances. Refactor chains/keys reload.

jorgecuesta requested a review from nodiesBlade March 31, 2023 21:25

jorgecuesta self-assigned this Mar 31, 2023

jorgecuesta added the enhancement New feature or request label Mar 31, 2023

nodiesBlade reviewed Apr 3, 2023

View reviewed changes

app/cmd/rpc/mesh.go Outdated Show resolved Hide resolved

nodiesBlade reviewed Apr 3, 2023

View reviewed changes

app/cmd/rpc/mesh.go Outdated Show resolved Hide resolved

nodiesBlade reviewed Apr 3, 2023

View reviewed changes

app/cmd/rpc/mesh.go Outdated Show resolved Hide resolved

nodiesBlade reviewed Apr 3, 2023

View reviewed changes

Fixed minor issues.

777831e

Added FullNode worker dynamic resize on servicers change. Updated servicers reload to only run the modification on maps when there is something new/removed.

jorgecuesta added 2 commits April 4, 2023 15:31

Replace sync.Map with xsync.Map to use Generic in favor of performanc…

4a36b14

…e and better readability of the code without so many casts. Refactor fullNode.Servicer to be a map instead of a slice. Enhance a bit more the logs and bootstrap time information.

Added worker metrics.

6488675

Added metrics config support. Refactor code to split in files. Bump pond version to 1.8.3 (patch). Clean up the code.

jorgecuesta changed the title ~~Initial rework to speed up bootstrap times of mesh client.~~ Boot time speed up & metrics Apr 5, 2023

jorgecuesta added 16 commits April 5, 2023 19:26

Update docs.

c698fb5

Update config to handle rpc timeout for different things like chains, client and pocket node calls with a different value.

Fixed metrics and cyclic dependencies.

dc6410c

Fixed a bug that disallow the ability to reuse http 1.x connections.

f8ffe45

Added support to Chains & Servicer http client options to be configur…

24ec29f

…able by config file.

Updated rpc-spec.yaml.

2923c6b

Ensure that http response body is read even on errored request to reuse connections.

Fixed issue with /v1/private/mesh/updatechains endpoint.

c7e3a24

Enhanced chains reload logs. Enhanced startup logs.

Added specialized metrics for mesh instead of reuse the pocket client.

c3911ce

Updated docs.

e8b4483

Added option to allow debug logs omit request/response payloads.

1828ac4

Added json schema support to additional properties. Those will not im…

b418680

…pact on mesh code, but help node runners to keep internal track.

Rework metrics to handle less metrics with more labels. Enhance group…

27a2a2b

…/queries on prometheus. Added chains name map so those metrics could contain the chain name you wish. ChainsNameMap could work with a local file or remote endpoint (GET)

Set chains_name_map default as empty so avoid to read it and write an…

9d14cf3

… error log. Does not affect the code but is unnecessary.

Added metrics_moniker.

49de5f2

Added status_type and status_code labels to error metrics. Added internal, notify and chain error metrics.

Fixed issue with error metrics

f52c861

Rename status_label to status_code.

57260e7

Added metrics docs and basic geo-mesh grafana dashboard.

nodiesBlade reviewed Apr 21, 2023

View reviewed changes

app/cmd/rpc/mesh/metrics.go Outdated Show resolved Hide resolved

nodiesBlade reviewed Apr 21, 2023

View reviewed changes

app/cmd/rpc/mesh/metrics.go Show resolved Hide resolved

nodiesBlade reviewed Apr 21, 2023

View reviewed changes

doc/specs/mesh.md Outdated Show resolved Hide resolved

nodiesBlade reviewed Apr 21, 2023

View reviewed changes

doc/specs/mesh.md Show resolved Hide resolved

jorgecuesta added 3 commits April 21, 2023 16:12

Moved authtoken to Authorization header to avoid the security token c…

056746b

…ould be leaked on logs. Moved Grafana Dashboard to a file to easily compare/copy from raw github files. Updated mesh.md

Enhance code based on blade suggestions.

e1563e2

Enhance docs based on blade suggestions.

Added golang-set dependency link to docs.

faf01cb

jorgecuesta requested a review from nodiesBlade April 21, 2023 21:13

jorgecuesta added 4 commits April 22, 2023 14:58

Added missing chain request log.

986b5be

Added support to configure more http client that handle mesh request …

37291db

…and the one dispatching to chains.

Fixed issue with log chain requests.

aee7d2b

Allow to remove non-printable characters from chain path

5feb931

nodiesBlade reviewed Apr 24, 2023

View reviewed changes

app/mesh.go Show resolved Hide resolved

jorgecuesta added 2 commits April 24, 2023 13:36

Updated default values.

eb81713

Updated mesh.md

551c5a6

jorgecuesta requested a review from nodiesBlade April 24, 2023 19:29

Updated mesh.md and default value of chain_request_path_cleanup

db473f8

nodiesBlade approved these changes Apr 24, 2023

View reviewed changes

jorgecuesta merged commit def24c3 into geo-mesh Apr 24, 2023

jorgecuesta deleted the initial-enhancement branch April 24, 2023 21:30

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Boot time speed up & metrics #1

Boot time speed up & metrics #1

jorgecuesta commented Mar 31, 2023 •

edited

nodiesBlade left a comment

jorgecuesta commented Apr 4, 2023

nodiesBlade commented Apr 24, 2023 •

edited

nodiesBlade left a comment

Boot time speed up & metrics #1

Boot time speed up & metrics #1

Conversation

jorgecuesta commented Mar 31, 2023 • edited

High-Level Summary

Breaking Changes

Configuration Changes

Core Changes

New:

Fix:

Rework:

Dependencies:

Bump Version

Docs

nodiesBlade left a comment

Choose a reason for hiding this comment

jorgecuesta commented Apr 4, 2023

nodiesBlade commented Apr 24, 2023 • edited

High Level Summary

Breaking Changes

Configuration Changes

Core Changes

New:

Fix:

Rework:

Dependencies:

Bump Version

Docs

nodiesBlade left a comment

Choose a reason for hiding this comment

jorgecuesta commented Mar 31, 2023 •

edited

nodiesBlade commented Apr 24, 2023 •

edited