netmon

A basic monitoring service for WAVES blockchain networks (mainnet, stagenet, testnet). The service stores all data in RAM and has no persistent storage, meaning that after a restart it loses all accumulated statistics and assessments.

Command line parameters

Below are the parameters that can be passed to the executable at startup to change its default behavior. Environment variables can also be used to override the default parameters.

Basic command line parameters

Basic settings:

• --log-level — logging level. Supported levels: DEV, DEBUG, INFO, WARN, ERROR, FATAL. Default: INFO. Environment variable: LOG_LEVEL.
• --bind-addr — IP address and port on which the service will run. Default: 0.0.0.0:2048. Environment variable: BIND_ADDR.
• --network-scheme — WAVES network byte to be monitored. Supported networks: W (mainnet), T (testnet), S (stagenet), E (custom). Default: W. Environment variable: NETWORK_SCHEME.
• --stats-url — URL from which node statistics will be collected. Default: https://waves-nodes-get-height.wavesnodes.com/. Environment variable: STATS_URL.
• --stats-poll-interval — interval at which network node statistics are collected and updated. Default: 1m. Environment variable: STATS_POLL_INTERVAL.
• --max-poll-response-size — maximum response size in bytes when fetching statistics. Default: 131072. Environment variable: MAX_POLL_RESPONSE_SIZE.
• --stats-history-size — the number of most recent stored statistics snapshots. Must be > 0. Default: 10. Environment variable: STATS_HISTORY_SIZE.
• --network-errors-streak — number of consecutive errors after which the network is considered degraded. Default: 5. Environment variable: NETWORK_ERRORS_STREAK.
• --initial-mon-state — monitoring state at startup. Possible values: active, frozen_operates_stable, frozen_degraded. Default: active. Environment variable: INITIAL_MON_STATE.
• --http-auth-header — HTTP header in which the token for access to private URLs will be checked. Default: X-Waves-Monitor-Auth. Environment variable: HTTP_AUTH_HEADER.
• --http-auth-token — access token for private URLs. REQUIRED parameter. No default value. Environment variable: HTTP_AUTH_TOKEN.

Monitoring criteria

Below are the options (criteria) that directly affect error monitoring. Network state is evaluated based on a series of consecutive errors generated by the service. When an error is generated, the consecutive error counter increases; however, if a statistics collection cycle succeeds without generating an error, the counter resets.

Down nodes criterion

• --criterion-down-total-part — threshold at which an error is generated. Calculated as the ratio of unavailable nodes to all monitored nodes. Value range: from 0.0 (exclusive) to 1.0 (inclusive). Default: 0.3. Environment variable: CRITERION_DOWN_TOTAL_PART.

Nodes height criterion

• --criterion-height-diff — allowed height difference between network nodes before an error is generated. Default: 10 blocks. Environment variable: CRITERION_HEIGHT_DIFF.
• --criterion-height-require-min-nodes-on-same-height — required number of nodes at the same height. Default: 2 nodes. Environment variable: CRITERION_HEIGHT_REQUIRE_MIN_NODES_ON_SAME_HEIGHT.

Statehash criterion

A group here refers to a group of nodes at the same height that have identical state hashes.

• --criterion-statehash-min-groups-on-same-height — minimum number of node groups with different state hashes at a single height. Default: 2 groups. Environment variable: CRITERION_STATEHASH_MIN_GROUPS_ON_SAME_HEIGHT.
• --criterion-statehash-min-valuable-groups — minimum number of valuable groups at a single height. Default: 2 groups. Environment variable: CRITERION_STATEHASH_MIN_VALUABLE_GROUPS.
• --criterion-statehash-min-nodes-in-valuable-group — minimum number of nodes in a group for it to be considered valuable. Default: 2 nodes. Environment variable: CRITERION_STATEHASH_MIN_NODES_IN_VALUABLE_GROUP.
• --criterion-statehash-require-min-nodes-on-same-height — required number of nodes at the same height. Default: 4 nodes. Environment variable: CRITERION_STATEHASH_REQUIRE_MIN_NODES_ON_SAME_HEIGHT.

HTTP API

Public URLs

1. GET /health — returns the monitored network byte, the current network state, the maximum height, and the timestamp of the last statistics update. If height cannot be obtained from at least one monitored node, -1 is returned instead of height.

   • Possible HTTP response codes:

     • 200 OK
     • 405 Method Not Allowed
     • 500 Internal Server Error

   • Response examples:

     • {"updated":"2021-12-02T19:35:24.144994Z","network":"W","status":true,"height":2882018} — network is healthy
     • {"updated":"2021-12-02T19:35:24.144994Z","network":"W","status":false,"height":2882018} — network is degraded, but at least one node is available
     • {"updated":"2021-12-02T19:35:24.144994Z","network":"W","status":false,"height":-1} — network is degraded and all nodes are unavailable

   • Example request: curl http://localhost:2048/health

Private URLs

1. POST /state — sets the monitoring state. If the new state differs from the old one, switching states resets the consecutive error counter.

   • Possible HTTP response codes:

     • 200 OK
     • 400 Bad Request
     • 403 Forbidden
     • 405 Method Not Allowed
     • 500 Internal Server Error

   • Response body: none

   • Possible request bodies:

     • {"state":"active"} — switch monitoring to normal (active) mode
     • {"state":"frozen_operates_stable"} — frozen mode: GET /health always returns {"status":true}
     • {"state":"frozen_degraded"} — frozen mode: GET /health always returns {"status":false}

   • Example request: curl -X POST -H "Content-Type: application/json" -d '{"state":"active"}' http://localhost:2048/state

Build

Requirements: the machine must have Make, the Go compiler, and the Go standard library installed. Build the executable using make build. After building, the executable will be located in the build directory.

Docker

The project is built with CGO_ENABLED=0, so the service binary does not require dynamic libraries and contains all necessary dependencies.

The monitoring service inside the container runs under the user netmon. The time zone inside the container is set to Etc/UTC.

Monitoring arguments can be passed into the container by appending them to the docker run command. Example:

$ docker build -t netmon -f Dockerfile . # build container
$ docker run --rm -ti --name netmon -p 2048:2048 netmon --http-auth-token="your-token" # run and pass CLI arguments