Monitoring & Snapshot tool for Ark Core
This tool is based on the monitoring and snapshot capabilities of Noah, but adjusted to work with v2 of Ark.
The reason for not going with a core plugin but a bash script instead, is because the script can still notify you if core goes down for whatever reason, while the plugin would go down with it. Nonetheless, it is possible that a core plugin with a subset of
hermod features might be created sometime in the future.
git clone https://github.com/ItsANameToo/hermod.git cd ~/hermod bash hermod.sh install bash hermod.sh config # See Configuration for more details bash hermod.sh start
That's it! Hermod is now monitoring your logs
bash hermod.sh install, there will be a
.hermod file available where you can change the configuration to your preference.
At a minimum, you should go over and fill in some of the following configuration:
# Notification settings, by default these should be fine, but you can decide to increase / reduce it to your liking monitor_lines=10 # if you have debug logs enabled, 20 lines is better monitor_interval=3 monitor_sleep_after_notif=10 monitor_lines_halted=5 # Delegate info to properly filter the log messages. Both should be filled in with your delegate details. delegate_username='' delegate_public_key='' # Snapshot settings, enabling the snapshot module will automatically take snapshots and append blocks to existing snapshots. # Note: only enable automatic snapshots on relay nodes! snapshots_enable=true snapshots_rounds=5 # Every round is ~6 minutes # Core settings, set the path to core (usually $HOME/core or $HOME/ark-core), and the network (devnet or mainnet). core_path=$HOME/ark-core core_network=mainnet core_log_path="$HOME/.local/state/ark-core/$core_network" # How you would like to be notified. Don't forget to change this line to one (or more) ways in which you would like to be updated. notification_drivers=(log slack discord)
When rounds are saved on core, hermod will take snapshots, or append to the most recent one. It will keep the 5 most recent snapshots and older snapshots are deleted. Make sure that you have enough storage space on your node!
With the introduction of core 2.6, snapshots can only be taken when core is shut down, which
hermod will handle for you. Due to the nature of having to shut down core, it is STRONGLY advised to only enable automatic snapshots on relay nodes and it will be disabled by default. Of course it is always possible to manually create a snapshot with
hermod snapshot, but please remember that this will temporarily shutdown core too!
If you need share a snapshot with someone,
hermod facilitates it. By running
hermod share, the command will archive the most recent snapshot, and host it on an URL that you can share with others. To be able to use this command, you must have at least one snapshot (you can run
hermod snapshot to take one), and port
8080 must be open.
There are a couple of services that you can use to get notified by the script. These are currently the following:
- Discord, with the use of webhooks
- Slack, with the use of webhooks
- Postfix (email)
- Twillio -
twilio_messagewill send you SMS, and
twilio_callwill call you when there's a problem with your node
You can check whether you have confired your notifications correctly by running
bash hermod.sh test notify MESSAGE. This will send
MESSAGE to each of the notification drivers you've configured in the config.
Usage: hermod.sh [options] options: help Show this help. version Show the installed version. start Start the hermod process. stop Stop the hermod process. restart Restart the hermod process. reload Reload the hermod process. delete Delete the hermod process. monitor Temporarily monitor the log. install Setup hermod interactively. update Update the hermod installation. config Configure the hermod installation. log Show the hermod log. test [method] [params] Test the specified method. alias Create a bash alias for hermod. snapshot Take a new snapshot, or append to the most recent one. rollback Roll back to the most recent snapshot. share Hosts the most recent snapshot on a temporary URL.
When upgrading from v2.x.x to v2.1.0, you should double check the following:
- in earlier
hermodversions, automatic snapshots are enabled by default. Please disable this on forging nodes, as core 2.6 will require a full shutdown to be able to take a snapshot.
- you will have to set a
snapshots_roundsvariable in your
.hermodto indicate after how many rounds a snapshot should be taken
When upgrading from v1.x.x to v2.x.x, you should make the following changes:
- In your
.hermodconfiguration file, you should add
2(if you run
ark-forgeras separate processes, or
1if you run the combined
ark-coreprocess). You can see an example of this in the
Make sure to restart
hermod to have the changes take effect!
MIT © ItsANameToo / vmunich / Brian Faust