Merge branch 'master' of https://github.com/DamjanOstrelic/guild-oper…

…ators
cardano-community · Sep 19, 2020 · f1f4329 · f1f4329
2 parents 35d2ad6 + 4d821de
commit f1f4329
Show file tree

Hide file tree

Showing 25 changed files with 1,471 additions and 198 deletions.
diff --git a/docs/Build/dbsync.md b/docs/Build/dbsync.md
@@ -22,6 +22,10 @@ You can use the instructions below to build the cardano-db-sync, same steps can
 ``` bash
 git fetch --tags --all
 git pull
+# Include the cardano-crypto-praos and libsodium components for db-sync
+# On CentOS 7 (GCC 4.8.5) we should also do
+# echo -e "package cryptonite\n  flags: -use_target_attributes" >> cabal.project.local
+echo -e "package cardano-crypto-praos\n flags: -external-libsodium-vrf" > cabal.project.local
 # Replace master with appropriate tag if you'd like to avoid compiling against master
 git checkout master
 $CNODE_HOME/scripts/cabal-build-all.sh
@@ -42,7 +46,7 @@ scripts/postgresql-setup.sh --createdb
 ##### Start cardano-db-sync-tool
 ``` bash
 cd ~/git/cardano-db-sync
-PGPASSFILE=$CNODE_HOME/priv/.pgpass cardano-db-sync-extended --config $CNODE_HOME/files/config.json --genesis-file $CNODE_HOME/files/genesis.json --socket-path $CNODE_HOME/sockets/node0.socket --schema-dir schema/
+PGPASSFILE=$CNODE_HOME/priv/.pgpass cardano-db-sync-extended --config $CNODE_HOME/files/config.json --socket-path $CNODE_HOME/sockets/node0.socket --schema-dir schema/
 ```
 
 You can use same instructions above to repeat and execute `cardano-db-sync` as well, but [cardano-graphql](Build/graphql.md) uses `cardano-db-sync-extended`, so we'll stick to it

diff --git a/docs/Build/graphql.md b/docs/Build/graphql.md
@@ -1,4 +1,4 @@
-!> - An average pool operator may not require cardano-wallet at all. Please verify if it is required for your use as mentioned [here](build.md#components)
+!> We have temporarily disabled updating build documentation for Cardano-GraphQL. The specific component does not follow the process/technology/language (requires npm, yarn) used by other components (cabal/stack), and the value provided by `cardano-graphql` over the (haskell-based) hasura instance has been negligible. Also, an average pool operator may not require cardano-graphql at all, please verify if it is required for your use as mentioned [here](build.md#components). The instructions below are `out of date`.
 
 > Ensure the [Pre-Requisites](basics.md#pre-requisites) are in place before you proceed.
 

diff --git a/docs/Build/node-cli.md b/docs/Build/node-cli.md
@@ -18,11 +18,14 @@ You can use the instructions below to build the cardano-node, same steps can be
 
 ``` bash
 git fetch --tags --all
-# Replace release 1.18.1 with the version/branch/tag you'd like to build
-git checkout tags/1.18.1
+# Replace release 1.19.0 with the version/branch/tag you'd like to build
 git pull
+git checkout 1.19.0
 
 echo -e "package cardano-crypto-praos\n  flags: -external-libsodium-vrf" > cabal.project.local
+# On CentOS 7 (GCC 4.8.5) we should also do
+# echo -e "package cryptonite\n  flags: -use_target_attributes" >> cabal.project.local
+
 $CNODE_HOME/scripts/cabal-build-all.sh
 ```
 
@@ -34,19 +37,69 @@ Execute cardano-cli and cardano-node to verify output as below:
 
 ```bash
 cardano-cli version
-# cardano-cli 1.18.1 - linux-x86_64 - ghc-8.6
-# git rev a4b6dae699fa21dc3c025c8a83d1718475cb3afc
+# cardano-cli 1.19.0 - linux-x86_64 - ghc-8.6
+# git rev 4814003f14340d5a1fc02f3ac15437387a7ada9f
 cardano-node version
-# cardano-node 1.18.1 - linux-x86_64 - ghc-8.6
-# git rev a4b6dae699fa21dc3c025c8a83d1718475cb3afc
+# cardano-node 1.19.0 - linux-x86_64 - ghc-8.6
+# git rev 4814003f14340d5a1fc02f3ac15437387a7ada9f
+```
+
+##### Update port number or pool name for relative paths
+
+Before you go ahead with starting your node, you may want to update values for CNODE_PORT in `$CNODE_HOME/scripts/cnode.sh`. Note that it is imperative for operational relays and pools to ensure that the port mentioned is opened via firewall to the destination your node is supposed to connect from. Update your network/firewall configuration accordingly. Future executions of prereqs.sh will preserve and not overwrite these values.
+
+```bash
+## Static (content that will not be overwritten by prereqs.sh)
+## Begin
+
+POOL_NAME="GUILD"
+CNODE_PORT=6000
+POOL_DIR="$CNODE_HOME/priv/pool/$POOL_NAME"
 ```
 
-##### Start a passive node
+> POOL_NAME is the name of folder that you will use when registering pools and starting node in core mode. This folder would typically contain your `hot.skey`,`vrf.skey` and `op.cert` files required. If the mentioned files are absent, the node will automatically start in a passive mode.
+
+##### Start the node
 
-To start the node in passive mode, you can use the pre-built script below:
+To test starting the node in interactive mode, you can use the pre-built script below (note that the config now uses `SimpleView` so you may not see much output):
 
 ```bash
 cd $CNODE_HOME/scripts
 ./cnode.sh
 ```
 
+##### Run as systemd service
+
+The preferred way to run the node is through a service manager like systemd. This section explains how to setup a systemd service file.
+
+**1. Deploy as a systemd service**  
+Execute the below command to deploy your node as a systemd service (from the respective scripts folder):
+```bash
+cd $CNODE_HOME/scripts
+./deploy-as-systemd.sh
+```
+
+**2. Start the node**  
+Run below commands to enable automatic start of service on startup and start it.
+``` bash
+sudo systemctl start cnode.service
+```
+
+**3. Check status and stop/start commands** 
+Replace `status` with `stop`/`start`/`restart` depending on what action to take.
+``` bash
+sudo systemctl status cnode.service
+```
+
+?> In case you see the node exit unsuccessfully upon checking status, please verify you've followed the transition process correctly as documented below, and that you do not have another instance of node already running. It would help to check your system logs (/var/log/syslog for debian-based and /var/log/messages for redhat/CentOS/Fedora systems) for any errors while starting node.
+
+You can use [gLiveView](Scripts/gliveview.md) to monitor your pool that was started as systemd, if you miss the LiveView functionality.
+
+##### Steps to transition from LiveView in tmux to systemd setup
+
+If you've followed guide from this repo previously and would like to transfer to systemd usage, please checkout the steps below:
+
+1. Stop previous instance of node if already running (eg: in tmux)
+2. Run `prereqs.sh`, but remember to preserve your customisations to cnode.sh, topology.json, env files (you can also compare and update cnode.sh and env files from github repo).
+3. Follow the instructions [above](#run-as-systemd-service) to setup your node as a service and start it using systemctl as directed.
+4. If you need to monitor via interactive terminal as before, use [gLiveView](Scripts/gliveview.md).
diff --git a/docs/Contributors.md b/docs/Contributors.md
diff --git a/docs/Scripts/cntools-blocks.md b/docs/Scripts/cntools-blocks.md
@@ -17,7 +17,7 @@ In this example normal output from the `cntoolsBlockCollector.sh` script is igno
 **1. Create systemd service file**  
 Replace `$USER` with the correct user for your system. Copy & paste all code below to create the service file.
 ``` bash
-sudo bash -c 'cat <<EOF > /etc/systemd/system/cntools-blockcollector.service
+sudo bash -c "cat << 'EOF' > /etc/systemd/system/cntools-blockcollector.service
 [Unit]
 Description=CNTools - Block Collector
 After=network.target
@@ -28,15 +28,15 @@ Restart=on-failure
 RestartSec=10
 User=$USER
 WorkingDirectory=/opt/cardano/cnode/scripts
-ExecStart=/opt/cardano/cnode/scripts/cntoolsBlockCollector.sh
+ExecStart=/bin/bash -l -c 'exec \"\$@\"' _ /opt/cardano/cnode/scripts/cntoolsBlockCollector.sh
 SuccessExitStatus=143
 StandardOutput=null
 StandardError=syslog
 SyslogIdentifier=cntools-blockcollector
 
 [Install]
 WantedBy=multi-user.target
-EOF'
+EOF"
 ```
 
 **2. Reload systemd**  

diff --git a/docs/Scripts/cntools-changelog.md b/docs/Scripts/cntools-changelog.md
@@ -5,19 +5,15 @@ All notable changes to this tool will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
-## [5.3.4] - 2020-08-18
+## [5.4.1] - 2020-09-10
 
-##### Changed
-- Use manual calculation based on slot tip to get KES period
-
-## [5.3.3] - 2020-08-14
+##### Fixed
+- KES Expiry to use KES Period instead of Epoch duration
 
-##### Added
-- Use secure remove (`srm`) when available when deleting files.
 
-## [5.3.2] - 2020-08-05
+## [5.4.0] - 2020-08-23
 
-> If you're coming version 5.2.1 (not required if you're on 5.3.0), We have made quite a few changes to not use ptn0 in our scripts and source github structures (except template files), alongwith other changes listed beneath. Please follow steps below for upgrade:  
+> A non-breaking change have been made to files outside of CNTools. Internal update function is not enough to update all files.  
 > - Execute the below (by default it will set you up against mainnet network), do not overwrite config please:  
 >    `cd "$CNODE_HOME"/scripts`  
 >    `curl -sS -o prereqs.sh https://raw.githubusercontent.com/cardano-community/guild-operators/master/scripts/cnode-helper-scripts/prereqs.sh`  
@@ -27,6 +23,54 @@ and this adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
 =======
 
+##### Added
+- Sanity check before launching a second instance of cnode.sh
+- Doc update to run cnode.sh as a systemd service
+
+##### Removed
+- `Pool >> Delegators` removed.
+  - If/when a better option than dumping and parsing ledger-state dump arise re-adding it will be considered. 
+  - Utilize the community explorers listed at https://cardano-community.github.io/support-faq/#/explorers 
+
+##### Fixed
+- Block Collector script adapted for cardano-node 1.19.0.
+  - Block hash is now truncated in log, issue https://github.com/input-output-hk/cardano-node/issues/1738
+- High cpu usage reported in a few cases when running Block Collector
+  - Depending on log level, parsing and byte64 enc each entry with jq could potentially put high load on weaker systems. Replaced with grep to only parse entries containing specific traces.
+- Docs for creating systemd block collector service file updated to include user env in run command
+
+
+## [5.3.6] - 2020-08-22
+
+##### Fixed
+- cardano-node 1.19.0 introduced an issue that required us to use KES as current - 1 while rotating.
+
+
+## [5.3.5] - 2020-08-20
+
+##### Changed
+- CNTools now uses and works with `cardano-node 1.19.0`, please upgrade if you're not using this version.
+
+##### Fixed
+- A new getPoolID helper function added to extract both hex and bech32 pool ID
+  - Added `--output-format hex` when extracting pool ID in hex format
+  - A new pool.id-bech32 file gets created if cold.vkey is available and decrypted
+- Added error check to see if cardano-cli is in $PATH before continuing.
+
+
+## [5.3.4] - 2020-08-18
+
+##### Changed
+- Use manual calculation based on slot tip to get KES period
+
+## [5.3.3] - 2020-08-14
+
+##### Added
+- Use secure remove (`srm`) when available when deleting files.
+
+
+## [5.3.2] - 2020-08-05
+
 #### Fixed
 - Backup & Restore paths were failing on machines due to alnum class availability on certain interpreters.
 - Rewards were not counted in stake and pledge
@@ -45,6 +89,7 @@ and this adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 - Balance check added to `Funds >> Withdraw` for base address as this is used to pay the withdraw transaction fee
 - Resolve issue with Multi Owner causing an error with new pool registration (error was due to quotes)
 
+
 ## [5.3.0] - 2020-08-03
 ##### Added
 - Ability to select a different pool owner and reward wallet

diff --git a/docs/Scripts/gliveview.md b/docs/Scripts/gliveview.md
@@ -0,0 +1,66 @@
+!> Ensure the [Pre-Requisites](basics.md#pre-requisites) are in place before you proceed.
+
+**Guild LiveView - gLiveView** is a utility to display an equivalent subset of LiveView interface that cardano-node users have grown accustomed to. This is especially useful when moving to a systemd deployment - if you haven't done so already - while looking for a familiar UI to monitor the node status.
+
+The tool is independent from other files and can run as a standalone utility that can be stopped/started without affecting the status of cardano-node.
+
+##### Download & Setup
+
+The tool in itself should only require a single file. If you've used [prereqs.sh](basics.md#pre-requisites), you can skip this part , as this is already set up for you.  
+To get current epoch blocks, [cntoolsBlockCollector.sh](Scripts/cntools-blocks.md) script is needed. This is optional and **Guild LiveView** will function without it.
+
+?> For those who follow guild's [folder structure](basics.md#folder-structure) and do not wish to run prereqs.sh, you can run the below in `$CNODE_HOME/scripts` folder
+
+To download the script:
+```bash
+curl -s -o gLiveView.sh https://raw.githubusercontent.com/cardano-community/guild-operators/master/scripts/cnode-helper-scripts/gLiveView.sh
+chmod 755 gLiveView.sh
+```
+
+##### Startup
+
+For most standard deployments, this should lead you to a stage where you can now start running `./gLiveView.sh` in the folder you downloaded the script (default location for cntools users would be `$CNODE_HOME/scripts`). Note that the script is smart enough to automatically detect when you're running as a Core or Relay and will show fields accordingly.  
+
+The tool can be run in legacy mode with only standard ASCII characters for terminals with trouble displaying the box-drawing characters. Run `./gLiveView.sh -h` to show available command-line parameters or permanently set it directly in script.
+
+A sample output from both core and relay(with peer analysis):
+
+![Core](https://raw.githubusercontent.com/cardano-community/guild-operators/images/gliveview-core.png)
+
+![Relay](https://raw.githubusercontent.com/cardano-community/guild-operators/images/gliveview-relay.png)
+
+##### Description
+
+**Upper main section**  
+Displays live metrics gathered from EKG. Epoch number and progress is live from node while date calculation until epoch boundary is based on offline genesis parameters. Reference tip is also an offline calculation based on genesis values used to compare against the node tip to see how far of the tip(diff value) the node is. With current parameters a slot diff up to 40 from reference tip is considered good but it ussually stay below 30. In/Out peers show how many connections the node have established in and out.
+
+**Core section**  
+If the node is run as a core, identified by the 'forge-about-to-lead' EKG parameter, a second core section is displayed. This section contain current and remaining KES periods as well as a calculated date for the expiration. When getting close to expire date the values will change color. Blocks created by the node since node start is another metric shown in this section. If [cntoolsBlockCollector.sh](Scripts/cntools-blocks.md) is running for the core node a second row with current epoch blocks is displayed.
+
+**Peer analysis**  
+A manual peer analysis can be triggered by key press `p`. A latency test will be done on incoming and outgoing connections to the node. For outgoing connections a normal ICMP ping is done as a first try. If this is blocked, tcptraceroute program is used to do a tcp ping against the cardano-node port of the remote peer. For incoming connections only ICMP ping is used as remote peer port is unknown. It's not uncommon to see many unreachable peers for incoming connections as it's a good security practice to disable ICMP in firewall.
+
+Once run it will display RTT for the peers and group them in the ranges 0-50, 50-100, 100-200, 200<. The analysis filter out multiple connections to the same IP. Unique Peers + Skipped should match the total number seen in top section. The analysis is **NOT** live. Last update timestamp is shown to know when it was last run. Press `p` for update or `h` to hide it.
+
+##### Troubleshooting/Customisations
+
+In case you run into trouble while running the script, you might want to edit `gLiveView.sh` and look at User Variables section shown below. You can override the values if the automatic detection do not provide the right information, but we would appreciate if you could also notify us by raising an issue against github repo:
+
+```bash
+######################################
+# User Variables - Change as desired #
+######################################
+
+#CNODE_HOME="/opt/cardano/cnode"          # Override default CNODE_HOME path
+#CNODE_PORT=6000                          # Override automatic detection of node port
+NODE_NAME="Cardano Node"                  # Change your node's name prefix here, keep at or below 19 characters!
+REFRESH_RATE=2                            # How often (in seconds) to refresh the view
+#CONFIG="${CNODE_HOME}/files/config.json" # Override automatic detection of node config path
+EKG_HOST=127.0.0.1                        # Set node EKG host
+#EKG_PORT=12788                           # Override automatic detection of node EKG port
+#PROTOCOL="Cardano"                       # Default: Combinator network (leave commented if unsure)
+#BLOCK_LOG_DIR="${CNODE_HOME}/db/blocks"  # CNTools Block Collector block dir set in cntools.config, override path if enabled and using non standard path
+legacy_mode=false                         # (true|false) If enabled unicode box-drawing characters will be replaced by standard ASCII characters
+THEME="dark"                              # dark  = suited for terminals with a dark background
+                                          # light = suited for terminals with a bright background
+```
diff --git a/docs/Scripts/itnrewards.md b/docs/Scripts/itnrewards.md
@@ -28,15 +28,6 @@ graph TB
 cd $CNODE_HOME/scripts
 ./itnRewards.sh MyITNWallet ~/jormu/account/priv/owner.sk ~/jormu/account/priv/owner.pk
 ```
-- However, if an extended secret key was used(ed25519e_sk), a recent version (min 1.18.x) of cardano-cli is needed to convert the key.
-  - Follow these instructions to build this version of cardano-cli:
-``` bash
-cd ~/git/cardano-node # Go to folder where you normally build cardano-node
-git checkout release/1.18.x
-git pull
-$CNODE_HOME/scripts/cabal-build-all.sh
-```
-  - Re-run `itnRewards.sh`
 - Start CNTools and verify that correct balance is shown in the wallet reward address
 - Fund base address of wallet with enough funds to pay for withdraw tx fee
 - Use FUNDS >> WITHDRAW to move rewards to base address of wallet

diff --git a/docs/_sidebar.md b/docs/_sidebar.md
@@ -11,6 +11,7 @@
       - [Common tasks](Scripts/cntools-common.md)
       - [Block Collector](Scripts/cntools-blocks.md)
       - [Changelog](Scripts/cntools-changelog.md)
+    * [Guild LiveView](Scripts/gliveview.md)
     * [ITN Metadata Proof](Scripts/itnwitness.md)
     * [ITN Rewards](Scripts/itnrewards.md)
     * [Topology Updater](Scripts/topologyupdater.md)

diff --git a/docs/basics.md b/docs/basics.md
@@ -27,13 +27,11 @@ chmod 755 prereqs.sh
 # -i    Interactive mode (Default: silent mode)
 # -g    Connect to guild network instead of public network (Default: connect to public cardano network)
 # -p    Copy Transitional Praos config as default instead of Combinator networks (Default: copies combinator network)
-
+# -t    Alternate name for top level folder
 # You can use one of the options above, if you'd like to defer from defaults (below).
 # Running without any parameters will run script in silent mode with OS Dependencies, and overwriting existing files.
 
 ./prereqs.sh
-
-## Follow the prompts for execution. To make sure environment variables are available for session you're running, make sure to source bashrc
 . "${HOME}/.bashrc"
 ```