Merge branch 'master' of github.com:cardano-community/guild-operators

.github/workflows/node-latest.yml

@@ -11,7 +11,7 @@ jobs:
           token: ${{ secrets.REPO_SCOPED_TOKEN }}
       - name: Fetch node release version
         run: |
-          curl -sL https://github.com/input-output-hk/cardano-node/releases/latest | grep "tag" | head -1 | sed -n -e 's/^.*tag\///p' | cut -d "\"" -f 1 > files/docker/node/release-versions/cardano-node-latest.txt
+          curl -sL https://api.github.com/repos/input-output-hk/cardano-node/releases/latest | grep tag_name | awk '{print $2}' | cut -d\" -f 2 > files/docker/node/release-versions/cardano-node-latest.txt
       - name: Assigns release version
         run: |
           VERSION=$(cat ./files/docker/node/release-versions/cardano-node-latest.txt)

docs/Appendix/postgres.md

@@ -23,7 +23,7 @@ elif [ -z "${OS_ID##*rhel*}" ]; then
   #CentOS/RHEL/Fedora
   sudo yum install -y postgresql-server postgresql-server-devel postgresql-contrib postgresql-devel libpq-devel
   sudo postgresql-setup initdb
-  sudo systemctl start postgresql
+  sudo systemctl restart postgresql
   sudo systemctl enable postgresql
 else
   echo "We have no automated procedures for this ${DISTRO} system"
@@ -63,3 +63,29 @@ psql postgres
 # 
 # postgres=#
 ```
+
+#### Tuning your instance
+
+Before you start populating your DB instance using dbsync data, now might be a good time to put some thought on to baseline configuration of your postgres instance by editing `/etc/postgresql/13/main/postgresql.conf`.
+Typically, you might find a lot of common standard practices parameters available in tuning guides. For our consideration, it would be nice to start with some baselines - for which we will use inputs from example [here](https://pgtune.leopard.in.ua/#/).
+You might want to fill in some sample information as per below to fill in the form:
+
+| Option         | Value |
+|----------------|-------|
+| DB Version     | 13    |
+| OS Type        | Linux |
+| DB Type        | Online Transaction Processing System|
+| Total RAM      | 32 (or as per your server) |
+| Number of CPUs | 8 (or as per your server)  |
+| Number of Connections | 200 |
+| Data Storage   | HDD Storage |
+
+In addition to above, due to the nature of usage by dbsync (restart of instance does a rollback to start of epoch), and data retention on blockchain - we're not affected by loss of volatile information upon a restart of instance. Thus, we can relax some of the data retention and protection against corruption related settings, as those are IOPs/CPU Load Average impacts that the instance does not need to spend. We'd recommend setting 3 of those below in your `/etc/postgresql/13/main/postgresql.conf`:
+
+| Parameter          | Value   |
+|--------------------|---------|
+| wal_level          | minimal |
+| max_wal_senders    | 0       |
+| synchronous_commit | off     |
+
+Once your changes are done, ensure to restart postgres service using `sudo systemctl restart postgresql`.

docs/Build/dbsync.md

@@ -45,16 +45,38 @@ scripts/postgresql-setup.sh --createdb
 # Password:
 # Password:
 # All good!
-## Verify you can see "All good!" as above
 ```
 
-#### Start Synching
+Verify you can see "All good!" as above!
+
+#### Create Symlink to schema folder
+
+DBSync instance requires the schema files from the git repository to be present and available to the dbsync instance. You can either clone the `~/git/cardano-db-sync/schema` folder OR create a symlink to the folder and make it available to the startup command we will be using. We will use the latter in sample below:
+
 ``` bash
-cd ~/git/cardano-db-sync
-cardano-db-sync-extended --config $CNODE_HOME/files/dbsync.json --socket-path $CNODE_HOME/sockets/node0.socket --state-dir $CNODE_HOME/guild-db/ledger-state --schema-dir schema/
+ln -s ~/git/cardano-db-sync/schema $CNODE_HOME/guild-db/schema
+```
+
+#### Test running dbsync manually at terminal
+
+In order to verify that you can run dbsync, before making a start - you'd want to ensure that you can run it interactively once. To do so, try the commands below:
+
+``` bash
+cd $CNODE_HOME/scripts
+export PGPASSFILE=$CNODE_HOME/priv/.pgpass
+./dbsync.sh
+```
+
+You can monitor logs if needed via parallel session using `tail -10f $CNODE_HOME/logs/dbsync.json`. If there are no error, you would want to press Ctrl-C to stop the dbsync.sh execution and deploy it as a systemd service. To do so, use the commands below (the creation of file is done using `sudo` permissions, but you can always deploy it manually):
+
+``` bash
+cd $CNODE_HOME/scripts
+./dbsync.sh -d
+# Deploying cnode-dbsync.service as systemd service..
+# cnode-dbsync.service deployed successfully!!
 ```
 
-You can use same instructions above to repeat and execute `cardano-db-sync` as well, but `cardano-db-sync-extended` provides additional views/useful data ready to be consumed by query layer. Note that synching entire blockchain from scratch could take a few hours.
+Now to start dbsync instance, you can run `sudo systemctl start cnode-dbsync`
 
 ### Validation
 

docs/Build/graphql.md

@@ -1,4 +1,4 @@
-!> We have stopped maintaining documentation for Cardano-GraphQL, and prefer use of [PostgREST](../Build/pgrest.md) instead. The specific component does not follow the process/technology/language (requires npm, yarn) used by other components (cabal/stack), and the value provided by `cardano-graphql` over the (haskell-based) hasura instance has been negligible. Also, an average pool operator may not require cardano-graphql at all, please verify if it is required for your use as mentioned [here](../build.md#components). The instructions below are `out of date`.
+!> We have stopped maintaining documentation for Cardano-GraphQL, and prefer use of [PostgREST](../Build/grest.md) instead. The specific component does not follow the process/technology/language (requires npm, yarn) used by other components (cabal/stack), and the value provided by `cardano-graphql` over the (haskell-based) hasura instance has been negligible. Also, an average pool operator may not require cardano-graphql at all, please verify if it is required for your use as mentioned [here](../build.md#components). The instructions below are `out of date`.
 
 > Ensure the [Pre-Requisites](../basics.md#pre-requisites) are in place before you proceed.
 

docs/Build/grest.md

@@ -0,0 +1,133 @@
+!!! important
+
+    - An average pool operator may not require this component at all. Please verify if it is required for your use as mentioned [here](../build.md#components)
+    - Ensure that you have setup [DBSync](../Build/dbsync.md) and that it is in sync atleast to Mary fork before you proceed. *IF* you're participating in Koios services, ensure that you're using [latest dbsync release](https://github.com/input-output-hk/cardano-db-sync/releases/latest)
+
+### What is gRest
+
+gRest is an open source implementation of a `query layer built over dbsync using PostgREST and HAProxy`. The package is built as part of [Koios](https://www.koios.rest) team's efforts to unite community individual stream of work together and give back a more aligned structure to query dbsync and adopt standardisation to queries utilising open-source tooling as well as collaboration. In addition to these, there are also accessibility features to deploy rules for failover, do healthchecks, set up priorities, have ability to prevent DDoS attacks, provide timeouts, report tips for analysis over a longer period, etc - which can prove to be really useful when performing any analysis for instances.
+
+### Components
+
+1. [PostgREST](https://postgrest.org/en/latest):  
+    An RPC JSON interface for any PostgreSQL database (in our case, database served via `cardano-db-sync`) to provide a RESTful Web Service. The endpoints of PostgREST in itself are essentially the table/functions defined in elected schema via grest config file. You can read more about advanced query syntax using PostgREST API [here](https://postgrest.org/en/latest/api.html), but we will provide a simpler view using examples towards the end of the page. It is an easy alternative - with almost no overhead as it directly serves the underlying database as an API, as compared to `Cardano GraphQL` component (which may often have lags). Some of the other advantages of PostgREST over graphql based projects are also performance, being stateless, 0 overhead, support for JWT / native Postgres DB authentication against the Rest Interface as well.
+
+2. [HAProxy](http://cbonte.github.io/haproxy-dconv/2.4/configuration.html):  
+    An easy gateway proxy that automatically provides failover/basic DDoS protection, specify rules management for load balancing, setup multiple frontend/backends, provide easy means to have TLS enabled for public facing instances, etc. You may alter the settings for proxy layer as per your SecOps preferences. This component is optional (eg: if you prefer to expose your PostgREST server itself, you can do so using similar steps below).
+
+### Setup gRest services {: id="setup"}
+
+To start with you'd want to ensure your current shell session has access to Postgres credentials, continuing from examples from the above mentioned [Sample Postgres deployment guide](../Appendix/postgres.md).
+
+``` bash
+cd $CNODE_HOME/priv
+PGPASSFILE=$CNODE_HOME/priv/.pgpass
+psql cexplorer
+```
+
+Ensure that you can connect to your Postgres DB fine using above (quit from psql once validated using `\q`). As part of `prereqs.sh` execution, you'd find setup-grest.sh file made available in `${CNODE_HOME}/scripts` folder, which will help you automate installation of PostgREST, HAProxy as well as brings in latest queries/functions provided via Koios to your instances.
+
+!!! warning "Warning"
+    As of now, gRest services are in alpha stage - while can be utilised, please remember there may be breaking changes and every collaborator is expected to work with the team to keep their instances up-to-date using alpha branch.
+
+Familiarise with the usage options for the setup script , the syntax can be viewed as below:
+
+``` bash
+cd "${CNODE_HOME}"/scripts
+./setup-grest.sh -h
+#
+# Usage: setup-grest.sh [-f] [-i [p][r][m][c][d]] [-u] [-b <branch>]
+# 
+# Install and setup haproxy, PostgREST, polling services and create systemd services for haproxy, postgREST and dbsync
+# 
+# -f    Force overwrite of all files including normally saved user config sections
+# -i    Set-up Components individually. If this option is not specified, components will only be installed if found missing (eg: -i prcd)
+#     p    Install/Update PostgREST binaries by downloading latest release from github.
+#     r    (Re-)Install Reverse Proxy Monitoring Layer (haproxy) binaries and config
+#     m    Install/Update Monitoring agent scripts
+#     c    Overwrite haproxy, postgREST configs
+#     d    Overwrite systemd definitions
+# -u    Skip update check for setup script itself
+# -q    Run all DB Queries to update on postgres (includes creating grest schema, and re-creating views/genesis table/functions/triggers and setting up cron jobs)
+# -b    Use alternate branch of scripts to download - only recommended for testing/development (Default: master)
+#
+```
+
+To run the setup with typical options, you may want to use:
+``` bash
+./setup-grest.sh -f -q
+```
+
+Similarly - if instead, you'd like to re-install all components as well as force overwrite all configs and queries, you may run:
+``` bash
+./setup-grest.sh -f -i prmcd -q
+```
+
+Another example could be to preserve your config, but only update queries using an alternate branch (eg: `alpha`). To do so, you may run:
+``` bash
+./setup-grest.sh -q -b alpha
+```
+
+Please ensure to follow the on-screen instructions, if any (for example restarting deployed services, or updating configs to specify correct target postgres URLs/enable TLS/add peers etc in `${CNODE_HOME}/priv/grest.conf` and `${CNODE_HOME}/files/haproxy.cfg`).
+
+The default ports used will make haproxy instance available at port 8053 or 8453 if TLS is enabled (you might want to enable firewall rule to open this port to services you would like to access). If you want to prevent unauthenticated access to grest schema, uncomment the jwt-secret and specify a custom `secret-token`.
+
+### Enable TLS on HAProxy {: id="tls"}
+
+In order to enable SSL on your haproxy, all you need to do is edit the file `${CNODE_HOME}/files/haproxy.cfg` and update the *frontend app* section to disable normal bind and enable ssl bind.
+
+!!! info ""
+
+    - server.pem referred below should be a chain containing server TLS certificate, signing certificates (intermediate/root) and private key.
+    - Make sure to replace the hostname to the CNAME/SAN used to create your TLS certificate.
+
+If you're not familiar with how to configure TLS OR would not like to buy one, you can find tips on how to create a TLS certificate for free via LetsEncrypt using tutorials [here](https://letsencrypt.org/getting-started/). Once you do have a TLS Certificate generated, you need to chain the private key and full chain cert together in a file - `/etc/ssl/server.pem` in example below:
+
+```
+frontend app
+  bind 0.0.0.0:8053
+  http-request replace-value Host (.*):8053 servername.koios.rest:8453
+  redirect scheme https code 301 if !{ ssl_fc }
+  
+frontend app-secured
+  bind :8453 ssl crt /etc/ssl/server.pem no-sslv3
+  http-request track-sc0 src table flood_lmt_rate
+  http-request deny deny_status 429 if { sc_http_req_rate(0) gt 100 }
+  default_backend grest_core
+```
+Restart haproxy service for changes to take effect.
+
+### Performance Tuning Considerations
+
+While the defaults on your system should not cause you any trouble, note that haproxy relies on ephemeral ports available on your system to be able to redirect frontend requests to backend servers. The four important configuration settings in your `/etc/sysctl.conf` files would be:
+
+```
+net.ipv4.ip_local_port_range="1024 65534"
+net.core.somaxconn=65534
+net.ipv4.tcp_rmem=4096 16060 64060
+net.ipv4.tcp_wmem=4096 16384 262144
+```
+Again, defaults should be fine for minimal usage, you do not need to tinker with above unless you expect a very high amount of load on your frontend. We intentionally do not set these for you as system tuning activities should be done very carefully, understanding what you're modifying and their impacts - these would be part of your SysOps + SecOps practices.
+
+### Validation
+
+With the setup, you also have a `checkstatus.sh` script, which will query the Postgres DB instance via haproxy (coming through postgREST), and only show an instance up if the latest block in your DB instance is within 180 seconds.
+
+!!! warning "Important"
+    While currently the HAProxy config only checks for tip, there will be test cases added for validating each endpoint in future. If you'd like to participate in joining to the elastic cluster via Koios, please raise a PR request by editing topology files in [this folder](https://github.com/cardano-community/guild-operators/tree/alpha/files/grest) to do so!!
+
+If you were using `guild` network, you could do a couple of very basic sanity checks as per below:
+
+1. To query active stake for pool `pool1z2ry6kxywgvdxv26g06mdywynvs7jj3uemnxv273mr5esukljsr` in epoch `122`, we can execute the below:
+``` bash
+curl -d _pool_bech32=pool1z2ry6kxywgvdxv26g06mdywynvs7jj3uemnxv273mr5esukljsr -d _epoch_no=122 -s http://localhost:8053/rpc/pool_active_stake
+## {"active_stake_sum" : 19409732875}
+```
+
+2. To check latest owner key(s) for a given pool `pool1z2ry6kxywgvdxv26g06mdywynvs7jj3uemnxv273mr5esukljsr`, you can execute the below:
+``` bash
+curl -d _pool_bech32=pool1z2ry6kxywgvdxv26g06mdywynvs7jj3uemnxv273mr5esukljsr -s http://localhost:8050/rpc/pool_owners
+## [{"owner" : "stake_test1upx5p04dn3t6dvhfh27744su35vvasgaaq565jdxwlxfq5sdjwksw"}, {"owner" : "stake_test1uqak99cgtrtpean8wqwp7d9taaqkt9gkkxga05m5azcg27chnzfry"}]
+```
+
+You may want to explore what all endpoints come out of the box, and test them out, to do so - refer to [API documentation](https://api.koios.rest) for OpenAPI3 documentation. Each endpoint has a pre-filled example for mainnet and connects by default to primary Koios endpoint, allowing you to test endpoints and if needed - grab the `curl` commands to start testing yourself against your local or remote instances.