From a5e71cb8f85af8473ee8bd9fb274c67cbd9830fa Mon Sep 17 00:00:00 2001 From: blockchainluffy Date: Mon, 17 Mar 2025 12:49:12 +0530 Subject: [PATCH 1/7] added loki logging configuration for api keypers --- README.md | 5 +++++ docker-compose.loki.yml | 14 ++++++++++++++ example-api.env | 18 ++++++++++++++++++ 3 files changed, 37 insertions(+) create mode 100644 docker-compose.loki.yml diff --git a/README.md b/README.md index c1ef5d4..be7e69d 100644 --- a/README.md +++ b/README.md @@ -82,6 +82,11 @@ You start your Keyper node by running docker compose up -d ``` +### To run with pushing logs to loki +``` +docker compose -f docker-compose.yml -f docker-compose.loki.yml up +``` + ## Backups Once your Keyper is up and running, you should regularly back up the following: diff --git a/docker-compose.loki.yml b/docker-compose.loki.yml new file mode 100644 index 0000000..6b98a4d --- /dev/null +++ b/docker-compose.loki.yml @@ -0,0 +1,14 @@ +services: + chain: + logging: !override + driver: loki + options: + loki-url: ${LOKI_URL} + loki-external-labels: keyper_name=${KEYPER_NAME} + + keyper: + logging: !override + driver: loki + options: + loki-url: ${LOKI_URL} + loki-external-labels: keyper_name=${KEYPER_NAME} \ No newline at end of file diff --git a/example-api.env b/example-api.env index 1756570..5efcc19 100644 --- a/example-api.env +++ b/example-api.env @@ -47,6 +47,24 @@ METRICS_LISTEN_IP=0.0.0.0 # Replace and above with the credentials you receive. # Multiple targets and their credentials can be specified by separating them with a comma. +## Push Logging ## + +# To enable push logs, loki docker driver needs to be installed first by following cmd. +# `docker plugin install grafana/loki-docker-driver:3.3.2-amd64 --alias loki --grant-all-permissions` +# Please change `amd64` to `arm64` based on machine service is hosted on + +#Uncomment to enable push logs +# Target URL(s) for push logs in Loki remote write format +#LOKI_URL=https://:@logs.metrics.shutter.network/insert/loki/api/v1/push + +# The default URL above points to a loki server by Shutter Network (which we +# would appreciate you send logs to), please ask for an access key in the +# Keyper section of the Shutter Network discourse forum. +# Replace and above with the credentials you receive. + +# To run with push logs to loki, the sevice should be run by command: +# `docker compose -f docker-compose.yml -f docker-compose.loki.yml up` + ###################################### ## These values MUST not be touched! From a7b73884e5c3a72a325fef08fdfe45411e47deb4 Mon Sep 17 00:00:00 2001 From: blockchainluffy Date: Mon, 17 Mar 2025 15:36:45 +0530 Subject: [PATCH 2/7] updated readme to include step to install lock docker driver --- README.md | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/README.md b/README.md index be7e69d..92e1c51 100644 --- a/README.md +++ b/README.md @@ -83,6 +83,13 @@ docker compose up -d ``` ### To run with pushing logs to loki + +Firstly, it needs loki docker driver installed, which can be done by following command: +``` +docker plugin install grafana/loki-docker-driver:3.3.2-amd64 --alias loki --grant-all-permissions +``` + +Now, after setting env variable accordingly, to start keyper with push logs enabled, run following command: ``` docker compose -f docker-compose.yml -f docker-compose.loki.yml up ``` From 20cd594d78e8c68574a95b5d328fdb4fcdc1deb7 Mon Sep 17 00:00:00 2001 From: blockchainluffy Date: Mon, 17 Mar 2025 17:16:23 +0530 Subject: [PATCH 3/7] updated 'update' section of readme to include push logs cmd --- README.md | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index 92e1c51..4258245 100644 --- a/README.md +++ b/README.md @@ -91,7 +91,7 @@ docker plugin install grafana/loki-docker-driver:3.3.2-amd64 --alias loki --gran Now, after setting env variable accordingly, to start keyper with push logs enabled, run following command: ``` -docker compose -f docker-compose.yml -f docker-compose.loki.yml up +docker compose -f docker-compose.yml -f docker-compose.loki.yml up -d ``` ## Backups @@ -113,6 +113,11 @@ git checkout shutter-api/ docker compose up -d ``` +If using loki push logs then instead on runing `docker compose up -d`, run the following +``` +docker compose -f docker-compose.yml -f docker-compose.loki.yml up -d +``` + ## Version History ### `shutter-api-keyper/2025.02.01` - initial public release From 5067a4cec2a50a352b1e01de26b5f9104b8348fb Mon Sep 17 00:00:00 2001 From: blockchainluffy Date: Mon, 17 Mar 2025 18:09:48 +0530 Subject: [PATCH 4/7] added readme section for update to setup loki --- README.md | 16 +++++++++++++++- 1 file changed, 15 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index 4258245..00894fb 100644 --- a/README.md +++ b/README.md @@ -71,6 +71,9 @@ git checkout shutter-api - Define the target(s) for the pushgateway with `PUSHGATEWAY_URL` (multiple targets can be separated by commas). The default value points to a pushgateway operated by the Shutter Network team. To gain access please ask for credentials in the Shutter Network Discourse forum. + - Logging (optional): + - To push logs to loki/vmlogs server, use the `docker-compose.loki.yml` file, which overrides logging. + - Define the url for the server to push metrics to, with `LOKI_URL`. Default value points to the logging server operated by Shutter Network Team. > *) **NOTE**: The `example-mainnet.env` file is a template for Gnosis mainnet deployment. If you want to deploy a Keyper for the Chiado testnet instead, use the `example-chiado.env` file. @@ -113,11 +116,22 @@ git checkout shutter-api/ docker compose up -d ``` -If using loki push logs then instead on runing `docker compose up -d`, run the following +If using loki push logs already then instead on runing `docker compose up -d`, run the following ``` docker compose -f docker-compose.yml -f docker-compose.loki.yml up -d ``` +## Update to use push logs to loki for the first time + +1. Update the environment variables ( or .env) to include the following: +`LOKI_URL=https://:@logs.metrics.shutter.network/insert/loki/api/v1/push` + +2. Install docker driver for loki, by the following cmd: +`docker plugin install grafana/loki-docker-driver:3.3.2-amd64 --alias loki --grant-all-permissions` + +3. Run docker compose with additional configuration file: +`docker compose -f docker-compose.yml -f docker-compose.loki.yml up -d` + ## Version History ### `shutter-api-keyper/2025.02.01` - initial public release From eb966b5d907c32fc525506e849d25e6921694872 Mon Sep 17 00:00:00 2001 From: blockchainluffy Date: Mon, 17 Mar 2025 18:14:01 +0530 Subject: [PATCH 5/7] added kpr prefix for keyper name --- docker-compose.loki.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docker-compose.loki.yml b/docker-compose.loki.yml index 6b98a4d..43a4b26 100644 --- a/docker-compose.loki.yml +++ b/docker-compose.loki.yml @@ -11,4 +11,4 @@ services: driver: loki options: loki-url: ${LOKI_URL} - loki-external-labels: keyper_name=${KEYPER_NAME} \ No newline at end of file + loki-external-labels: keyper_name=kpr-${KEYPER_NAME} \ No newline at end of file From 1639df5468a7ec0de4540903a753b7889859fd18 Mon Sep 17 00:00:00 2001 From: ylembachar Date: Tue, 18 Mar 2025 15:00:00 +0100 Subject: [PATCH 6/7] update readme --- README.md | 106 +++++++++++++++++++++------------------- docker-compose.loki.yml | 2 +- 2 files changed, 58 insertions(+), 50 deletions(-) diff --git a/README.md b/README.md index 00894fb..bbd3c34 100644 --- a/README.md +++ b/README.md @@ -6,7 +6,7 @@ This repository contains the docker compose configuration to run a shutter api k ### Chain execution clients -Keypers are required to have access to a the Chain's execution client API, where the shutter registry and keyperset manager contracts are deployed. +Keypers are required to have access to the Chain's execution client API, where the shutter registry and keyperset manager contracts are deployed. ### System requirements @@ -27,14 +27,15 @@ Shutter Network does operate a prometheus compatible monitoring system. See belo ### Software -- You will need a recent version of `docker` and the `docker compose` cli plugin. +- You will need a recent version of `docker` and the `docker compose` cli plugin. - For cloning the repository you will need `git`. The Keyper node is distributed as a docker-compose stack consisting of multiple services. This repository contains all necessary files. -One component is opt-in monitoring, which by default requires opening a port for scrape access to the metrics endpoints, -as well as a public IP address that needs to be shared with the Shutter team if you wish to participate in system-wide +One component is opt-in monitoring, which by default requires opening a port for scrape access to the metrics endpoints, +as well as a public IP address that needs to be shared with the Shutter team if you wish to participate in system-wide monitoring of the nodes. + If you would rather not open up the metrics endpoints we also provide support for push based monitoring. See the `Metrics` section in the .env file for more information. @@ -53,47 +54,62 @@ git checkout shutter-api ``` 2. Copy the `example-api.env`(*) file to `.env` and fill in your information: - - **Required values** - - Your Ethereum account key (hex-encoded *without* `0x` prefix): `SIGNING_KEY` + - **Required values** + - Your Ethereum account key (hex-encoded *without* `0x` prefix): `SIGNING_KEY` - **IMPORTANT**: Please double-check that you are using the key associated with the address that you provided during the Keyper application process. Otherwise, your Keyper node will not be able to join the network. - - A name of your choice for your keyper node: `KEYPER_NAME` + **IMPORTANT**: Please double-check that you are using the key associated with the address that you provided during the Keyper application process. Otherwise, your Keyper node will not be able to join the network. + - A name of your choice for your keyper node: `KEYPER_NAME` - (Please use only letters, numbers, and underscores. No spaces or special characters.) - - Your **public** IP address: `PUBLIC_IP` + (Please use only letters, numbers, and underscores. No spaces or special characters.) + - Your **public** IP address: `PUBLIC_IP` - It is important that this is the address your node is reachable under from the internet since it is used for the P2P network between the nodes. - - A Chain execution JSON RPC API endpoint (WebSocket): `CHAIN_EXECUTION_RPC_WS_URL` - - Metrics (optional): - - To enable metrics, set `METRICS_ENABLED` to `true` (the default) - - Define the interface the metrics ports (`:9200` and `:27660`) should be exposed on with `METRICS_INTERFACE` (defaults to `0.0.0.0`, i.e. the public interface) - - If you rather not publicly expose the metrics and would like to push metrics instead, uncomment the `COMPOSE_PROFILES=pushmetrics` line and set `METRICS_INTERFACE` to `127.0.0.1`. - - Define the target(s) for the pushgateway with `PUSHGATEWAY_URL` (multiple targets can be separated by commas). - - The default value points to a pushgateway operated by the Shutter Network team. To gain access please ask for credentials in the Shutter Network Discourse forum. - - Logging (optional): - - To push logs to loki/vmlogs server, use the `docker-compose.loki.yml` file, which overrides logging. - - Define the url for the server to push metrics to, with `LOKI_URL`. Default value points to the logging server operated by Shutter Network Team. + It is important that this is the address your node is reachable under from the internet since it is used for the P2P network between the nodes. + - A Chain execution JSON RPC API endpoint (WebSocket): `CHAIN_EXECUTION_RPC_WS_URL` + - Metrics (optional): + - To enable metrics, set `METRICS_ENABLED` to `true` (the default) + - Define the interface the metrics ports (`:9200` and `:27660`) should be exposed on with `METRICS_INTERFACE` (defaults to `0.0.0.0`, i.e. the public interface) + - If you rather not publicly expose the metrics and would like to push metrics instead, uncomment the `COMPOSE_PROFILES=pushmetrics` line and set `METRICS_INTERFACE` to `127.0.0.1`. + - Define the target(s) for the pushgateway with `PUSHGATEWAY_URL` (multiple targets can be separated by commas). -> *) **NOTE**: The `example-mainnet.env` file is a template for Gnosis mainnet deployment. If you want to deploy a Keyper for the Chiado testnet instead, use the `example-chiado.env` file. + The default value points to a pushgateway operated by the Shutter Network team. To gain access please ask for credentials in the Shutter Network Discourse forum. + - Logging (optional): + - To push logs to loki/vmlogs server: + - Define the url for the server to push logs to, with `LOKI_URL`. The default value points to the logging server operated by the Shutter Network Team. To gain access please ask for credentials in the Shutter Network Discourse forum. + - Use the `docker-compose.loki.yml` file, which overrides logging, as shown under the running and update sections. -## Running +> **NOTE**: The `example-api.env` file is a template for the Shutter API Keyper deployment on Gnosis Mainnet. -You start your Keyper node by running +## Setting Up Logging -``` -docker compose up -d -``` +To enable log pushing to a **Loki/VmLogs server**, follow these steps: -### To run with pushing logs to loki +1. **Update `.env` to include Loki logs configuration using the credentials provided by the Shutter team:** + ```sh + LOKI_URL=https://:@logs.metrics.shutter.network/insert/loki/api/v1/push + ``` +2. **Install the Loki Docker driver (if not installed):** + ```sh + docker plugin install grafana/loki-docker-driver:3.3.2-amd64 --alias loki --grant-all-permissions + ``` -Firstly, it needs loki docker driver installed, which can be done by following command: -``` -docker plugin install grafana/loki-docker-driver:3.3.2-amd64 --alias loki --grant-all-permissions -``` +> **Note:** For **ARM64 hosts**, add `-arm64` to the image tag. + +For more details, refer to the [Docker driver client | Grafana Loki documentation](https://grafana.com/docs/loki/latest/send-data/docker-driver/). + + +> **IMPORTANT:** If logging is enabled, make sure to use the correct `docker compose` commands as noted below. + +## Running the Keyper Node -Now, after setting env variable accordingly, to start keyper with push logs enabled, run following command: +### **Without Pushing Logs** +To start the Keyper node run: +```sh +docker compose up -d ``` + +### **With Log Pushing Enabled** +> **IMPORTANT:** Logging requires additional configuration steps. Follow the [Logging Setup](#setting-up-logging) section before running the following command: +```sh docker compose -f docker-compose.yml -f docker-compose.loki.yml up -d ``` @@ -107,7 +123,9 @@ Once your Keyper is up and running, you should regularly back up the following: These files will allow you to re-build your Keyper in case of data loss. -## Updating +## **Updating** + +### **Without Pushing Logs** ```shell cd shutter-keyper-deployment @@ -116,22 +134,12 @@ git checkout shutter-api/ docker compose up -d ``` -If using loki push logs already then instead on runing `docker compose up -d`, run the following -``` +### **Updating with Log Pushing Enabled** +> **IMPORTANT:** If logging is enabled, ensure you follow the [Logging Setup](#setting-up-logging) section first. Then, use the correct command: +```sh docker compose -f docker-compose.yml -f docker-compose.loki.yml up -d ``` -## Update to use push logs to loki for the first time - -1. Update the environment variables ( or .env) to include the following: -`LOKI_URL=https://:@logs.metrics.shutter.network/insert/loki/api/v1/push` - -2. Install docker driver for loki, by the following cmd: -`docker plugin install grafana/loki-docker-driver:3.3.2-amd64 --alias loki --grant-all-permissions` - -3. Run docker compose with additional configuration file: -`docker compose -f docker-compose.yml -f docker-compose.loki.yml up -d` - ## Version History ### `shutter-api-keyper/2025.02.01` - initial public release @@ -143,4 +151,4 @@ docker compose -f docker-compose.yml -f docker-compose.loki.yml up -d RegistryContract: 0x7D18359c2f49e4aEBc0df761B1152c31DE044e83 KeysetManagerContract: 0xFaB842d8Ff826E93D31f839AD47218d8c9511cC6 KeyBroadcastContract: 0x474079EFa5D93bb48a1cA27Ebbf46d32E273A28b -``` +``` \ No newline at end of file diff --git a/docker-compose.loki.yml b/docker-compose.loki.yml index 43a4b26..0c76d37 100644 --- a/docker-compose.loki.yml +++ b/docker-compose.loki.yml @@ -4,7 +4,7 @@ services: driver: loki options: loki-url: ${LOKI_URL} - loki-external-labels: keyper_name=${KEYPER_NAME} + loki-external-labels: keyper_name=kpr-${KEYPER_NAME} keyper: logging: !override From f41e9df3f8d7f9764f693b5efe5256a207693285 Mon Sep 17 00:00:00 2001 From: ylembachar <123627984+ylembachar@users.noreply.github.com> Date: Tue, 18 Mar 2025 15:38:43 +0100 Subject: [PATCH 7/7] Apply suggestions from code review Co-authored-by: konradkonrad --- README.md | 4 ++-- example-api.env | 2 +- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index bbd3c34..ebd7787 100644 --- a/README.md +++ b/README.md @@ -6,7 +6,7 @@ This repository contains the docker compose configuration to run a shutter api k ### Chain execution clients -Keypers are required to have access to the Chain's execution client API, where the shutter registry and keyperset manager contracts are deployed. +Keypers are required to have access an execution client's JSON RPC API, of the chain, where the shutter registry and keyperset manager contracts are deployed. ### System requirements @@ -72,7 +72,7 @@ git checkout shutter-api - Define the target(s) for the pushgateway with `PUSHGATEWAY_URL` (multiple targets can be separated by commas). The default value points to a pushgateway operated by the Shutter Network team. To gain access please ask for credentials in the Shutter Network Discourse forum. - - Logging (optional): + - Logging collection (optional): - To push logs to loki/vmlogs server: - Define the url for the server to push logs to, with `LOKI_URL`. The default value points to the logging server operated by the Shutter Network Team. To gain access please ask for credentials in the Shutter Network Discourse forum. - Use the `docker-compose.loki.yml` file, which overrides logging, as shown under the running and update sections. diff --git a/example-api.env b/example-api.env index 5efcc19..f4e60b8 100644 --- a/example-api.env +++ b/example-api.env @@ -63,7 +63,7 @@ METRICS_LISTEN_IP=0.0.0.0 # Replace and above with the credentials you receive. # To run with push logs to loki, the sevice should be run by command: -# `docker compose -f docker-compose.yml -f docker-compose.loki.yml up` +# `docker compose -f docker-compose.yml -f docker-compose.loki.yml up -d` ######################################