From bfd786ebbcd4f943a3400d8da45d004a4b3ddc13 Mon Sep 17 00:00:00 2001 From: Michael Buntarman Date: Tue, 22 Jul 2025 22:27:07 +0700 Subject: [PATCH 1/4] docs: view reset procedure guide --- docs/node-operator-guide.md | 88 ++++++++++++++++++++++++++++++++++++- docs/node-upgrade-guide.md | 22 +++++++++- 2 files changed, 108 insertions(+), 2 deletions(-) diff --git a/docs/node-operator-guide.md b/docs/node-operator-guide.md index 82eb2f864..59b35ee44 100644 --- a/docs/node-operator-guide.md +++ b/docs/node-operator-guide.md @@ -776,6 +776,89 @@ To view logs since the last system boot: sudo journalctl -u kwild -b ``` +## Resetting Your Node Instance + +Sometimes you may need to reset your node to sync from a specific point or recover from an inconsistent state. This is different from a complete removal - you preserve your configuration but reset the blockchain data. + +### When to Reset Your Node + +- Network fork or critical incident requiring new genesis +- Database corruption or inconsistent state +- Joining the network from a specific block height +- Following network recovery procedures + +### Reset Procedure + +1. **Stop and disable services**: + ```bash + # For Linux + sudo systemctl stop kwild tn-postgres + sudo systemctl disable kwild tn-postgres + sudo systemctl daemon-reload + + # For macOS + launchctl stop com.trufnetwork.kwild + launchctl stop com.trufnetwork.tn-postgres + ``` + +2. **Backup your node configuration** (optional but recommended): + ```bash + cp -r ~/truf-node-operator/my-node-config ~/truf-node-operator/my-node-config.backup.$(date +%Y%m%d_%H%M%S) + ``` + +3. **Remove PostgreSQL data**: + ```bash + # Remove Docker resources + docker stop tn-postgres + docker rm tn-postgres + docker volume rm tn-pgdata + ``` + +4. **Update genesis file** (required for network forks): + ```bash + # Pull latest changes + cd ~/truf-node-operator + git pull + + # Copy the new genesis file to your node config (only if there's a new genesis) + cp ./configs/network/v2/genesis.json ./my-node-config/genesis.json + ``` + +5. **Recreate PostgreSQL**: + ```bash + docker run -d -p 127.0.0.1:5432:5432 --name tn-postgres \ + -e "POSTGRES_HOST_AUTH_METHOD=trust" \ + -v tn-pgdata:/var/lib/postgresql/data \ + --shm-size=1gb \ + kwildb/postgres:latest + ``` + +6. **Re-enable and start services**: + ```bash + # For Linux + sudo systemctl enable tn-postgres kwild + sudo systemctl start tn-postgres + sudo systemctl start kwild + + # For macOS + launchctl start com.trufnetwork.tn-postgres + launchctl start com.trufnetwork.kwild + ``` + +7. **Monitor synchronization**: + ```bash + kwild admin status + ``` + Wait for `syncing: false` and ensure your `best_block_height` is advancing. + +### Important Notes + +- This process preserves your node identity (nodekey.json) and configuration files +- Your validator status (if applicable) is preserved +- The node will resync from the genesis block specified in the new genesis.json +- State sync (if enabled in your config) allows fast resynchronization +- For critical network incidents, always check the [Node Upgrade Guide](node-upgrade-guide.md#breaking-changes--migrations) for specific instructions + ## Clean Removal > **Warning**: The following steps will completely remove your node setup, including all data and configuration. This is irreversible and should only be done if: @@ -800,7 +883,8 @@ docker stop tn-postgres docker rm tn-postgres docker volume rm tn-pgdata -# Remove node configuration +# Remove node configuration (optional - only if you want to regenerate nodekey) +# This will create a new node identity on next init rm -rf $HOME/truf-node-operator/my-node-config ``` @@ -822,6 +906,8 @@ docker stop tn-postgres docker rm tn-postgres docker volume rm tn-pgdata +# Remove node configuration (optional - only if you want to regenerate nodekey) +# This will create a new node identity on next init rm -rf $HOME/truf-node-operator/my-node-config ``` diff --git a/docs/node-upgrade-guide.md b/docs/node-upgrade-guide.md index b17d2f872..fe4644ec8 100644 --- a/docs/node-upgrade-guide.md +++ b/docs/node-upgrade-guide.md @@ -98,11 +98,31 @@ If the release notes specify a new official Postgres image (e.g. `kwildb/postgre --- -## Breaking Changes & Migrations (TBD) +## Breaking Changes & Migrations When a new version introduces consensus-breaking changes, a simple binary replacement is **not sufficient**. These scenarios require a **network migration** (offline or zero-downtime) and coordination with the core team. +### Network Fork Recovery + +When the network requires a fork due to consensus changes or critical issues, you'll need to reset your node with a new genesis file. + +**Prerequisites:** +- Check official announcements for specific migration instructions +- Ensure you have the latest `truf-node-operator` repository (`git pull`) + +**Recovery Steps:** + +Follow the **[Resetting Your Node Instance](node-operator-guide.md#resetting-your-node-instance)** procedure in the Node Operator Guide. The process preserves your node identity and configuration while syncing with the new network state. + +**What's Preserved During Network Forks:** +- Node identity (nodekey) and validator status +- All custom configuration settings +- Chain ID continuity +- Historical data up to the fork point + +### Other Migration Types + See: * [Migrations overview](/docs/node/migrations) * [Offline migrations guide](/docs/node/migrations/offline-migrations) From e3bf522cc89baaa8cad877d666262548b3e2691c Mon Sep 17 00:00:00 2001 From: Michael Buntarman Date: Tue, 22 Jul 2025 23:52:15 +0700 Subject: [PATCH 2/4] docs: guide to preserve node identity and config --- docs/node-operator-guide.md | 45 ++++++++++++++++++++++++++++--------- 1 file changed, 35 insertions(+), 10 deletions(-) diff --git a/docs/node-operator-guide.md b/docs/node-operator-guide.md index 59b35ee44..9aaca137a 100644 --- a/docs/node-operator-guide.md +++ b/docs/node-operator-guide.md @@ -801,30 +801,55 @@ Sometimes you may need to reset your node to sync from a specific point or recov launchctl stop com.trufnetwork.tn-postgres ``` -2. **Backup your node configuration** (optional but recommended): +2. **Backup node identity and configuration** (to preserve them): ```bash - cp -r ~/truf-node-operator/my-node-config ~/truf-node-operator/my-node-config.backup.$(date +%Y%m%d_%H%M%S) + # Create backup directory + mkdir -p ~/truf-node-operator/backup.$(date +%Y%m%d_%H%M%S) + + # Backup nodekey to preserve node identity + cp ~/truf-node-operator/my-node-config/nodekey.json ~/truf-node-operator/backup.$(date +%Y%m%d_%H%M%S)/ + + # Backup config.toml if you want to preserve custom settings (optional) + cp ~/truf-node-operator/my-node-config/config.toml ~/truf-node-operator/backup.$(date +%Y%m%d_%H%M%S)/ ``` -3. **Remove PostgreSQL data**: +3. **Remove PostgreSQL data and node configuration**: ```bash # Remove Docker resources docker stop tn-postgres docker rm tn-postgres docker volume rm tn-pgdata + + # Remove entire node configuration directory + rm -rf ~/truf-node-operator/my-node-config ``` -4. **Update genesis file** (required for network forks): +4. **Regenerate node configuration**: ```bash - # Pull latest changes + # Pull latest changes (for network forks) cd ~/truf-node-operator git pull - # Copy the new genesis file to your node config (only if there's a new genesis) - cp ./configs/network/v2/genesis.json ./my-node-config/genesis.json + # Reinitialize configuration + kwild setup init \ + --genesis ./configs/network/v2/genesis.json \ + --root ./my-node-config \ + --p2p.bootnodes "4e0b5c952be7f26698dc1898ff3696ac30e990f25891aeaf88b0285eab4663e1#ed25519@node-1.mainnet.truf.network:26656,0c830b69790eaa09315826403c2008edc65b5c7132be9d4b7b4da825c2a166ae#ed25519@node-2.mainnet.truf.network:26656" \ + --state-sync.enable \ + --state-sync.trusted-providers "0c830b69790eaa09315826403c2008edc65b5c7132be9d4b7b4da825c2a166ae#ed25519@node-2.mainnet.truf.network:26656" \ + --rpc.private + ``` + +5. **Restore node identity and custom configuration**: + ```bash + # Restore nodekey to preserve node identity + cp ~/truf-node-operator/backup.*/nodekey.json ~/truf-node-operator/my-node-config/ + + # Restore config.toml if you want to keep custom settings (optional) + cp ~/truf-node-operator/backup.*/config.toml ~/truf-node-operator/my-node-config/ ``` -5. **Recreate PostgreSQL**: +6. **Recreate PostgreSQL**: ```bash docker run -d -p 127.0.0.1:5432:5432 --name tn-postgres \ -e "POSTGRES_HOST_AUTH_METHOD=trust" \ @@ -833,7 +858,7 @@ Sometimes you may need to reset your node to sync from a specific point or recov kwildb/postgres:latest ``` -6. **Re-enable and start services**: +7. **Re-enable and start services**: ```bash # For Linux sudo systemctl enable tn-postgres kwild @@ -845,7 +870,7 @@ Sometimes you may need to reset your node to sync from a specific point or recov launchctl start com.trufnetwork.kwild ``` -7. **Monitor synchronization**: +8. **Monitor synchronization**: ```bash kwild admin status ``` From cf44c2bfe53ca6bc3de20295749ea16a1557e60f Mon Sep 17 00:00:00 2001 From: Michael Buntarman Date: Wed, 23 Jul 2025 11:30:23 +0700 Subject: [PATCH 3/4] docs: add update kwild binary step to node reset guide --- docs/node-operator-guide.md | 38 ++++++++++++++++++++++++++++--------- 1 file changed, 29 insertions(+), 9 deletions(-) diff --git a/docs/node-operator-guide.md b/docs/node-operator-guide.md index 9aaca137a..8601d473f 100644 --- a/docs/node-operator-guide.md +++ b/docs/node-operator-guide.md @@ -163,6 +163,7 @@ You have two options to get the `kwild` binary. ```bash sudo mv .build/kwild /usr/local/bin/ + sudo chmod +x /usr/local/bin/kwild ``` For Linux users, apply new docker group: @@ -804,13 +805,13 @@ Sometimes you may need to reset your node to sync from a specific point or recov 2. **Backup node identity and configuration** (to preserve them): ```bash # Create backup directory - mkdir -p ~/truf-node-operator/backup.$(date +%Y%m%d_%H%M%S) + mkdir -p ~/truf-node-operator/backup-reset # Backup nodekey to preserve node identity - cp ~/truf-node-operator/my-node-config/nodekey.json ~/truf-node-operator/backup.$(date +%Y%m%d_%H%M%S)/ + cp ~/truf-node-operator/my-node-config/nodekey.json ~/truf-node-operator/backup-reset/ # Backup config.toml if you want to preserve custom settings (optional) - cp ~/truf-node-operator/my-node-config/config.toml ~/truf-node-operator/backup.$(date +%Y%m%d_%H%M%S)/ + cp ~/truf-node-operator/my-node-config/config.toml ~/truf-node-operator/backup-reset/ ``` 3. **Remove PostgreSQL data and node configuration**: @@ -840,16 +841,35 @@ Sometimes you may need to reset your node to sync from a specific point or recov --rpc.private ``` -5. **Restore node identity and custom configuration**: +5. **Update kwild binary** *(optional but recommended)*: + ```bash + # Option 1: Download latest release + curl -L "https://github.com/trufnetwork/node/releases/latest/download/tn__linux_amd64.tar.gz" \ + -o kwild.tgz + tar -xzf kwild.tgz kwild + sudo mv kwild /usr/local/bin/kwild + sudo chmod +x /usr/local/bin/kwild + + # Option 2: Build from source + cd ~/node # assuming you cloned the node repository here, please adjust if different + git pull && task build + sudo mv .build/kwild /usr/local/bin/kwild + sudo chmod +x /usr/local/bin/kwild + + # Verify version + kwild version + ``` + +6. **Restore node identity and custom configuration**: ```bash # Restore nodekey to preserve node identity - cp ~/truf-node-operator/backup.*/nodekey.json ~/truf-node-operator/my-node-config/ + cp ~/truf-node-operator/backup-reset/nodekey.json ~/truf-node-operator/my-node-config/ # Restore config.toml if you want to keep custom settings (optional) - cp ~/truf-node-operator/backup.*/config.toml ~/truf-node-operator/my-node-config/ + cp ~/truf-node-operator/backup-reset/config.toml ~/truf-node-operator/my-node-config/ ``` -6. **Recreate PostgreSQL**: +7. **Recreate PostgreSQL**: ```bash docker run -d -p 127.0.0.1:5432:5432 --name tn-postgres \ -e "POSTGRES_HOST_AUTH_METHOD=trust" \ @@ -858,7 +878,7 @@ Sometimes you may need to reset your node to sync from a specific point or recov kwildb/postgres:latest ``` -7. **Re-enable and start services**: +8. **Re-enable and start services**: ```bash # For Linux sudo systemctl enable tn-postgres kwild @@ -870,7 +890,7 @@ Sometimes you may need to reset your node to sync from a specific point or recov launchctl start com.trufnetwork.kwild ``` -8. **Monitor synchronization**: +9. **Monitor synchronization**: ```bash kwild admin status ``` From 900c3fd29020e236be8d4e22ce69503099ce9688 Mon Sep 17 00:00:00 2001 From: Michael Buntarman Date: Wed, 23 Jul 2025 11:48:20 +0700 Subject: [PATCH 4/4] docs: add notes about preserve node identity --- docs/node-operator-guide.md | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/docs/node-operator-guide.md b/docs/node-operator-guide.md index 8601d473f..fe0834eeb 100644 --- a/docs/node-operator-guide.md +++ b/docs/node-operator-guide.md @@ -860,7 +860,10 @@ Sometimes you may need to reset your node to sync from a specific point or recov kwild version ``` -6. **Restore node identity and custom configuration**: +6. **Restore node identity and custom configuration** *(optional)*: + + > **Note**: Skip this step if you want to start with a completely fresh node identity and default configuration. Only restore if you need to preserve your existing validator status or custom settings. + ```bash # Restore nodekey to preserve node identity cp ~/truf-node-operator/backup-reset/nodekey.json ~/truf-node-operator/my-node-config/