diff --git a/docs/node-operator-guide.md b/docs/node-operator-guide.md index 82eb2f864..fe0834eeb 100644 --- a/docs/node-operator-guide.md +++ b/docs/node-operator-guide.md @@ -163,6 +163,7 @@ You have two options to get the `kwild` binary. ```bash sudo mv .build/kwild /usr/local/bin/ + sudo chmod +x /usr/local/bin/kwild ``` For Linux users, apply new docker group: @@ -776,6 +777,136 @@ To view logs since the last system boot: sudo journalctl -u kwild -b ``` +## Resetting Your Node Instance + +Sometimes you may need to reset your node to sync from a specific point or recover from an inconsistent state. This is different from a complete removal - you preserve your configuration but reset the blockchain data. + +### When to Reset Your Node + +- Network fork or critical incident requiring new genesis +- Database corruption or inconsistent state +- Joining the network from a specific block height +- Following network recovery procedures + +### Reset Procedure + +1. **Stop and disable services**: + ```bash + # For Linux + sudo systemctl stop kwild tn-postgres + sudo systemctl disable kwild tn-postgres + sudo systemctl daemon-reload + + # For macOS + launchctl stop com.trufnetwork.kwild + launchctl stop com.trufnetwork.tn-postgres + ``` + +2. **Backup node identity and configuration** (to preserve them): + ```bash + # Create backup directory + mkdir -p ~/truf-node-operator/backup-reset + + # Backup nodekey to preserve node identity + cp ~/truf-node-operator/my-node-config/nodekey.json ~/truf-node-operator/backup-reset/ + + # Backup config.toml if you want to preserve custom settings (optional) + cp ~/truf-node-operator/my-node-config/config.toml ~/truf-node-operator/backup-reset/ + ``` + +3. **Remove PostgreSQL data and node configuration**: + ```bash + # Remove Docker resources + docker stop tn-postgres + docker rm tn-postgres + docker volume rm tn-pgdata + + # Remove entire node configuration directory + rm -rf ~/truf-node-operator/my-node-config + ``` + +4. **Regenerate node configuration**: + ```bash + # Pull latest changes (for network forks) + cd ~/truf-node-operator + git pull + + # Reinitialize configuration + kwild setup init \ + --genesis ./configs/network/v2/genesis.json \ + --root ./my-node-config \ + --p2p.bootnodes "4e0b5c952be7f26698dc1898ff3696ac30e990f25891aeaf88b0285eab4663e1#ed25519@node-1.mainnet.truf.network:26656,0c830b69790eaa09315826403c2008edc65b5c7132be9d4b7b4da825c2a166ae#ed25519@node-2.mainnet.truf.network:26656" \ + --state-sync.enable \ + --state-sync.trusted-providers "0c830b69790eaa09315826403c2008edc65b5c7132be9d4b7b4da825c2a166ae#ed25519@node-2.mainnet.truf.network:26656" \ + --rpc.private + ``` + +5. **Update kwild binary** *(optional but recommended)*: + ```bash + # Option 1: Download latest release + curl -L "https://github.com/trufnetwork/node/releases/latest/download/tn__linux_amd64.tar.gz" \ + -o kwild.tgz + tar -xzf kwild.tgz kwild + sudo mv kwild /usr/local/bin/kwild + sudo chmod +x /usr/local/bin/kwild + + # Option 2: Build from source + cd ~/node # assuming you cloned the node repository here, please adjust if different + git pull && task build + sudo mv .build/kwild /usr/local/bin/kwild + sudo chmod +x /usr/local/bin/kwild + + # Verify version + kwild version + ``` + +6. **Restore node identity and custom configuration** *(optional)*: + + > **Note**: Skip this step if you want to start with a completely fresh node identity and default configuration. Only restore if you need to preserve your existing validator status or custom settings. + + ```bash + # Restore nodekey to preserve node identity + cp ~/truf-node-operator/backup-reset/nodekey.json ~/truf-node-operator/my-node-config/ + + # Restore config.toml if you want to keep custom settings (optional) + cp ~/truf-node-operator/backup-reset/config.toml ~/truf-node-operator/my-node-config/ + ``` + +7. **Recreate PostgreSQL**: + ```bash + docker run -d -p 127.0.0.1:5432:5432 --name tn-postgres \ + -e "POSTGRES_HOST_AUTH_METHOD=trust" \ + -v tn-pgdata:/var/lib/postgresql/data \ + --shm-size=1gb \ + kwildb/postgres:latest + ``` + +8. **Re-enable and start services**: + ```bash + # For Linux + sudo systemctl enable tn-postgres kwild + sudo systemctl start tn-postgres + sudo systemctl start kwild + + # For macOS + launchctl start com.trufnetwork.tn-postgres + launchctl start com.trufnetwork.kwild + ``` + +9. **Monitor synchronization**: + ```bash + kwild admin status + ``` + Wait for `syncing: false` and ensure your `best_block_height` is advancing. + +### Important Notes + +- This process preserves your node identity (nodekey.json) and configuration files +- Your validator status (if applicable) is preserved +- The node will resync from the genesis block specified in the new genesis.json +- State sync (if enabled in your config) allows fast resynchronization +- For critical network incidents, always check the [Node Upgrade Guide](node-upgrade-guide.md#breaking-changes--migrations) for specific instructions + ## Clean Removal > **Warning**: The following steps will completely remove your node setup, including all data and configuration. This is irreversible and should only be done if: @@ -800,7 +931,8 @@ docker stop tn-postgres docker rm tn-postgres docker volume rm tn-pgdata -# Remove node configuration +# Remove node configuration (optional - only if you want to regenerate nodekey) +# This will create a new node identity on next init rm -rf $HOME/truf-node-operator/my-node-config ``` @@ -822,6 +954,8 @@ docker stop tn-postgres docker rm tn-postgres docker volume rm tn-pgdata +# Remove node configuration (optional - only if you want to regenerate nodekey) +# This will create a new node identity on next init rm -rf $HOME/truf-node-operator/my-node-config ``` diff --git a/docs/node-upgrade-guide.md b/docs/node-upgrade-guide.md index b17d2f872..fe4644ec8 100644 --- a/docs/node-upgrade-guide.md +++ b/docs/node-upgrade-guide.md @@ -98,11 +98,31 @@ If the release notes specify a new official Postgres image (e.g. `kwildb/postgre --- -## Breaking Changes & Migrations (TBD) +## Breaking Changes & Migrations When a new version introduces consensus-breaking changes, a simple binary replacement is **not sufficient**. These scenarios require a **network migration** (offline or zero-downtime) and coordination with the core team. +### Network Fork Recovery + +When the network requires a fork due to consensus changes or critical issues, you'll need to reset your node with a new genesis file. + +**Prerequisites:** +- Check official announcements for specific migration instructions +- Ensure you have the latest `truf-node-operator` repository (`git pull`) + +**Recovery Steps:** + +Follow the **[Resetting Your Node Instance](node-operator-guide.md#resetting-your-node-instance)** procedure in the Node Operator Guide. The process preserves your node identity and configuration while syncing with the new network state. + +**What's Preserved During Network Forks:** +- Node identity (nodekey) and validator status +- All custom configuration settings +- Chain ID continuity +- Historical data up to the fork point + +### Other Migration Types + See: * [Migrations overview](/docs/node/migrations) * [Offline migrations guide](/docs/node/migrations/offline-migrations)