Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
136 changes: 135 additions & 1 deletion docs/node-operator-guide.md
Original file line number Diff line number Diff line change
Expand Up @@ -163,6 +163,7 @@ You have two options to get the `kwild` binary.

```bash
sudo mv .build/kwild /usr/local/bin/
sudo chmod +x /usr/local/bin/kwild
```

For Linux users, apply new docker group:
Expand Down Expand Up @@ -776,6 +777,136 @@ To view logs since the last system boot:
sudo journalctl -u kwild -b
```

## Resetting Your Node Instance

Sometimes you may need to reset your node to sync from a specific point or recover from an inconsistent state. This is different from a complete removal - you preserve your configuration but reset the blockchain data.

### When to Reset Your Node

- Network fork or critical incident requiring new genesis
- Database corruption or inconsistent state
- Joining the network from a specific block height
- Following network recovery procedures

### Reset Procedure

1. **Stop and disable services**:
```bash
# For Linux
sudo systemctl stop kwild tn-postgres
sudo systemctl disable kwild tn-postgres
sudo systemctl daemon-reload

# For macOS
launchctl stop com.trufnetwork.kwild
launchctl stop com.trufnetwork.tn-postgres
```

2. **Backup node identity and configuration** (to preserve them):
```bash
# Create backup directory
mkdir -p ~/truf-node-operator/backup-reset

# Backup nodekey to preserve node identity
cp ~/truf-node-operator/my-node-config/nodekey.json ~/truf-node-operator/backup-reset/

# Backup config.toml if you want to preserve custom settings (optional)
cp ~/truf-node-operator/my-node-config/config.toml ~/truf-node-operator/backup-reset/
```

3. **Remove PostgreSQL data and node configuration**:
```bash
# Remove Docker resources
docker stop tn-postgres
docker rm tn-postgres
docker volume rm tn-pgdata

# Remove entire node configuration directory
rm -rf ~/truf-node-operator/my-node-config
```

4. **Regenerate node configuration**:
```bash
# Pull latest changes (for network forks)
cd ~/truf-node-operator
git pull

# Reinitialize configuration
kwild setup init \
--genesis ./configs/network/v2/genesis.json \
--root ./my-node-config \
--p2p.bootnodes "4e0b5c952be7f26698dc1898ff3696ac30e990f25891aeaf88b0285eab4663e1#ed25519@node-1.mainnet.truf.network:26656,0c830b69790eaa09315826403c2008edc65b5c7132be9d4b7b4da825c2a166ae#ed25519@node-2.mainnet.truf.network:26656" \
--state-sync.enable \
--state-sync.trusted-providers "0c830b69790eaa09315826403c2008edc65b5c7132be9d4b7b4da825c2a166ae#ed25519@node-2.mainnet.truf.network:26656" \
--rpc.private
```

5. **Update kwild binary** *(optional but recommended)*:
```bash
# Option 1: Download latest release
curl -L "https://github.com/trufnetwork/node/releases/latest/download/tn_<VERSION>_linux_amd64.tar.gz" \
-o kwild.tgz
tar -xzf kwild.tgz kwild
sudo mv kwild /usr/local/bin/kwild
sudo chmod +x /usr/local/bin/kwild

# Option 2: Build from source
cd ~/node # assuming you cloned the node repository here, please adjust if different
git pull && task build
sudo mv .build/kwild /usr/local/bin/kwild
sudo chmod +x /usr/local/bin/kwild

# Verify version
kwild version
```

6. **Restore node identity and custom configuration** *(optional)*:

> **Note**: Skip this step if you want to start with a completely fresh node identity and default configuration. Only restore if you need to preserve your existing validator status or custom settings.

```bash
# Restore nodekey to preserve node identity
cp ~/truf-node-operator/backup-reset/nodekey.json ~/truf-node-operator/my-node-config/

# Restore config.toml if you want to keep custom settings (optional)
cp ~/truf-node-operator/backup-reset/config.toml ~/truf-node-operator/my-node-config/
```

7. **Recreate PostgreSQL**:
```bash
docker run -d -p 127.0.0.1:5432:5432 --name tn-postgres \
-e "POSTGRES_HOST_AUTH_METHOD=trust" \
-v tn-pgdata:/var/lib/postgresql/data \
--shm-size=1gb \
kwildb/postgres:latest
```

8. **Re-enable and start services**:
```bash
# For Linux
sudo systemctl enable tn-postgres kwild
sudo systemctl start tn-postgres
sudo systemctl start kwild

# For macOS
launchctl start com.trufnetwork.tn-postgres
launchctl start com.trufnetwork.kwild
```

9. **Monitor synchronization**:
```bash
kwild admin status
```
Wait for `syncing: false` and ensure your `best_block_height` is advancing.

### Important Notes

- This process preserves your node identity (nodekey.json) and configuration files
- Your validator status (if applicable) is preserved
- The node will resync from the genesis block specified in the new genesis.json
- State sync (if enabled in your config) allows fast resynchronization
- For critical network incidents, always check the [Node Upgrade Guide](node-upgrade-guide.md#breaking-changes--migrations) for specific instructions

## Clean Removal

> **Warning**: The following steps will completely remove your node setup, including all data and configuration. This is irreversible and should only be done if:
Expand All @@ -800,7 +931,8 @@ docker stop tn-postgres
docker rm tn-postgres
docker volume rm tn-pgdata

# Remove node configuration
# Remove node configuration (optional - only if you want to regenerate nodekey)
# This will create a new node identity on next init
rm -rf $HOME/truf-node-operator/my-node-config
```

Expand All @@ -822,6 +954,8 @@ docker stop tn-postgres
docker rm tn-postgres
docker volume rm tn-pgdata

# Remove node configuration (optional - only if you want to regenerate nodekey)
# This will create a new node identity on next init
rm -rf $HOME/truf-node-operator/my-node-config
```

Expand Down
22 changes: 21 additions & 1 deletion docs/node-upgrade-guide.md
Original file line number Diff line number Diff line change
Expand Up @@ -98,11 +98,31 @@ If the release notes specify a new official Postgres image (e.g. `kwildb/postgre

---

## Breaking Changes & Migrations (TBD)
## Breaking Changes & Migrations

When a new version introduces consensus-breaking changes, a simple binary replacement is **not sufficient**.
These scenarios require a **network migration** (offline or zero-downtime) and coordination with the core team.

### Network Fork Recovery

When the network requires a fork due to consensus changes or critical issues, you'll need to reset your node with a new genesis file.

**Prerequisites:**
- Check official announcements for specific migration instructions
- Ensure you have the latest `truf-node-operator` repository (`git pull`)

**Recovery Steps:**

Follow the **[Resetting Your Node Instance](node-operator-guide.md#resetting-your-node-instance)** procedure in the Node Operator Guide. The process preserves your node identity and configuration while syncing with the new network state.

**What's Preserved During Network Forks:**
- Node identity (nodekey) and validator status
- All custom configuration settings
- Chain ID continuity
- Historical data up to the fork point

### Other Migration Types

See:
* [Migrations overview](/docs/node/migrations)
* [Offline migrations guide](/docs/node/migrations/offline-migrations)
Expand Down
Loading