Fast Stellar Core Catch Up
Starting a full stellar core validator from scratch takes a while, as the node needs to download and process a lot of data during the "Catch up" phase.
The catching up phase usually takes more than a month.
This script helps to make the process significantly shorter, as it allows to perform the catch up of multiple ledgers blocks in parallel.
On a powerful server the full catch up using this Python script can be done in less than a day.
Another implementation of the parallel catch up idea is a shell script by SatoshiPay.
Due to the parallel nature of the process, the sync time can be significantly decreased with bigger instance type and more CPU and RAM.
So it's recommended to scale up a server instance until the catch up process is complete. On the instance similar to c5.12xlarge the full catch up can be completed within 24 hours (please feel free to share your specs and results).
The process also requires a lot of disk space, so we recommend to connect a temporary SSD disk around 2TB in size. This temporary disk will be used to store copies of ledger chunks, merging them together in the database.
This script is also designed to publish history archives to a cloud archive like S3.
- Make sure stellar core is not running:
sudo service stellar-core stop
- Tweak default settings of postgresql for high performance:
sudo nano /etc/postgresql/10/main/postgresql.conf
max_locks_per_transaction 5000 max_pred_locks_per_transaction 5000 max_connections 1500 shared_buffers 10GB # (allocate about 25% of total RAM)
- Grant superuser permissions to the user:
sudo -i -u postgres psql -c "alter user <db_user> with superuser;"
- Connect a large disk to the server (in early 2020 about 2TB would be enough for history and databases)
lsblk # format & create partition if required sudo mkdir /mnt/storage sudo mount /dev/sdb1 /mnt/storage
- Move postgresql database to the temporary disk:
sudo service postgresql stop sudo mv sudo mv /var/lib/postgresql/10/main /mnt/storage/ sudo ln -s /mnt/storage/main /var/lib/postgresql/10/main sudo service postgresql start
- Clone repository:
git clone https://github.com/Lobstrco/stellar-core-parallel-catchup-py.git /mnt/storage/core-parallel-catchup sudo chown stellar /mnt/storage/core-parallel-catchup
- Login as stellar user:
sudo -i -u stellar cd src
- Install python script requirements from provided requirements.txt
pip install -r requirements.txt
- Generate your secret key - it will be needed for next step:
- Initialize folders structure, daemonize workers monitor and merge process. This will take a while:
export DB_HOST=localhost export DB_PORT=5432 export DB_USER=<db_user> export DB_PASSWORD=<db_password> export NODE_SECRET_KEY=<secret_key> python cli.py initialize nohup python cli.py monitor > monitor.log & nohup python cli.py merge > merge.log &
- Great! You're almost done. Now rename database:
ALTER DATABASE "catchup-stellar-result" RENAME TO "stellar-core";
- Move folders to their real destinations:
sudo mv result/data/buckets /var/lib/stellar/ sudo service postgresql stop sudo rm /var/lib/postgresql/10/main sudo mv /mnt/storage/main /var/lib/postgresql/10/ sudo service postgresql start
- You're ready to start stellar core node
sudo service stellar-core start
- (Optional) If you're not going to use stellar horizon, remove HORIZON cursor from core database to re-enable db maintenance
- Publish history archives to the cloud using stellar-archivist:
sudo apt-get install stellar-archivist sudo service stellar-core stop sudo -i -u stellar /bin/bash export $(cat /etc/stellar/stellar-core | xargs) && stellar-core new-hist s3 --conf /etc/stellar/stellar-core.cfg exit sudo service stellar-core start sudo -i -u stellar /bin/bash export $(cat /etc/stellar/stellar-core | xargs) && nohup stellar-archivist repair file:///mnt/storage/parallel-catchup/result/vs/ s3://<aws_s3_history_bucket_name> --s3region=$AWS_DEFAULT_REGION > archivist-repair.out & exit
- It's all done! Now you can detach the temporary disk from the instance, and scale it down to the usual size.