Skip to content
This repository has been archived by the owner on May 1, 2023. It is now read-only.


Switch branches/tags

Name already in use

A tag already exists with the provided branch name. Many Git commands accept both tag and branch names, so creating this branch may cause unexpected behavior. Are you sure you want to create this branch?

Latest commit


Git stats


Failed to load latest commit information.
Latest commit message
Commit time

Polkadot snapshot generator

This is a self-container Kuberneted cluster to generate Polkadot snapshots, deployable with just one command.

When running your own polkadot validation operations, it is essential to be able to quickly recover from a disaster. Snapshots help you to get a node fully synchronized sooner.

These snapshots are available at Polkashots, but you may want to deploy the entire snapshot generation engine yourself, so your disaster recovery plan does not depend on any third-party services.


  • runs a Kubernetes node
  • leverages the Kubernetes Persistent Volume Snapshot feature: takes a snapshot of the storage at filesystem level before generating the polkadot snapshot
  • runs the snapshot generation job on a configurable cron schedule
  • generates markdown metadata and a Jekyll static webpage describing the snapshots
  • deploys snapshot and static webpage to Firebase

How to deploy


  1. Download and install Terraform

  2. Download, install, and configure the Google Cloud SDK.

  3. Install the kubernetes CLI (aka kubectl)

  4. Download and install the Firebase CLI.


Using your Google account, active your Google Cloud access.

Login to gcloud using gcloud auth login

Set up Google Default Application Credentials by issuing the command:

gcloud auth application-default login

Populate terraform variables

All custom values unique to your deployment are set as terraform variables. You must populate these variables manually before deploying the setup.

A simple way is to populate a file called terraform.tfvars.

First, go to terraform folder:

cd terraform

Below is a list of variables you must set in terraform.tfvars using the variable=value format.


Name Description Type Default Required
billing_account Billing account ID. string "" no
chain chain (polkadot, kusama) string "polkadot" no
cluster_ca_certificate kubernetes cluster certificate string "" no
cluster_name name of the kubernetes cluster string "" no
database the database backend to use string "RocksDb" no
firebase_project name of the firebase project for the snapshot website string "" no
firebase_subdomain name of the firebase subdomain to generate proper urls in examples string "dot" no
firebase_token firebase token (secret) to publish to the polkashots website string "" no
kubernetes_access_token name of the kubernetes endpoint string "" no
kubernetes_endpoint name of the kubernetes endpoint string "" no
kubernetes_name_prefix kubernetes name prefix to prepend to all resources (should be short, like dot) string "dot" no
kubernetes_namespace kubernetes namespace to deploy the resource into string "tzshots" no
kubernetes_pool_name when kubernetes cluster has several node pools, specify which ones to deploy the baking setup into. only effective when deploying on an external cluster with terraform_no_cluster_create string "blockchain-pool" no
node_locations Zones in which to create the nodes list
org_id Organization ID. string "" no
polkadot_version The polkadot container software version string "latest-release" no
project Project ID where Terraform is authenticated to run to create additional projects. If provided, Terraform will great the GKE and Polkadot cluster inside this project. If not given, Terraform will generate a new project. string "" no
region Region in which to create the cluster, or region where the cluster exists. string "us-central1" no
snapshot_cron_schedule the schedule on which to generate snapshots, in cron format string "7 13 * * *" no
snapshot_url url of the snapshot of type full to download string "" no
terraform_service_account_credentials path to terraform service account file, created following the instructions in string "~/.config/gcloud/application_default_credentials.json" no

Google Cloud project

A default Google Cloud project should have been created when you activated your account. Verify its ID with gcloud projects list. You may also create a dedicated project to deploy the cluster.

Set the project id in the project terraform variable.

Polkadot version (optional)

Set the polkadot_version variable to the desired branch of the polkadot software release.

Database format

Set the database parameter to either ParityDb or RocksDb.

Snapshot url (optional)

Yes, the snapshot engine also can take a snapshot to sync faster. Pass the snapshot URL as snapshot_url parameter.

Firebase args

Note: I tried to make the firebase project and the token automatically with terraform, but there was a bug. See terraform/

For now, the terraform project must be created separately, and a CI token must be created with the firebase login:ci command.

Then pass the project id as firebase_project and the token as firebase_token.

Also pass firebase_subdomain so the snapshot webpage example points to the correct URL.

Snapshot cron schedule (optional)

Fill the snapshot_cron_schedule variable if you want to alter how often or when the snapshot generation cronjob runs.

Full example of terraform.tfvars

More variables, can be configured, see terraform/ file for details. Below is a full example:

project = "My Project"
firebase_project = "polkashots-dot"
firebase_subdomain = "dot"
firebase_token = "1//0xxxxxxxxxxxx"
snapshot_url = ""
snapshot_cron_schedule = "43 2,14 * * *"
database = "ParityDb"
node_storage_size = "60"


  1. Run the following:
terraform init
terraform plan -out plan.out
terraform apply plan.out

This will take time as it will:

  • create a Google Cloud project
  • create a Kubernetes cluster
  • build the necessary containers
  • push the kubernetes configuration, which will spin up a node a start synchronization

In case of error, run the plan and apply steps again:

terraform plan -out plan.out
terraform apply plan.out

Connect to the cluster

Once the command returns, you can verify that the pods are up by running:

kubectl get pods

How to resize

Resizing the storage of the node used for snapshotting has two components: the volume itself needs resizing, and the snapshot-engine cronjob env variable must change.

If these 2 actions are not taken together, the snapshot generation will stop due to size mismatch.

  1. Change the node_storage_size value. This value is passed to the VolumeSnapshot kubernetes object.
  2. Change the persistentvolumeclaim size manually with kubectl edit or k9s.

Wrapping up

To delete everything and terminate all the charges, issue the command:

terraform destroy

Alternatively, go to the GCP console and delete the project.