Cirrus lets you manage your Snowflake node IDs.
High-level clouds, called cirrus clouds, can reach heights of 20,000 feet (6,000 meters) and are typically thin. They do not produce rain and often indicate fair weather. They are usually made up of ice.
Not Production Ready
See TODO list below.
Each Snowflake ID is a 63 bit int.
- 41 bits for a timestamp (milliseconds since a custom epoch)
- 10 bits for a node ID (ranging from 0 to 1023)
- 12 bits for a sequence number (ranging from 0 to 4095)
If you're planning on horizontally scaling (i.e., replicating) your app, you'll need a way of doling out node IDs.
In the era of Docker and Kubernetes, containers and pods are expected to be ephemeral. They can go down at any moment. If that happens, they have no way of relinquishing their node IDs, and so that node ID is essentially lost.
Fortunately, checking for "stale" IDs is super easy, so we can ensure we have a dynamic and reliable way of re-requisitioning and reallocating node IDs.
What does Cirrus do?
Cirrus maps node IDs to app IDs.
An app ID is a string that can be anything (e.g., an int, a UUID, a hash, etc).
A node ID is an int ranging from 0 to 1023.
Apps must periodically send heartbeats to Cirrus, otherwise their node IDs get requisitioned.
- How do we know if a node has actually died? What if it's just busy and stays alive? There might be a moment where 2 apps generate PKs with the same node ID? That would be disastrous.
Single point of failure
Cirrus needs to be able scale out so it's not a single point of failure.
We need a couple things:
AvailableNodeIdschannel with a distributed queue system (e.g., RabbitMQ, Cherami, etc)
Heartbeatsmap with a distributed store (e.g., etcd)
- Usage section in README
- Configurable durations
- Helm chart