Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP
Unique sortable shardable readable timestampeable incredible ID generator.
JavaScript
branch: master

Fetching latest commit…

Cannot retrieve the latest commit at this time

Failed to load latest commit information.
test
.gitignore
README.md
index.js
package.json
project.sublime-project

README.md

unik

unik is a unique identifier generator for creating shardable and sortable IDs (non-RFC compliant).

warning: unik is experimental software

Install

npm install unik --save

Make sure you specify the unik version in your package.json.

Usage

var Unik = require('unik')
   , db  = require('mydb')

var unik = new Unik()

db.put(unik.flake(), data, function (err, res) { ... })

The Unik constructor also has a .create shortcut for convenience:

var unik = require('unik').create()

var id = unik.flake()

The constructor takes an options argument:

new Unik({
    // base: use `10` to get the ID as an integer
    base: 36
    // unique: unique identifier < 1024 for `unik.flake()`. Use `.workerID` in a cluster, for example.
    unique: process.pid % 1024
    // seed: random seed OR unique identifier for `unik.bigflake()`
    seed: random 0-1048576
    // separator. Set to '' to get a string w/o breaks (makes timestamp not extractable)
    sep: '-'
    // epoch: custom start epoch. If you change this the resulting dates must fit in 41 bits
    // Defaults to Sun, 01 Jan 2012 00:00:00 GMT
    epoch: 1325376000000
    // If set, will be appended to the end of the ID using `.separator`; useful if you need to store
    // some data in the key itself. Resulting key length will be dependent on this value.
    append: ''
})

unik.flake()

Snowflake-inspired 64-bit IDs. Uses a custom epoch start, the .unique option along with an internal counter for generating up to 8191 ids/ms.

Input Size
Time (milliseconds) 41 bits
Counter sequence 13 bits (*8192)
Process ID / unique 10 bits (*1024)

Due to javascript's 53-bit integer limit, the ID is split in two parts, time-counter|pid.

.flake can generate a total of 18,446,744,073,709,551,616 values.

unik.bigflake()

96 bit IDs. Uses full unix epoch, node process.hrtime nanosecond resolution timer and a random seed, doesn't require a unique identifier.

Input Size
Time (milliseconds) 53 bits
Time (nanoseconds) 23 bits (*8388608)
Random / unique 20 bits (*1048576)

.bigflake can generate a total of 79,228,162,514,264,337,593,543,950,336 values.

Sharding

Set the unique option to the shard's ID. .flake can have up to 1024 shards. If needed, the value can be extracted from the ID by reading the last 10 bits, e.g.:

var id = unik.flake()
var shard_id = parseInt(id.split('-')[1], 36) // assuming base 36 and '-' separator

shard_id = parseInt(shard_id.toString(2).slice(-10), 2)

For .bigflake use the seed option, up to 1048576 shards.

Performance and collisions

unik.flake() can generate 700k ids/second using two workers on a C2D 2.4ghz. It is collision-free as long as the .unique value is different for each process. Here is it after generating over a billion IDs:

A billion unik flakes

unik.bigflake() is about 3x slower. It should be collision-free up to a few million ids/second, if you somehow manage that. There is an infinitesimal possibility of generating a duplicate, if two IDs are generated by separate processes at the exact same nanosecond and they contain the same random seed. Even though it uses the native Math.random it should be pretty safe. To guarantee no collisions, give it a known unique value using the seed option.

It has built-in protection for backwards-flowing time by holding back ID generation until the next clock tick.

Test

npm test or node test/sharding.js [workers] [burst_size] is designed to test for collisions using multiple processes. The main process will break when a collision is found.

References

Something went wrong with that request. Please try again.